Steel bar binding method, steel bar binding system, storage medium and electronic equipment

Abstract

The invention provides a reinforcing steel bar binding method, a reinforcing steel bar binding system, a storage medium and electronic equipment, and relates to the field of industrial automation, and the method comprises the following steps: obtaining a color image and a depth image of a reinforcing steel bar scene; inputting the preprocessed color image into a key point detection model to obtain pixel information; the pixel information comprises a binding point area pixel frame and binding point pixel coordinates; extracting point cloud data of a binding point area according to the pixel information and the depth image; setting a template point cloud, and performing registration on the binding point area point cloud data by using the template point cloud to obtain a transformation matrix of the actual binding point point cloud and the template point cloud; and according to the transformation matrix, the binding posture of the reinforcing steel bar is determined, so that a mechanical arm is guided to bind the reinforcing steel bar according to the binding posture. The reinforcing steel bar binding pose can be estimated, so that the mechanical arm is guided to complete reinforcing steel bar binding operation, and automation of the reinforcing steel bar binding process is achieved.

Description

Technical Field

[0001] The present application relates to the field of industrial automation, and in particular to a steel bar tying method, a steel bar tying system, a storage medium and an electronic device. Background Art

[0002] Rebar is an important structural material widely used in civil engineering and is used in various projects, such as houses, bridges, and water conservancy projects. In the process of steel bar construction, binding is one of the indispensable processes, which is directly related to the firmness and service life of the structure. At present, the steel bar binding process is mainly completed manually. During this process, workers need to bend over and kneel for a long time to operate, which can easily cause worker fatigue, muscle soreness, and even chronic injuries, thus affecting work efficiency and work quality. In addition, due to the large number of steel bars, traditional manual binding requires a lot of time and manpower, resulting in extended construction period and increased project costs.

[0003] In recent years, robotics technology has developed rapidly, and it has shown great potential in automated construction. In the existing technology, image information can be used and combined with computer vision technology to identify the coordinates of the binding points to realize automated steel bar binding operations. However, image information is greatly affected by the shooting angle, resulting in poor accuracy of the estimated spatial coordinates of the binding points, and the end-binding device of the robotic arm is prone to collision with the surrounding steel bars during execution. In addition, in robot kinematics, the position and posture of the end effector are crucial to the execution of the task, where the position information describes the displacement of the object on the x, y, and z axes, while the posture information describes the rotation angle of the object around the x, y, and z axes. Only identifying the binding point position information can only guide the end effector to the target position, and cannot guide the rotation of the binder, making it difficult to ensure that the binding wire of the binder can correctly bypass the steel bars and perform the binding task. Summary of the invention

[0004] The purpose of this application is to provide a steel bar tying method, a steel bar tying system, a storage medium and an electronic device that can accurately perform steel bar identification and tying tasks.

[0005] In order to solve the above technical problems, the present application provides a steel bar binding method, and the specific technical solution is as follows:

[0006] Obtain color images and depth images of the steel bar scene;

[0007] Input the preprocessed color image into a key point detection model to obtain pixel information; the pixel information includes a pixel border of a binding point area and pixel coordinates of the binding point;

[0008] Extracting point cloud data of the binding point area according to the pixel information and the depth image;

[0009] Setting a template point cloud, applying the template point cloud to align the point cloud data of the binding point area, and obtaining a transformation matrix between the actual binding point point cloud and the template point cloud;

[0010] The binding posture of the steel bars is determined according to the transformation matrix, so as to guide the robot arm to bind the steel bars according to the binding posture.

[0011] Optionally, the step of inputting the preprocessed color image into a key point detection model to obtain pixel information includes:

[0012] Using the key point detection model, a rectangular frame and a key point are marked for each binding point target in the color image; the width and height of the rectangular frame are positively correlated with the diameter of the steel bar; the key point is the cross center point of the steel bar;

[0013] The pixel border of the binding point area is determined according to the rectangular border, and the pixel coordinates of the binding point are determined according to the key points.

[0014] Optionally, the generation process of the key point detection model includes:

[0015] A color training image and a depth training image of untied steel bars are collected using a depth camera; the color training image contains visible light information of the steel bar scene; the depth training image contains distance information in the steel bar scene;

[0016] Preprocessing the color image by depth threshold segmentation to obtain image data containing only surface layer steel bar information;

[0017] Marking the cross center points of the steel bars in the image data to obtain marked data; in the marking process, the width and height of the binding point boundary box are the same;

[0018] The key point detection model is obtained by training a convolutional neural network using the labeled data.

[0019] Optionally, the preprocessing of the color image by depth threshold segmentation to obtain image data containing only surface layer steel bar information includes:

[0020] Assuming the distance d between the depth camera and the steel bar surface, according to the depth value information in the depth image, filter out the target coordinate points with depth values ​​in the range of [d-50, d+50];

[0021] In the color image, the color of the target coordinate point is set to a first color, and the colors of other points are set to a second color.

[0022] Optionally, extracting point cloud data of the binding point area according to the pixel information includes:

[0023] Determine the shooting parameters of the camera; the shooting parameters include the horizontal focal length, the vertical focal length and the pixel coordinates of the optical center projection point on the imaging plane;

[0024] Determine the pixel coordinates of the upper left corner and the lower right corner of the pixel frame of the binding point area according to the shooting parameters, and for any pixel point in the frame, determine the spatial coordinate value of the pixel point in the camera coordinate system according to the depth value in the depth image corresponding to the pixel point;

[0025] Traverse all the pixels in the pixel frame of the binding point area, obtain the spatial coordinates of all the pixels in the pixel frame in the camera coordinate system, and combine all the spatial coordinate points to obtain the point cloud data of the binding point area.

[0026] Optionally, the applying the template point cloud to register the point cloud data of the binding point area includes:

[0027] Construct a point cloud model of the lashing point template;

[0028] Performing global registration on the point cloud model based on geometric features;

[0029] The point cloud model is locally registered based on adaptive displacement correction.

[0030] Optionally, the point cloud model for constructing the lashing point template includes:

[0031] Two mutually perpendicular cylinders are used as template point clouds of the binding point cloud for registration; the two mutually perpendicular cylinders include a horizontal cylinder and a vertical cylinder perpendicular to the horizontal plane;

[0032] Determine a camera coordinate system; the X-axis of the camera coordinate system points to the right side of the camera, the Y-axis points to the top of the camera, and the Z-axis coincides with the optical axis of the camera;

[0033] The template point cloud is kept parallel to an XOY plane formed by the X-axis and the Y-axis; wherein the horizontal cylinder is kept parallel to the X-axis and passes through the origin of the camera coordinate system; and the vertical cylinder is kept parallel to the Y-axis.

[0034] Optionally, the performing global registration on the point cloud model based on geometric features includes:

[0035] The real spatial coordinates of the lashing points are obtained by key point detection method, and compared with the spatial coordinates of the lashing points in the template to obtain the initial translation value;

[0036] The point cloud of actual lashing points is processed using point cloud plane fitting technology to determine the plane where the point cloud of actual lashing points is located;

[0037] Compare the plane where the actual binding point is located with the plane where the template point cloud is located to obtain the initial rotation value;

[0038] The point cloud model of the actual binding point is translated and rotated according to the initial translation value and the initial rotation value, so that the point cloud model of the actual binding point is aligned with the template point cloud.

[0039] Optionally, the performing local registration on the point cloud model based on adaptive displacement correction includes:

[0040] Use the set point cloud precise registration algorithm to perform precise registration of the scene point cloud and the template point cloud, and obtain the transformation matrix from the scene point cloud to the template point cloud;

[0041] Determining whether the transformation matrix satisfies a root mean square error threshold;

[0042] If not, apply multiple corrected displacement values ​​in the direction of the axis of the outer steel bar, calculate the root mean square error after the displacement correction, compare the root mean square errors in the positive and negative directions, and use the displacement correction corresponding to the smaller root mean square error value to re-execute the step of using the set point cloud precise registration algorithm to perform fine registration of the scene point cloud and the template point cloud, until the transformation matrix meets the root mean square error threshold, and obtain the final transformation matrix from the template coordinate system to the camera coordinate system.

[0043] Wherein, determining the binding posture of the steel bars according to the transformation matrix includes:

[0044] Determine the lashing posture in the template coordinate system, the orientation of the lashing posture is -45° rotated around the Z axis, and the position is the center point of the intersection area of ​​the two cylinders on the outer edge cylinder;

[0045] Determine the transformation matrix between the camera coordinate system and the robot end coordinate system through hand-eye calibration technology;

[0046] According to the robot arm teaching device, the transformation matrix from the robot arm end coordinates to the base coordinate system is obtained;

[0047] The binding posture in the base coordinate system is obtained by using the transformation matrix from the end of the robot arm to the base coordinate system, the transformation matrix from the camera coordinate system to the end of the robot arm coordinate system, the transformation matrix from the template coordinate system to the camera coordinate system, and the binding posture in the template coordinate system.

[0048] The present application also provides a steel bar tying system, comprising:

[0049] An image acquisition module, used to acquire color images and depth images of the steel bar scene;

[0050] A pixel extraction module, used to input the preprocessed color image into a key point detection model to obtain pixel information; the pixel information includes a binding point area pixel border and binding point pixel coordinates;

[0051] A point cloud data extraction module, used to extract point cloud data of the binding point area according to the pixel information and the depth image;

[0052] A registration module is used to set a template point cloud, apply the template point cloud to register the point cloud data of the binding point area, and obtain a transformation matrix between the actual binding point point cloud and the template point cloud;

[0053] The binding posture determination module is used to determine the binding posture of the steel bars according to the transformation matrix, so as to guide the robot arm to bind the steel bars according to the binding posture.

[0054] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described above are implemented.

[0055] The present application also provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned method when calling the computer program in the memory.

[0056] The present application provides a steel bar binding method, comprising: acquiring a color image and a depth image of a steel bar scene; inputting the preprocessed color image into a key point detection model to obtain pixel information; the pixel information comprises a binding point area pixel border and binding point pixel coordinates; extracting binding point area point cloud data according to the pixel information and the depth image; setting a template point cloud, applying the template point cloud to align the binding point area point cloud data, and obtaining a transformation matrix between the actual binding point point cloud and the template point cloud; determining the binding posture of the steel bar according to the transformation matrix, so as to guide a robotic arm to perform steel bar binding according to the binding posture.

[0057] This application uses point cloud data to estimate the coordinates of the binding points. Point cloud data is not affected by viewing angles and can directly represent the geometric shape of the steel bars in three-dimensional space, so it can achieve higher spatial positioning accuracy of the binding points. This application uses a point cloud registration method to estimate and determine the binding posture, which has lower computational complexity than computer vision technology. In addition, the point cloud registration method is based on mathematical models for matching. For example, the geometric information of the point cloud can be used for feature matching or surface features can be used for registration. When faced with a clear geometric structure and specific feature scenes such as the binding point cloud, the algorithm can more accurately identify and match these features, thereby obtaining more accurate posture estimation results.

[0058] The present application also provides a steel bar binding system, a computer-readable storage medium and an electronic device, which have the above-mentioned beneficial effects and are not described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0060] Figure 1 A flowchart of a steel bar binding method provided in an embodiment of the present application;

[0061] Figure 2 This is a steel bar binding point detection effect diagram provided in the embodiment of the present application;

[0062] Figure 3 A front view of a point cloud model of a steel bar scene provided in an embodiment of the present application;

[0063] Figure 4 A side view of a point cloud model of a steel bar scene provided in an embodiment of the present application;

[0064] Figure 5 A front schematic diagram of a point cloud model of a lashing point template provided in an embodiment of the present application;

[0065] Figure 6 A side view of a point cloud model of a lashing point template provided in an embodiment of the present application;

[0066] Figure 7 A front view schematic diagram of the global registration process provided in an embodiment of the present application;

[0067] Figure 8 A side schematic diagram of the global registration process provided in an embodiment of the present application;

[0068] Fig. 9 A front schematic diagram of a local optimal solution provided by an embodiment of the present application;

[0069] Fig.10 A side schematic diagram of a local optimal solution provided by an embodiment of the present application;

[0070] Fig.11 A front view schematic diagram of the displacement correction effect provided by an embodiment of the present application;

[0071] Fig.12 A side schematic diagram of the displacement correction effect provided by an embodiment of the present application;

[0072] Fig.13 A structural schematic diagram of a steel bar binding system provided in an embodiment of the present application;

[0073] Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0075] In order to solve the shortcomings of existing automated binding technology, such as poor positioning accuracy of steel bar binding points and inability to estimate the execution posture of the end of the robotic arm, this application will use point cloud data combined with point cloud registration technology to estimate the spatial point coordinates and binding posture of the steel bar binding, that is, to estimate the binding posture. Point cloud data is not affected by viewing angle and can directly represent the geometric shape of the steel bar in three-dimensional space, so it can achieve higher spatial positioning accuracy of the binding point. At the same time, compared with computer vision technology, the point cloud registration method has lower computational complexity. In addition, the point cloud registration method is based on mathematical models for matching, such as using the geometric information of the point cloud for feature matching or using surface features for registration. When faced with a clear geometric structure and specific feature scenes such as the binding point cloud model, the algorithm can more accurately identify and match these features, thereby obtaining more accurate posture estimation results.

[0076] However, since the point cloud information collected when shooting with a depth camera usually contains multiple binding points, the amount of calculation is large when directly using the template point cloud to align with the steel scene point cloud data. Therefore, the present application proposes a steel bar binding pose estimation framework based on key point detection and point cloud alignment. A depth camera is used to collect color images and depth images in the steel bar scene, where the color image is the visible light information of the steel bar scene captured by the camera, which provides visual information such as the color and texture of each part of the scene, while the depth map provides the distance information from the point in the scene to the camera for each pixel, which records the distance difference between different objects or different parts in the steel bar scene, reflecting the three-dimensional structure of the scene. According to the results of the key point detection algorithm, the steel bar scene point cloud is divided into multiple subsets of binding point point clouds, and each individual target is used as a template for alignment.

[0077] Please refer to Figure 1 , Figure 1 A flowchart of a steel bar binding method provided in an embodiment of the present application, the method comprising:

[0078] In the process of rough point cloud registration, this application proposes a global registration method based on geometric features to preliminarily align the binding point point cloud model to an initial position relatively close to the template point cloud, so that the fine registration algorithm can converge faster to obtain the optimal solution. In the process of fine point cloud registration, a local registration based on adaptive displacement correction is proposed for the point cloud model, which adaptively corrects the displacement value in the axial direction of the horizontal steel bar to avoid the algorithm from falling into the local optimal solution and ensure the accuracy of point cloud registration. The binding point point cloud model can be extracted first. The detailed steps are as follows:

[0079] S101: Acquire a color image and a depth image of a steel bar scene;

[0080] S102: Input the preprocessed color image into a key point detection model to obtain pixel information; the pixel information includes a binding point area pixel border and binding point pixel coordinates;

[0081] S103: extracting point cloud data of the binding point area according to the pixel information and the depth image;

[0082] S104: setting a template point cloud, applying the template point cloud to align the point cloud data of the binding point area, and obtaining a transformation matrix between the actual binding point point cloud and the template point cloud;

[0083] S105: Determine the binding posture of the steel bars according to the transformation matrix, so as to guide the robot arm to bind the steel bars according to the binding posture.

[0084] The point cloud data collected by the depth camera usually contains multiple binding points. The steel bar scene point cloud needs to be divided into multiple subsets of binding point point clouds, and each individual target is aligned with the template to improve the efficiency of the alignment. At the same time, it is also necessary to obtain the coordinates of the key points to provide a better initial displacement value for the subsequent point cloud alignment algorithm.

[0085] Since the point cloud information collected by the depth camera usually contains multiple binding points, the amount of calculation is large when the template point cloud is directly used to align with the steel scene point cloud data. Therefore, the structured light camera can be used to collect color images and depth images in the steel scene. The color image is the visible light information of the steel scene captured by the camera, which provides visual information such as the color and texture of each part of the scene, while the depth image provides the distance information from the point in the scene to the camera for each pixel. It records the distance difference between different objects or different parts in the steel scene and reflects the three-dimensional structure of the scene.

[0086] Key point detection is one of the computer vision tasks, which aims to identify important feature points in an image or video. These feature points are usually locations with significant properties in the image, such as corners, edge points, or areas with specific textures. In recent years, deep learning-based methods have made significant progress in the field of key point detection. They use convolutional neural networks (CNNs) to directly learn feature representations and learn key points in specific tasks by training image data. Therefore, the present application can use a CNN-based key point detection algorithm to simultaneously determine the bounding box and binding point coordinates of the target. Specifically, the key point detection model can be used to annotate a rectangular bounding box and a key point for each binding point target in the color image, and then determine the binding point area pixel bounding box based on the rectangular bounding box, and determine the binding point pixel coordinates based on the key point. The width and height of the rectangular bounding box are positively correlated with the diameter of the steel bar, and the key point refers to the center point of the steel bar intersection.

[0087] The specific steps are as follows: In actual production, the steel cage usually involves multiple layers of steel bars. During the image capture process, it is difficult to capture the detailed three-dimensional shape of the target due to the relatively limited geometric information provided by the color image. This causes the binding points to show the characteristics of steel bar crossing in the image, and steel bars at different levels may also appear to be crossed in the image. The "false points" at the intersections of different layers of steel bars are difficult to distinguish from the real binding points that need to be tied in the color image, which affects the accuracy and robustness of the steel bar binding point recognition algorithm based on color images. To solve this problem, the application obtains a key point detection model through training, and uses a deep threshold segmentation method to pre-process the color image to retain only the surface steel bar information.

[0088] The generation process of the key point detection model can include the following steps:

[0089] The first step is to use a depth camera to collect color training images and depth training images of untied steel bars; the color training images contain visible light information of the steel bar scene; the depth training images contain distance information in the steel bar scene;

[0090] The second step is to pre-process the color image through depth threshold segmentation to obtain image data containing only the surface layer steel bar information;

[0091] Step 3: Mark the cross center points of the steel bars in the image data to obtain the marked data; during the marking process, the width and height of the bounding box of the binding point are the same;

[0092] The fourth step is to use the labeled data to apply the convolutional neural network for training to obtain the key point detection model.

[0093] Assume that the distance between the depth camera and the steel bar surface is d. According to the depth value information in the depth image, the target coordinate points with depth values ​​between the range of [d-50, d+50] are screened out.

[0094] In a color image, the color of the target coordinate point is set to the first color, and the colors of other points are set to the second color. The first color and the second color are not specifically limited and are only used to distinguish different coordinate points.

[0095] A specific implementation process can be: given the distance d between the camera and the surface of the steel bar, based on the depth value information in the depth map, filter out the coordinate points with depth values ​​between the range of [d-50, d+50]. In the color map, the colors of these coordinate points are set to black, and the colors of other points are set to white. In this way, a preprocessed color image is obtained, which only contains the pixel information of the surface steel bars. Thereafter, the image data is annotated using Labelme software, and each binding point target is annotated with a rectangular bounding box and a key point, where the binding point bounding box has the same width and height, and is approximately 3 times the length of the steel bar diameter. After the annotation is completed, the model is trained. Finally, the trained key point detection model can be used to predict the pixel borders and pixel coordinates of each binding point area in different scenes. The results are shown below. Figure 2 As shown, Figure 2 This is a steel bar binding point detection effect diagram provided in the embodiment of the present application. Figure 2 It shows where the key points crosspoints are and where the bounding boxes are.

[0096] Extraction of point cloud of binding point area. The above steel bar binding point detection step has obtained the pixel border and pixel coordinates of each binding point. This application will extract the point cloud data of the binding point area based on the pixel information.

[0097] In a feasible implementation manner, the method for extracting point cloud data of the binding point area may include the following steps:

[0098] The first step is to determine the shooting parameters of the camera; the shooting parameters include the horizontal focal length, the vertical focal length and the pixel coordinates of the optical center projection point on the imaging plane;

[0099] The second step is to determine the pixel coordinates of the upper left corner and the lower right corner of the pixel frame of the binding point area according to the prediction results of the key point detection model. For any pixel point in the frame, the spatial coordinate value of the pixel point in the camera coordinate system is determined according to the depth value in the depth map corresponding to the pixel point and the shooting parameters;

[0100] The third step is to traverse all the pixels within the pixel border of the binding point area, obtain the spatial coordinates of all the pixels within the pixel border in the camera coordinate system, and combine all the spatial coordinate points to obtain the point cloud data of the binding point area.

[0101] The corresponding specific implementation process is: the internal parameters of the camera are known: horizontal focal length , vertical focal length , pixel coordinates of the projection point of the optical center on the imaging plane The pixel coordinates of the upper left corner of the pixel border of any binding point area are , the pixel coordinates of the lower right corner are , for any point within the bounding box , the pixel coordinates are ,in , The depth value of the pixel in the depth map is , then its spatial coordinate value in the camera coordinate system is:

[0102] ;

[0103] ;

[0104] ;

[0105] By traversing all the pixels within the pixel border of the binding point area, the spatial coordinates of all the pixels within the pixel border in the camera coordinate system can be obtained. By combining all the spatial coordinate points, the point cloud model of each binding point can be constructed, such as Figure 3 and Figure 4 As shown, Figure 3 This is a front view of the steel bar scene point cloud model provided in the embodiment of the present application. Figure 4 This is a side view of the steel bar scene point cloud model provided in the embodiment of the present application. At this point, the steel bar scene point cloud has been segmented into a plurality of subsets of binding point point clouds, and the binding point area point cloud data has been obtained.

[0106] After that, the template point cloud needs to be applied to register the point cloud data of the binding point area. It is necessary to build a point cloud model of the binding point template, first perform global registration on the point cloud model based on geometric features, and then perform local registration on the point cloud model based on adaptive displacement correction.

[0107] First, a template point cloud model of the binding point is generated. The actual binding point point cloud structure shows two crisscrossed cylinder features. Therefore, a method is proposed to use two mutually perpendicular cylinders as template point clouds for binding point point cloud registration. In addition, the camera coordinate system is usually a right-handed coordinate system, with its origin at the optical center of the camera. The axis points to the right of the camera, The axis points upwards from the camera. The axis coincides with the optical axis of the camera and points to the front of the camera (the direction of the steel bar being photographed). During the camera shooting process, the steel bar plane is usually nearly parallel to Plane, horizontal stirrups and The axis is nearly parallel, and the longitudinal force reinforcement is The axes are close to parallel. Therefore, the point cloud of the steel bar binding point template proposed in this application is consistent with The planes remain parallel, and the horizontal cylinder is The axes remain parallel and pass through the origin of the coordinate system. The longitudinal cylinder and The axes remain parallel. The details are as follows: Assume that the diameters of the horizontal and longitudinal steel bars in actual production are and , then the center position of the horizontal cylinder in the template point cloud model is , the axis vector is , diameter is , the height range of the cylinder is , the center position of the longitudinal cylinder is , the axis vector is , diameter is , the height range of the cylinder is , the point cloud of the binding point template is shown in Figure and Figure 6 As shown, Figure 5 This is a front view of the point cloud model of the binding point template provided in the embodiment of the present application. Figure 6 This is a side schematic diagram of the point cloud model of the binding point template provided in an embodiment of the present application.

[0108] Perform global registration of the point cloud model based on geometric features. The specific steps are as follows:

[0109] Before fine registration, it is very important to perform global registration, because global registration can provide a good initial alignment state and provide a good starting point for subsequent fine registration. Since the binding point point cloud model is a smooth surface and lacks obvious feature points, traditional feature extraction methods may not be able to effectively extract the binding point point cloud features, which makes it difficult to find the correspondence between point clouds, making registration difficult. Therefore, this application proposes a global registration method based on geometric features. First, the spatial coordinates of the real binding points are obtained by the key point detection method, and the initial translation value is obtained by comparing the spatial coordinates of the binding points in the template. Next, the actual binding point point cloud is processed using the point cloud plane fitting technology to determine the plane where the actual binding point point cloud is located. Then, the plane where the actual binding point is located is compared with the plane where the template point cloud is located to obtain the initial rotation value. Finally, the point cloud model of the actual binding point is rotated and translated to align it with the template point cloud. This method can accurately align point cloud data in a global range by combining key point detection and point cloud plane fitting technology, providing a good initial state for subsequent fine registration. The specific steps are as follows:

[0110] From the previous step, we can know that the diameter of horizontal and longitudinal reinforcement is and The geometric characteristics and position of the steel bar template point cloud are taken as follows: The spatial point located on the horizontal cylindrical surface and at the center of the intersection area of ​​the horizontal and longitudinal cylinders is taken as the steel bar binding point in the template point cloud, and its coordinates are obtained as follows: In the first step of this application, the key point detection algorithm can be used to identify the pixel coordinates of the binding point. If a pixel coordinate is According to the calculation method described in the first step, the spatial coordinate value of the binding point in the camera coordinate system can be obtained as At this point, the initial displacement value is obtained by subtracting the coordinates of the binding points in the template point cloud from the coordinates of the binding points in the actual scene point cloud. ,Right now .

[0111] For the calculation of the initial rotation matrix, the parameters of the plane where the actual binding point cloud is located are estimated by using a plane fitting algorithm (such as RANSAC, least squares method, etc.). Suppose the normal vector of the plane where the binding point cloud is located is , it is known that the planes of the vertical and horizontal cylinders in the template point cloud are parallel to , then its normal vector is To simplify the calculation, we can determine whether the value of c is positive or negative and convert the normal vector Adjust it to point to the positive direction of the Z axis and normalize it. The normalized normal vector is: ,in , and then use the cross product operation to calculate the rotation axis, which is: , whose direction is perpendicular to both given vectors and conforms to the right-hand rule. Then, the rotation angle is obtained by calculating the angle between the normal vectors. Finally, the matrix is ​​constructed by the rotation axis and rotation angle to get the initial rotation value ,for:

[0112] ;

[0113] Through the above calculation, we can get the initial displacement value and initial rotation value, transformation matrix For each point P in the actual binding point cloud, after translation and rotation, the actual binding point cloud model can be initially aligned to an initial position relatively close to the template point cloud, so that the precise registration algorithm can converge faster to obtain the optimal solution. The effect is as follows: Figure 7 and Figure 8 As shown, Figure 7 This is a front view schematic diagram of the global registration process provided in the embodiment of the present application. Figure 8 A side schematic diagram of the global registration process provided in an embodiment of the present application.

[0114] Afterwards, local registration of the point cloud model based on adaptive displacement correction is performed. The specific steps are as follows:

[0115] First registration. After completing the rough registration in the previous step, although preliminary alignment has been achieved, there are still certain deviations and misalignments between the scene point cloud and the template point cloud. These deviations mainly come from two aspects: first, the initial displacement value is calculated by a key point detection algorithm based on 2D images. The positioning accuracy of this method is easily affected by the shooting angle; second, in the above-mentioned rough registration stage, the rotation error around the Z axis is not considered, which will also lead to inaccurate registration. In this case, it is particularly important to perform fine registration to minimize the differences between the point clouds so that they can be completely aligned geometrically. Therefore, this application proposes to use the ICP (Iterative Closest Point) algorithm to perform precise registration of the scene point cloud and the template point cloud. At the same time, the kd tree (K-Dimensional Tree, a data structure used to organize data points for spatial search) is combined to accelerate the search process of corresponding points, greatly improving the calculation efficiency of the registration. After completing the registration, the transformation matrix from the scene point cloud to the template point cloud after rough registration can be obtained:

[0116] ;

[0117] Secondary registration after displacement correction. In the process of registering the binding area point cloud and the template point cloud, the ICP algorithm is prone to fall into a local optimal solution. The main reason is that since the vertical steel bars are blocked by the horizontal steel bars, most of the point cloud data in the binding area point cloud is the horizontal steel bars. Therefore, when the horizontal steel bar point cloud is aligned with the horizontal cylinder in the template point cloud, the algorithm may converge to the local optimal solution too early, and fail to fully consider the geometric information and position relationship of the vertical steel bars. This local optimal solution may cause the registration result to deviate from the actual position, affecting the subsequent pose estimation accuracy, such as Fig. 9 and Fig.10 As shown, Fig. 9 This is a front view schematic diagram of a local optimal solution provided in an embodiment of the present application. Fig.10 A side schematic diagram of a local optimal solution provided in an embodiment of the present application.

[0118] Therefore, this application proposes an adaptive displacement correction registration method, which improves the registration accuracy by adaptively correcting the displacement value in the direction of the outer steel bar axis. The specific steps are: After the above registration is completed, the root mean square error (RMSE) is used as an indicator to measure the registration accuracy. RMSE is used to measure the distance error between two point clouds, that is, the difference between the actual point cloud and the target point cloud. Set an RMSE threshold to determine whether the registration accuracy meets the requirements. When the RMSE exceeds the threshold, it means that the registration result is not accurate enough and needs to be corrected. At this time, apply an RMSE correction in the X-axis direction. or The displacement value is used to adjust the position of the actual point cloud, and ICP registration is performed again. The iteration is continued until the accuracy requirement is met or the maximum number of iterations is reached, and the transformation matrix is ​​finally obtained. , the registration effect is as follows Fig.11 and Fig.12 As shown, Fig.11 This is a front view schematic diagram of the displacement correction effect provided by the embodiment of the present application. Fig.12 A side schematic diagram of the displacement correction effect provided by an embodiment of the present application.

[0119] After completing the above point cloud registration process, the transformation matrix from the actual binding point cloud to the template point cloud can be obtained. , this transformation matrix describes how to align the binding point cloud with the template point cloud so that they overlap as much as possible. However, since the result of the registration is relative to the camera coordinate system. However, for the robot, it needs to know the binding posture in the robot coordinate system (usually the robot base) to accurately operate the end tie for binding. The transformation relationship between the steel bar binding posture in the robot base coordinate system and the steel bar binding posture in the camera coordinate system is:

[0120] ;

[0121] in, is the steel bar binding posture in the template coordinate system. Since the geometric characteristics and spatial position of the template are known, is a known item. In the second step, the spatial coordinates of the template binding point are set to The principle for setting the binding direction is that the actuator does not collide with the longitudinal and transverse reinforcements. The plane of the cylinder point cloud in the template coordinate system is perpendicular to the Z axis, and the direction of the cylinder is parallel to the X axis and the Y axis respectively. Therefore, the binding direction can be set to the original coordinate system rotated around the Z axis -45° direction. At this time, the binding posture in the template coordinate system is:

[0122] ;

[0123] in, is the transformation matrix from the template point cloud to the actual binding point cloud. Through the above calculation, it is known that , through Taking the inverse, we can get ,Right now .and It is the transformation matrix between the camera coordinate system and the end coordinate system of the robot arm. At this time, the end of the robot arm is the end point of the binding actuator. Hand-Eye Calibration is a hand-eye calibration technology, which is used to determine the exact position and posture of the end effector (hand) of the robot arm relative to the visual system (eye). This application uses the Eye-in-HandCalibration program for calibration. The specific implementation steps are as follows: First, select a calibration plate with calibration points of known features arranged on it. Install the camera (eye) on the end effector (hand) of the robot arm to ensure that the camera can accurately observe the calibration plate. Then, by controlling the robot arm, the camera can observe the calibration plate in different positions and postures and collect corresponding image data. The collected image data is processed, and the position information of the feature points on the calibration workpiece is extracted using computer vision technology. These are usually pre-designed, known, and easy-to-detect points, such as corner points, center points, etc. The collected image data and the corresponding robot arm motion trajectory are used to calculate the accurate relative position and posture relationship between the camera (eye) and the robot arm end effector (hand) through the spatial transformation model, that is, the transformation matrix of the hand-eye calibration . It is the transformation matrix from the robot end coordinate system to the base coordinate system, which can be calculated through the end effector pose obtained by the robot arm teach pendant and the pose in the base coordinate system.

[0124] At this point, the above calculations are completed, and the rebar binding posture in the robot arm base coordinate system can be obtained. Using this posture, the robot arm can perform rebar binding operations.

[0125] It can be seen that the present application uses point cloud data to estimate the coordinates of the binding points. Point cloud data is not affected by the viewing angle and can directly represent the geometric shape of the steel bar in three-dimensional space, thus achieving higher spatial positioning accuracy of the binding points.

[0126] At the same time, this application proposes to use the point cloud registration method to estimate the binding posture. Compared with computer vision technology, this method has lower computational complexity. In addition, the point cloud registration method is based on mathematical models for matching, such as using the geometric information of the point cloud for feature matching or using surface features for registration. When faced with the clear geometric structure and specific feature scenes of the binding point cloud, the algorithm can more accurately identify and match these features, thereby obtaining more accurate posture estimation results.

[0127] In addition, the existing automated tying technology only obtains the spatial position information of the tying point, which can only enable the robot arm to simply reach the execution position, and cannot guide the end effector in the direction required to tie the steel bars. The orientation of the end effector is crucial to ensure that the tying wire correctly bypasses the steel bars, avoids collisions, and ensures the accuracy and stability of the tying. This application uses point cloud registration technology to not only obtain the spatial position information of the tying point, but also calculate the spatial posture of the tying point, ensuring that the tying wire correctly bypasses the steel bars and completes the tying task.

[0128] Since the point cloud information collected when shooting with a depth camera usually contains multiple binding points, directly matching the entire steel bar point cloud may require a lot of computing resources and time, and may also cause local features to be interfered with by other objects, thereby reducing the matching accuracy. To solve this problem, the present application proposes a steel bar binding pose estimation framework based on key point detection and point cloud registration. According to the results of the key point detection algorithm, the steel bar scene point cloud is divided into multiple subsets of binding point point clouds, and each subset contains only one binding point point cloud. Therefore, when performing registration on each subset, the search space is smaller, and the calculation amount of the registration algorithm is correspondingly reduced. At the same time, it can also better focus the attention of the matching algorithm and improve the matching accuracy.

[0129] In the process of point cloud matching, since the binding point cloud model lacks obvious edge points or corner points, it becomes very difficult to extract features for matching. Traditional feature extraction algorithms may not be able to effectively capture the features of the model, resulting in reduced registration accuracy. To solve this problem, this application proposes a global registration method based on geometric features in the process of rough point cloud registration, which preliminarily aligns the scene point cloud model to an initial position that is relatively close to the template point cloud, so that the fine registration algorithm can converge faster to obtain the optimal solution. At the same time, in the process of fine point cloud registration, this application adaptively corrects the displacement value in the direction of the axis of the outer steel bars to avoid the algorithm from falling into a local optimal solution and ensure the accuracy of point cloud registration.

[0130] A steel bar tying system provided in an embodiment of the present application is introduced below. The steel bar tying system described below and the steel bar tying method described above can refer to each other.

[0131] See also Fig.13 , Fig.13 A structural schematic diagram of a steel bar binding system provided in an embodiment of the present application, the system comprising:

[0132] An image acquisition module, used to acquire color images and depth images of the steel bar scene;

[0133] A pixel extraction module, used to input the preprocessed color image into a key point detection model to obtain pixel information; the pixel information includes a binding point area pixel border and binding point pixel coordinates;

[0134] A point cloud data extraction module, used to extract point cloud data of the binding point area according to the pixel information;

[0135] A registration module is used to set a template point cloud, apply the template point cloud to register the point cloud data of the binding point area, and obtain a transformation matrix between the actual binding point point cloud and the template point cloud;

[0136] The binding posture determination module is used to determine the binding posture of the steel bars according to the transformation matrix, so as to guide the robot arm to bind the steel bars according to the binding posture.

[0137] Based on the above embodiment, as a preferred embodiment, the pixel extraction module is a module for performing the following steps:

[0138] Using the key point detection model, a rectangular frame and a key point are marked for each binding point target in the color image; the width and height of the rectangular frame are positively correlated with the diameter of the steel bar; the key point is the cross center point of the steel bar;

[0139] The pixel border of the binding point area is determined according to the rectangular border, and the pixel coordinates of the binding point are determined according to the key points.

[0140] Based on the above embodiment, as a preferred embodiment, a key point detection model training module is further included, which is used to perform the following steps:

[0141] A color training image and a depth training image of untied steel bars are collected using a depth camera; the color training image contains visible light information of the steel bar scene; the depth training image contains distance information in the steel bar scene;

[0142] Preprocessing the color image by depth threshold segmentation to obtain image data containing only surface layer steel bar information;

[0143] Marking the cross center points of the steel bars in the image data to obtain marked data; in the marking process, the width and height of the binding point boundary box are the same;

[0144] The key point detection model is obtained by using the labeled data to apply a convolutional neural network for training.

[0145] Based on the above embodiment, as a preferred embodiment, the point cloud data extraction module includes:

[0146] A parameter determination unit, used to determine the shooting parameters of the camera; the shooting parameters include the horizontal focal length, the vertical focal length and the pixel coordinates of the optical center projection point on the imaging plane;

[0147] A coordinate determination unit, used to determine the pixel coordinates of the upper left corner and the lower right corner of the pixel frame of the binding point area according to the shooting parameters, and for any pixel point in the frame, determine the spatial coordinate value of the pixel point in the camera coordinate system according to the depth value in the depth image corresponding to the pixel point;

[0148] The point cloud extraction unit is used to traverse all pixel points within the pixel frame of the binding point area, obtain the spatial coordinates of all pixel points within the pixel frame in the camera coordinate system, and combine all spatial coordinate points to obtain the point cloud data of the binding point area.

[0149] Based on the above embodiment, as a preferred embodiment, the registration module includes:

[0150] A point cloud model building unit, used to build a point cloud model of a lashing point template;

[0151] A global registration unit, used for performing global registration of the point cloud model based on geometric features;

[0152] The local registration unit is used to perform local registration on the point cloud model based on adaptive displacement correction.

[0153] Based on the above embodiment, as a preferred embodiment, the point cloud model building unit is a unit for performing the following steps:

[0154] Two mutually perpendicular cylinders are used as template point clouds of the binding point cloud for registration; the two mutually perpendicular cylinders include a horizontal cylinder and a vertical cylinder perpendicular to the horizontal plane;

[0155] Determine a camera coordinate system; the X-axis of the camera coordinate system points to the right side of the camera, the Y-axis points to the top of the camera, and the Z-axis coincides with the optical axis of the camera;

[0156] The template point cloud is kept parallel to an XOY plane formed by the X-axis and the Y-axis; wherein the horizontal cylinder is kept parallel to the X-axis and passes through the origin of the camera coordinate system; and the vertical cylinder is kept parallel to the Y-axis.

[0157] Based on the above embodiment, as a preferred embodiment, the global registration unit is a unit for performing the following steps:

[0158] The real spatial coordinates of the lashing points are obtained by key point detection method, and compared with the spatial coordinates of the lashing points in the template to obtain the initial translation value;

[0159] The point cloud of actual lashing points is processed using point cloud plane fitting technology to determine the plane where the point cloud of actual lashing points is located;

[0160] Compare the plane where the actual binding point is located with the plane where the template point cloud is located to obtain the initial rotation value;

[0161] The point cloud model of the actual binding point is translated and rotated according to the initial translation value and the initial rotation value, so that the point cloud model of the actual binding point is aligned with the template point cloud.

[0162] Based on the above embodiment, as a preferred embodiment, the local registration unit is a unit for performing the following steps:

[0163] Use the set point cloud precise registration algorithm to perform precise registration of the scene point cloud and the template point cloud, and obtain the transformation matrix from the scene point cloud to the template point cloud;

[0164] Determining whether the transformation matrix satisfies a root mean square error threshold;

[0165] If not, apply multiple corrected displacement values ​​in the direction of the axis of the outer steel bar, calculate the root mean square error after the displacement correction, compare the root mean square errors in the positive and negative directions, and use the displacement correction corresponding to the smaller root mean square error value to re-execute the step of using the set point cloud precise registration algorithm to perform fine registration of the scene point cloud and the template point cloud, until the transformation matrix meets the root mean square error threshold, and obtain the final transformation matrix from the template coordinate system to the camera coordinate system.

[0166] Based on the above embodiment, as a preferred embodiment, the binding posture determination module is a module for performing the following steps:

[0167] Determine the lashing posture in the template coordinate system, the orientation of the lashing posture is -45° rotated around the Z axis, and the position is the center point of the intersection area of ​​the two cylinders on the outer edge cylinder;

[0168] Determine the transformation matrix between the camera coordinate system and the robot end coordinate system through hand-eye calibration technology;

[0169] According to the robot arm teaching device, the transformation matrix from the robot arm end coordinates to the base coordinate system is obtained;

[0170] The binding posture in the base coordinate system is obtained by using the transformation matrix from the end of the robot arm to the base coordinate system, the transformation matrix from the camera coordinate system to the end of the robot arm coordinate system, the transformation matrix from the template coordinate system to the camera coordinate system, and the binding posture in the template coordinate system.

[0171] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed, the steps provided in the above embodiment can be implemented. The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0172] The present application also provides an electronic device, see Fig.14 , a structural diagram of an electronic device provided in an embodiment of the present application, such as Fig.14 As shown, a processor 1410 and a memory 1420 may be included.

[0173] Among them, the processor 1410 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1410 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1410 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1410 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1410 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0174] The memory 1420 may include one or more computer-readable storage media, which may be non-transitory. The memory 1420 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 1420 is at least used to store the following computer program 1421, wherein, after the computer program is loaded and executed by the processor 1410, it can implement the relevant steps in the method performed by the electronic device side disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 1420 may also include an operating system 1422 and data 1423, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 1422 may include Windows, Linux, Android, etc.

[0175] In some embodiments, the electronic device may further include a display screen 1430 , an input / output interface 1440 , a communication interface 1450 , a sensor 1460 , a power source 1470 , and a communication bus 1480 .

[0176] certainly, Fig.14 The structure of the electronic device shown does not constitute a limitation on the electronic device in the embodiments of the present application. In actual applications, the electronic device may include Fig.14 More or fewer components than shown, or combinations of certain components.

[0177] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the system provided in the embodiment, since it corresponds to the method provided in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0178] Specific examples are used herein to illustrate the principles and implementation methods of the present application, and the description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.

[0179] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A method for tying steel bars, characterized in that: include: Obtain color images and depth images of the steel bar scene; Inputting the preprocessed color image into a key point detection model to obtain pixel information; The pixel information includes the pixel border of the binding point area and the pixel coordinates of the binding point; Extracting point cloud data of the binding point area according to the pixel information and the depth image; Setting a template point cloud, applying the template point cloud to align the point cloud data of the binding point area, and obtaining a transformation matrix between the actual binding point point cloud and the template point cloud; The binding posture of the steel bars is determined according to the transformation matrix, so as to guide the robot arm to bind the steel bars according to the binding posture.

2. The steel bar binding method according to claim 1, characterized in that: The inputting the preprocessed color image into the key point detection model to obtain pixel information includes: Using the key point detection model, a rectangular frame and a key point are marked for each binding point target in the color image; the width and height of the rectangular frame are positively correlated with the diameter of the steel bar; the key point is the cross center point of the steel bar; The pixel border of the binding point area is determined according to the rectangular border, and the pixel coordinates of the binding point are determined according to the key points.

3. The steel bar binding method according to claim 2, characterized in that: The generation process of the key point detection model includes: A color training image and a depth training image of untied steel bars are collected using a depth camera; the color training image contains visible light information of the steel bar scene; the depth training image contains distance information in the steel bar scene; Preprocessing the color image by depth threshold segmentation to obtain image data containing only surface layer steel bar information; Marking the cross center points of the steel bars in the image data to obtain marked data; in the marking process, the width and height of the binding point boundary box are the same; The key point detection model is obtained by training a convolutional neural network using the labeled data.

4. The steel bar binding method according to claim 3, characterized in that: The preprocessing of the color image by depth threshold segmentation to obtain image data containing only surface layer steel bar information includes: Assuming the distance d between the depth camera and the steel bar surface, according to the depth value information in the depth image, filter out the target coordinate points with depth values ​​in the range of [d-50, d+50]; In the color image, the color of the target coordinate point is set to a first color, and the colors of other points are set to a second color.

5. The steel bar binding method according to claim 1, characterized in that: The extracting of point cloud data of the binding point area according to the pixel information comprises: Determine the shooting parameters of the camera; the shooting parameters include the horizontal focal length, the vertical focal length and the pixel coordinates of the optical center projection point on the imaging plane; Determine the pixel coordinates of the upper left corner and the lower right corner of the pixel border of the binding point area according to the prediction results of the key point detection model, and for any pixel point in the border, determine the spatial coordinate value of the pixel point in the camera coordinate system according to the depth value in the depth map corresponding to the pixel point and the shooting parameters; Traverse all the pixels in the pixel border of the binding point area, obtain the spatial coordinates of all the pixels in the pixel border in the camera coordinate system, and combine all the spatial coordinate points to obtain the point cloud data of the binding point area.

6. The steel bar binding method according to claim 1, characterized in that: The applying the template point cloud to register the point cloud data of the binding point area comprises: Construct a point cloud model of the lashing point template; Performing global registration on the point cloud model based on geometric features; The point cloud model is locally registered based on adaptive displacement correction.

7. The steel bar binding method according to claim 6, characterized in that: The point cloud model for constructing the lashing point template includes: Two mutually perpendicular cylinders are used as template point clouds of the binding point cloud for registration; the two mutually perpendicular cylinders include a horizontal cylinder and a vertical cylinder perpendicular to the horizontal plane; Determine a camera coordinate system; the X-axis of the camera coordinate system points to the right side of the camera, the Y-axis points to the top of the camera, and the Z-axis coincides with the optical axis of the camera; The template point cloud is kept parallel to the XOY plane formed by the X-axis and the Y-axis; wherein the horizontal cylinder is kept parallel to the X-axis and passes through the origin of the camera coordinate system; and the vertical cylinder is kept parallel to the Y-axis.

8. The steel bar binding method according to claim 6, characterized in that: The performing global registration of the point cloud model based on geometric features comprises: The real spatial coordinates of the lashing points are obtained by key point detection method, and compared with the spatial coordinates of the lashing points in the template to obtain the initial translation value; The point cloud of actual lashing points is processed using point cloud plane fitting technology to determine the plane where the point cloud of actual lashing points is located; Compare the plane where the actual binding point is located with the plane where the template point cloud is located to obtain the initial rotation value; The point cloud model of the actual binding point is translated and rotated according to the initial translation value and the initial rotation value, so that the point cloud model of the actual binding point is aligned with the template point cloud.

9. The steel bar binding method according to claim 6, characterized in that: The local registration of the point cloud model based on adaptive displacement correction includes: Use the set point cloud precise registration algorithm to perform precise registration of the scene point cloud and the template point cloud, and obtain the transformation matrix from the scene point cloud to the template point cloud; Determining whether the transformation matrix satisfies a root mean square error threshold; If not, apply multiple corrected displacement values ​​in the direction of the axis of the outer steel bar, calculate the root mean square error after the displacement correction, compare the root mean square errors in the positive and negative directions, and use the displacement correction corresponding to the smaller root mean square error value to re-execute the step of using the set point cloud precise registration algorithm to perform fine registration of the scene point cloud and the template point cloud, until the transformation matrix meets the root mean square error threshold, and obtain the final transformation matrix from the template coordinate system to the camera coordinate system.

10. The steel bar binding method according to claim 1, characterized in that: Determining the binding posture of the steel bars according to the transformation matrix includes: Determine the lashing posture in the template coordinate system, the orientation of the lashing posture is -45° rotated around the Z axis, and the position is the center point of the intersection area of ​​the two cylinders on the outer edge cylinder; Determine the transformation matrix between the camera coordinate system and the robot end coordinate system through hand-eye calibration technology; According to the robot arm teaching device, the transformation matrix from the robot arm end coordinates to the base coordinate system is obtained; The binding posture in the base coordinate system is obtained by using the transformation matrix from the end of the robot arm to the base coordinate system, the transformation matrix from the camera coordinate system to the end of the robot arm coordinate system, the transformation matrix from the template coordinate system to the camera coordinate system, and the binding posture in the template coordinate system.

11. A steel bar tying system, characterized in that: include: An image acquisition module, used to acquire color images and depth images of the steel bar scene; A pixel extraction module, used to input the preprocessed color image into a key point detection model to obtain pixel information; the pixel information includes a binding point area pixel border and binding point pixel coordinates; A point cloud data extraction module, used to extract point cloud data of the binding point area according to the pixel information and the depth image; A registration module is used to set a template point cloud, apply the template point cloud to register the point cloud data of the binding point area, and obtain a transformation matrix between the actual binding point point cloud and the template point cloud; The binding posture determination module is used to determine the binding posture of the steel bars according to the transformation matrix, so as to guide the robot arm to bind the steel bars according to the binding posture.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

13. An electronic device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps of the method according to any one of claims 1 to 10 are implemented.

Images