Steel bar framework size measuring method and device based on steel bar binding robot

Abstract

The invention provides a steel bar framework size measuring method and device based on a steel bar binding robot. The method comprises the steps that the steel bar binding robot is controlled to search for the boundary of a to-be-measured component; if the boundary of the to-be-detected component is searched, recording a first horizontal distance from the tail end of the mechanical arm to the base; the steel bar binding robot is controlled to search the upper boundary of the to-be-detected component upwards; calculating a first height from the upper boundary to the base; the steel bar binding robot is controlled to move rightwards to search for the right boundary of the to-be-detected component; if the right boundary is searched, recording a second horizontal distance from the tail end of the mechanical arm to the base; calculating the running distance of the steel bar binding robot; according to the first horizontal distance, the second horizontal distance, the first height, the running distance and the ground clearance of the base, the steel reinforcement framework size of the to-be-measured component is calculated, the calculated steel reinforcement framework size is applied to path planning of the steel reinforcement binding robot, and the path planning effect of the steel reinforcement binding robot is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for measuring the size of a steel bar skeleton based on a steel bar tying robot. Background Art

[0002] Rebar binding can effectively fix and enhance the stability of the steel bars, preventing them from shifting, deforming, and breaking during construction. After the steel bars are tied, they can be reasonably distributed and support the concrete structure, thereby ensuring the firmness and stability of the entire building structure.

[0003] With the development of rebar tying robots, rebar tying robots are gradually being used in sites and factories for automatic tying. However, when working, the rebar tying robots often plan while walking due to the lack of global information of the entire component, which leads to poor path planning effect of the rebar tying robots. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method and device for measuring the size of a steel bar skeleton based on a steel bar tying robot, so as to solve the problem that the path planning effect of the steel bar tying robot is poor when the steel bar tying robot plans while working.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] A first aspect of an embodiment of the present invention discloses a method for measuring the size of a steel bar skeleton based on a steel bar tying robot, the method comprising:

[0007] Controlling a steel bar tying robot to search for the boundary of a component to be tested, the steel bar tying robot at least comprising: a mechanical arm, a base, and a camera arranged on the mechanical arm;

[0008] If the boundary of the component to be measured is found, a first horizontal distance between the end of the robot arm and the base is recorded;

[0009] Controlling the steel bar tying robot to search upward for the upper boundary of the component to be tested;

[0010] Calculating a first height of the upper boundary from the base;

[0011] Controlling the steel bar tying robot to move rightward to search for the right boundary of the component to be tested;

[0012] If the right boundary is found, a second horizontal distance between the end of the robot arm and the base is recorded;

[0013] Calculating the running distance of the steel bar tying robot;

[0014] The steel skeleton size of the component to be measured is calculated according to the first horizontal distance, the second horizontal distance, the first height, the running distance, and the height of the base from the ground.

[0015] Preferably, controlling the steel bar tying robot to search for the boundary of the component to be tested comprises:

[0016] Controlling the steel bar tying robot to move along the edge of the component to be measured from one side of the component to be measured, and shooting a first video stream of the component to be measured in real time;

[0017] Inputting the first image of the first video stream into a preset boundary recognition model for recognition to obtain a category of the first image, wherein the boundary recognition model is obtained by training a neural network model based on a sample data set, and the category of the first image is boundary, corner point, interior, or non-reinforced cage;

[0018] If the category of the first image is identified as a boundary, it is determined that the boundary of the component to be detected is found.

[0019] Preferably, controlling the steel bar tying robot to search upward for the upper boundary of the component to be tested comprises:

[0020] Controlling the mechanical arm of the steel bar tying robot to move upward, and collecting a second video stream of the component to be tested in real time;

[0021] Inputting the second image of the second video stream into a preset boundary recognition model for recognition to obtain a category of the second image, wherein the boundary recognition model is obtained by training a neural network model based on a sample data set, and the category of the second image is boundary, corner point, interior, or non-reinforced cage;

[0022] If the category of the second image is identified as a corner point, it is determined that the upper boundary of the component to be measured is found.

[0023] Preferably, controlling the steel bar tying robot to move rightward to search for the right boundary of the component to be tested comprises:

[0024] Controlling the steel bar tying robot to move rightward, and collecting a third video stream of the component to be tested in real time;

[0025] Inputting the third image of the third video stream into a preset boundary recognition model for recognition to obtain a category of the third image, wherein the boundary recognition model is obtained by training a neural network model based on a sample data set, and the category of the third image is boundary, corner point, interior, or non-reinforced cage;

[0026] If the category of the third image is identified as a corner point, it is determined that the right boundary of the component to be measured is found.

[0027] Preferably, calculating the first height of the upper boundary from the base includes:

[0028] parsing the second image to obtain pixel coordinates of the upper boundary;

[0029] Convert the pixel coordinates of the upper boundary to a camera coordinate system corresponding to a camera disposed on the robotic arm;

[0030] Convert the pixel coordinates of the upper boundary converted to the camera coordinate system to the robot base coordinate system corresponding to the robot arm;

[0031] A first height of the upper boundary from the base is extracted from the pixel coordinates of the upper boundary that have been converted into the robot base coordinate system.

[0032] Preferably, calculating the running distance of the steel bar tying robot comprises:

[0033] Based on the visual inertial ranging algorithm, the running distance of the steel bar tying robot is calculated.

[0034] A second aspect of an embodiment of the present invention discloses a steel bar skeleton size measuring device based on a steel bar tying robot, the device comprising:

[0035] A first control unit is used to control a steel bar tying robot to search for a boundary of a component to be tested, wherein the steel bar tying robot comprises at least: a mechanical arm, a base, and a camera arranged on the mechanical arm;

[0036] A first recording unit, configured to record a first horizontal distance between the end of the robot arm and the base if a boundary of the component to be measured is found;

[0037] A second control unit is used to control the steel bar tying robot to search upward for the upper boundary of the component to be tested;

[0038] A first calculation unit, used for calculating a first height of the upper boundary from the base;

[0039] A third control unit is used to control the steel bar tying robot to move rightward to search for the right boundary of the component to be tested;

[0040] A second recording unit, for recording a second horizontal distance between the end of the robot arm and the base if the right boundary is found;

[0041] A second calculation unit, used for calculating the running distance of the steel bar tying robot;

[0042] The third calculation unit is used to calculate the steel skeleton size of the component to be measured according to the first horizontal distance, the second horizontal distance, the first height, the running distance, and the height of the base from the ground.

[0043] Preferably, the first control unit includes:

[0044] A first control module is used to control the steel bar tying robot to move along the edge of the component to be measured from one side of the component to be measured, and to shoot a first video stream of the component to be measured in real time;

[0045] A first recognition module is used to input the first image of the first video stream into a preset boundary recognition model for recognition to obtain a category of the first image, wherein the boundary recognition model is obtained by training a neural network model based on a sample data set, and the category of the first image is boundary, corner point, interior, or non-reinforced cage;

[0046] The first determination module is used to determine that a boundary of the component to be detected has been searched if the category of the first image is identified as a boundary.

[0047] Preferably, the second control unit comprises:

[0048] A second control module is used to control the mechanical arm of the steel bar tying robot to move upward and collect a second video stream of the component to be tested in real time;

[0049] A second recognition module is used to input the second image of the second video stream into a preset boundary recognition model for recognition to obtain a category of the second image, wherein the boundary recognition model is obtained by training a neural network model based on a sample data set, and the category of the second image is boundary, corner point, interior or non-reinforced cage;

[0050] The second determination module is used to determine that the upper boundary of the component to be measured is searched if the category of the second image is identified as a corner point.

[0051] Preferably, the third control unit includes:

[0052] A third control module is used to control the steel bar tying robot to move rightward and collect a third video stream of the component to be tested in real time;

[0053] A third recognition module is used to input the third image of the third video stream into a preset boundary recognition model for recognition to obtain a category of the third image, wherein the boundary recognition model is obtained by training a neural network model based on a sample data set, and the category of the third image is boundary, corner point, interior or non-reinforced cage;

[0054] The third determination module is used to determine that the right boundary of the component to be measured is found if the category of the third image is identified as a corner point.

[0055] Based on the above-mentioned embodiment of the present invention, a steel bar skeleton size measurement method and device based on a steel bar binding robot is provided. The method is as follows: control the steel bar binding robot to search for the boundary of the component to be measured; if the boundary of the component to be measured is found, record the first horizontal distance between the end of the mechanical arm and the base; control the steel bar binding robot to search upward for the upper boundary of the component to be measured; calculate the first height of the upper boundary from the base; control the steel bar binding robot to move right to search for the right boundary of the component to be measured; if the right boundary is found, record the second horizontal distance between the end of the mechanical arm and the base; calculate the running distance of the steel bar binding robot; according to the first horizontal distance, the second horizontal distance, the first height, the running distance, and the height of the base from the ground, calculate the steel bar skeleton size of the component to be measured. In this scheme, the steel bar binding robot is controlled to search for the boundary, upper boundary and right boundary of the component to be measured, and the corresponding horizontal distance and running distance are obtained. The steel bar skeleton size of the component to be measured is determined by the obtained horizontal distance and running distance, and the determined steel bar skeleton size is applied to the path planning of the steel bar binding robot to improve the path planning effect of the steel bar binding robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0057] Figure 1 A diagram showing the principle of a hand-eye calibration method provided by an embodiment of the present invention;

[0058] FIG2(a) is an example of an image of a boundary category provided by an embodiment of the present invention; FIG2(b) is an example of an image of a corner category provided by an embodiment of the present invention; FIG2(c) is an example of an image of an interior category provided by an embodiment of the present invention;

[0059] Figure 3 An example diagram of segmenting a steel cage provided by an embodiment of the present invention;

[0060] Figure 4 An example diagram of the recognition effect of the boundary recognition model provided by the embodiment of the present invention;

[0061] Figure 5 An example diagram of the iterative least squares fitting method for reducing the influence of mutation positions provided for the implementation of the present invention;

[0062] Figure 6 A flowchart of a method for measuring the size of a steel bar skeleton based on a steel bar tying robot provided in an embodiment of the present invention;

[0063] Figure 7 An example diagram of the relationship between a pixel coordinate system and an image coordinate system provided in an embodiment of the present invention;

[0064] Figure 8 An example diagram of the relationship between the camera coordinate system and the image coordinate system provided in an embodiment of the present invention;

[0065] Fig. 9 An example diagram of a motion trajectory provided by an embodiment of the present invention;

[0066] Fig.10 Another flow chart of a method for measuring the size of a steel bar skeleton based on a steel bar tying robot provided in an embodiment of the present invention;

[0067] Fig.11 A structural block diagram of a steel bar skeleton size measuring device based on a steel bar tying robot provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0070] As can be seen from the background technology, steel bar binding can effectively fix and enhance the stability of steel bars, and prevent the steel bars from shifting, deforming and breaking during the construction process; after the steel bars are tied, the steel bars can be reasonably distributed and support the concrete structure, thereby ensuring the firmness and stability of the entire building structure.

[0071] The traditional form of steel bar binding is time-consuming and labor-intensive manual binding. With the development of steel bar binding robots, steel bar binding robots are gradually being used on site and in factories for automatic binding. However, the steel bar binding robot is only used for plane plate reinforcement binding and lacks global information of the entire component. When binding components of unknown size in an unknown environment, the steel bar binding robot often plans while walking, which results in poor path planning of the steel bar binding robot.

[0072] To solve the above problems, this scheme proposes a steel bar skeleton size measurement method and device based on a steel bar tying robot, which controls the steel bar tying robot to search for the boundary, upper boundary and right boundary of the component to be measured, and obtains the corresponding horizontal distance and running distance. The steel bar skeleton size of the component to be measured is determined by the obtained horizontal distance and running distance, and the determined steel bar skeleton size is applied to the path planning of the steel bar tying robot, thereby improving the path planning effect of the steel bar tying robot.

[0073] In practical applications, this scheme can estimate the size of components of unknown size with regular working surfaces, and provide the steel skeleton size for the steel tying robot in subsequent steel tying work, thereby improving the path planning effect of the steel tying robot and optimizing the path planning effect of the steel tying robot and the spacing step length of the movement during tying.

[0074] For example: When the rebar tying robot is tying the rebars of a vertically placed component (the side of the component is a rectangle), it can estimate the size of the vertically placed component and provide the rebar skeleton size of the vertically placed component to the rebar tying robot in subsequent rebar tying work.

[0075] In order to better understand the contents of this plan, some of the terms involved in this plan are explained here.

[0076] Rebar tying robot: A base (i.e., the chassis of the mobile vehicle) is used to carry a robotic arm, which is controlled to perform rebar tying. A camera (such as an RGB-D binocular camera) and a rebar tying device are installed at the end of the robotic arm.

[0077] Image classification: The process of identifying the category to which an image belongs from a given image.

[0078] Semantic segmentation: Assigning each pixel in an image to different semantic categories to understand the scene and content of the image.

[0079] ISAT Annotation Tool: An interactive semi-automatic image segmentation and annotation tool that integrates segment-anything technology to quickly perform image segmentation and annotation.

[0080] XLD (Extended Line Descriptions): XLD is a contour function, usually called sub-pixel; XLD is extracted not along the intersection of pixel and pixel boundaries, but the position (a description) obtained by Halcon after some calculation, where Threshold is the threshold for extracting XLD, and the grayscale value is extracted based on the threshold.

[0081] Halcon operator fit_line_contour_xld: The function is fit_line_contour_xld(Contours::Algorithm,MaxNumPoints,ClippingEndPoints,Iterations,ClippingFactor:RowBegin,ColBegin,RowEnd,ColEnd,Nr,Nc,Dist), which performs approximate straight line calculation on the XLD of some line segments.

[0082] Some types in the parameter Algorithm are explained as follows:

[0083] regression: regression, standard least squares fitting;

[0084] huber: weighted least squares fitting, the influence of outliers is reduced based on the Huber method;

[0085] tukey: weighted least squares fitting, the influence of outliers is reduced based on Tukey's method;

[0086] drop: weighted least squares fitting, the influence of outliers is eliminated;

[0087] gauss: weighted least squares fitting, the influence of outliers is reduced based on the mean and standard deviation of the distance of all its contour points on the closest approximation line;

[0088] ClippingFactor controls the number of outliers that are weakened. The smaller the value, the more outliers are detected;

[0089] The straight line fitting method can fit the main steel bar contour at the boundary position in a weighted manner while ignoring the abnormal protrusions caused by the lap joints.

[0090] Weighted iterative least squares fitting: Iteratively reweighted least squares (also called iteratively reweighted least squares).

[0091] RANSAC fitting method: Random sample consensus (RANSAC), RANSAC is a robust model fitting method that can handle data with a lot of noise and outliers.

[0092] Visual-inertial odometry algorithm: VIO (visual-inertialodometry), also known as visual-inertial system (VINS), is an algorithm that integrates camera and IMU data to achieve SLAM.

[0093] Eye-in-hand calibration: Eye-in-hand calibration method; such as Figure 1 The principle example diagram of the hand-eye calibration method shown in the figure can be seen. Figure 1 It includes the following coordinate systems: the basic coordinate system (expressed as base) is the base coordinate system of the robot arm (called the robot base coordinate system), the end coordinate system (expressed as end) is the end coordinate system of the robot arm (called the robot end coordinate system), the camera coordinate system (expressed as cam) is the coordinate system of the camera fixed on the robot arm, and the calibration object coordinate system (expressed as cal) is the coordinate system where the calibration plate is located.

[0094] Move the robot arm twice arbitrarily. Since the calibration plate and the base of the robot arm are stationary, for a certain world point, its coordinate value in the base coordinate system and the cal coordinate system remains unchanged, while its coordinate value in the end coordinate system and the cam coordinate system changes with the movement of the robot arm. Based on the above relationship, the transformation matrix between the end coordinate system and the cam coordinate system can be solved.

[0095] The above content is an explanation of some of the terms involved in this plan.

[0096] It is worth noting that in the process of implementing this solution, when controlling the movement of the steel bar binding robot, a video stream of the component to be tested (such as a steel bar cage) will be collected in real time, and then the category of each image in the video stream will be identified through the pre-trained boundary recognition model. The category of each image is boundary, corner point, interior or non-steel cage; that is, the boundary recognition model applied in this solution can identify the boundary, corner point, interior and non-steel cage of the steel bar cage, and the boundary recognition model is obtained by training a neural network model based on a sample data set.

[0097] In order to better understand the main content of this subsequent solution, the following first explains the process of how to train the boundary recognition model; among them, the source of the sample data set used for training the boundary recognition model is: prefabricated components (such as steel cages) placed vertically in the factory; when the prefabricated components are processed, a steel positioning mold is used to form a steel skeleton with binding.

[0098] Explanation of the process of training the boundary recognition model:

[0099] (1) Image collection:

[0100] The local images of the steel cage are taken, and the images (RGB images) obtained are divided into three categories: boundary, corner, and inner. The images obtained by taking the steel cage and the images of the non-steel cage are combined to construct a sample data set. That is, the sample data set contains four categories of images: boundary, corner, inner, and non-steel cage.

[0101] The images in the sample data set are enhanced by one or more of the following data enhancement methods: scaling the image by a certain proportion, cropping the image at a random position, randomly flipping the image horizontally and vertically, rotating the image at a random angle, and randomly changing the brightness, contrast, and color of the image.

[0102] After image enhancement processing is performed on the images in the sample data set, the images in the sample data set are classified and divided into images with boundary categories, images with corner categories, images with internal categories, and images with non-reinforced cage categories.

[0103] The specific rules for classifying images in the sample data set are as follows:

[0104] Images classified as boundaries: only one edge is >1 / 6 away from the edge of the image;

[0105] Images with the corner category: the distance between the two margins and the edge of the image is >1 / 6;

[0106] Images classified as internal: any margin is < 1 / 6 from the edge of the image.

[0107] Taking into account the accuracy of determining the categories of images in the sample data set, for images with a small amount of white space at the boundary, it is still determined that the line of sight in the direction corresponding to the image is located inside, which can strengthen the feature determination of boundaries and corner points.

[0108] For example, Figure 2 (a) is an image of the boundary category obtained by photographing the steel cage, Figure 2 (b) is an image of the corner category obtained by photographing the steel cage, and Figure 2 (c) is an image of the interior category obtained by photographing the steel cage.

[0109] (2) Classify the images in the sample dataset based on deep learning and roughly find the boundary location:

[0110] The ISAT annotation tool is used to divide the steel bar parts in the images in the sample data set. The label of the steel bar part is set to rebar, and the mask mode is to take the maximum outer contour. After the above processing, the initial label file is obtained in json format. After processing all the images in the sample data set, the label format is converted into the required format such as coco, VOC, yolo, etc.

[0111] For example Figure 3 As shown in the example figure of segmenting the steel cage, the ISAT annotation tool is used to segment the steel bar part in the image in the sample data set, and the label of the steel bar part is set to rebar.

[0112] According to the ratio of "8:2", the sample data set is divided into a training set and a validation set. The neural network model is trained and validated using the image classification algorithm, deep learning algorithm, the labels of each image in the training set, and the labels of each image in the validation set to obtain the optimal weight parameters, thereby obtaining a boundary recognition model that can distinguish the category of the image; through the boundary recognition model, the part of the steel cage where the camera's line of sight is located can be identified, and the boundary of the steel cage can be found.

[0113] Specifically, the image is input into a boundary recognition model, which outputs the category of the image, which is boundary, corner, inner, or non-reinforced cage.

[0114] like Figure 4 As shown in the example diagram of the recognition effect of the boundary recognition model, multiple images are input into the boundary recognition model, and the boundary recognition model outputs the category of each image. The category of a recognized image is boundary, corner or inner.

[0115] (3) Combine the depth map with depth filtering to obtain shallow steel bars:

[0116] When using the boundary recognition model to identify the category of an image, due to the influence of the shooting angle, the rear row of steel bars may appear at the boundary position, which will affect the judgment of the boundary position; therefore, when using the boundary recognition model to identify the category of an image, it is necessary to process the image (depth image) taken by the camera (RGB camera).

[0117] Specifically, the depth information of the location of the mask predicted by semantic segmentation is extracted in the depth image, the depth information is converted into spatial point cloud information using open3d, and the RANSAC multi-plane fitting method is used to obtain the front and back two layers of steel bar planes.

[0118] The expressions of the RANSAC multi-plane fitting method are shown in formula (1) and formula (2).

[0119] (1)

[0120] (2)

[0121] Substituting formula (2) into formula (1) yields formula (3).

[0122] (3)

[0123] In formula (1) to formula (3), x, y and z are the three-dimensional coordinates of the point cloud obtained by the depth image relative to the camera, and A, B, C and D are the coefficients of the fitting plane. The steel bar plane closest to the camera coordinate origin can be obtained through the above formula (1) to formula (3).

[0124] If there are n points (n is greater than 3), the specific process of fitting the plane equation through these n points is:

[0125] 1. Randomly select 3 points from the point cloud data obtained by depth image conversion. These 3 randomly selected points constitute a hypothetical plane; calculate the distance from other points (except the 3 randomly selected points) to the screen; if the distance from any other point to the plane is less than the error threshold, the other point is marked as an inlier;

[0126] 2. Repeat step 1 above; if the number of inliers found in this round exceeds the largest number of inliers found previously, the optimal plane is updated to the plane obtained by fitting in this round; after completing the set number of iterations, the final optimal plane is obtained. The final optimal plane is: find the plane with the most inliers.

[0127] It should be noted that after a depth image is converted into a point cloud, there may be multiple fitted planes. The distance from the origin of the camera coordinate system to each plane is calculated separately. The plane closest to the origin of the camera coordinate system is the first layer of steel bars.

[0128] Extract the coordinates of the first layer of steel bars, and then perform reverse calculation to obtain the location of the first layer of steel bars on the depth image;

[0129] The boundary position in the depth image is found through image segmentation, that is, the segmented boundary position is extracted; wherein, the standard for judging the boundary position (equivalent to the boundary judgment method) is: respectively intercepting several rows or columns of pixels of the four boundaries of the depth image, if there are no pixels judged as steel bars, it can be considered that a boundary appears on this side.

[0130] (4) Dealing with the problem of convex boundaries:

[0131] It should be noted that iteratively reweighted least squares (IRLS) is an iterative algorithm for solving weighted least squares problems, which is often used to deal with regression problems with heteroscedasticity and robust regression problems. IRLS converts the problem into a series of weighted least squares problems by iteratively updating weights.

[0132] According to iterative weighted least squares fitting, the influence of bulges at the reinforcement joints is avoided, for example Figure 5The example figure showing how iterative least squares fitting reduces the influence of mutation positions shows that the influence of bulges at the steel bar lap joints can be effectively avoided by iterative weighted least squares fitting.

[0133] The above (1)-(4) are explanations about the boundary recognition model and other contents.

[0134] See also Figure 6 , shows a flow chart of a steel bar skeleton size measurement method based on a steel bar tying robot provided by an embodiment of the present invention, the steel bar skeleton size measurement method comprising:

[0135] Step S601: Control the steel bar tying robot to search for the boundary of the component to be tested.

[0136] It should be noted that the component to be tested is a steel cage, and the steel bar tying robot includes at least: a mechanical arm, a base (such as the chassis of a mobile vehicle), and a camera (such as an RGB-D binocular camera) arranged on the mechanical arm.

[0137] In the process of implementing step S601, the camera on the mechanical arm of the rebar tying robot is directed toward the component to be measured, and the rebar tying robot is controlled to move along the edge of the component to be measured starting from one side (such as the left side) of the component to be measured, and the first video stream of the component to be measured is captured in real time. The first video stream can be used to analyze which part of the component to be measured is within the current field of view of the camera.

[0138] Specifically, the mobile trolley under the base of the steel bar tying robot is controlled to move along the edge of the component to be tested from one side (such as the left side) of the component to be tested, and the camera on the mechanical arm of the steel bar tying robot is controlled to capture the first video stream of the component to be tested.

[0139] The first image of the first video stream is input into a preset boundary recognition model for recognition to obtain a category of the first image, where the category of the first image is boundary, corner point, interior or non-reinforced cage; wherein the boundary recognition model is obtained by training a neural network model based on a sample data set, and the principle of the boundary recognition model has been described in detail in the above content and will not be repeated here.

[0140] If the category of the first image is identified as a boundary (the category of the boundary can be recorded as category 0), it is determined that the boundary of the component to be tested is found, and step S602 is executed.

[0141] That is to say, in the process of controlling the rebar tying robot to move along the edge of the component to be tested starting from one side (such as the left side) of the component to be tested, the boundary recognition model is used to parse the first video stream shot in real time. If the boundary is recognized, it is determined that the rebar tying robot is in a horizontal boundary position.

[0142] In some embodiments, during the process of parsing the first video stream shot in real time, if the boundary of the component to be measured is not found, it is determined whether the picture in the first image is inside or outside the component to be measured;

[0143] If the picture in the first image is the interior of the component to be measured, the steel bar tying robot is controlled to move leftward, and the process returns to step S601 to continue searching for the boundary of the component to be measured;

[0144] If the picture in the first image is the exterior of the component to be measured, the steel bar tying robot is controlled to move rightward and returns to execute step S601 to continue searching for the boundary of the component to be measured.

[0145] It should be noted that, when executing step S601, the steel bar tying robot starts from the left side of the component to be measured and moves along the edge of the component to be measured, so the boundary searched in step S601 is the left boundary.

[0146] Step S602: If the boundary of the component to be measured is found, a first horizontal distance between the end of the robot arm and the base is recorded.

[0147] In the specific implementation of step S602, if the boundary of the component to be measured is found, the first horizontal distance between the end of the mechanical arm of the steel bar tying robot and the base at the moment of "finding the boundary of the component to be measured" is calculated and recorded.

[0148] In some embodiments, it can be seen from the content of the above step S601 that the steel bar binding robot starts from the left side of the component to be measured and moves along the edge of the component to be measured, then the boundary searched in step S601 is the left boundary; the specific method of calculating the first horizontal distance is: through the position of the left boundary in the first image, combined with the camera parameters of the camera on the robotic arm, the horizontal distance from the left boundary to the base is converted, and the horizontal distance from the left boundary to the base is the first horizontal distance (denoted as x1).

[0149] Step S603: Control the steel bar binding robot to search upward for the upper boundary of the component to be tested.

[0150] In the specific implementation of step S603, after searching for the boundary of the component to be measured and recording the first horizontal distance, the mechanical arm of the steel bar tying robot is controlled to move upward, and the second video stream of the component to be measured is collected in real time;

[0151] Inputting the second image of the second video stream into the boundary recognition model for recognition to obtain a category of the second image, wherein the category of the second image is boundary, corner point, interior, or non-reinforced cage;

[0152] If the category of the second image is identified as a corner point (the category of the corner point can be recorded as category 1), it is determined that the upper boundary of the component to be measured is searched.

[0153] That is, after searching for the boundary of the component to be measured and recording the first horizontal distance, the mechanical arm of the steel bar tying robot is controlled to move upward to continue searching for the corner point, and during the upward movement of the mechanical arm, the camera on the mechanical arm is controlled to capture the second video stream of the component to be measured in real time;

[0154] The boundary recognition model is used to analyze the second video stream shot in real time. If a corner point is recognized, it is determined that the camera is at a corner point position, and the upper boundary of the component to be measured is determined to be searched;

[0155] If it is identified that the category of the second image is not a corner point (equivalent to not finding the upper boundary), the camera on the robot arm is moved upward and the second video stream is continuously analyzed to continue searching for the upper boundary.

[0156] Step S604: Calculate a first height of the upper boundary from the base.

[0157] In the specific implementation of step S604, after controlling the robot arm to move upward to search for the upper boundary of the component to be measured, the second image is parsed to obtain the pixel coordinates of the upper boundary (coordinates in the pixel coordinate system);

[0158] The pixel coordinates of the upper boundary are converted into a camera coordinate system corresponding to a camera disposed on the robot arm. Specifically, the pixel coordinates of the upper boundary are first converted into an image coordinate system, and then the pixel coordinates of the upper boundary converted into the image coordinate system are converted into the camera coordinate system;

[0159] The pixel coordinates of the upper boundary that have been converted to the camera coordinate system are converted to the robot base coordinate system corresponding to the robot arm;

[0160] From the pixel coordinates of the upper boundary converted to the robot base coordinate system, extract the first height of the upper boundary from the base. The first height of the upper boundary of the component to be measured from the base is: the y coordinate of the pixel coordinates of the upper boundary converted to the robot base coordinate system; the first height is recorded as y1.

[0161] More specifically, after the mechanical arm of the steel bar binding robot is controlled to move upward and the corner point is found (the corner point is identified from the second image), the second image with the corner point identified is semantically segmented to obtain the target steel bar, and the position of the steel bar at the horizontal and vertical boundaries of the component to be tested is obtained through the "boundary determination method" mentioned in the above content about the boundary recognition model. The depth information of the second image is then used to calculate the three-dimensional coordinates of the steel bar at the upper boundary of the component to be tested relative to the camera coordinate system (equivalent to the pixel coordinates converted to the camera coordinate system), and then the three-dimensional coordinates are converted to the mechanical arm base coordinate system, thereby obtaining the above-mentioned "pixel coordinates of the upper boundary converted to the mechanical arm base coordinate system".

[0162] In the process of obtaining the pixel coordinates of the upper boundary that have been converted to the robot base coordinate system, the relationship between the pixel coordinate system and the image coordinate system is as follows: Figure 7 As shown in the figure, the relationship between the camera coordinate system and the image coordinate system is shown in Figure 8 As shown; Figure 7 and Figure 8 In, O p is the origin of the image coordinate system, u0 and v0 represent O p The coordinates in the pixel coordinate system, P is any point, O C is the origin of the camera coordinate system, P C is any point in the camera coordinate system, x c ,y c and z c is the camera coordinate.

[0163] Based on the aforementioned “relationship between the pixel coordinate system and the image coordinate system” and “relationship between the camera coordinate system and the image coordinate system”, the pixel coordinates of the upper boundary are converted to the camera coordinate system using the pixel coordinate system and camera coordinate system conversion equation provided by formula (4).

[0164] (4)

[0165] In formula (4), u and v are pixel coordinates, X, Y, and Z are camera coordinates, dx and dy indicate how many millimeters a pixel is in each row (or column), and f is the focal length; in other words, dx and dy are equivalent to the conversion ratio between pixels and actual size.

[0166] After the pixel coordinates of the upper boundary are converted to the camera coordinate system, the pixel coordinates of the upper boundary that have been converted to the camera coordinate system are converted to the robot arm base coordinate system through the hand-eye calibration (eye on hand) calculation method.

[0167] Specifically, the specific method of converting the coordinates in the camera coordinate system to the robot base coordinate system is: first perform hand-eye calibration on a point p=[x, y, z] in the camera coordinate system to obtain , The transformation matrix from the camera coordinate system to the robot base coordinate system;

[0168] The coordinates of the object to be measured in the robot base coordinate system are denoted as p', then p'= *p; where = * , is the transformation matrix from the robot end coordinate system to the robot base coordinate system, It is the transformation matrix from the camera coordinate system to the robot end coordinate system.

[0169] Step S605: Control the steel bar tying robot to move rightward to search for the right boundary of the component to be tested.

[0170] In the specific implementation of step S605, after obtaining the first height of the upper boundary from the base, the steel bar tying robot is controlled to move rightward, and a third video stream of the component to be tested is collected in real time;

[0171] Inputting the third image of the third video stream into a preset boundary recognition model for recognition to obtain a category of the third image, where the category of the third image is boundary, corner point, interior, or non-reinforced cage;

[0172] If the category of the third image is identified as a corner point, it is determined that the right boundary of the component to be measured is found.

[0173] Specifically, the mobile cart under the base of the rebar tying robot is controlled to move to the right, and the camera on the mechanical arm of the rebar tying robot is controlled to shoot the third video stream of the component to be tested, and then the boundary recognition model is used to identify the third image in the third video stream. If the category of the third image is identified as a corner point, it is determined that the right boundary of the component to be tested has been found.

[0174] If the right boundary of the component to be tested is found, execute step S606; if the right boundary of the component to be tested is not found, control the steel bar binding robot to continue moving to the right and continue analyzing the third video stream to continue searching for the right boundary.

[0175] Through the processing of the above steps S603 and S605, the two corner points located at the upper boundary of the upper part of the component to be measured can be found, and then the positions of the two corner points relative to the robot base coordinate system can be found.

[0176] Step S606: If the right boundary is found, record the second horizontal distance between the end of the robot arm and the base.

[0177] In the specific implementation of step S606, if the right boundary is found, the second horizontal distance between the end of the robot arm and the base is calculated and recorded.

[0178] In some embodiments, the specific method for calculating the second horizontal distance is: through the position of the right boundary in the third image, combined with the camera parameters of the camera on the robotic arm, the horizontal distance from the right boundary to the base is converted, and the horizontal distance from the right boundary to the base is the second horizontal distance (denoted as x2).

[0179] Step S607: Calculate the running distance of the steel bar tying robot.

[0180] In the specific implementation of step S607, when the rebar tying robot searches from the left boundary to the right boundary of the component to be tested, the running distance (denoted as x3) of the mobile trolley under the base of the rebar tying robot is calculated based on the visual inertial ranging algorithm (vio).

[0181] Specifically, the visual inertial odometry algorithm and the SLAM algorithm are used for local mapping. The initial position of the mobile car under the base of the steel bar tying robot is used as the origin, and the robot base coordinate system is used as the local map coordinate system. The visual inertial odometry algorithm is used to estimate the running distance of the mobile car. The process is as follows (the MSCKF algorithm is used as an example):

[0182] 1. Initialize camera parameters, MSCKF parameters, and construct MSCKF state vector;

[0183] 2. Read IMU data and estimate new MSCKF state variables and corresponding covariance matrices;

[0184] 3. Add the camera pose of the current frame to the MSCKF state vector. If the number of poses is greater than the range of the sliding window size, remove the camera pose corresponding to the earliest view in the state variable;

[0185] 4. Extract image features and match them to remove outliers;

[0186] 5. Process all extracted features;

[0187] 6. Loop through the tracks in featureTracksToResidualize to update the state variables of MSCKF;

[0188] 7. Calculate the covariance matrix of the new MSCKF state vector;

[0189] 8. Draw Fig. 9 The motion trajectory shown in FIG. 1 is used to calculate the running distance between the starting point and the end point of the moving car. It should be noted that for Fig. 9 The different colors on the motion trajectory shown change according to the color scale. Different colors represent errors of different sizes in the motion trajectory. Red indicates the largest error and blue indicates the smallest error.

[0190] Step S608: Calculate the steel skeleton size of the component to be measured according to the first horizontal distance, the second horizontal distance, the first height, the running distance, and the height of the base from the ground.

[0191] It should be noted that the steel bar skeleton dimensions of the component to be tested include height and length.

[0192] In the process of implementing step S608, after obtaining the first horizontal distance (denoted as x1), the second horizontal distance (denoted as x2), the first height (denoted as y1), and the running distance (denoted as x3) through the above steps S601-S607, the height of the base of the steel bar tying robot from the ground (denoted as y2) is obtained.

[0193] The sum of the first horizontal distance, the second horizontal distance and the running distance is calculated to obtain the length of the component to be measured; that is, the length of the component to be measured = x1 + x2 + x3.

[0194] The sum of the first height and the height of the base from the ground is calculated to obtain the height of the component to be measured; that is, the height of the component to be measured = y1 + y2.

[0195] The above steps S601 to S608 are explanations on how to measure the steel bar skeleton size of the component to be measured;

[0196] In practical applications, the steel bar tying robot can search from the left boundary to the right boundary of the component to be tested through the above-mentioned embodiment of the present invention. Figure 6 Similarly, the steel bar tying robot can also use the above-mentioned embodiment of the present invention to measure the steel bar skeleton size of the component to be tested from the right boundary to the left boundary of the component to be tested. Figure 6 The content in is used to measure the steel skeleton size of the component to be tested, and the implementation principle is similar.

[0197] In the embodiment of the present invention, the steel bar tying robot is controlled to search for the boundary, upper boundary and right boundary of the component to be measured, and obtain the corresponding horizontal distance and running distance. The steel bar skeleton size of the component to be measured is determined by the obtained horizontal distance and running distance, and the determined steel bar skeleton size is applied to the path planning of the steel bar tying robot, thereby improving the path planning effect of the steel bar tying robot.

[0198] To better understand the above embodiments of the present invention Figure 6 In the content, take the component to be tested as a steel cage, and the steel bar tying robot is located on the left side of the steel cage as an example. Fig.10 Another flowchart of a method for measuring the size of a steel bar skeleton based on a steel bar tying robot is shown as an example. Fig.10 The following steps are involved:

[0199] Step S1001: Adjust the posture of the steel bar tying robot so that the camera on the robot arm faces the plane to be processed of the steel bar cage for parallel shooting.

[0200] It should be noted that the steel bar tying robot is located on the left side of the steel bar cage.

[0201] Step S1002: Determine whether it is a boundary; if not a boundary, execute step S1003; if it is a boundary, execute step S1004.

[0202] It should be noted that when performing boundary determination, the boundary is identified based on the boundary recognition model; the subsequent steps are also based on the boundary recognition model to identify the upper boundary and the right boundary.

[0203] Step S1003: Identify whether the steel bar tying robot is inside or outside the steel bar cage, and control the steel bar tying robot to move, and return to execute step S1002.

[0204] In a specific implementation, if the steel bar tying robot is inside the steel bar cage, the steel bar tying robot is controlled to move to the left, and the process returns to step S1002;

[0205] If the steel bar tying robot is outside the steel bar cage, control the steel bar tying robot to move right and return to execute step S1002.

[0206] Step S1004: Record the first horizontal distance x1 between the end of the robot arm and the base.

[0207] Step S1005: The robot arm moves upward and searches for corner points using algorithm 1.

[0208] It should be noted that Algorithm 1 contains the following contents:

[0209] 1. Collect sample data sets; that is, the sample data sets contain four types of images: boundary, corner, interior, and non-reinforced cage;

[0210] 2. Use the sample data set to train the neural network model to obtain a boundary recognition model. The boundary recognition model can identify the category of the image, which is boundary, corner, inner or non-reinforced cage;

[0211] 3. During the upward movement of the robotic arm, the boundary recognition model is used to analyze the second video stream captured by the camera. After finding the corner point, it is considered that the steel bar binding robot has reached the initial position, and the second image (depth map) in the second video stream is processed, and depth filtering is performed to obtain the surface steel bars. At the same time, the coordinates of the boundary of the surface steel bars relative to the camera can also be obtained, and the coordinates of the boundary relative to the camera are converted to the base coordinate system of the robotic arm.

[0212] Step S1006: Determine whether the upper boundary is found; if the upper boundary is found, execute step S1007; if the upper boundary is not found, move the camera on the robotic arm upward and continue to analyze the second video stream, and return to execute step S1006 to continue to determine whether the upper boundary is found.

[0213] Step S1007: Calculate the height of the steel cage using Algorithm 2.

[0214] It should be noted that Algorithm 2 contains the following contents:

[0215] 1. Extract the first height y1 of the upper boundary from the base from the pixel coordinates of the upper boundary that have been converted to the robot base coordinate system;

[0216] 2. Obtain the height y2 of the base of the steel bar tying robot from the ground;

[0217] 3. The height of the steel cage = y1+y2.

[0218] Step S1008: Control the steel bar tying robot to move right.

[0219] Step S1009: Determine whether the right boundary is found; if the right boundary is found, execute step S1010; if the right boundary is not found, control the steel bar binding robot to continue moving to the right and continue analyzing the third video stream, and return to step S1009 to continue to determine whether the right boundary is found.

[0220] Step S1010: Record the second horizontal distance x2 between the end of the robotic arm and the base.

[0221] Step S1011: Use the visual inertial ranging algorithm to calculate the running distance x3 of the mobile trolley under the base of the steel bar tying robot.

[0222] Step S1012: Calculate the length of the steel cage using Algorithm 3.

[0223] It should be noted that Algorithm 3 includes the following contents:

[0224] 1. Get the first horizontal distance x1 and the second horizontal distance x2;

[0225] 2. Obtain the running distance x3 of the mobile trolley under the base of the steel bar tying robot;

[0226] 3. The length of the steel cage = x1+x2+x3.

[0227] The above embodiments of the present invention Fig.10 For the execution principle of Figure 6 The contents in will not be repeated here.

[0228] In summary, in practical application, this solution has the following beneficial effects:

[0229] There is no need to add additional equipment to the rebar tying robot to estimate the size of the component's steel skeleton; the estimated steel skeleton size can be used to help the rebar tying robot to plan its path efficiently; and deep learning image classification and image segmentation technology can be used to accurately find boundaries.

[0230] Corresponding to the steel bar skeleton size measurement method based on the steel bar tying robot provided in the above embodiment of the present invention, see Fig.11 , an embodiment of the present invention also provides a structural block diagram of a steel bar skeleton size measuring device based on a steel bar tying robot, the steel bar skeleton size measuring device includes: a first control unit 100, a first recording unit 200, a second control unit 300, a first calculation unit 400, a third control unit 500, a second recording unit 600, a second calculation unit 700 and a third calculation unit 800;

[0231] The first control unit 100 is used to control the steel bar tying robot to search for the boundary of the component to be measured. The steel bar tying robot at least includes: a mechanical arm, a base, and a camera arranged on the mechanical arm.

[0232] The first recording unit 200 is used to record a first horizontal distance between the end of the robot arm and the base if the boundary of the component to be measured is found.

[0233] The second control unit 300 is used to control the steel bar tying robot to search upward for the upper boundary of the component to be tested.

[0234] The first calculation unit 400 is used to calculate a first height of the upper boundary from the base.

[0235] The third control unit 500 is used to control the steel bar tying robot to move rightward to search for the right boundary of the component to be tested.

[0236] The second recording unit 600 is used to record the second horizontal distance between the end of the robot arm and the base if the right boundary is found.

[0237] The second calculation unit 700 is used to calculate the running distance of the steel bar tying robot.

[0238] In a specific implementation, the second calculation unit 700 is specifically used to calculate the running distance of the steel bar tying robot based on a visual inertial ranging algorithm.

[0239] The third calculation unit 800 is used to calculate the steel skeleton size of the component to be measured according to the first horizontal distance, the second horizontal distance, the first height, the running distance, and the height of the base from the ground.

[0240] In the embodiment of the present invention, the steel bar tying robot is controlled to search for the boundary, upper boundary and right boundary of the component to be measured, and obtain the corresponding horizontal distance and running distance. The steel bar skeleton size of the component to be measured is determined by the obtained horizontal distance and running distance, and the determined steel bar skeleton size is applied to the path planning of the steel bar tying robot, thereby improving the path planning effect of the steel bar tying robot.

[0241] Preferably, combined Fig.11 As shown in the figure, the first control unit 100 includes a first control module, a first identification module and a first determination module; the execution principle of each module is as follows:

[0242] The first control module is used to control the steel bar binding robot to move along the edge of the component to be measured from one side of the component to be measured, and to shoot a first video stream of the component to be measured in real time.

[0243] The first recognition module is used to input the first image of the first video stream into a preset boundary recognition model for recognition to obtain the category of the first image. The boundary recognition model is obtained by training a neural network model based on a sample data set. The category of the first image is boundary, corner point, interior or non-reinforced cage.

[0244] The first determination module is used to determine that a boundary of the component to be detected has been searched if the category of the first image is identified as a boundary.

[0245] Preferably, combined Fig.11 As shown in the figure, the second control unit 300 includes a second control module, a second identification module and a second determination module; the execution principle of each module is as follows:

[0246] The second control module is used to control the mechanical arm of the steel bar binding robot to move upward and collect the second video stream of the component to be tested in real time.

[0247] The second recognition module is used to input the second image of the second video stream into a preset boundary recognition model for recognition to obtain the category of the second image. The boundary recognition model is obtained by training a neural network model based on a sample data set. The category of the second image is boundary, corner point, interior or non-reinforced cage.

[0248] The second determination module is used to determine that the upper boundary of the component to be measured has been searched if the category of the second image is identified as a corner point.

[0249] Preferably, combined Fig.11 As shown in the figure, the third control unit 500 includes a third control module, a third identification module and a third determination module; the execution principle of each module is as follows:

[0250] The third control module is used to control the steel bar binding robot to move rightward and collect the third video stream of the component to be tested in real time.

[0251] The third recognition module is used to input the third image of the third video stream into a preset boundary recognition model for recognition to obtain the category of the third image. The boundary recognition model is obtained by training a neural network model based on a sample data set. The category of the third image is boundary, corner point, interior or non-reinforced cage.

[0252] The third determination module is used to determine that the right boundary of the component to be measured has been searched if the category of the third image is identified as a corner point.

[0253] Preferably, combined Fig.11 As shown in the figure, the first computing unit 400 includes a parsing module, a first conversion module, a second conversion module and an extraction module; the execution principle of each module is as follows:

[0254] The parsing module is used to parse the second image to obtain the pixel coordinates of the upper boundary.

[0255] The first conversion module is used to convert the pixel coordinates of the upper boundary into a camera coordinate system corresponding to the camera arranged on the robotic arm.

[0256] The second conversion module is used to convert the pixel coordinates of the upper boundary that have been converted to the camera coordinate system into the robotic arm base coordinate system corresponding to the robotic arm.

[0257] The extraction module is used to extract the first height of the upper boundary from the base from the pixel coordinates of the upper boundary that have been converted into the robot arm base coordinate system.

[0258] In summary, the embodiment of the present invention provides a steel bar skeleton size measurement method and device based on a steel bar tying robot, which controls the steel bar tying robot to search for the boundary, upper boundary and right boundary of the component to be measured, and obtains the corresponding horizontal distance and running distance. The steel bar skeleton size of the component to be measured is determined by the obtained horizontal distance and running distance, and the determined steel bar skeleton size is applied to the path planning of the steel bar tying robot, thereby improving the path planning effect of the steel bar tying robot.

[0259] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0260] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0261] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for measuring the size of a steel bar skeleton based on a steel bar tying robot, characterized in that: The method comprises: Controlling a steel bar tying robot to search for the boundary of a component to be tested, the steel bar tying robot at least comprising: a mechanical arm, a base, and a camera arranged on the mechanical arm; If the boundary of the component to be measured is found, a first horizontal distance between the end of the robot arm and the base is recorded; Controlling the steel bar tying robot to search upward for the upper boundary of the component to be tested; Calculating a first height of the upper boundary from the base; Controlling the steel bar tying robot to move rightward to search for the right boundary of the component to be tested; If the right boundary is found, a second horizontal distance between the end of the robot arm and the base is recorded; Calculating the running distance of the steel bar tying robot; The steel skeleton size of the component to be measured is calculated according to the first horizontal distance, the second horizontal distance, the first height, the running distance, and the height of the base from the ground.

2. The method according to claim 1, characterized in that Control the steel bar tying robot to search for the boundary of the component to be tested, including: Controlling the steel bar tying robot to move along the edge of the component to be measured from one side of the component to be measured, and shooting a first video stream of the component to be measured in real time; Inputting the first image of the first video stream into a preset boundary recognition model for recognition to obtain a category of the first image, wherein the boundary recognition model is obtained by training a neural network model based on a sample data set, and the category of the first image is boundary, corner point, interior, or non-reinforced cage; If the category of the first image is identified as a boundary, it is determined that the boundary of the component to be detected is found.

3. The method according to claim 1, characterized in that: Controlling the steel bar tying robot to search upward for the upper boundary of the component to be tested comprises: Controlling the mechanical arm of the steel bar tying robot to move upward, and collecting a second video stream of the component to be tested in real time; Inputting the second image of the second video stream into a preset boundary recognition model for recognition to obtain a category of the second image, wherein the boundary recognition model is obtained by training a neural network model based on a sample data set, and the category of the second image is boundary, corner point, interior, or non-reinforced cage; If the category of the second image is identified as a corner point, it is determined that the upper boundary of the component to be measured is found.

4. The method according to claim 1, characterized in that Controlling the steel bar tying robot to move rightward to search for the right boundary of the component to be tested comprises: Controlling the steel bar tying robot to move rightward, and collecting a third video stream of the component to be tested in real time; Inputting the third image of the third video stream into a preset boundary recognition model for recognition to obtain a category of the third image, wherein the boundary recognition model is obtained by training a neural network model based on a sample data set, and the category of the third image is boundary, corner point, interior, or non-reinforced cage; If the category of the third image is identified as a corner point, it is determined that the right boundary of the component to be measured is found.

5. The method according to claim 3, characterized in that: Calculating a first height of the upper boundary from the base includes: parsing the second image to obtain pixel coordinates of the upper boundary; Convert the pixel coordinates of the upper boundary to a camera coordinate system corresponding to a camera disposed on the robotic arm; Convert the pixel coordinates of the upper boundary converted to the camera coordinate system to the robot base coordinate system corresponding to the robot arm; A first height of the upper boundary from the base is extracted from the pixel coordinates of the upper boundary that have been converted into the robot base coordinate system.

6. The method according to claim 1, characterized in that Calculating the running distance of the steel bar tying robot includes: Based on the visual inertial ranging algorithm, the running distance of the steel bar tying robot is calculated.

7. A steel bar skeleton size measuring device based on a steel bar tying robot, characterized in that: The device comprises: A first control unit is used to control a steel bar tying robot to search for a boundary of a component to be tested, wherein the steel bar tying robot comprises at least: a mechanical arm, a base, and a camera arranged on the mechanical arm; A first recording unit, configured to record a first horizontal distance between the end of the robot arm and the base if a boundary of the component to be measured is found; A second control unit is used to control the steel bar tying robot to search upward for the upper boundary of the component to be tested; A first calculation unit, used for calculating a first height of the upper boundary from the base; A third control unit is used to control the steel bar tying robot to move rightward to search for the right boundary of the component to be tested; A second recording unit, for recording a second horizontal distance between the end of the robot arm and the base if the right boundary is found; A second calculation unit, used for calculating the running distance of the steel bar tying robot; The third calculation unit is used to calculate the steel skeleton size of the component to be measured according to the first horizontal distance, the second horizontal distance, the first height, the running distance, and the height of the base from the ground.

8. The device according to claim 7, characterized in that The first control unit comprises: A first control module is used to control the steel bar tying robot to move along the edge of the component to be measured from one side of the component to be measured, and to shoot a first video stream of the component to be measured in real time; A first recognition module is used to input the first image of the first video stream into a preset boundary recognition model for recognition to obtain a category of the first image, wherein the boundary recognition model is obtained by training a neural network model based on a sample data set, and the category of the first image is boundary, corner point, interior, or non-reinforced cage; The first determination module is used to determine that a boundary of the component to be detected has been searched if the category of the first image is identified as a boundary.

9. The device according to claim 7, characterized in that The second control unit comprises: A second control module is used to control the mechanical arm of the steel bar tying robot to move upward and collect a second video stream of the component to be tested in real time; A second recognition module is used to input the second image of the second video stream into a preset boundary recognition model for recognition to obtain a category of the second image, wherein the boundary recognition model is obtained by training a neural network model based on a sample data set, and the category of the second image is boundary, corner point, interior or non-reinforced cage; The second determination module is used to determine that the upper boundary of the component to be measured is searched if the category of the second image is identified as a corner point.

10. The device according to claim 7, characterized in that The third control unit comprises: A third control module is used to control the steel bar tying robot to move rightward and collect a third video stream of the component to be tested in real time; A third recognition module is used to input the third image of the third video stream into a preset boundary recognition model for recognition to obtain a category of the third image, wherein the boundary recognition model is obtained by training a neural network model based on a sample data set, and the category of the third image is boundary, corner point, interior or non-reinforced cage; The third determination module is used to determine that the right boundary of the component to be measured is found if the category of the third image is identified as a corner point.

Images