Steel bar binding decision-making method, device, equipment and storage medium based on deep learning

Abstract

The present application discloses a steel bar binding decision-making method, device, equipment and storage medium based on deep learning, which relates to the field of civil engineering and architecture and is applied to a steel bar binding robot. The method includes: collecting a target 2D map and a target depth map of the current binding area, processing the target 2D map and the target depth map by using a surface steel bar mesh extraction algorithm to obtain a surface steel bar mesh image, and inputting the image into a binding recognition and positioning model to determine the coordinates of the bound points, unbound points and key points; the key points include the bound points and the unbound points; sorting the coordinates of the key points to obtain a target binding path, and binding the unbound points based on the target binding path; wherein, the surface steel bar mesh extraction algorithm is an algorithm constructed by using a random sample consensus algorithm improved based on a dynamic Bayesian network; the binding recognition and positioning model is a model constructed based on a key point detection deep learning model and a neighborhood depth averaging algorithm. The present application improves the working efficiency of the steel bar binding robot.

Description

Technical Field

[0001] The present invention relates to the field of civil engineering, and particularly relates to a steel bar binding decision-making method, device, equipment and storage medium based on deep learning. Background Art

[0002] Currently, steel bar binding is mainly completed manually, and there are problems such as high labor intensity, high production cost, and low work efficiency. The emergence of steel bar binding robots has brought hope for solving this difficult problem. However, currently, the steel bar binding robots have low efficiency in identifying and positioning steel bar binding points during operation, and have low applicability in making binding decisions for multi-layer steel bar cages, thereby reducing the work efficiency of the steel bar binding robots and having low practicality.

[0003] In summary, how to improve the identification and positioning efficiency of the binding points of the binding robot to improve its work efficiency is an urgent problem to be solved at present. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a steel bar binding decision-making method based on deep learning, which can improve the identification and positioning efficiency of the binding points of the binding robot to improve its work efficiency. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses a steel bar binding decision-making method based on deep learning, which is applied to a steel bar binding robot and includes:

[0006] Using a preset 3D camera to collect a target 2D map and a target depth map of the current binding area of the target steel bar cage, and using a preset surface steel bar mesh extraction algorithm to process the target 2D map and the target depth map to obtain a surface steel bar mesh image corresponding to the target steel bar cage;

[0007] Inputting the surface steel bar mesh image into a preset binding recognition and positioning model to classify key points in the surface steel bar mesh image to determine bound points, unbound points, and key point coordinates and depth values corresponding to the key points; the key points include the bound points and the unbound points, and the key point coordinates are the pixel coordinates of the key points in the target depth map and the target 2D map;

[0008] Performing path sorting on all the key point coordinates based on a preset binding path planning method and a preset binding method to obtain a target binding path, and binding the unbound points of the target steel bar cage based on the target binding path;

[0009] Among them, the surface steel bar mesh extraction algorithm is an algorithm constructed by using the Random Sample Consensus (RANSAC) algorithm improved based on the Dynamic Bayesian Network; the binding recognition and positioning model is a model constructed based on the deep learning model for key point detection and the neighborhood depth averaging algorithm.

[0010] Optionally, before processing the target 2D image and the target depth image by using the preset surface steel bar mesh extraction algorithm, it further includes:

[0011] Using the Dynamic Bayesian Network to iterate the initial Random Sample Consensus (RANSAC) algorithm based on a preset stopping criterion; the preset stopping criterion is a stopping criterion based on a preset inlier probability;

[0012] Determining a target Random Sample Consensus (RANSAC) algorithm and a target inlier / outlier classification from all the Random Sample Consensus (RANSAC) algorithms based on the number of inliers and outliers corresponding to each Random Sample Consensus (RANSAC) algorithm obtained during the iteration process;

[0013] Constructing the surface steel bar mesh extraction algorithm by using the target Random Sample Consensus (RANSAC) algorithm.

[0014] Optionally, the iteration of the initial Random Sample Consensus (RANSAC) algorithm by using the Dynamic Bayesian Network based on a preset stopping criterion includes:

[0015] Performing weighted sampling on each data point based on the inlier / outlier probabilities corresponding to each data point under the current Random Sample Consensus (RANSAC) algorithm to determine a minimum data set, and generating a new Random Sample Consensus (RANSAC) algorithm corresponding to the minimum data set based on the minimum data set;

[0016] Determining the inlier / outlier classification corresponding to the current Random Sample Consensus (RANSAC) algorithm, and updating the inlier / outlier probabilities corresponding to each data point according to the inlier / outlier classification to obtain new inlier / outlier probabilities;

[0017] If there are data points with inlier probabilities less than the preset inlier probability in the current Random Sample Consensus (RANSAC) algorithm, stop the iteration.

[0018] Optionally, processing the target 2D image and the target depth image by using the preset surface steel bar mesh extraction algorithm to obtain the surface steel bar mesh image corresponding to the target steel reinforcement cage includes:

[0019] Aligning pixel points of the target 2D image and the target depth image by using the surface steel bar mesh extraction algorithm to obtain a target three-dimensional point cloud, and performing plane fitting on the target mesh in the target three-dimensional point cloud to obtain the fitting planes where each target mesh is located;

[0020] Determining the surface steel bar mesh image corresponding to the target steel reinforcement cage based on the distances between each fitting plane and the origin of the camera coordinate system.

[0021] Optionally, the step of determining the surface steel mesh image corresponding to the target steel cage based on the distances between the fitting planes and the origin of the camera coordinate system includes:

[0022] Determining a target fitting plane corresponding to the target steel cage based on the distances between the fitting planes and the origin of the camera coordinate system, and performing an image dilation operation on the target fitting plane when there are pixel depth values on the target fitting plane that are less than a first preset depth range, so as to obtain the surface steel mesh image.

[0023] Optionally, the deep learning-based steel bar binding decision-making method further includes:

[0024] If the depth value corresponding to the key point coordinate in the target depth map is not within the second preset depth range, processing the depth value corresponding to the key point coordinate based on the neighborhood depth averaging algorithm to determine a new depth value corresponding to the key point coordinate.

[0025] Optionally, the step of performing path sorting on all the key point coordinates based on a preset binding path planning method and a preset binding method to obtain a target binding path includes:

[0026] If the preset binding method is the full binding method, performing path sorting on the key point coordinates corresponding to all the unbound points based on the preset binding path planning method to obtain the target binding path;

[0027] If the preset binding method is the skip binding method, performing path sorting on all the key point coordinates based on the preset binding path planning method to determine a to-be-optimized binding path, and determining a binding starting point of the current binding area based on the binding information of the previous binding area, so as to optimize the to-be-optimized binding path based on the binding starting point to determine the target binding path.

[0028] In a second aspect, the present application discloses a deep learning-based steel bar binding decision-making device, which is applied to a steel bar binding robot and includes:

[0029] An image extraction module, configured to collect a target 2D image and a target depth image of a current binding area of a target steel cage by using a preset 3D camera, and process the target 2D image and the target depth image by using a preset surface steel mesh extraction algorithm, so as to obtain a surface steel mesh image corresponding to the target steel cage;

[0030] An identification and positioning module, configured to input the surface steel bar mesh image into a preset binding identification and positioning model to classify key points in the surface steel bar mesh image, so as to determine the bound points, unbound points, and the key point coordinates and depth values corresponding to the key points; the key points include the bound points and the unbound points, and the key point coordinates are the pixel coordinates of the key points in the target depth map and the target 2D map;

[0031] A steel bar binding module, configured to perform path sorting on all the key point coordinates based on a preset binding path planning method and a preset binding method to obtain a target binding path, and bind the unbound points of the target steel bar cage based on the target binding path;

[0032] Wherein, the surface steel bar mesh extraction algorithm is an algorithm constructed by using a random sample consensus algorithm improved based on a dynamic Bayesian network; the binding identification and positioning model is a model constructed based on a key point detection deep learning model and a neighborhood depth averaging algorithm.

[0033] In a third aspect, the present application discloses an electronic device, including:

[0034] A memory, configured to store a computer program;

[0035] A processor, configured to execute the computer program to implement the foregoing deep learning-based steel bar binding decision-making method.

[0036] In a fourth aspect, the present application discloses a computer-readable storage medium, configured to store a computer program, wherein the computer program, when executed by a processor, implements the foregoing deep learning-based steel bar binding decision-making method.

[0037] In this application, when the robot performs steel bar binding, it uses a preset 3D camera to collect the target 2D image and the target depth image of the current binding area of the target steel cage, and uses a preset surface steel bar mesh extraction algorithm to process the target 2D image and the target depth image to obtain the surface steel bar mesh image corresponding to the target steel cage; inputs the surface steel bar mesh image into a preset binding recognition and positioning model to classify the key points in the surface steel bar mesh image to determine the bound points, unbound points, and the key point coordinates and depth values corresponding to the key points; the key points include the bound points and the unbound points; sorts the paths of all the key point coordinates based on a preset binding path planning method and a preset binding method to obtain a target binding path, and binds the unbound points of the target steel cage based on the target binding path; wherein, the surface steel bar mesh extraction algorithm is an algorithm constructed by using the random sample consensus algorithm improved based on the dynamic Bayesian network; the binding recognition and positioning model is a model constructed based on a key point detection deep learning model and a neighborhood depth averaging algorithm. It can be seen that this application uses the surface steel bar mesh extraction algorithm based on the improved random sample consensus algorithm to extract the surface steel bar mesh from the grayscale image with a complex background, and inputs the surface steel bar mesh image into the binding recognition and positioning model. The binding recognition and positioning model can accurately classify the steel bar intersection points (i.e., key points) in the picture into bound points and unbound points and obtain the pixel coordinates and depth values of all key points. Using the pixel coordinates, depth values, and the internal parameters of the preset 3D camera, the three-dimensional space coordinates corresponding to the key points can be determined. Then, based on the preset binding path planning method and the preset binding method, the paths of all key point coordinates can be sorted to obtain the target binding path. Finally, the unbound points in the current binding area of the target steel cage are bound using the target binding path, improving the working efficiency of the binding robot. Description of the Drawings

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on the provided drawings.

[0039] Figure 1 Flowchart of a steel bar binding decision-making method based on deep learning disclosed in this application;

[0040] Figure 2 Schematic diagram of the extraction result of the surface steel bar mesh disclosed in this application;

[0041] Figure 3 Pseudo-code schematic diagram of the iterative process of the random sample consensus algorithm disclosed in this application;

[0042] Figure 4 Schematic diagram of a specific process for extracting the surface steel bar mesh disclosed in this application;

[0043] Figure 5 Comparison diagram of the expansion results of the surface steel bar mesh image disclosed in this application;

[0044] Figure 6 Surface steel bar mesh image obtained by processing the original image taken at an inclination angle disclosed in this application;

[0045] Figure 7 Schematic diagram of a specific process for training the YOLOv9-pose algorithm disclosed in this application;

[0046] Figure 8 Schematic diagram of the key point detection results disclosed in this application;

[0047] Figure 9 Schematic diagram of the key point identification and positioning process disclosed in this application;

[0048] Figure 10 Schematic diagram of the steel bar binding path provided by this application, where (a) is the loop-shaped binding path, (b) is the zigzag binding path, and (c) is the bow-shaped binding path;

[0049] Figure 11 Schematic diagram of the steel bar binding method provided by this application, where (a) is the full binding method and (b) is the skip binding method;

[0050] Figure 12 Schematic diagram of a specific steel bar binding process disclosed in this application;

[0051] Figure 13 Schematic diagram of the positional relationship between the current photographing area and the previous binding area disclosed in this application;

[0052] Figure 14 Another schematic diagram of the positional relationship between the current photographing area and the previous binding area disclosed in this application;

[0053] Figure 15 Schematic diagram of a specific camera photographing path disclosed in this application;

[0054] Figure 16 Another schematic diagram of a specific camera photographing path disclosed in this application;

[0055] Figure 17 Schematic diagram of a specific skip binding decision-making process disclosed in this application;

[0056] Figure 18Schematic diagram of the working conditions in the binding area disclosed in this application, where (a) is a working condition where the camera moves downward, (b) is another working condition where the camera moves downward, (c) is a working condition where the camera moves upward, (d) is another working condition where the camera moves upward, (e) is a working condition where the camera moves to the right, and (f) is another working condition where the camera moves to the right;

[0057] Figure 19 Schematic diagram of the structure of a steel bar binding decision-making device based on deep learning disclosed in this application;

[0058] Figure 20 Schematic diagram of the structure of an electronic device disclosed in this application. Specific implementation manners

[0059] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0060] Currently, when a steel bar binding robot is operating, the recognition and positioning efficiency of the steel bar binding points is relatively low, and the applicability is not high when making binding decisions for multi-layer steel bar cages, thereby reducing the working efficiency of the steel bar binding robot and having low practicability. To solve the above technical problems, this application discloses a steel bar binding decision-making method based on deep learning, which can improve the recognition and positioning efficiency of the steel bar binding points of the binding robot to improve its working efficiency.

[0061] See Figure 1 As shown, the embodiments of the present invention disclose a steel bar binding decision-making method based on deep learning, which is applied to a steel bar binding robot and includes:

[0062] Step S11: Use a preset 3D camera to collect the target 2D map and the target depth map of the current binding area of the target steel bar cage, and use the preset surface steel bar mesh extraction algorithm to process the target 2D map and the target depth map to obtain the surface steel bar mesh image corresponding to the target steel bar cage.

[0063] In this embodiment, as Figure 2As shown, a preset 3D camera can be used to collect the target 2D image and the target depth image of the current binding area of the target steel cage, and then the surface steel mesh image corresponding to this area can be obtained by processing the target 2D image and the target depth image. Specifically, a preset surface steel mesh extraction algorithm can be used to extract the surface steel mesh from a grayscale image with a complex background. Among them, the surface steel mesh extraction algorithm is an algorithm constructed by using the random sample consensus algorithm improved based on DBN (Dynamic Bayesian Network).

[0064] In a specific implementation manner, as Figure 3 shown, the specific process of using DBN to improve the RANSAC algorithm (Random Sample Consensus) can include: using the dynamic Bayesian network to iterate the initial random sample consensus algorithm based on a preset stopping criterion, determining the target random sample consensus algorithm and the target inlier-outlier classification from all random sample consensus algorithms based on the number of inliers and outliers corresponding to each random sample consensus algorithm obtained during the iteration process, and using the target random sample consensus algorithm to construct the surface steel mesh extraction algorithm. Among them, the preset stopping criterion is a stopping criterion based on a preset inlier probability. The specific iteration process can include: performing weighted sampling on each data point based on the inlier-outlier probability corresponding to each data point under the current random sample consensus algorithm to determine the minimum data set, and generating a new random sample consensus algorithm corresponding to the minimum data set based on the minimum data set; determining the inlier-outlier classification corresponding to the current random sample consensus algorithm, and updating the inlier-outlier probability corresponding to each data point according to the inlier-outlier classification to obtain a new inlier-outlier probability. If there are data points with an inlier probability less than the preset inlier probability in the current random sample consensus algorithm, stop the iteration.

[0065] That is to say, in this embodiment, first, the data point probabilities of the initial random sample consensus algorithm are determined , and these probabilities can be scores obtained through prior calculations or can be uniform. Then, in each iteration, a set of data points is selected through weighted sampling , and the weights during sampling are based on the inlier probability of each data point . According to the selected minimum data set , a new random sample consensus algorithm is generated , and the inlier-outlier classification is determined . After each iteration, the inlier-outlier probabilities of the data points are updated according to the current inlier-outlier classification These updates will affect the weights during the next sampling. If the current inlier probability is lower than the preset inlier probability, the iteration is stopped, thus avoiding inaccurate algorithm fitting caused by excessive iteration and increasing the calculation time. When the iteration ends, the best model will be returned. and the corresponding inlier and outlier classifications .

[0066] In this embodiment, as Figure 4 shown, by inputting the target 2D map and the target depth map of the target steel cage into the surface steel mesh extraction algorithm, the point cloud plane corresponding to the target 2D map and the target depth map in the original three-dimensional point cloud can be fitted. Since the collected target three-dimensional point cloud contains two layers of mesh, at this time, the surface steel mesh extraction algorithm can fit two spatial planes. Therefore, it is necessary to screen these two planes to extract the plane where the surface steel mesh is located. It can be understood that since the distance between the plane where the surface steel mesh is located and the origin of the camera coordinate system is smaller, the distances between the two spatial planes and the origin of the camera coordinate system are calculated respectively, and the plane where the surface steel mesh is located with the smaller distance is extracted. At this time, the specific process of using the preset surface steel mesh extraction algorithm to process the target 2D map and the target depth map to obtain the surface steel mesh image corresponding to the target steel cage may include: using the surface steel mesh extraction algorithm to align the pixel points of the target 2D map and the target depth map to obtain the target three-dimensional point cloud, and performing plane fitting on the target mesh in the target three-dimensional point cloud to obtain the fitting planes where each target mesh is located; determining the surface steel mesh image corresponding to the target steel cage based on the distances between the fitting planes and the origin of the camera coordinate system. At this time, the extracted surface steel mesh image is as Figure 2 shown.

[0067] In addition, since there will be pixel points with depth disappearance in the extracted surface steel mesh image ( Figure 5 in the left rectangle of the figure), this will make the surface steel bars incomplete. For this phenomenon, image dilation operation can be used to further process the surface steel mesh image. The processed image is as Figure 5As shown in the right figure. That is, determining the surface steel mesh image corresponding to the target steel cage based on the distances between the fitting planes and the origin of the camera coordinate system may specifically include: determining the target fitting plane corresponding to the target steel cage based on the distances between the fitting planes and the origin of the camera coordinate system, and performing an image dilation operation on the target fitting plane when there are pixel depth values less than the first preset depth range in the target fitting plane to obtain the surface steel mesh image. That is, the initial surface steel mesh image corresponding to the target steel cage can be determined based on the distances between the fitting planes and the origin of the camera coordinate system. If there are pixel points with depth values less than the first preset depth range in the obtained surface steel mesh image at this time, the image dilation operation can be performed on the surface steel mesh image to obtain the final surface steel mesh image. In addition, there is no requirement for the angle of the preset 3D camera to photograph the target steel cage in this application. Even if the camera is tilted to photograph the target steel cage, the surface steel mesh without depth vanishing points can still be extracted well, as Figure 6 shown.

[0068] Step S12: Input the surface steel mesh image into a preset binding recognition and positioning model to classify the key points in the surface steel mesh image to determine the bound points, unbound points, and the key point coordinates and depth values corresponding to the key points; the key points include the bound points and the unbound points, and the key point coordinates are the pixel coordinates of the key points in the target depth map and the target 2D map.

[0069] In this embodiment, the binding recognition and positioning model is a model constructed based on a key point detection deep learning model and a neighborhood depth averaging algorithm. In a specific implementation, the key point detection deep learning model is a model obtained by training the YOLOv9-pose (an application version of the YOLOv9 (You Only Look Once version 9, a target detection algorithm) series of models in the pose estimation task). The training, validation, and testing processes are as Figure 7 shown. First, establish a data set. Collect surface steel mesh images, with a total of three types of labels: bound points, unbound points, and key points. Specifically, the ratio of the number of images in the training set, validation set, and test set can be 8:1:1. For example, the number of images in the three image sets are 160, 20, and 20 respectively. Secondly, train the YOLOv9-pose model, and set the training parameters according to experience, such as the learning rate, batch size (the number of samples used for each parameter update when training a neural network), epoch (the process of the entire training data set being completely traversed by the neural network once), etc. Finally, perform model performance inspection. Use the trained YOLOv9-pose model to predict 20 images in the test set. Some prediction results are as Figure 8As shown in the figure, after statistics, the model can predict all the key points in these 20 images and classify these key points into tied points and untied points (the label of tied points is bz, and the label of untied points is wbz).

[0070] In this embodiment, as Figure 9 shown, after obtaining the surface steel bar mesh image, the image can be input into the trained tying recognition and positioning model for key point classification and obtaining the corresponding key point coordinates and depth values. The key points are also the steel bar intersection points. Specifically, if the depth value corresponding to the key point coordinate in the target depth map is not within the second preset depth range, the depth value corresponding to the key point coordinate can be processed based on the neighborhood depth averaging algorithm to determine the new depth value corresponding to the key point. That is to say, after obtaining the surface steel bar mesh image, the tying recognition and positioning model will accurately identify and classify all the key points in the image, and locate the untied points and tied points. If the depth value corresponding to the key point is a normal value (that is, the depth value is within the effective shooting distance range of the camera, that is, within the second preset depth range, such as 300 - 600 mm), the three-dimensional space coordinates corresponding to the key point can be determined based on the pixel coordinates and depth value of the key point and the internal parameters of the preset 3D camera for subsequent tying operations; if the depth value corresponding to the key point is an invalid value, it means that the key point is a depth disappearance point, and the neighborhood depth averaging algorithm is used to process it, and then the new depth value corresponding to the key point is obtained, so as to determine the three-dimensional space coordinates corresponding to the key point based on the pixel coordinates and the newly determined depth value of the key point and the internal parameters of the preset 3D camera, and perform subsequent tying operations.

[0071] Step S13: Sort the coordinates of all the key points based on the preset tying path planning method and preset tying method to obtain the target tying path, and tie the untied points of the target steel cage based on the target tying path.

[0072] In this embodiment, as Figure 10 shown, during the tying operation, multiple different tying paths can be selected for tying path planning, such as the zigzag shape ( Figure 10 (a)), the z - shape ( Figure 10 (b)), and the bow - shape ( Figure 10(c)), assuming the distance between adjacent key points is 1, the total length of the square-shaped path is 35, the total length of the zigzag path is greater than 35, and the total length of the bow-shaped path is 35. Compared with the zigzag path, the total lengths of the square-shaped path and the bow-shaped path are shorter, the moving distance of the tying robot is shorter, and the tying operation efficiency is higher. The square-shaped path is more suitable for use when the global information of the steel reinforcement cage is known, and the path planning algorithm for the square-shaped path is relatively complex. Therefore, the bow-shaped path can be preferentially considered as the preset tying path planning method to sort the coordinates of all key points.

[0073] In this embodiment, in addition to determining the preset tying path planning method, it is also necessary to determine the preset tying method. For different components, the full tying or skip tying method can be adopted. The schematic diagrams of the two tying methods are as Figure 11 shown. Full tying means tying all the un-tied points; skip tying means tying one and leaving one empty. Figure 11 In (a) is full tying, and (b) is skip tying. If the full tying method is adopted, the tied points in the bow-shaped path can be removed first, and then the subsequent tying process can be executed; if the skip tying method is adopted, when tying the current area, it is necessary to determine the tying starting point of the current tying area according to the information of the previous tying area. Therefore, as Figure 12 shown, the specific process of sorting the coordinates of all key points based on the preset tying path planning method and the preset tying method to obtain the target tying path may include: if the preset tying method is the full tying method, sort the coordinates of the key points corresponding to all un-tied points based on the preset tying path planning method to obtain the target tying path; if the preset tying method is the skip tying method, sort the coordinates of all the key points based on the preset tying path planning method to determine the tying path to be optimized, and determine the tying starting point of the current tying area based on the tying information of the previous tying area, so as to optimize the tying path to be optimized based on the tying starting point to determine the target tying path. It can be understood that if the preset tying method is skip tying, when planning the camera photographing path, it is necessary to ensure that each photographing area of the camera (except the first photographing position of the camera) can always cover a steel bar in the previous tying area. As Figure 13 and Figure 14 shown, the specific camera photographing path may include Figure 15 and Figure 16 , which will not be listed one by one here.

[0074] In a specific implementation manner, as Figure 17As shown, when sorting the coordinates of all key points according to a preset binding path planning method and a preset binding method to obtain a target binding path, if a zigzag path or a skipping binding method is adopted, first obtain a list of key points sorted according to the zigzag path, and respectively extract the list of key points list1 with odd indexes and the list of key points list2 with even indexes; secondly, screen out the list of key points list3 that contains all the already bound points; finally, remove all the already bound points in list3 to obtain the target list list0, and perform subsequent binding processes.

[0075] It can be understood that assuming that the binding robot starts operating from the upper left corner area of a large-size planar steel bar cage, and the camera photographing path is as Figure 15 shown, first "from top to bottom" and then "from left to right", then during the movement of the camera, there will be 6 working conditions of photographing areas as Figure 18 shown, which are the camera moving downwards ( Figure 18 (a) and Figure 18 (b)), the camera moving upwards ( Figure 18 (c) and Figure 18 (d)) and the camera moving to the right ( Figure 18 (e) and Figure 18 (f)). The skipping binding decision method proposed above can solve the skipping binding decision problems of the above 6 working conditions.

[0076] It can be seen that this application uses a surface steel bar mesh extraction algorithm based on an improved random sample consensus algorithm to extract the surface steel bar mesh from a grayscale image with a complex background, and inputs the surface steel bar mesh image into a binding recognition and positioning model. The binding recognition and positioning model can accurately classify the steel bar intersection points (i.e., key points) in the picture into already bound points and unbound points and obtain the pixel coordinates of all key points. Then, based on a preset binding path planning method and a preset binding method, the coordinates of all key points can be sorted and screened to obtain a target binding path. Finally, the unbound points in the current binding area of the target steel bar cage are bound using the target binding path, improving the working efficiency of the binding robot.

[0077] See Figure 19 shown, this application discloses a steel bar binding decision device based on deep learning, which is applied to a steel bar binding robot and includes:

[0078] An image extraction module 11, configured to use a preset 3D camera to collect a target 2D map and a target depth map of the current binding area of a target steel bar cage, and use a preset surface steel bar mesh extraction algorithm to process the target 2D map and the target depth map to obtain a surface steel bar mesh image corresponding to the target steel bar cage;

[0079] The recognition and positioning module 12 is configured to input the surface steel bar mesh image into a preset binding recognition and positioning model to classify key points in the surface steel bar mesh image, so as to determine the tied points, untied points, and the corresponding key point coordinates and depth values of the key points; the key points include the tied points and the untied points, and the key point coordinates are the pixel coordinates of the key points in the target depth map and the target 2D map;

[0080] The steel bar binding module 13 is configured to perform path sorting on all the key point coordinates based on a preset binding path planning method and a preset binding method to obtain a target binding path, and bind the untied points of the target steel bar cage based on the target binding path;

[0081] Wherein, the surface steel bar mesh extraction algorithm is an algorithm constructed by using the random sample consensus algorithm improved based on the dynamic Bayesian network; the binding recognition and positioning model is a model constructed based on the key point detection deep learning model and the neighborhood depth averaging algorithm.

[0082] It can be seen that this application uses the surface steel bar mesh extraction algorithm based on the improved random sample consensus algorithm to extract the surface steel bar mesh from the grayscale image with a complex background, and inputs the surface steel bar mesh image into the binding recognition and positioning model. The binding recognition and positioning model can accurately classify the steel bar intersection points (i.e., key points) in the picture into tied points and untied points and obtain the coordinates of all key points. Then, based on the preset binding path planning method and the preset binding method, path sorting can be performed on all key point coordinates to obtain the target binding path. Finally, the untied points in the current binding area of the target steel bar cage are bound by using the target binding path, which improves the working efficiency of the binding robot.

[0083] In a specific embodiment, the device may further include:

[0084] The algorithm iteration module is configured to use the dynamic Bayesian network to iterate the initial random sample consensus algorithm based on a preset stopping criterion; the preset stopping criterion is a stopping criterion based on a preset inlier probability;

[0085] The algorithm determination module is configured to determine a target random sample consensus algorithm and a target inlier-outlier classification from all the random sample consensus algorithms based on the number of inliers and outliers corresponding to each random sample consensus algorithm obtained during the iteration process;

[0086] The algorithm construction module is configured to construct the surface steel bar mesh extraction algorithm by using the target random sample consensus algorithm.

[0087] In a specific embodiment, the algorithm iteration module may specifically include:

[0088] An algorithm generation sub-module, configured to perform weighted sampling on each of the data points based on the inlier and outlier probabilities corresponding to the data points under the current Random Sample Consensus (RANSAC) algorithm to determine a minimum data set, and generate a new RANSAC algorithm corresponding to the minimum data set based on the minimum data set;

[0089] A probability update sub-module, configured to determine the inlier and outlier classification corresponding to the current RANSAC algorithm, and update the inlier and outlier probabilities corresponding to each of the data points according to the inlier and outlier classification to obtain new inlier and outlier probabilities;

[0090] An iteration stop sub-module, configured to stop iteration if there are data points with inlier probabilities less than a preset inlier probability in the current RANSAC algorithm.

[0091] In a specific embodiment, the image extraction module 11 may specifically include:

[0092] A mesh fitting sub-module, configured to align pixel points of the target 2D map and the target depth map by using the surface steel bar mesh extraction algorithm to obtain a target three-dimensional point cloud, and perform plane fitting on the target mesh in the target three-dimensional point cloud to obtain a fitting plane where each target mesh is located;

[0093] A target image determination sub-module, configured to determine the surface steel bar mesh image corresponding to the target steel reinforcement cage based on the distance between each of the fitting planes and the origin of the camera coordinate system.

[0094] In a specific embodiment, the target image determination sub-module may specifically include:

[0095] A target image determination unit, configured to determine a target fitting plane corresponding to the target steel reinforcement cage based on the distance between each of the fitting planes and the origin of the camera coordinate system, and perform an image dilation operation on the target fitting plane when the depth value of the pixel points in the target fitting plane is less than a first preset depth range to obtain the surface steel bar mesh image.

[0096] In a specific embodiment, the apparatus may further include:

[0097] A depth value update module, configured to, if the depth value corresponding to the key point coordinate in the target depth map is not within a second preset depth range, process the depth value corresponding to the key point coordinate based on a neighborhood depth averaging algorithm to determine a new depth value corresponding to the key point coordinate.

[0098] In a specific embodiment, the steel bar binding module 13 may specifically include:

[0099] The first path determination sub-module is configured to, if the preset binding method is the full-binding method, perform path sorting on the key point coordinates corresponding to all the unbound points based on the preset binding path planning method to obtain the target binding path;

[0100] The second path determination sub-module is configured to, if the preset binding method is the skip-binding method, perform path sorting on all the key point coordinates based on the preset binding path planning method to determine the binding path to be optimized, and determine the binding starting point of the current binding area based on the binding information of the previous binding area, so as to optimize the binding path to be optimized based on the binding starting point to determine the target binding path.

[0101] Furthermore, an embodiment of the present application also discloses an electronic device, Figure 20 which is a structural diagram of an electronic device 20 shown according to an exemplary embodiment. The content in the figure should not be considered as any limitation to the scope of use of the present application.

[0102] Figure 20 This is a schematic structural diagram of an electronic device 20 provided by an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the decision-making method for steel bar binding based on deep learning disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0103] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows is any communication protocol applicable to the technical solution of the present application, and no specific limitation is imposed on it here; the input / output interface 25 is used to obtain external input data or output data to the outside, and its specific interface type can be selected according to specific application requirements, and no specific limitation is made here.

[0104] In addition, as a carrier for resource storage, the memory 22 may be a read-only memory, a random access memory, a magnetic disk, or an optical disc, etc. The resources stored thereon may include an operating system 221, a computer program 222, etc., and the storage method may be short-term storage or permanent storage.

[0105] Among them, the operating system 221 is used to manage and control each hardware device and computer program 222 on the electronic device 20, and it can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the deep learning-based steel bar binding decision-making method executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs that can be used to complete other specific tasks.

[0106] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the foregoing disclosed deep learning-based steel bar binding decision-making method. For the specific steps of this method, reference can be made to the corresponding content disclosed in the foregoing embodiments, and details will not be repeated here.

[0107] In this specification, the various embodiments are described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. For the same or similar parts among the various embodiments, reference can be made to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and reference can be made to the description in the method part for related parts.

[0108] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in this article can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0109] The steps of the method or algorithm described in combination with the embodiments disclosed in this article can be directly implemented by hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.

[0110] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising said element.

[0111] The technical solutions provided in this application have been introduced in detail above. Specific examples are used in this text to elaborate on the principles and implementation manners of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application. At the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to this application.

Claims

1. A steel bar tying decision method based on deep learning, characterized in that: Applied to steel bar tying robots, including: A target 2D image and a target depth image of the current binding area of ​​the target steel cage are collected by using a preset 3D camera, and the target 2D image and the target depth image are processed by using a preset surface steel mesh extraction algorithm to obtain a surface steel mesh image corresponding to the target steel cage; The surface steel mesh image is input into a preset binding identification and positioning model to classify the key points in the surface steel mesh image to determine the key point coordinates and depth values ​​corresponding to the bound points, the unbound points and the key points; the key points include the bound points and the unbound points, and the key point coordinates are the pixel coordinates of the key points in the target depth map and the target 2D map; Based on a preset tying path planning method and a preset tying method, all the key point coordinates are sorted by path to obtain a target tying path, and the untied points of the target steel cage are tied based on the target tying path; Wherein, the path sorting of all the key point coordinates based on the preset lashing path planning method and the preset lashing mode to obtain the target lashing path includes: If the preset tying method is the skip tying method, all the key point coordinates are sorted based on the preset tying path planning method to determine the tying path to be optimized, and the tying starting point of the current tying area is determined based on the tying information of the previous tying area, so as to optimize the tying path to be optimized based on the tying starting point to determine the target tying path; The step of determining the tying starting point of the current tying area based on the tying information of the previous tying area, and optimizing the tying path to be optimized based on the tying starting point to determine the target tying path, includes: If the preset binding path planning method is a bow-shaped path, extract a first key point list of odd indexes and a second key point list of even indexes from the binding path to be optimized, determine a third key point list and a binding starting point of the current binding area based on the first key point list, the second key point list and the binding information of the previous binding area, and remove all the bound points in the third key point list to determine the target binding path; the binding path to be optimized is a key point list obtained by sorting the key points based on the bow-shaped path; Among them, the surface steel mesh extraction algorithm is an algorithm constructed using a random sample consensus algorithm improved based on a dynamic Bayesian network; the binding identification and positioning model is a model constructed based on a key point detection deep learning model and a neighborhood deep averaging algorithm, and the key point detection deep learning model is a model obtained by training the YOLOv9-pose model.

2. The deep learning-based steel bar binding decision-making method according to claim 1 is characterized in that: Before the target 2D image and the target depth image are processed by using the preset surface steel mesh extraction algorithm, the method further includes: The initial random sample consensus algorithm is iterated using a dynamic Bayesian network based on a preset stopping criterion; the preset stopping criterion is a stopping criterion based on a preset interior point probability; Determine a target random sample consensus algorithm and a target internal and external point classification from all the random sample consensus algorithms based on the number of internal and external points corresponding to each random sample consensus algorithm obtained in the iteration process; The surface steel mesh extraction algorithm is constructed using the target random sample consensus algorithm.

3. The steel bar binding decision-making method based on deep learning according to claim 2 is characterized in that: The method of iterating the initial random sample consensus algorithm based on a preset stopping criterion using a dynamic Bayesian network includes: Perform weighted sampling on each of the data points based on the inlier and inlier point probabilities corresponding to each data point under the current random sample consensus algorithm to determine a minimum data set, and generate a new random sample consensus algorithm corresponding to the minimum data set based on the minimum data set; Determine the inlier and outlier point classification corresponding to the current random sample consensus algorithm, and update the inlier and outlier point probabilities corresponding to each of the data points according to the inlier and outlier point classification to obtain new inlier and outlier point probabilities; If the current random sample consensus algorithm has the data point whose inlier probability is less than the preset inlier probability, the iteration is stopped.

4. The steel bar tying decision method based on deep learning according to claim 1 is characterized in that: The method of processing the target 2D image and the target depth image using a preset surface steel mesh extraction algorithm to obtain a surface steel mesh image corresponding to the target steel cage includes: Using the surface steel mesh extraction algorithm, pixel alignment is performed on the target 2D image and the target depth image to obtain a target three-dimensional point cloud, and plane fitting is performed on the target meshes in the target three-dimensional point cloud to obtain a fitting plane where each target mesh is located; The surface steel mesh image corresponding to the target steel cage is determined based on the distance between each fitting plane and the origin of the camera coordinate system.

5. The deep learning-based steel bar tying decision-making method according to claim 4 is characterized in that: The step of determining the surface steel mesh image corresponding to the target steel cage based on the distance between each fitting plane and the origin of the camera coordinate system includes: Based on the distance between each fitting plane and the origin of the camera coordinate system, the target fitting plane corresponding to the target steel cage is determined, and when the depth value of the pixel point on the target fitting plane is less than the first preset depth range, the image expansion operation is performed on the target fitting plane to obtain the surface steel mesh image.

6. The steel bar tying decision-making method based on deep learning according to claim 1 is characterized in that: Also includes: If the depth value corresponding to the key point coordinate in the target depth map is not within a second preset depth range, the depth value corresponding to the key point coordinate is processed based on a neighborhood depth averaging algorithm to determine a new depth value corresponding to the key point coordinate.

7. The steel bar tying decision method based on deep learning according to any one of claims 1 to 6, characterized in that: The method of performing path sorting on all the key point coordinates based on the preset lashing path planning method and the preset lashing mode to obtain the target lashing path includes: If the preset binding method is the full binding method, the key point coordinates corresponding to all the unbound points are sorted based on the preset binding path planning method to obtain the target binding path.

8. A steel bar tying decision-making device based on deep learning, characterized in that: Applied to steel bar tying robots, including: An image extraction module is used to collect a target 2D image and a target depth image of the current binding area of ​​the target steel cage using a preset 3D camera, and process the target 2D image and the target depth image using a preset surface steel mesh extraction algorithm to obtain a surface steel mesh image corresponding to the target steel cage; A recognition and positioning module, used for inputting the surface steel mesh image into a preset binding recognition and positioning model to classify the key points in the surface steel mesh image, so as to determine the key point coordinates and depth values ​​corresponding to the bound points, the unbound points and the key points; the key points include the bound points and the unbound points, and the key point coordinates are the pixel coordinates of the key points in the target depth map and the target 2D map; A steel bar binding module, used for performing path sorting on all the key point coordinates based on a preset binding path planning method and a preset binding method to obtain a target binding path, and binding the unbound points of the target steel bar cage based on the target binding path; Wherein, the steel bar binding module comprises: A second path determination submodule is used for, if the preset tying method is the skipping method, to sort the coordinates of all the key points based on the preset tying path planning method to determine the tying path to be optimized, and to determine the tying starting point of the current tying area based on the tying information of the previous tying area, so as to optimize the tying path to be optimized based on the tying starting point to determine the target tying path; Wherein, the second path determination submodule is specifically used for: If the preset binding path planning method is a bow-shaped path, extract a first key point list of odd indexes and a second key point list of even indexes from the binding path to be optimized, determine a third key point list and a binding starting point of the current binding area based on the first key point list, the second key point list and the binding information of the previous binding area, and remove all the bound points in the third key point list to determine the target binding path; the binding path to be optimized is a key point list obtained by sorting the key points based on the bow-shaped path; Among them, the surface steel mesh extraction algorithm is an algorithm constructed using a random sample consensus algorithm improved based on a dynamic Bayesian network; the binding identification and positioning model is a model constructed based on a key point detection deep learning model and a neighborhood deep averaging algorithm, and the key point detection deep learning model is a model obtained by training the YOLOv9-pose model.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the deep learning-based steel bar binding decision-making method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the deep learning-based steel bar binding decision-making method as described in any one of claims 1 to 7 is implemented.

Images