Reinforcing steel bar binding decision-making method, device and equipment based on deep learning and storage medium

Abstract

The invention discloses a steel bar binding decision-making method, device and equipment based on deep learning and a storage medium, relates to the field of civil construction, is applied to a steel bar binding robot, and comprises the following steps: collecting a target 2D map and a target depth map of a current binding area, processing the target 2D map and the target depth map by using a surface layer reinforcing mesh extraction algorithm to obtain a surface layer reinforcing mesh image, and inputting the surface layer reinforcing mesh image into a binding identification positioning model to determine coordinates of bound points, non-bound points and key points; the key points comprise bound points and unbound points; path sorting is carried out on the key point coordinates to obtain a target binding path, and non-binding points are bound based on the target binding path; wherein the surface reinforcing mesh extraction algorithm is an algorithm constructed by using a random sample consistency algorithm improved based on a dynamic Bayesian network; the binding identification positioning model is a model constructed based on a key point detection deep learning model and a neighborhood depth average algorithm. The working efficiency of the steel bar binding robot is improved.

Description

Technical Field

[0001] The present invention relates to the field of civil engineering, and in particular to a deep learning-based steel bar binding decision-making method, device, equipment and storage medium. Background Art

[0002] At present, steel bar binding is mainly done manually, and there are problems such as high labor intensity, high production cost and low work efficiency. The emergence of steel bar binding robots has brought hope for solving this problem. However, the current steel bar binding robots have low efficiency in identifying and locating steel bar binding points during operation, and are not very applicable when making binding decisions for multi-layer steel cages, which reduces the work efficiency of steel bar binding robots and their practicality is low.

[0003] In summary, how to improve the recognition and positioning efficiency of the steel bar binding points of the binding robot to improve its work efficiency is a problem that needs to be solved urgently. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a steel bar tying decision method based on deep learning, which can improve the efficiency of the tying robot in identifying and locating the steel bar tying points to improve its work efficiency. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses a deep learning-based steel bar tying decision-making method, which is applied to a steel bar tying robot, comprising:

[0006] A target 2D image and a target depth image of the current binding area of ​​the target steel cage are collected by using a preset 3D camera, and the target 2D image and the target depth image are processed by using a preset surface steel mesh extraction algorithm to obtain a surface steel mesh image corresponding to the target steel cage;

[0007] The surface steel mesh image is input into a preset binding identification and positioning model to classify the key points in the surface steel mesh image to determine the key point coordinates and depth values ​​corresponding to the bound points, the unbound points and the key points; the key points include the bound points and the unbound points, and the key point coordinates are the pixel coordinates of the key points in the target depth map and the target 2D map;

[0008] Based on a preset tying path planning method and a preset tying method, all the key point coordinates are sorted by path to obtain a target tying path, and the untied points of the target steel cage are tied based on the target tying path;

[0009] Among them, the surface steel mesh extraction algorithm is an algorithm constructed using a random sample consensus algorithm improved based on a dynamic Bayesian network; the binding identification and positioning model is a model constructed based on a key point detection deep learning model and a neighborhood depth averaging algorithm.

[0010] Optionally, before processing the target 2D image and the target depth image using a preset surface steel mesh extraction algorithm, the method further includes:

[0011] The initial random sample consensus algorithm is iterated using a dynamic Bayesian network based on a preset stopping criterion; the preset stopping criterion is a stopping criterion based on a preset interior point probability;

[0012] Determine a target random sample consensus algorithm and a target internal and external point classification from all the random sample consensus algorithms based on the number of internal and external points corresponding to each random sample consensus algorithm obtained in the iteration process;

[0013] The surface steel mesh extraction algorithm is constructed using the target random sample consensus algorithm.

[0014] Optionally, the iterating the initial random sample consensus algorithm based on a preset stopping criterion using a dynamic Bayesian network includes:

[0015] Perform weighted sampling on each of the data points based on the inlier and inlier point probabilities corresponding to each data point under the current random sample consensus algorithm to determine a minimum data set, and generate a new random sample consensus algorithm corresponding to the minimum data set based on the minimum data set;

[0016] Determine the inlier and outlier point classification corresponding to the current random sample consensus algorithm, and update the inlier and outlier point probabilities corresponding to each of the data points according to the inlier and outlier point classification to obtain new inlier and outlier point probabilities;

[0017] If the current random sample consensus algorithm has the data point whose inlier probability is less than the preset inlier probability, the iteration is stopped.

[0018] Optionally, the process of processing the target 2D image and the target depth image using a preset surface steel mesh extraction algorithm to obtain a surface steel mesh image corresponding to the target steel cage includes:

[0019] Using the surface steel mesh extraction algorithm, pixel alignment is performed on the target 2D image and the target depth image to obtain a target three-dimensional point cloud, and plane fitting is performed on the target meshes in the target three-dimensional point cloud to obtain a fitting plane where each target mesh is located;

[0020] The surface steel mesh image corresponding to the target steel cage is determined based on the distance between each fitting plane and the origin of the camera coordinate system.

[0021] Optionally, determining the surface steel mesh image corresponding to the target steel cage based on the distance between each fitting plane and the origin of the camera coordinate system includes:

[0022] Based on the distance between each fitting plane and the origin of the camera coordinate system, the target fitting plane corresponding to the target steel cage is determined, and when the depth value of the pixel point on the target fitting plane is less than the first preset depth range, the image expansion operation is performed on the target fitting plane to obtain the surface steel mesh image.

[0023] Optionally, the steel bar binding decision-making method based on deep learning further includes:

[0024] If the depth value corresponding to the key point coordinate in the target depth map is not within the second preset depth range, the depth value corresponding to the key point coordinate is processed based on a neighborhood depth averaging algorithm to determine a new depth value corresponding to the key point coordinate.

[0025] Optionally, the path sorting of all the key point coordinates based on a preset lashing path planning method and a preset lashing mode to obtain a target lashing path includes:

[0026] If the preset tying method is the full tying method, then based on the preset tying path planning method, the key point coordinates corresponding to all the untied points are sorted by path to obtain the target tying path;

[0027] If the preset binding method is the skipping method, all the key point coordinates are sorted based on the preset binding path planning method to determine the binding path to be optimized, and the binding starting point of the current binding area is determined based on the binding information of the previous binding area, so as to optimize the binding path to be optimized based on the binding starting point to determine the target binding path.

[0028] In a second aspect, the present application discloses a deep learning-based steel bar tying decision-making device, which is applied to a steel bar tying robot, comprising:

[0029] An image extraction module is used to collect a target 2D image and a target depth image of the current binding area of ​​the target steel cage using a preset 3D camera, and process the target 2D image and the target depth image using a preset surface steel mesh extraction algorithm to obtain a surface steel mesh image corresponding to the target steel cage;

[0030] A recognition and positioning module, used for inputting the surface steel mesh image into a preset binding recognition and positioning model to classify the key points in the surface steel mesh image, so as to determine the key point coordinates and depth values ​​corresponding to the bound points, the unbound points and the key points; the key points include the bound points and the unbound points, and the key point coordinates are the pixel coordinates of the key points in the target depth map and the target 2D map;

[0031] A steel bar binding module, used for performing path sorting on all the key point coordinates based on a preset binding path planning method and a preset binding method to obtain a target binding path, and binding the unbound points of the target steel bar cage based on the target binding path;

[0032] Among them, the surface steel mesh extraction algorithm is an algorithm constructed using a random sample consensus algorithm improved based on a dynamic Bayesian network; the binding identification and positioning model is a model constructed based on a key point detection deep learning model and a neighborhood depth averaging algorithm.

[0033] In a third aspect, the present application discloses an electronic device, comprising:

[0034] Memory, used to store computer programs;

[0035] A processor is used to execute the computer program to implement the aforementioned steel bar binding decision-making method based on deep learning.

[0036] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned deep learning-based steel bar binding decision-making method.

[0037] In the present application, when the robot is tying steel bars, it uses a preset 3D camera to collect a target 2D image and a target depth image of the current tying area of ​​the target steel cage, and uses a preset surface steel mesh extraction algorithm to process the target 2D image and the target depth map to obtain a surface steel mesh image corresponding to the target steel cage; the surface steel mesh image is input into a preset binding identification and positioning model to classify the key points in the surface steel mesh image to determine the key point coordinates and depth values ​​corresponding to the tied points, untied points and the key points; the key points include the tied points and the untied points; based on a preset binding path planning method and a preset binding method, all the key point coordinates are sorted by path to obtain a target binding path, and the untied points of the target steel cage are tied based on the target binding path; wherein the surface steel mesh extraction algorithm is an algorithm constructed using a random sample consensus algorithm improved based on a dynamic Bayesian network; the binding identification and positioning model is a model constructed based on a key point detection deep learning model and a neighborhood depth averaging algorithm. It can be seen that the present application utilizes a surface steel mesh extraction algorithm based on an improved random sample consensus algorithm to extract the surface steel mesh from a grayscale image with a complex background, and inputs the surface steel mesh image into a binding identification and positioning model. The binding identification and positioning model can accurately classify the steel bar intersection points (i.e., key points) in the image into bound points and unbound points and obtain the pixel coordinates and depth values ​​of all key points. The three-dimensional spatial coordinates corresponding to the key point can be determined using the pixel coordinates and depth values ​​and the internal parameters of the preset 3D camera. After that, the coordinates of all key points can be sorted based on the preset binding path planning method and the preset binding method, thereby obtaining the target binding path. Finally, the target binding path is used to bind the unbound points in the current binding area of ​​the target steel cage, thereby improving the working efficiency of the binding robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0039] Figure 1 A flow chart of a steel bar binding decision-making method based on deep learning disclosed in this application;

[0040] Figure 2 A schematic diagram of a surface steel mesh extraction result disclosed in this application;

[0041] Figure 3 A pseudo code diagram of an iterative process of a random sample consensus algorithm disclosed in this application;

[0042] Figure 4 A schematic diagram of a specific surface steel mesh extraction process disclosed in this application;

[0043] Figure 5 A comparison diagram of the expansion results of a surface steel mesh image disclosed in this application;

[0044] Figure 6 The surface steel mesh image disclosed in the present application is obtained by processing an original image taken at an inclined angle;

[0045] Figure 7 A schematic diagram of a specific process of training the YOLOv9-pose algorithm disclosed in this application;

[0046] Figure 8 A schematic diagram of a key point detection result disclosed in this application;

[0047] Fig. 9 A schematic diagram of a key point identification and positioning process disclosed in this application;

[0048] Fig.10 Schematic diagram of the steel bar binding path provided for this application, wherein (a) is a U-shaped binding path, (b) is a zigzag binding path, and (c) is a bow-shaped binding path;

[0049] Fig.11 Schematic diagram of the steel bar tying method provided for this application, where (a) is the full tying method and (b) is the skip tying method;

[0050] Fig.12 A schematic diagram of a specific steel bar binding process disclosed in this application;

[0051] Fig.13 A schematic diagram of the positional relationship between a current photographing area and a previous binding area disclosed in the present application;

[0052] Fig.14 Another schematic diagram of the positional relationship between the current photographing area and the previous binding area disclosed in this application;

[0053] Fig.15 A schematic diagram of a specific camera shooting path disclosed in this application;

[0054] Fig.16 This is another specific camera shooting path schematic diagram disclosed in this application;

[0055] Fig.17 A schematic diagram of a specific decision-making process for skipping is disclosed in this application;

[0056] Fig.18Schematic diagram of the working conditions of the lashing area disclosed in the present application, wherein (a) is a working condition in which the camera moves downward, (b) is another working condition in which the camera moves downward, (c) is a working condition in which the camera moves upward, (d) is another working condition in which the camera moves upward, (e) is a working condition in which the camera moves rightward, and (f) is another working condition in which the camera moves rightward;

[0057] Fig.19 A schematic diagram of the structure of a steel bar binding decision-making device based on deep learning disclosed in this application;

[0058] Fig. 20 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] At present, the efficiency of steel bar tying robot in identifying and locating steel bar tying points during operation is low, and the applicability is not high when making tying decisions for multi-layer steel cages, which reduces the working efficiency of steel bar tying robots and has low practicality. In order to solve the above technical problems, the present application discloses a steel bar tying decision method based on deep learning, which can improve the efficiency of tying robots in identifying and locating steel bar tying points to improve their working efficiency.

[0061] See also Figure 1 As shown, an embodiment of the present invention discloses a steel bar tying decision method based on deep learning, which is applied to a steel bar tying robot, comprising:

[0062] Step S11, using a preset 3D camera to collect a target 2D image and a target depth image of the current binding area of ​​the target steel cage, and using a preset surface steel mesh extraction algorithm to process the target 2D image and the target depth image to obtain a surface steel mesh image corresponding to the target steel cage.

[0063] In this embodiment, Figure 2As shown, the preset 3D camera can be used to collect the target 2D image and target depth image of the current binding area of ​​the target steel cage, and then the surface steel mesh image corresponding to the area is obtained by processing the target 2D image and the target depth image. Specifically, the surface steel mesh can be extracted from the grayscale image with complex background using the preset surface steel mesh extraction algorithm. The surface steel mesh extraction algorithm is an algorithm constructed using a random sample consensus algorithm improved based on DBN (Dynamic Bayesian Network).

[0064] In a specific embodiment, Figure 3 As shown in FIG. 1 , the specific process of improving the RANSAC algorithm (Random Sample Consensus) by using DBN may include: iterating the initial random sample consensus algorithm based on a preset stopping criterion by using a dynamic Bayesian network, determining the target random sample consensus algorithm and the target internal and external point classification from all random sample consensus algorithms based on the number of internal and external points corresponding to each random sample consensus algorithm obtained during the iteration, and constructing the surface steel mesh extraction algorithm by using the target random sample consensus algorithm. Among them, the preset stopping criterion is a stopping criterion based on the preset internal point probability, and the specific iterative process may include: weighted sampling of each data point based on the internal and external point probabilities corresponding to each data point under the current random sample consensus algorithm to determine the minimum data set, and generating a new random sample consensus algorithm corresponding to the minimum data set based on the minimum data set; determining the internal and external point classification corresponding to the current random sample consensus algorithm, and updating the internal and external point probabilities corresponding to each data point according to the internal and external point classification to obtain new internal and external point probabilities. If the current random sample consensus algorithm has a data point whose internal point probability is less than the preset internal point probability, the iteration is stopped.

[0065] That is, in this embodiment, the probability of the data point of the initial random sample consensus algorithm is first determined , these probabilities can be scores calculated a priori or uniform. Then at each iteration, a set of data points is selected by weighted sampling , the sampling weight is based on the inlier probability of each data point , based on the selected minimum data set , generate a new random sample consensus algorithm , and determine Classification of internal and external points After each iteration, the inlier / inlier probability of the data point is updated according to the current inlier / inlier classification These updates will affect the weights of the next sampling. If the current inlier probability is lower than the preset inlier probability, the iteration will be stopped to avoid inaccurate algorithm fitting caused by excessive iteration and increase calculation time. When the iteration is completed, the best model will be returned. And the corresponding internal and external point classification .

[0066] In this embodiment, Figure 4 As shown, by inputting the target 2D image and the target depth image of the target steel cage into the surface steel mesh extraction algorithm, the point cloud plane in the original three-dimensional point cloud corresponding to the target 2D image and the target depth image can be fitted. Since the collected target three-dimensional point cloud contains two layers of mesh, the surface steel mesh extraction algorithm can fit two spatial planes at this time. Therefore, these two planes need to be screened to extract the plane where the surface steel mesh is located. It can be understood that since the distance between the plane where the surface steel mesh is located and the origin of the camera coordinate system is smaller, the distances between the two spatial planes and the origin of the camera coordinate system are calculated respectively, and the plane where the surface steel mesh is located with a smaller distance is extracted. At this time, the target 2D image and the target depth image are processed using the preset surface steel mesh extraction algorithm to obtain the surface steel mesh image corresponding to the target steel cage. The specific process may include: using the surface steel mesh extraction algorithm to align the pixels of the target 2D image and the target depth image to obtain the target three-dimensional point cloud, and performing plane fitting on the target mesh in the target three-dimensional point cloud to obtain the fitting plane where each target mesh is located; based on the distance between each fitting plane and the origin of the camera coordinate system, the surface steel mesh image corresponding to the target steel cage is determined. At this time, the extracted surface steel mesh image is as follows: Figure 2 shown.

[0067] In addition, there are pixels with vanishing depth in the extracted surface steel mesh image ( Figure 5 The image is in the rectangular box on the left of the middle image. This will make the surface steel bars incomplete. To address this phenomenon, the image expansion operation can be used to further process the surface steel mesh image. The processed image is as follows Figure 5As shown in the right figure. That is to say, determining the surface steel mesh image corresponding to the target steel cage based on the distance between each fitting plane and the origin of the camera coordinate system can specifically include: determining the target fitting plane corresponding to the target steel cage based on the distance between each fitting plane and the origin of the camera coordinate system, and performing an image expansion operation on the target fitting plane when there is a pixel point in the target fitting plane with a depth value less than a first preset depth range, so as to obtain the surface steel mesh image. That is to say, the initial surface steel mesh image corresponding to the target steel cage can be determined based on the distance between each fitting plane and the origin of the camera coordinate system. If there are pixels with a depth value less than the first preset depth range in the surface steel mesh image obtained at this time, an image expansion operation can be performed on the surface steel mesh image to obtain the final surface steel mesh image. In addition, in the present application, there is no requirement for the angle at which the preset 3D camera shoots the target steel cage. Even if the camera is tilted to the target steel cage for shooting, the surface steel mesh without a depth vanishing point can still be better extracted, such as Figure 6 shown.

[0068] Step S12, input the surface steel mesh image into a preset binding identification and positioning model to classify the key points in the surface steel mesh image to determine the key point coordinates and depth values ​​corresponding to the bound points, unbound points and the key points; the key points include the bound points and the unbound points, and the key point coordinates are the pixel coordinates of the key points in the target depth map and the target 2D map.

[0069] In this embodiment, the binding identification and positioning model is a model constructed based on a key point detection deep learning model and a neighborhood deep averaging algorithm. In a specific implementation, the key point detection deep learning model is a model obtained by training the YOLOv9-pose (YOLOv9 (You Only Look Once version9, a target detection algorithm) series model for posture estimation tasks) model. The training, verification and testing process is as follows: Figure 7 As shown, first, establish a data set. Collect surface steel mesh images, there are three types of labels, tied points, untied points and key points. Specifically, the ratio of the number of images in the training set, validation set and test set can be 8:1:1. For example, the number of images in the three image sets are 160, 20 and 20 respectively. Secondly, train the YOLOv9-pose model, and set the training parameters according to experience, such as learning rate, batchsize (batch size, that is, the number of samples used for each parameter update when training the neural network), epoch (the process in which the entire training data set is completely traversed by the neural network once), etc. Finally, test the model performance. Use the trained YOLOv9-pose model to predict the 20 images in the test set. Some of the prediction results are shown in the following figure. Figure 8As shown in the figure, according to statistics, the model can predict all the key points in these 20 images and classify these key points into bound points and unbound points (the bound points are labeled bz and the unbound points are labeled wbz).

[0070] In this embodiment, Fig. 9 As shown, after obtaining the surface steel mesh image, the image can be input into the trained binding identification and positioning model to classify key points and obtain the corresponding key point coordinates and depth values. The key points are also the intersection points of the steel bars. Specifically, if the depth value corresponding to the key point coordinates in the target depth map is not within the second preset depth range, the depth value corresponding to the key point coordinates can be processed based on the neighborhood depth averaging algorithm to determine the new depth value corresponding to the key point. In other words, after obtaining the surface steel mesh image, the binding identification and positioning model will accurately identify and classify all key points in the image, and locate the unbound points and bound points. If the depth value corresponding to the key point is a normal value (that is, the depth value is within the effective shooting distance range of the camera, that is, within the second preset depth range, such as 300~600mm), the three-dimensional space coordinates corresponding to the key point can be determined based on the pixel coordinates and depth value of the key point and the internal parameters of the preset 3D camera to perform subsequent binding procedures; if the depth value corresponding to the key point is an invalid value, indicating that the key point is a depth vanishing point, the neighborhood depth averaging algorithm is used to process it, and then a new depth value corresponding to the key point is obtained, so as to determine the three-dimensional space coordinates corresponding to the key point based on the pixel coordinates corresponding to the key point and the newly determined depth value and the internal parameters of the preset 3D camera, and perform subsequent binding procedures.

[0071] Step S13: sorting the coordinates of all the key points based on a preset binding path planning method and a preset binding method to obtain a target binding path, and binding the unbound points of the target steel cage based on the target binding path.

[0072] In this embodiment, Fig.10 As shown in the figure, when performing tying operations, a variety of different tying paths can be selected for tying path planning, such as the U-shaped ( Fig.10 (a)), zigzag ( Fig.10 (b)) and bow-shaped ( Fig.10(c)), assuming that the distance between adjacent key points is 1, the total length of the U-shaped path is 35, the total length of the zigzag path is greater than 35, and the total length of the bow-shaped path is 35. Compared with the zigzag path, the total length of the U-shaped path and the bow-shaped path is shorter, the moving distance of the tying robot is shorter, and the tying operation efficiency is higher. The U-shaped path is more suitable for the situation where the global information of the steel cage is known, and the path planning algorithm of the U-shaped path is more complicated. Therefore, it is preferred to use the bow-shaped path as the preset tying path planning method to sort all the key point coordinates.

[0073] In this embodiment, in addition to determining the preset lashing path planning method, it is also necessary to determine the preset lashing method. For different components, full lashing or skip lashing methods can be used. The schematic diagrams of the two lashing methods are as follows: Fig.11 As shown in the figure, full tying means tying all the untied points; skip tying means tying one point and leaving the other point empty. Fig.11 (a) in the figure is full tying, and (b) is skip tying. If the full tying method is adopted, the tied points in the bow-shaped path can be removed first, and then the subsequent tying process can be performed; if the skip tying method is adopted, when tying the current area, the tying starting point of the current tying area needs to be determined based on the information of the previous tying area. Therefore, if Fig.12 As shown, the specific process of sorting the path of all key point coordinates based on the preset binding path planning method and the preset binding method to obtain the target binding path may include: if the preset binding method is the full binding method, then based on the preset binding path planning method, the path of all key point coordinates corresponding to the unbound points is sorted to obtain the target binding path; if the preset binding method is the skipping method, then based on the preset binding path planning method, the path of all the key point coordinates is sorted to determine the binding path to be optimized, and based on the binding information of the previous binding area, the binding starting point of the current binding area is determined, so as to optimize the binding path to be optimized based on the binding starting point to determine the target binding path. It can be understood that if the preset binding method is skipping, when planning the camera shooting path, it is necessary to ensure that each shooting area of ​​the camera (except the first shooting position of the camera) can always cover a steel bar in the previous binding area, such as Fig.13 and Fig.14 As shown, the specific camera shooting path may include Fig.15 and Fig.16 , which are not listed here one by one.

[0074] In a specific embodiment, Fig.17As shown, when all key point coordinates are sorted by path based on the preset binding path planning method and the preset binding method to obtain the target binding path, if the bow-shaped path and the skipping method are used, first obtain the key point list sorted according to the bow-shaped path, and extract the odd-indexed key point list list1 and the even-indexed key point list list2 respectively; secondly, filter out the key point list list3 containing all the bound points; finally, eliminate all the bound points in list3, which is the target list list0, and execute the subsequent binding process.

[0075] It can be understood that, assuming that the tying robot starts working from the upper left corner of the large-sized flat steel cage, and the camera takes pictures along the path shown in Fig.15 As shown, first "from top to bottom", then "from left to right", the following will appear during the camera movement process Fig.18 The six conditions of the photographing area shown are: the camera moves downward ( Fig.18 (a) and Fig.18 (b)), the camera moves up ( Fig.18 (c) and Fig.18 (d)) and the camera moves to the right ( Fig.18 (e) and Fig.18 (f)), the above-mentioned jumping decision method can solve the jumping decision problem of the above 6 working conditions.

[0076] It can be seen that the present application utilizes a surface steel mesh extraction algorithm based on an improved random sample consensus algorithm to extract the surface steel mesh from a grayscale image with a complex background, and inputs the surface steel mesh image into a binding identification and positioning model. The binding identification and positioning model can accurately classify the steel bar intersection points (ie, key points) in the image into bound points and unbound points and obtain the pixel coordinates of all key points. After that, the path of all key point coordinates can be sorted and screened based on a preset binding path planning method and a preset binding method, thereby obtaining a target binding path. Finally, the target binding path is used to bind the unbound points in the current binding area of ​​the target steel cage, thereby improving the working efficiency of the binding robot.

[0077] See also Fig.19 As shown, the present application discloses a steel bar tying decision-making device based on deep learning, which is applied to a steel bar tying robot, comprising:

[0078] An image extraction module 11 is used to collect a target 2D image and a target depth image of a current binding area of ​​a target steel cage using a preset 3D camera, and process the target 2D image and the target depth image using a preset surface steel mesh extraction algorithm to obtain a surface steel mesh image corresponding to the target steel cage;

[0079] The identification and positioning module 12 is used to input the surface steel mesh image into a preset binding identification and positioning model to classify the key points in the surface steel mesh image to determine the key point coordinates and depth values ​​corresponding to the bound points, the unbound points and the key points; the key points include the bound points and the unbound points, and the key point coordinates are the pixel coordinates of the key points in the target depth map and the target 2D map;

[0080] A steel bar binding module 13, used for performing path sorting on all the key point coordinates based on a preset binding path planning method and a preset binding method to obtain a target binding path, and binding the unbound points of the target steel bar cage based on the target binding path;

[0081] Among them, the surface steel mesh extraction algorithm is an algorithm constructed using a random sample consensus algorithm improved based on a dynamic Bayesian network; the binding identification and positioning model is a model constructed based on a key point detection deep learning model and a neighborhood depth averaging algorithm.

[0082] It can be seen that the present application utilizes the surface steel mesh extraction algorithm based on the improved random sample consistency algorithm to extract the surface steel mesh from the complex background grayscale image, and inputs the surface steel mesh image into the binding identification and positioning model. The binding identification and positioning model can accurately classify the steel bar intersection points (ie, key points) in the image into bound points and unbound points and obtain the coordinates of all key points. After that, the path of all key point coordinates can be sorted based on the preset binding path planning method and the preset binding method, so as to obtain the target binding path. Finally, the target binding path is used to bind the unbound points in the current binding area of ​​the target steel cage, thereby improving the working efficiency of the binding robot.

[0083] In a specific embodiment, the device may further include:

[0084] An algorithm iteration module, used to iterate the initial random sample consensus algorithm based on a preset stopping criterion using a dynamic Bayesian network; the preset stopping criterion is a stopping criterion based on a preset interior point probability;

[0085] An algorithm determination module, used to determine a target random sample consensus algorithm and a target internal and external point classification from all the random sample consensus algorithms based on the number of internal and external points corresponding to each random sample consensus algorithm obtained in an iterative process;

[0086] An algorithm construction module is used to construct the surface steel mesh extraction algorithm using the target random sample consistency algorithm.

[0087] In a specific implementation, the algorithm iteration module may specifically include:

[0088] An algorithm generation submodule, for performing weighted sampling on each data point based on the inlier and inlier point probabilities corresponding to each data point under the current random sample consensus algorithm to determine a minimum data set, and generating a new random sample consensus algorithm corresponding to the minimum data set based on the minimum data set;

[0089] A probability update submodule, used to determine the inlier and lier point classification corresponding to the current random sample consensus algorithm, and to update the inlier and lier point probabilities corresponding to each of the data points according to the inlier and lier point classification to obtain new inlier and lier point probabilities;

[0090] The iteration stop submodule is used to stop iteration if the current random sample consensus algorithm has the data point whose inlier probability is less than the preset inlier probability.

[0091] In a specific implementation, the image extraction module 11 may specifically include:

[0092] A mesh fitting submodule is used to perform pixel alignment on the target 2D image and the target depth image using the surface steel mesh extraction algorithm to obtain a target three-dimensional point cloud, and perform plane fitting on the target meshes in the target three-dimensional point cloud to obtain a fitting plane where each target mesh is located;

[0093] The target image determination submodule is used to determine the surface steel mesh image corresponding to the target steel cage based on the distance between each fitting plane and the origin of the camera coordinate system.

[0094] In a specific implementation, the target image determination submodule may specifically include:

[0095] A target image determination unit is used to determine the target fitting plane corresponding to the target steel cage based on the distance between each fitting plane and the origin of the camera coordinate system, and to perform an image expansion operation on the target fitting plane when the depth value of a pixel point on the target fitting plane is less than a first preset depth range, so as to obtain the surface steel mesh image.

[0096] In a specific embodiment, the device may further include:

[0097] A depth value updating module is used to process the depth value corresponding to the key point coordinates based on a neighborhood depth averaging algorithm to determine a new depth value corresponding to the key point coordinates if the depth value corresponding to the key point coordinates in the target depth map is not within a second preset depth range.

[0098] In a specific implementation, the steel bar binding module 13 may specifically include:

[0099] A first path determination submodule, for, if the preset tying method is the full tying method, performing path sorting on the key point coordinates corresponding to all the untied points based on the preset tying path planning method to obtain the target tying path;

[0100] The second path determination submodule is used for, if the preset binding method is the skipping method, to sort the path of all the key point coordinates based on the preset binding path planning method to determine the binding path to be optimized, and to determine the binding starting point of the current binding area based on the binding information of the previous binding area, so as to optimize the binding path to be optimized based on the binding starting point to determine the target binding path.

[0101] Furthermore, the present application also discloses an electronic device. Fig. 20 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0102] Fig. 20 A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the deep learning-based steel bar binding decision method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0103] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0104] In addition, the memory 22 as a carrier for resource storage may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon may include an operating system 221, a computer program 222, etc., and the storage method may be temporary storage or permanent storage.

[0105] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, and may be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the deep learning-based steel bar binding decision method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 may further include computer programs that can be used to complete other specific tasks.

[0106] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned deep learning-based steel bar binding decision method disclosed above is implemented. The specific steps of the method can be referred to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.

[0107] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0108] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0109] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0110] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0111] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A steel bar tying decision method based on deep learning, characterized in that: Applied to steel bar tying robots, including: A target 2D image and a target depth image of the current binding area of ​​the target steel cage are collected by using a preset 3D camera, and the target 2D image and the target depth image are processed by using a preset surface steel mesh extraction algorithm to obtain a surface steel mesh image corresponding to the target steel cage; The surface steel mesh image is input into a preset binding identification and positioning model to classify the key points in the surface steel mesh image to determine the key point coordinates and depth values ​​corresponding to the bound points, the unbound points and the key points; the key points include the bound points and the unbound points, and the key point coordinates are the pixel coordinates of the key points in the target depth map and the target 2D map; Based on a preset tying path planning method and a preset tying method, all the key point coordinates are sorted by path to obtain a target tying path, and the untied points of the target steel cage are tied based on the target tying path; Among them, the surface steel mesh extraction algorithm is an algorithm constructed using a random sample consensus algorithm improved based on a dynamic Bayesian network; the binding identification and positioning model is a model constructed based on a key point detection deep learning model and a neighborhood depth averaging algorithm.

2. The deep learning-based steel bar binding decision-making method according to claim 1 is characterized in that: Before the target 2D image and the target depth image are processed by using the preset surface steel mesh extraction algorithm, the method further includes: The initial random sample consensus algorithm is iterated using a dynamic Bayesian network based on a preset stopping criterion; the preset stopping criterion is a stopping criterion based on a preset interior point probability; Determine a target random sample consensus algorithm and a target internal and external point classification from all the random sample consensus algorithms based on the number of internal and external points corresponding to each random sample consensus algorithm obtained in the iteration process; The surface steel mesh extraction algorithm is constructed using the target random sample consensus algorithm.

3. The steel bar binding decision-making method based on deep learning according to claim 2 is characterized in that: The method of iterating the initial random sample consensus algorithm based on a preset stopping criterion using a dynamic Bayesian network includes: Perform weighted sampling on each of the data points based on the inlier and inlier point probabilities corresponding to each data point under the current random sample consensus algorithm to determine a minimum data set, and generate a new random sample consensus algorithm corresponding to the minimum data set based on the minimum data set; Determine the inlier and outlier point classification corresponding to the current random sample consensus algorithm, and update the inlier and outlier point probabilities corresponding to each of the data points according to the inlier and outlier point classification to obtain new inlier and outlier point probabilities; If the current random sample consensus algorithm has the data point whose inlier probability is less than the preset inlier probability, the iteration is stopped.

4. The steel bar tying decision-making method based on deep learning according to claim 1 is characterized in that: The method of processing the target 2D image and the target depth image using a preset surface steel mesh extraction algorithm to obtain a surface steel mesh image corresponding to the target steel cage includes: Using the surface steel mesh extraction algorithm, pixel alignment is performed on the target 2D image and the target depth image to obtain a target three-dimensional point cloud, and plane fitting is performed on the target meshes in the target three-dimensional point cloud to obtain a fitting plane where each target mesh is located; The surface steel mesh image corresponding to the target steel cage is determined based on the distance between each fitting plane and the origin of the camera coordinate system.

5. The deep learning-based steel bar tying decision-making method according to claim 4 is characterized in that: The step of determining the surface steel mesh image corresponding to the target steel cage based on the distance between each fitting plane and the origin of the camera coordinate system includes: Based on the distance between each fitting plane and the origin of the camera coordinate system, the target fitting plane corresponding to the target steel cage is determined, and when the depth value of the pixel point on the target fitting plane is less than the first preset depth range, the image expansion operation is performed on the target fitting plane to obtain the surface steel mesh image.

6. The steel bar tying decision-making method based on deep learning according to claim 1 is characterized in that: Also includes: If the depth value corresponding to the key point coordinate in the target depth map is not within a second preset depth range, the depth value corresponding to the key point coordinate is processed based on a neighborhood depth averaging algorithm to determine a new depth value corresponding to the key point coordinate.

7. The steel bar tying decision method based on deep learning according to any one of claims 1 to 6, characterized in that: The method of performing path sorting on all the key point coordinates based on the preset lashing path planning method and the preset lashing mode to obtain the target lashing path includes: If the preset tying method is the full tying method, then based on the preset tying path planning method, the key point coordinates corresponding to all the untied points are sorted by path to obtain the target tying path; If the preset binding method is the skipping method, all the key point coordinates are sorted based on the preset binding path planning method to determine the binding path to be optimized, and the binding starting point of the current binding area is determined based on the binding information of the previous binding area, so as to optimize the binding path to be optimized based on the binding starting point to determine the target binding path.

8. A steel bar tying decision-making device based on deep learning, characterized in that: Applied to steel bar tying robots, including: An image extraction module is used to collect a target 2D image and a target depth image of the current binding area of ​​the target steel cage using a preset 3D camera, and process the target 2D image and the target depth image using a preset surface steel mesh extraction algorithm to obtain a surface steel mesh image corresponding to the target steel cage; A recognition and positioning module, used for inputting the surface steel mesh image into a preset binding recognition and positioning model to classify the key points in the surface steel mesh image, so as to determine the key point coordinates and depth values ​​corresponding to the bound points, the unbound points and the key points; the key points include the bound points and the unbound points, and the key point coordinates are the pixel coordinates of the key points in the target depth map and the target 2D map; A steel bar binding module, used for performing path sorting on all the key point coordinates based on a preset binding path planning method and a preset binding method to obtain a target binding path, and binding the unbound points of the target steel bar cage based on the target binding path; Among them, the surface steel mesh extraction algorithm is an algorithm constructed using a random sample consensus algorithm improved based on a dynamic Bayesian network; the binding identification and positioning model is a model constructed based on a key point detection deep learning model and a neighborhood depth averaging algorithm.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the deep learning-based steel bar binding decision-making method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the deep learning-based steel bar binding decision-making method as described in any one of claims 1 to 7 is implemented.

Images