Automatic operation method of steel bar tying robot and steel bar tying robot

Abstract

The present application discloses an autonomous operation method of a steel bar binding robot and a steel bar binding robot, and relates to the field of robot technology. The method comprises: generating a three-dimensional point cloud model based on a global image of a steel bar cage; dividing the three-dimensional point cloud model into blocks according to the positions of the identified steel bar nodes to obtain a robot positioning point sequence; executing a traversal step of the robot positioning point sequence; the traversal step of the robot positioning point sequence comprises: generating a moving path guided to the currently traversed positioning point through a surrounding environment image, moving to the currently traversed positioning point based on the moving path, generating a corresponding local photo point sequence, and executing a traversal step of the local photo point sequence; the traversal step of the local photo point sequence comprises: obtaining a local image of the steel bar cage corresponding to the currently traversed local photo point, generating a binding path based on the local image to locally bind the steel bar cage; and realizing the automated construction of a three-dimensional steel bar skeleton.

Description

Technical Field

[0001] The present application relates to the field of robot technology, and in particular to an automatic operation method of a steel bar tying robot and a steel bar tying robot. Background Art

[0002] In the field of construction engineering, steel bar construction has always been one of the most time-consuming links in the construction process. The main feature of this link is a large number of manual repetitive operations, which is not only inefficient, but also very likely to lead to unstable construction quality due to human factors.

[0003] Although the industrialization of steel bar operations has made some progress in recent years, such as the steel cage automation equipment developed by a foreign company, which can automatically arrange and tie under the guidance of a three-dimensional model, this type of equipment is still limited by many factors in practical application. Although the steel frame intelligent production line of a large domestic construction company has achieved automated welding in a factory environment, it is limited to horizontal steel mesh and is still insufficient for complex three-dimensional steel frame processing.

[0004] In terms of mechanized construction, although there are products such as automatic plate reinforcement layout equipment from a foreign company and automatic plate reinforcement binding equipment from another domestic construction company, they are mostly limited to the laying and binding of flat steel bars. For on-site construction of three-dimensional steel bar skeletons, it is impossible to achieve automated steel bar binding. Summary of the invention

[0005] In view of this, the main purpose of the present application is to provide an automatic operation method of a steel bar tying robot and a steel bar tying robot, with the aim of realizing the automated construction of a three-dimensional steel bar skeleton.

[0006] The first aspect of the present application provides an automatic operation method of a steel bar tying robot, which is applied to the steel bar tying robot, and the method comprises:

[0007] The global image of the steel cage is obtained through the camera group on the production line;

[0008] Generate a 3D point cloud model of the steel cage based on the global image;

[0009] Identify the steel bar nodes in the 3D point cloud model, and divide the 3D point cloud model into blocks according to the positions of the steel bar nodes to obtain the robot positioning point sequence;

[0010] Execute the traversal step of the robot positioning point sequence to globally tie the steel cage;

[0011] The traversal step of the robot positioning point sequence includes: by acquiring the surrounding environment image of the steel cage, generating a moving path for guiding the steel bar tying robot to move to the currently traversed positioning point, moving to the currently traversed positioning point based on the moving path, generating a local photographing point sequence corresponding to the currently traversed bundling positioning point, and executing the traversal step of the local photographing point sequence to locally tie the steel cage;

[0012] The traversal step of the local photographing point sequence to locally bind the steel cage includes: obtaining a local image of the steel cage corresponding to the currently traversed local photographing point, generating a binding path based on the local image, and locally binding the steel cage based on the binding path.

[0013] In some implementations of the first aspect of the present application, after generating a three-dimensional point cloud model of the steel cage based on the global image, the method further includes:

[0014] Extracting geometric information of the steel cage from the three-dimensional point cloud model, the geometric information including at least one of the following information: steel bar spacing, steel cage size, and steel bar diameter;

[0015] Compare the geometric information with the pre-set Building Information Model (BIM) model to determine whether the steel cage meets the standard specifications indicated by the BIM model;

[0016] Identify steel bar nodes in the 3D point cloud model, including: when the steel cage meets the standard specifications indicated by the BIM model, identify the steel bar nodes in the 3D point cloud model.

[0017] In some implementations of the first aspect of the present application, when the spacing between the steel bars of the steel cage does not meet the standard specifications indicated by the BIM model, the spacing between the steel bars of the steel cage is adjusted until it meets the standard specifications indicated by the BIM model.

[0018] In some implementations of the first aspect of the present application, after executing the step of traversing the robot positioning point sequence, the method further includes:

[0019] Obtain a new global image of the steel cage, and determine whether there are any missing binding points in the steel cage based on the new global image;

[0020] If there are missing binding points in the steel cage, a local image corresponding to the missing binding point is obtained, a supplementary binding path is generated based on the local image corresponding to the missing binding point, and the missing binding point is supplemented based on the supplementary binding path.

[0021] In some implementations of the first aspect of the present application, by acquiring an image of the surrounding environment of the steel cage, generating a moving path to guide the steel bar tying robot to move to the currently traversed positioning point, including:

[0022] The surrounding environment image of the steel cage is obtained, and a moving path is generated through a liquid neural network to guide the steel tying robot to move to the currently traversed positioning point.

[0023] In some implementations of the first aspect of the present application, an image of the surrounding environment of the steel cage is obtained, and a moving path for guiding the steel bar tying robot to move to the currently traversed positioning point is generated through a liquid neural network, including:

[0024] Acquire the surrounding environment image of the steel cage;

[0025] Inputting the surrounding environment image into the input layer of the liquid neural network so that the input layer outputs an image code of the surrounding environment image;

[0026] The image code is input into the liquid layer of the liquid neural network, so that the liquid layer performs nonlinear processing and pattern recognition on the image code through dense connections and dynamic behaviors of neurons to obtain information code, wherein the information code includes: feature information extracted from the image code and a decision corresponding to the feature information;

[0027] The information encoding is input into the output layer of the liquid neural network, so that the output layer converts the feature information in the information encoding and the decision corresponding to the feature information into a series of action navigation instructions to obtain a moving path to guide the steel bar binding robot to move to the currently traversed positioning point.

[0028] In some implementations of the first aspect of the present application, the training process of the liquid neural network includes:

[0029] A training set is obtained, where the training set includes: sensor data during the operation of the steel bar tying robot by the operator and action instructions received by the steel bar tying robot;

[0030] Sensor data is used as the input of the liquid neural network, and action instructions are used as the output of the liquid neural network. The model is trained through the dynamic adaptability and time-varying weight adjustment capabilities of the liquid neural network. During the training process, the directional propagation algorithm is used to optimize the network parameters until the liquid neural network completes the model training.

[0031] In some implementations of the first aspect of the present application, obtaining a local image of the steel cage corresponding to the local photographing point currently traversed, and generating a binding path based on the local image, includes:

[0032] Obtain the local image of the steel cage corresponding to the local photo point currently traversed;

[0033] Inputting the local image into the neural descriptor model so that the neural descriptor model recognizes and outputs the position and posture of the steel bar node in the local image;

[0034] The binding path is generated based on the position and posture of the steel bar node in the local image.

[0035] In some implementations of the first aspect of the present application, the neural descriptor model is pre-trained by annotating a plurality of sets of three-dimensional point cloud data of a set of steel bar nodes for model training, and the local image is a local point cloud image including a plurality of local steel bar nodes;

[0036] The local image is input into the neural descriptor model so that the neural descriptor model recognizes and outputs the position and posture of the steel bar node, including:

[0037] Input the local image to the neural descriptor model;

[0038] The coordinates of each local steel bar node in the local image are mapped to the spatial descriptor through the neural descriptor model to obtain the neural descriptor corresponding to each local steel bar node. The coordinates of each local steel bar node are the coordinates of each local steel bar node in the local coordinate system established based on the local image. The neural descriptor is used to describe the position of the local steel bar node in the local image.

[0039] A set of rigid query points are selected to represent the local coordinate frame of the steel bar node. The posture of each local steel bar node in the local coordinate frame is estimated through a neural description model to obtain a posture descriptor of each local steel bar node. The posture descriptor is used to describe the posture of the local steel bar node in the image.

[0040] A second aspect of the present application provides a steel bar tying robot, the steel bar tying robot comprising: a sensing module, a terminal control module, a tying module, a chassis module and a communication module;

[0041] The terminal control module is used to control the perception module to obtain and return the global image of the steel cage through the camera group on the production line, generate a three-dimensional point cloud model of the steel cage based on the global image, identify the steel nodes in the three-dimensional point cloud model, and divide the three-dimensional point cloud model into blocks according to the positions of the steel nodes to obtain the robot positioning point sequence;

[0042] The terminal control module is also used to execute the traversal steps of the robot positioning point sequence to control the binding module to globally bind the steel cage;

[0043] The traversal steps of the robot positioning point sequence include: controlling the perception module to obtain and return the surrounding environment image of the steel cage, generating a moving path to guide the steel cage tying robot to move to the currently traversed positioning point, so as to control the chassis module to move the steel cage tying robot to the currently traversed positioning point; generating a local photo point sequence corresponding to the currently traversed positioning point, and executing the traversal steps of the local photo point sequence, so as to control the tying module to perform local tying of the steel cage;

[0044] The traversal step of the local photo point sequence includes: controlling the perception module to obtain and return a local image of the steel cage corresponding to the currently traversed local photo point, and generating a binding path based on the local image;

[0045] A perception module is used to obtain a global image, a surrounding environment image, and a local image of the steel cage;

[0046] The tying module is used to perform local tying of the steel cage based on the tying path;

[0047] A chassis module, used for moving the steel cage tying robot to a currently traversed positioning point based on a moving path;

[0048] The communication module is used to establish communication between the perception module, the binding module and the chassis module and the terminal control module respectively.

[0049] The technical solution provided by this application has the following beneficial effects:

[0050] In the technical solution provided in the present application, first, a steel cage binding robot (hereinafter referred to as the binding robot) obtains a global image of a steel cage, and generates a three-dimensional point cloud model of the steel cage based on the global image to establish a three-dimensional skeleton of the steel cage; then, the steel cage binding robot identifies the steel nodes in the three-dimensional point cloud model, and blocks the three-dimensional point cloud model according to the positions of these nodes to obtain a robot positioning point sequence, so that the steel cage robot moves to each positioning point in the order determined by this sequence to carry out the binding operation, so as to complete the global binding of the steel cage. In the process of traversing the robot positioning point sequence, the binding robot generates a moving path by obtaining an image of the robot's surrounding environment, and completes automatic navigation and real-time obstacle avoidance based on the moving path to reach the current traversed positioning point. After arriving at the positioning point, the binding robot generates a local photo point sequence corresponding to the current positioning point, and carries out a local binding operation by traversing the local photo point sequence. Specifically, in the process of traversing the local photo point sequence, the binding robot will obtain the local image of the steel cage corresponding to the local photo point currently traversed, and generate a binding path based on the local image, and carry out a local binding operation through the binding path. This enables the automated construction of three-dimensional steel skeletons. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic flow chart of an automatic operation method of a steel bar tying robot provided in an embodiment of the present application;

[0052] Figure 2 A schematic flow chart of another automatic operation method of a steel bar tying robot provided in an embodiment of the present application;

[0053] Figure 3 A schematic diagram of an LTC neuron provided in an embodiment of the present application;

[0054] Figure 4 A schematic flow chart of another automatic operation method of a steel bar tying robot provided in an embodiment of the present application;

[0055] Figure 5 A schematic flow chart of another automatic operation method of a steel bar tying robot provided in an embodiment of the present application;

[0056] Figure 6 A schematic flow chart of another automatic operation method of a steel bar tying robot provided in an embodiment of the present application;

[0057] Figure 7 A schematic diagram of the structure of a neural network model provided in an embodiment of the present application;

[0058] Figure 8 A schematic flow chart of another automatic operation method of a steel bar tying robot provided in an embodiment of the present application;

[0059] Fig. 9 A workflow diagram of a steel bar tying robot provided in an embodiment of the present application;

[0060] Fig.10 A schematic diagram of the structure of a steel bar tying robot provided in an embodiment of the present application;

[0061] Fig.11 A schematic structural diagram of a universal steel bar tying robot provided in an embodiment of the present application;

[0062] Fig.12 A schematic structural diagram of a plate reinforcement binding robot provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0064] In order to realize the automated construction of a three-dimensional steel cage skeleton, an embodiment of the present application provides an automatic operation method of a steel bar tying robot and a steel bar tying robot.

[0065] See also Figure 1 As shown, in order to realize the automated construction of a three-dimensional steel bar skeleton, the embodiment of the present application provides an automatic operation method of a steel bar tying robot (hereinafter referred to as the tying robot), which specifically includes the following steps:

[0066] S101: The tying robot obtains a global image of the steel cage through a camera group on the production line.

[0067] In the embodiments of the present application, the global image of the steel cage refers to an image covering the entire steel cage, which includes information such as the position, structure and geometric shape of all steel bars; the image of the steel cage can be acquired from different angles by a group of cameras on the production line to establish a global image of the steel cage.

[0068] A steel bar tying robot refers to a robot specially used for steel bar tying operations in construction projects, which can automatically identify steel bar tying points and perform tying operations. The hardware system of the tying robot in the embodiment of the present application may include the following five modules, namely, a perception module, a tying module, a chassis module, a communication module, and a terminal control module, wherein the specific functions of each module are described in the subsequent embodiments.

[0069] In the hardware system of the tying robot, the perception module of the tying robot includes a camera group fixed on the workshop production line. The camera group can be composed of multiple binocular cameras or structured light cameras, which are used to simultaneously obtain a global image covering the entire steel cage from different angles.

[0070] S102: The tying robot generates a three-dimensional point cloud model of the steel cage based on the global image.

[0071] In the embodiments of the present application, the three-dimensional point cloud model refers to a point cloud model that represents the position, size, structure and shape of steel bars in a steel cage through a series of three-dimensional coordinate points.

[0072] In the hardware system of the tying robot, after the perception module obtains the global image through the camera group, the communication module supports transmitting the global image obtained by the perception module to the terminal control module via wired or wireless means. After receiving the global image, the terminal control module uses computer vision technology and 3D reconstruction algorithms to generate a 3D point cloud model of the steel cage.

[0073] S103: The tying robot identifies the steel bar nodes in the three-dimensional point cloud model, and divides the three-dimensional point cloud model into blocks according to the positions of the steel bar nodes to obtain a robot positioning point sequence.

[0074] In the embodiments of the present application, a steel bar node refers to an interaction point or a connection point formed by at least two steel bars in a steel cage, and these steel bar nodes need to be tied to ensure the stability of the entire steel cage structure.

[0075] After identifying the steel bar nodes in the three-dimensional point cloud model, the present application further divides the three-dimensional point cloud model into multiple local areas according to the positions of each steel bar node. Each local area contains a certain number of steel bar nodes, so as to facilitate the binding robot to complete the steel bar node binding operation in the local area at one time.

[0076] The positioning point sequence is used to determine the set of positioning points for the tying robot to complete the tying of all steel bar nodes in the local area after each block without moving again. Each positioning point in the positioning point sequence can be determined based on the single operation range of the tying robot.

[0077] In the hardware system of the tying robot, the terminal control module uses image processing technology to identify the steel bar nodes in the 3D point cloud model, and divides the 3D point cloud model into blocks according to the positions of the steel bar nodes to obtain the robot positioning point sequence.

[0078] S104: The tying robot executes the traversal step of the robot positioning point sequence to perform global tying of the steel cage; wherein, the traversal step of the robot positioning point sequence specifically includes: by acquiring the surrounding environment image of the steel cage, generating a moving path to guide the steel tying robot to move to the currently traversed positioning point; based on the moving path, moving to the currently traversed positioning point, generating a local photo point sequence corresponding to the currently traversed binding positioning point, and executing the traversal step of the local photo point sequence to perform local tying of the steel cage. wherein, the traversal step of the local photo point sequence to perform local tying of the steel cage includes: acquiring a local image of the steel cage corresponding to the currently traversed local photo point, generating a tying path based on the local image, and performing local tying of the steel cage based on the tying path.

[0079] In the hardware system of the tying robot, the terminal control module traverses the robot's positioning point sequence to control the tying module, chassis module and perception module to complete the global tying operation of the steel cage.

[0080] Among them, the image of the surrounding environment of the steel cage obtained during the robot's positioning point sequence traversal refers to the image data of the surrounding environment obtained by the tying robot through the camera installed on the robot during the working process. It is used for the tying robot to perceive the actual environment in the working area, including surrounding obstacles and other factors that may affect the movement or operation of the tying robot.

[0081] The surrounding environment image is obtained for the generation of the moving path, which refers to the route of the tying robot from the current position to the currently traversed positioning point. The generation of the moving path enables the tying robot to autonomously complete navigation and obstacle avoidance.

[0082] After the tying robot reaches the currently traversed positioning point based on the moving path, a local photo point sequence corresponding to the positioning point will be generated. The local photo point sequence refers to the predetermined position sequence for the tying robot to obtain local images. These photo points are used to obtain detailed images of the tying positions, that is, local images of the steel cage, so as to facilitate the subsequent generation of tying paths and perform precise tying operations.

[0083] In the hardware system of the tying robot, first, the terminal control module instructs the perception module to obtain the surrounding environment image, and the perception module includes the laser radar and camera carried by the tying robot to obtain the surrounding environment image for environmental mapping, autonomous navigation and real-time obstacle avoidance. Then, after receiving the surrounding environment image, the terminal control module generates a moving path to guide the tying robot to move to the currently traversed positioning point, for example, generates a moving path to move to the first positioning point. After that, the terminal control module sends the moving path to the chassis module, so that the chassis module drives the tying robot to move to the currently traversed positioning point according to the moving path. Finally, after the tying robot reaches the currently traversed positioning point, the terminal control module generates a local photo sequence corresponding to the currently traversed positioning point. Among them, the information interaction between the modules in the above steps is supported by the communication module.

[0084] The local image obtained during the traversal of the local photo point sequence refers to the detailed image of the local area of ​​the steel cage obtained by the tying robot at the local photo point currently traversed, and the local image provides the specific position and geometric features of the steel nodes in the area. The tying path is the specific path for the tying robot to perform tying operations based on the specific position and geometric features of the steel nodes identified by the local image.

[0085] In the hardware system of the tying robot, the perception module includes a camera mounted at the end of the execution end. The terminal control module issues instructions to the tying module based on the currently traversed local photo point to control the execution end to carry the terminal camera to reach the local photo point, and after reaching the local photo point, instructs the perception module to obtain and return the local image to the terminal control module. After receiving the local image, the terminal control module generates a tying path based on the local image and sends the tying path to the tying module to control the execution end to complete the local tying operation.

[0086] exist Figure 1In the process shown, first, the steel cage binding robot (hereinafter referred to as the binding robot) obtains the global image of the steel cage, and generates a three-dimensional point cloud model of the steel cage based on the global image to establish the three-dimensional skeleton of the steel cage; then, the steel cage binding robot identifies the steel nodes in the three-dimensional point cloud model, and divides the three-dimensional point cloud model into blocks according to the positions of these nodes to obtain a robot positioning point sequence, so that the steel cage robot moves to each positioning point in the order determined by this sequence to carry out the binding operation, so as to complete the global binding of the steel cage. In the process of traversing the robot positioning point sequence, the binding robot generates a moving path by obtaining an image of the robot's surrounding environment, and completes automatic navigation and real-time obstacle avoidance based on the moving path to reach the current traversed positioning point. After arriving at the positioning point, the binding robot generates a local photo point sequence corresponding to the current positioning point, and performs local binding operations by traversing the local photo point sequence. Specifically, in the process of traversing the local photo point sequence, the binding robot will obtain the local image of the steel cage corresponding to the local photo point currently traversed, and generate a binding path based on the local image, and carry out local binding operations through the binding path. This enables the automated construction of three-dimensional steel skeletons, which can achieve 24-hour uninterrupted construction, significantly improve project progress and efficiency, and provide support for large projects or projects that need to be completed quickly. At the same time, by reducing the demand for labor, especially in steel engineering, it helps to improve the level of unmanned operations on construction sites, which is in line with the current labor shortage situation and reduces dependence on human resources.

[0087] In some implementations of the embodiments of the present application, the moving path can be generated by a liquid neural network or an A* algorithm, wherein the liquid neural network is a time-connected neural network model that is used to process time series data and adapt to changing conditions, and can show strong adaptability and robustness in dynamic and complex environments. The A* algorithm is a heuristic algorithm for path search, which is widely used in the field of robot navigation and can find the optimal path from the starting point to the end point.

[0088] Preferably, considering that the continuous movement of workers and equipment in a concrete component prefabrication plant will cause large changes in the environment, the traditional A* algorithm will not be suitable for a fast-changing environment, so the embodiment of the present application preferentially uses a liquid neural network model to generate a moving path. On this basis, see Figure 2 As shown, the step of generating a moving path by a liquid neural network may specifically include the following steps:

[0089] S201: The tying robot obtains the surrounding environment image of the steel cage through the camera group on the production line.

[0090] S202: The tying robot inputs the surrounding environment image into the input layer of the liquid neural network, so that the input layer outputs the image code of the surrounding environment image.

[0091] In an embodiment of the present application, the surrounding environment image is converted into a data representation that can be understood and processed by the liquid neural network, that is, the image encoding of the surrounding environment image.

[0092] S203: The tying robot inputs the image code into the liquid layer of the liquid neural network, so that the liquid layer performs nonlinear processing and pattern recognition on the image code through dense connections and dynamic behaviors of neurons to obtain information code, and the information code includes: feature information extracted from the image code, and decisions corresponding to the feature information.

[0093] In an embodiment of the present application, the liquid layer is the core processing layer of the liquid neural network, which simulates the dynamic behavior and dense connections of biological neurons, performs in-depth analysis of image coding, and extracts important feature information in the environment. Here, the in-depth analysis of image coding includes linear processing and pattern recognition. Nonlinear processing is used to capture complex nonlinear relationships in image coding in order to extract more valuable feature information; pattern recognition is used to identify key feature information such as obstacles from feature information, and make decisions based on the identified key feature information, that is, decisions corresponding to the feature information.

[0094] In a specific implementation, the liquid layer is configured as a sparse network consisting of twenty Liquid Time-constant Networks (LTC) neurons equipped with convolutional heads, and the LTC neurons are composed of Figure 3 As shown. Among them, the state update of each neuron is usually expressed by the following differential equation:

[0095] ;

[0096] in Representing the moment, the neuron Status; represents the neuron time constant; is a nonlinear function describing synaptic connections, defining the Status To the neuron is a nonlinear function of the synaptic connections. is the synaptic weight parameter, which represents the strength or reversal potential of the connection.

[0097] S204: The tying robot inputs the information code into the output layer of the liquid neural network, so that the output layer converts the feature information in the information code and the decision corresponding to the feature information into a series of action navigation instructions to obtain a moving path to guide the steel bar tying robot to move to the currently traversed positioning point.

[0098] In the embodiments of the present application, the action instructions refer to the specific operation instructions required for the lashing robot to move in the environment, such as direction, speed, steering angle, moving distance, etc., and a series of action operation instructions constitute the moving path of the lashing robot, such as when and at what angle to switch the moving direction, and then at what speed and how much distance to move. Specifically, the output layer of the liquid neural network inputs the feature information output by the liquid layer and the decision corresponding to the feature information to understand the key feature information and corresponding decisions in the current environment, such as determining which obstacles are, whether to avoid the obstacles, etc.; then, convert it into specific action instructions, such as turning left 30 degrees and moving forward 1 meter to avoid obstacles.

[0099] In the hardware system of the tying robot, after obtaining the surrounding environment image returned by the perception module, the terminal control module inputs the surrounding environment image into the liquid neural network model, and generates a moving path based on the output of the liquid neural network model.

[0100] exist Figure 2 In the process shown, the model movement path is generated by the liquid neural network, which enables the tying robot to adapt to the rapidly changing working environment.

[0101] In the above steps S201-S204, the liquid neural network model is pre-trained. In the navigation and obstacle avoidance part of the lashing robot, the embodiment of the present application builds an autonomous navigation and actual obstacle avoidance framework based on end-to-end imitation learning, see Figure 4 , the specific training process of the liquid neural network model includes S401-S402:

[0102] S401: The tying robot obtains a training set, which includes: sensor data during the operation of the operator controlling the movement of the steel tying robot and action instructions received by the steel tying robot.

[0103] In an embodiment of the present application, an operator first controls the tying robot to move within its workshop and collects data from sensors on the tying robot, such as camera images and lidar data, as well as corresponding action instructions, such as steering angle and speed.

[0104] S402: The tying robot uses sensor data as the input of the liquid neural network and action instructions as the output of the liquid neural network. The model is trained through the dynamic adaptability and time-varying weight adjustment capabilities of the liquid neural network. During the training process, the directional propagation algorithm is used to optimize the network parameters until the liquid neural network completes the model training.

[0105] In an embodiment of the present application, after the training set is collected through the operator's manipulation, a liquid neural network is designed and trained. Specifically, the sensor data is used as the input of the liquid neural network, and the action instructions are used as the output of the liquid neural network. Through the dynamic adaptability and time-varying weight adjustment capabilities of the liquid neural network itself, the liquid neural network can learn end-to-end driving strategies in different environments. During the training process, the back propagation algorithm is used to optimize the network parameters of the liquid neural network, and the performance and robustness of the model are verified through simulation and actual testing. Finally, the trained liquid neural network is deployed to the terminal control module of the lashing robot to realize the autonomous navigation function, so that the lashing robot can drive safely and efficiently according to the real-time environmental perception and learned driving strategy.

[0106] In some implementations of the embodiments of the present application, the binding path is specifically generated by describing the position and posture of the steel bar node through a neural descriptor model (Neural Descriptor Fields, NDF), also known as a neural descriptor field model, so as to meet the steel bar node positioning and posture evaluation of various types of diameters and various types of crossover modes. Compared with traditional vision and template matching, the use of a neural descriptor model has the following advantages:

[0107] 1. SE(3)-Equivariance: The neural descriptor is equivariant to 3D rotation and translation, which means it can recognize and manipulate objects in any position and orientation, while template matching and traditional visual recognition methods can usually only handle transformations within the plane and have poor handling of 3D rotations and complex positions.

[0108] 2. Efficient feature matching: The neural descriptor model recognizes and locates objects through descriptor matching in a high-dimensional feature space, which is more robust than key point detection based on two-dimensional images, especially when the object shape or posture changes greatly.

[0109] 3. Small sample learning: Neural descriptor models only need a small number of demonstrations (such as 5-10 times) to learn tasks, while template matching and traditional visual recognition methods usually require a large amount of data to cover a variety of possible object positions and postures.

[0110] Based on the generation of binding paths through the neural descriptor model, see Figure 5 As shown, the step of generating the binding path may specifically include steps S501-S503:

[0111] S501: The tying robot obtains a local image of the steel cage corresponding to the local photographing point currently traversed.

[0112] S502: The tying robot inputs the local image into the neural descriptor model, so that the neural descriptor model recognizes and outputs the position and posture of the steel bar node in the local image.

[0113] In the embodiment of the present application, the neural descriptor model learns to identify the key features of the steel bar node through pre-training, and encodes the position and posture of the key features in three-dimensional space to identify and locate the position and posture of the steel bar node in the local image. The position in the local image refers to the specific coordinates of the steel bar node in the local image, and the posture in the local image refers to the spatial direction and angle of the steel bar node.

[0114] S503: The tying robot generates a tying path based on the position and posture of the steel bar node in the local image.

[0115] In the embodiment of the present application, an optimal or better tying path may be generated based on the position and posture through a path planning algorithm.

[0116] In the hardware system of the binding robot, after the terminal control module guides the binding module to control the structured light camera carried by the execution end to reach the local photo point, the terminal control module instructs the perception module to obtain and return the local point cloud image through the camera at the execution end; the terminal control module inputs the local image into the neural symbol description model, and generates a binding path based on the position and posture of the steel bar node output by the neural symbol description model, and then converts the binding path into the control command of the execution end to control the execution end to complete the specific binding operation.

[0117] In some implementations of the embodiments of the present application, the neural descriptor model is pre-trained by annotating multiple groups of three-dimensional point cloud data of a set of steel bar nodes for model training, and the local image is a local point cloud image including multiple local steel bar nodes; the training process of the neural descriptor model is usually self-supervised through a three-dimensional reconstruction task to learn the point-to-point correspondence within the object category. During the training process, it is ensured that the point descriptor has SE (3) variability to facilitate corresponding matching under different postures, so that the neural descriptor model can be generalized to different shapes and postures of steel bar nodes.

[0118] See also Figure 6 As shown, the step of identifying the position and posture of the steel bar node in the local image by using the neural descriptor model may specifically include S601-S603:

[0119] S601: The tying robot inputs the local image into the neural descriptor model.

[0120] In an embodiment of the present application, the tying robot inputs a local point cloud image including a plurality of local steel bar nodes into a neural descriptor model, wherein the local steel bar nodes refer to steel bar nodes in the local image.

[0121] S602: The tying robot maps the coordinates of each local steel bar node in the local image to a spatial descriptor through a neural descriptor model to obtain a neural descriptor corresponding to each local steel bar node. The coordinates of each local steel bar node are the coordinates of each local steel bar node in a local coordinate system established based on the local image. The neural descriptor is used to describe the position of the local steel bar node in the local image.

[0122] In an embodiment of the present application, the spatial descriptor is used to describe the spatial coordinates of the part of the steel bar cage contained in the local image. The neural descriptor model maps the coordinates of each local steel bar node to the spatial descriptor, and generates a neural descriptor corresponding to each local steel bar node. The neural descriptor is used to describe the position of the steel bar node in the local image.

[0123] S603: The tying robot selects a set of rigid query points to represent the local coordinate frame of the steel bar node, and estimates the posture of each local steel bar node in the local coordinate frame through a neural description model to obtain a posture descriptor of each local steel bar node. The posture descriptor is used to describe the posture of the local steel bar node in the image.

[0124] In an embodiment of the present application, the neural descriptor model estimates the posture of each local steel bar node in the local coordinate frame through a rigid query point. The rigid query point is a fixed reference point. The local coordinate frame of the steel bar node can be represented by selecting a set of rigid query points. The configuration of these rigid query points is represented by the SE (3) posture equivalent to the world coordinate system. The posture of the local coordinate frame of each local steel bar node is estimated by an optimization method so that its posture descriptor matches the posture descriptor of the demonstration instance.

[0125] Among them, the posture descriptor is used to describe the posture of each local reinforcement node in the image, including direction and rotation state.

[0126] Here, the specific structure diagram of the neural descriptor model can be found in Figure 7 As shown in the figure, the neural descriptor model generates point descriptors through a point cloud encoder and an occupancy network, encoding the spatial relationship between the steel node and the geometric features of the steel cage. In this structure, the object point cloud data is converted into an encoding format suitable for model processing through a point cloud encoder, and then combined with the three-dimensional coordinates of the query point (steel node) to generate a unified data representation and input it into the occupancy network, outputting features related to the query point, and finally merging the extracted point features to obtain a point descriptor, wherein the point descriptor includes: a neural descriptor and a spatial descriptor.

[0127] exist Figure 6 In the process shown, the neural descriptor model can generalize to the shapes and postures of different steel bar nodes to generate accurate tying paths to help the tying robot carry out tying operations.

[0128] See also Figure 8 The present application embodiment further provides a method for automatic steel bar binding. Figure 1 On the basis of the process shown, the compliance judgment of the 3D point cloud model and the solution for missing binding points are further added, which can specifically include the following steps:

[0129] S801: The tying robot obtains a global image of the steel cage through a camera group on the production line.

[0130] S802: The tying robot generates a three-dimensional point cloud model of the steel cage based on the global image.

[0131] S803: The tying robot extracts geometric information of the steel cage from the three-dimensional point cloud model.

[0132] In the embodiments of the present application, geometric information refers to various parameters extracted from the three-dimensional point cloud model to describe the structure and shape of the steel cage. Among them, the geometric information includes at least one of the following information: steel bar spacing, steel cage size, and steel bar diameter. Among them, the steel bar spacing refers to the distance between the centers of two adjacent steel bars; the steel cage size includes overall geometric parameters such as the length, width, and height of the steel cage. By comparing the above geometric information, it can be ensured that the spacing, size, and diameter of all steel bars meet the design requirements, and the overall quality of the steel cage can be guaranteed. At the same time, it can also reduce the time of manual inspection and further improve construction efficiency.

[0133] It should be noted that the geometric information in the embodiments of the present application includes but is not limited to the spacing between steel bars, the size of the steel cage and the diameter of the steel bars.

[0134] S804: The tying robot compares the geometric information with a preset Building Information Modeling (BIM) model to determine whether the steel cage meets the standard specifications indicated by the BIM model.

[0135] In the embodiments of the present application, the building information model BIM may also be referred to as a BIM model. The BIM model is a model that digitally represents the physical and functional effects of a building. The standard specifications indicated by the BIM model refer to the steel cage specification standards pre-defined in the BIM model. By comparing the geometric information of the three-dimensional point cloud model with the BIM model, the steel cage is tested for compliance to ensure that the steel cage is compliant before the binding operation is carried out.

[0136] In the hardware system of the tying robot, after extracting geometric information from the 3D point cloud model, the terminal control module compares the geometric information with the manually designed BIM, and checks the parameters such as the size of the steel cage, the spacing between steel bars, and the type of steel bars to determine whether the steel cage is compliant. If it is compliant, the steel bar nodes in the 3D point cloud model are identified. If it is not compliant, the non-compliant position is marked and the staff is guided to handle it.

[0137] In some implementations of the embodiments of the present application, when the spacing between the steel bars of the steel cage does not meet the standard specifications indicated by the BIM model, the tying robot adjusts the spacing between the steel bars of the steel cage until it meets the standard specifications indicated by the BIM model. Specifically, the tying robot sends a message to an execution end with the function of placing and adjusting steel bars under the same system to instruct the execution end to adjust the spacing between the steel bars in the steel cage to meet the standard specifications of the steel bar spacing indicated by the BIM model. In this way, when the spacing between the steel bars of the steel cage does not meet the specifications, the spacing can be automatically adjusted without manual operation. It should be noted that before the tying robot provided in the embodiments of the present application carries out tying operations on the steel cage, other execution ends with the ability to place and adjust steel bars under the same system on the production line place and adjust the steel bars to build a steel cage for subsequent tying operations.

[0138] Among them, the execution end for adjusting the spacing between steel bars can be an overhead crane-type robotic arm trained by a behavioral cloning algorithm; specifically, data demonstrated by human experts is collected and stored before training, and then the behavioral cloning algorithm is trained to enable the overhead crane-type robotic arm to imitate and learn the operations of human experts during construction based on the human expert demonstration data, thereby enabling it to master the ability to place and adjust steel bars.

[0139] S805: When the steel cage meets the standard specifications indicated by the BIM model, the tying robot identifies the steel nodes in the three-dimensional point cloud model, and divides the three-dimensional point cloud model into blocks according to the positions of the steel nodes to obtain a robot positioning point sequence.

[0140] S806: The binding robot executes the traversal step of the robot positioning point sequence to globally bind the steel cage; the traversal step of the robot positioning point sequence includes: by acquiring the surrounding environment image of the steel cage, generating a moving path to guide the steel binding robot to move to the currently traversed positioning point, moving to the currently traversed positioning point based on the moving path, generating a local photo point sequence corresponding to the currently traversed binding positioning point, and executing the traversal step of the local photo point sequence to locally bind the steel cage; the traversal step of the local photo point sequence to locally bind the steel cage includes: acquiring a local image of the steel cage corresponding to the currently traversed local photo point, generating a binding path based on the local image, and locally binding the steel cage based on the binding path.

[0141] S807: The tying robot obtains a new global image of the steel cage, and determines whether there are any missing tying points in the steel cage based on the new global image.

[0142] In an embodiment of the present application, after acquiring a new global image of the steel cage, the missed binding points in the global image are identified through image processing technology and computer vision algorithms. The missed binding points refer to steel bar nodes that have not been tied in the steel cage. They may be steel bar nodes that have not been successfully identified in the traversal steps of the local photographing point sequence during the global binding process, or unbound steel bar nodes caused by other operational errors or equipment failures.

[0143] S808: If there is a missing binding point in the steel cage, the binding robot obtains a local image corresponding to the missing binding point, generates a supplementary binding path based on the local image corresponding to the missing binding point, and supplements the missing binding point based on the supplementary binding path.

[0144] In the embodiment of the present application, the step of generating a supplementary binding path based on the local image corresponding to the missed binding point and performing the supplementary binding operation can refer to the step of generating a binding path based on the local image and performing local binding, which will not be elaborated in this application.

[0145] exist Figure 8 In the process shown, geometric information is extracted from the 3D point cloud data and compared with the BIM model to carry out the binding operation under the premise of ensuring the compliance of the steel cage. And the BIM model that does not require manual deepening can reduce the complexity and workload of the design work, thereby improving the overall engineering efficiency. In addition, the global binding of the steel cage can be guaranteed by identifying and supplementing the missed binding points.

[0146] In actual application scenarios, the workflow of the steel bar tying robot can be as follows: Fig. 9 shown.

[0147] S901: Use multiple fixed-position cameras to photograph the steel cage on the production line, generate a three-dimensional point cloud model of the steel cage, and execute S902.

[0148] S902: Compare the 3D point cloud model of the steel cage with the BIM model, conduct tests on the spacing and diameter of the steel bars, determine whether the steel cage is qualified, and ensure that the design of the steel cage meets the construction specifications. After that, identify the steel bar nodes, and process the 3D point cloud model in blocks according to the node positions of the steel bar nodes to generate a robot positioning point sequence, such as positioning points 1, 2, 3, ..., N, where N is the total number of robot positioning points, to guide the robot to carry out the binding operation block by block, and execute S903.

[0149] S903: Based on the robot positioning point sequence, trigger the rebar binding robot to autonomously navigate to the positioning point corresponding to the current traversal block, and execute S904.

[0150] S904: In response to the rebar tying robot arriving at the positioning point corresponding to the current traversal block, a sequence of local photo points corresponding to the current traversal block is generated according to the divided three-dimensional point cloud model, such as local photo points 1, 2, 3, ..., M, where M is the total number of local photo points, to further obtain detailed local image analysis, ensure the accuracy and quality of subsequent tying operations, and execute S905.

[0151] S905: Guide the robot arm to the local photo-taking point currently traversed, shoot the steel cage through the camera at the end of the robot arm, accurately locate the steel bar node, then generate the robot arm binding path, and carry out the binding operation based on the binding path, and execute S906.

[0152] S906: Determine whether the currently traversed local photographing point is the last local photographing point in the local photographing point sequence. If so, execute S907; if not, execute S905.

[0153] S907: Determine whether the currently traversed robot positioning point is the last robot positioning point in the robot positioning point sequence. If so, execute S908; if not, execute S903.

[0154] S908: A fixed position camera photographs the steel cage on the production line to determine whether there are any missing binding points in the steel cage. If so, execute S909; if not, execute S99.

[0155] S909: According to the missed binding points, the moving paths of the binding robot and the mechanical arm are autonomously planned, and the supplementary binding work is carried out, and S908 is executed.

[0156] S910: End the operation.

[0157] See also Fig.10 As shown, the embodiment of the present application also provides a steel bar tying robot, which includes: a terminal control module 1001, a sensing module 1002, a tying module 1003, a chassis module 1004 and a communication module 1005;

[0158] The terminal control module 1001 is used to control the perception module to obtain and return the global image of the steel cage through the camera group on the production line, generate a three-dimensional point cloud model of the steel cage based on the global image, identify the steel nodes in the three-dimensional point cloud model, and divide the three-dimensional point cloud model into blocks according to the positions of the steel nodes to obtain a robot positioning point sequence;

[0159] The terminal control module is also used to execute the traversal steps of the robot positioning point sequence to control the binding module to globally bind the steel cage;

[0160] The traversal steps of the robot positioning point sequence include: controlling the perception module to obtain and return the surrounding environment image of the steel cage, generating a moving path to guide the steel cage tying robot to move to the currently traversed positioning point, so as to control the chassis module to move the steel cage tying robot to the currently traversed positioning point; generating a local photo point sequence corresponding to the currently traversed positioning point, and executing the traversal steps of the local photo point sequence, so as to control the tying module to perform local tying of the steel cage;

[0161] The traversal step of the local photo point sequence includes: controlling the perception module to obtain and return a local image of the steel cage corresponding to the currently traversed local photo point, and generating a binding path based on the local image;

[0162] A perception module 1002 is used to obtain a global image, a surrounding environment image, and a local image of the steel cage;

[0163] The binding module 1003 is used to locally bind the steel cage based on the binding path;

[0164] The chassis module 1004 is used to move the steel cage tying robot to the currently traversed positioning point based on the moving path;

[0165] The communication module 1005 is used to establish communication between the sensing module, the binding module and the chassis module and the terminal control module respectively.

[0166] In some implementations of the embodiments of the present application, the terminal control module is further used to:

[0167] Extracting geometric information of the steel cage from the three-dimensional point cloud model, the geometric information including at least one of the following information: steel bar spacing, steel cage size, and steel bar diameter;

[0168] Compare the geometric information with the pre-set Building Information Model (BIM) model to determine whether the steel cage meets the standard specifications indicated by the BIM model;

[0169] When the reinforcement cage meets the standard specifications indicated by the BIM model, the reinforcement nodes in the 3D point cloud model are identified.

[0170] In some implementations of the embodiments of the present application, the terminal control module is further used to:

[0171] When the steel bar spacing of the steel cage does not meet the standard specifications indicated by the BIM model, adjust the steel bar spacing of the steel cage until it meets the standard specifications indicated by the BIM model.

[0172] In some implementations of the embodiments of the present application, the perception module is also used to obtain a new global image of the steel cage.

[0173] The terminal control module is also used to: determine whether there are any missing binding points in the steel cage based on the new global image; if there are any missing binding points in the steel cage, obtain the local image corresponding to the missing binding point, generate a supplementary binding path based on the local image corresponding to the missing binding point, and supplement the missing binding point based on the supplementary binding path.

[0174] In some implementations of the embodiments of the present application, the terminal control module is specifically used to: obtain the surrounding environment image of the steel cage, and generate a moving path to guide the steel bar binding robot to move to the currently traversed positioning point through a liquid neural network.

[0175] In some implementations of the embodiments of the present application, the terminal control module is specifically used to:

[0176] Inputting the surrounding environment image into the input layer of the liquid neural network so that the input layer outputs an image code of the surrounding environment image;

[0177] The image code is input into the liquid layer of the liquid neural network, so that the liquid layer performs nonlinear processing and pattern recognition on the image code through dense connections and dynamic behaviors of neurons to obtain information code, wherein the information code includes: feature information extracted from the image code and a decision corresponding to the feature information;

[0178] The information encoding is input into the output layer of the liquid neural network, so that the output layer converts the feature information in the information encoding and the decision corresponding to the feature information into a series of action navigation instructions to obtain a moving path to guide the steel bar binding robot to move to the currently traversed positioning point.

[0179] In some implementations of the embodiments of the present application, the training process of the liquid neural network includes:

[0180] A training set is obtained, where the training set includes: sensor data during the operation of the steel bar tying robot by the operator and action instructions received by the steel bar tying robot;

[0181] Sensor data is used as the input of the liquid neural network, and action instructions are used as the output of the liquid neural network. The model is trained through the dynamic adaptability and time-varying weight adjustment capabilities of the liquid neural network. During the training process, the directional propagation algorithm is used to optimize the network parameters until the liquid neural network completes the model training.

[0182] In some implementations of the embodiments of the present application, the terminal control module is specifically used to:

[0183] Obtain the local image of the steel cage corresponding to the local photo point currently traversed;

[0184] Inputting the local image into the neural descriptor model so that the neural descriptor model recognizes and outputs the position and posture of the steel bar node in the local image;

[0185] In some implementations of the embodiments of the present application, the neural descriptor model is pre-trained by annotating a plurality of sets of three-dimensional point cloud data of a set of steel bar nodes for model training, and the local image is a local point cloud image including a plurality of local steel bar nodes; the terminal control module is specifically used for:

[0186] Input the local image to the neural descriptor model;

[0187] The coordinates of each local steel bar node in the local image are mapped to the spatial descriptor through the neural descriptor model to obtain the neural descriptor corresponding to each local steel bar node. The coordinates of each local steel bar node are the coordinates of each local steel bar node in the local coordinate system established based on the local image. The neural descriptor is used to describe the position of the local steel bar node in the local image.

[0188] A set of rigid query points are selected to represent the local coordinate frame of the steel bar node. The posture of each local steel bar node in the local coordinate frame is estimated through a neural description model to obtain a posture descriptor of each local steel bar node. The posture descriptor is used to describe the posture of the local steel bar node in the image.

[0189] In some implementations of the embodiments of the present application, the steel bar tying robot is as follows Fig.11 The general purpose steel bar tying robot shown or Fig.12 The plate reinforcement binding robot shown is preferably used in large-surface operation scenarios.

[0190] In some implementations of the embodiments of the present application, the steel bar tying robot is Fig.11 The general-purpose rebar tying robot shown.

[0191] The perception module of the universal rebar tying robot includes a camera group fixed on the workshop production line, a laser radar and camera carried by the robot, and a camera carried at the end of the robotic arm. Among them, the camera group fixed on the workshop production line consists of 4 to 6 binocular cameras, which are used to obtain global information of the rebar cage on the production line. The laser radar and camera carried by the robot are mainly used for mapping, autonomous navigation and real-time obstacle avoidance. The camera carried at the end of the robotic arm is used to obtain local photo point information.

[0192] The binding module of the universal rebar binding robot has two functions. One function is to carry the terminal camera to the corresponding local photo point to take pictures after the trolley reaches the designated binding position. The other task is to obtain point cloud information with the camera carried at the end of the robotic arm. After processing by the terminal control module, the position and posture information of the corresponding rebar node is obtained, and the robotic arm is guided to carry the binding device to carry out the binding work. The binding device is modified from a traditional electric binding device. The signal receiving part and the signal output part inside the binding device are connected with signal lines respectively, and respectively connected to the signal input and output parts in the control cabinet of the robotic arm. The binding device control part can also be directly connected to the industrial control computer for control.

[0193] The chassis module of the universal rebar tying robot is mainly used to carry the mechanical arm to improve the flexibility of the entire mobile rebar tying robot. The chassis can be in various forms, such as universal wheel type, AGV type, and crawler type. This application takes the four-wheel drive universal wheel structure as an example. After the global camera captures the position of the rebar cage, the industrial computer calculates the specific positioning point of the car, and uses autonomous navigation and real-time obstacle avoidance algorithms to guide the car to the corresponding point.

[0194] The communication module and terminal control module of the universal rebar tying robot are partially in the same box, which is used to establish connections between various parts using wired or wireless methods. The three modules, namely the sensing module, tying module, and chassis module, can establish connections with the industrial control computer through wired or wireless methods. The sensor sends the received data to the industrial control computer for processing, and then transmits the processed instructions to the robotic arm, the end effector (rebar tying device), and the trolley chassis to guide them to carry out autonomous rebar tying operations.

[0195] The terminal control module of the general-purpose rebar tying robot is also called the industrial control computer module, which is used to receive and process data from the perception module and control the robotic arm, end effector and trolley chassis module.

[0196] In some implementations of the embodiments of the present application, the steel bar tying robot is Fig.12 The plate reinforcement tying robot shown is used to carry out large-plane reinforcement tying operations.

[0197] The chassis module of the plate reinforcement tying robot consists of wheels, 1 rod and 2 rods, among which the wheels are the horizontal moving parts of the plate reinforcement robot, and 1 rod and 2 rods are the lateral moving parts of the mechanism. Under the action of the wheels, the plate reinforcement robot can move horizontally on the plate reinforcement. When a row of reinforcement bars is tied, 1 rod and 2 rods can be used to change rows. Specifically, by lifting 1 rod, 2 rods can be suspended in the air, and then 2 rods can be moved to the left or right, and then 1 rod can be retracted to make the plate reinforcement robot fully supported on 3 rods, and then the plate reinforcement robot moves in the corresponding direction as a whole.

[0198] The execution module of the plate reinforcement binding robot consists of a first execution end and a second execution end. The camera on the first execution end recognizes the corresponding point, and the terminal control module controls the binding actuator on the first execution end to go to the corresponding point to carry out the binding work; the second execution end is used to bend the binding wire so that the binding wire does not reduce the thickness of the concrete protective layer.

[0199] The perception module of the plate reinforcement tying robot includes a camera group fixed on the workshop production line, a laser radar and camera carried by the robot, a camera carried by the first execution end, and the communication module and terminal control module of the plate reinforcement tying robot are integrated in the same box. It should be noted that since the perception module, terminal control module and communication module of the plate reinforcement tying robot are the same as those of the general-purpose steel bar tying robot, this application will not be repeated here.

[0200] Finally, it should be noted that, in the embodiments of the present application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements that are not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the statement "comprise one..." do not exclude the presence of other identical elements in the process, method, article or equipment including the elements.

[0201] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An automatic operation method of a steel bar tying robot, characterized in that: Applied to a steel bar tying robot, the method comprises: The global image of the steel cage is obtained through the camera group on the production line; Generate a three-dimensional point cloud model of the steel cage based on the global image; Identify steel bar nodes in the three-dimensional point cloud model, and process the three-dimensional point cloud model in blocks according to the positions of the steel bar nodes to obtain a robot positioning point sequence; Executing the traversal step of the robot positioning point sequence to globally bind the steel cage; The traversal step of the robot positioning point sequence includes: by acquiring the surrounding environment image of the steel cage, generating a moving path for guiding the steel bar tying robot to move to the currently traversed positioning point, moving to the currently traversed positioning point based on the moving path, generating a local photographing point sequence corresponding to the currently traversed bundling positioning point, and executing the traversal step of the local photographing point sequence to locally tie the steel cage; The traversal step of the local photographing point sequence to locally bind the steel cage comprises: obtaining a local image of the steel cage corresponding to the currently traversed local photographing point, generating a binding path based on the local image, and locally binding the steel cage based on the binding path; The obtaining of the local image of the steel cage corresponding to the local photographing point currently traversed, and generating the binding path based on the local image, comprises: obtaining the local image of the steel cage corresponding to the local photographing point currently traversed; inputting the local image into a neural descriptor model so that the neural descriptor model recognizes and outputs the position and posture of the steel bar node in the local image; generating the binding path based on the position and posture of the steel bar node in the local image; The neural descriptor model is obtained by pre-training a plurality of sets of three-dimensional point cloud data of steel bar node sets labeled for model training, and the local image is a local point cloud image including a plurality of local steel bar nodes; The method of inputting the local image into a neural descriptor model so that the neural descriptor model recognizes and outputs the position and posture of a steel bar node comprises: inputting the local image into the neural descriptor model; mapping the coordinates of each local steel bar node in the local image to a spatial descriptor through the neural descriptor model to obtain a neural descriptor corresponding to each local steel bar node, wherein the coordinates of each local steel bar node are the coordinates of each local steel bar node in a local coordinate system established based on the local image, and the neural descriptor is used to describe the position of the local steel bar node in the local image; selecting a group of rigid query points to represent a local coordinate frame of the steel bar node, estimating the posture of each local steel bar node in the local coordinate frame through the neural description model, to obtain a posture descriptor of each local steel bar node, and the posture descriptor is used to describe the posture of the local steel bar node in the image.

2. The method according to claim 1, characterized in that After generating the three-dimensional point cloud model of the reinforcement cage based on the global image, the method further comprises: Extracting geometric information of the steel cage from the three-dimensional point cloud model, wherein the geometric information includes at least one of the following information: steel bar spacing, steel cage size, and steel bar diameter; Comparing the geometric information with a pre-trained Building Information Model (BIM) model to determine whether the steel cage meets the standard specifications indicated by the BIM model; The identifying of the steel bar nodes in the three-dimensional point cloud model includes: when the steel cage meets the standard specifications indicated by the BIM model, identifying the steel bar nodes in the three-dimensional point cloud model.

3. The method according to claim 2, characterized in that The method further comprises: When the steel bar spacing of the steel cage does not meet the standard specifications indicated by the BIM model, the steel bar spacing of the steel cage is adjusted until it meets the standard specifications indicated by the BIM model.

4. The method according to claim 1, characterized in that: After executing the step of traversing the robot positioning point sequence, the method further includes: Acquire a new global image of the steel cage, and determine whether there are any missing binding points in the steel cage based on the new global image; If there is a missing binding point in the steel cage, a local image corresponding to the missing binding point is obtained, a supplementary binding path is generated based on the local image corresponding to the missing binding point, and the missing binding point is supplemented based on the supplementary binding path.

5. The method according to claim 1, characterized in that The method of acquiring the surrounding environment image of the steel cage and generating a moving path for guiding the steel bar tying robot to move to the currently traversed positioning point comprises: The surrounding environment image of the steel cage is acquired, and the moving path for guiding the steel bar tying robot to move to the currently traversed positioning point is generated through a liquid neural network.

6. The method according to claim 5, characterized in that The step of acquiring the surrounding environment image of the steel cage and generating a moving path for guiding the steel bar tying robot to move to the currently traversed positioning point through a liquid neural network includes: Acquire the surrounding environment image of the steel cage; Inputting the surrounding environment image into the input layer of the liquid neural network so that the input layer outputs the image code of the surrounding environment image; Inputting the image code into the liquid layer of the liquid neural network, so that the liquid layer performs nonlinear processing and pattern recognition on the image code through dense connections and dynamic behaviors of neurons to obtain information code, wherein the information code includes: feature information extracted from the image code, and a decision corresponding to the feature information; The information code is input into the output layer of the liquid neural network, so that the output layer converts the feature information in the information code and the decision corresponding to the feature information into a series of action navigation instructions, so as to obtain the moving path that guides the steel bar binding robot to move to the currently traversed positioning point.

7. The method according to claim 6, characterized in that The training process of the liquid neural network includes: Acquire a training set, wherein the training set includes: sensor data during the operation of the steel bar tying robot by an operator and action instructions received by the steel bar tying robot; The sensor data is used as the input of the liquid neural network, and the action instruction is used as the output of the liquid neural network. The model is trained through the dynamic adaptability and time-varying weight adjustment ability of the liquid neural network. During the training process, the directional propagation algorithm is used to optimize the network parameters until the liquid neural network completes the model training.

8. A steel bar tying robot, characterized in that: The steel bar tying robot comprises: a sensing module, a terminal control module, a tying module, a chassis module and a communication module; The terminal control module is used to control the perception module to acquire and return a global image of the steel cage through a camera group on the production line, generate a three-dimensional point cloud model of the steel cage based on the global image, identify steel bar nodes in the three-dimensional point cloud model, and perform block processing on the three-dimensional point cloud model according to the positions of the steel bar nodes to obtain a robot positioning point sequence; The terminal control module is also used to execute the traversal step of the robot positioning point sequence to control the binding module to globally bind the steel cage; The traversal step of the robot positioning point sequence includes: controlling the perception module to obtain and return the surrounding environment image of the steel cage, generating a moving path to guide the steel bar tying robot to move to the currently traversed positioning point, so as to control the chassis module to move the steel bar cage tying robot to the currently traversed positioning point; generating a local photo point sequence corresponding to the currently traversed positioning point, and executing the traversal step of the local photo point sequence, so as to control the tying module to perform local tying of the steel cage; The traversal step of the local photo point sequence includes: controlling the perception module to obtain and return a local image of the steel cage corresponding to the currently traversed local photo point, and generating a binding path based on the local image; The perception module is used to obtain the global image, the surrounding environment image and the local image of the steel cage; The binding module is used to locally bind the steel cage based on the binding path; The chassis module is used to move the steel cage tying robot to the currently traversed positioning point based on the moving path; The communication module is used to establish communication between the sensing module, the lashing module and the chassis module and the terminal control module respectively; The terminal control module is specifically used to: control the perception module to obtain and return the local image of the steel cage corresponding to the local photo point currently traversed; input the local image into the neural descriptor model so that the neural descriptor model recognizes and outputs the position and posture of the steel node in the local image; generate the binding path based on the position and posture of the steel node in the local image; The neural descriptor model is obtained by pre-training a plurality of sets of three-dimensional point cloud data of steel bar node sets labeled for model training, and the local image is a local point cloud image including a plurality of local steel bar nodes; The method of inputting the local image into a neural descriptor model so that the neural descriptor model recognizes and outputs the position and posture of a steel bar node comprises: inputting the local image into the neural descriptor model; mapping the coordinates of each local steel bar node in the local image to a spatial descriptor through the neural descriptor model to obtain a neural descriptor corresponding to each local steel bar node, wherein the coordinates of each local steel bar node are the coordinates of each local steel bar node in a local coordinate system established based on the local image, and the neural descriptor is used to describe the position of the local steel bar node in the local image; selecting a group of rigid query points to represent a local coordinate frame of the steel bar node, estimating the posture of each local steel bar node in the local coordinate frame through the neural description model, to obtain a posture descriptor of each local steel bar node, and the posture descriptor is used to describe the posture of the local steel bar node in the image.

Images