Robot unhooking method based on vision

Abstract

The invention provides a robot unhooking method based on vision, and belongs to the technical field of train unhooking, and the method comprises the steps: starting a laser radar when a carriage stops running, collecting the point cloud data of the surrounding environment of the laser radar, and calculating the position information of a coupler handle; position information of the coupler handle is converted into a robot coordinate system through hand-eye calibration, and the robot reaches a designated position according to coordinate information; the camera is started and collects an RGB image when the robot reaches a designated position, and whether the robot grabs a coupler handle or not is judged; the robot performs unhooking action according to points of the unhooking path; after unhooking is completed, an RGB image after unhooking is completed is collected, whether unhooking succeeds or not is judged, the carriage is unhooked, the unhooking action is completed, and the robot returns to the initial position. The situation that unhooking fails due to the complex surrounding environment is overcome, robot unhooking is more accurate and errorless, the safety risk of workers is reduced, and the automatic control level of a car dumper system is improved.

Description

Technical Field

[0001] The invention relates to the technical field of train hook removal, in particular to a vision-based robot hook removal method. Background Art

[0002] With the rapid development of my country's economy, the demand for coal in thermal power plants is increasing. Railway coal transportation is a common method of coal transportation for land-based thermal power plants. After the coal train arrives at the power plant, a car tipper is needed to unload coal from each carriage. When the car tipper system is working, the car that needs to be unloaded on the heavy vehicle line is first uncoupled, and then the heavy vehicle shunting locomotive pulls the uncoupled car into the car tipper room, and the car tipper unloads the cargo. Finally, the next group of uncoupled cars is pulled into the car tipper room, and the empty car after unloading is pushed to the empty car line, and the empty car shunting locomotive pushes the car traction platform.

[0003] In the car tipper unloading system, most of the processes have been automated, but most of the car unhooking operations before unloading are still done manually. The operating space is small when performing manual unhooking operations, and there is a risk of workers being drawn into the equipment. In addition, workers are accustomed to unhooking when the train has not stopped. When workers are not focused or cannot adjust according to the actual situation, it may cause casualties or damage to equipment.

[0004] Although there are also robot hook removal methods disclosed in the prior art, they all use cameras to take pictures and collect two-dimensional RGB images, and then use stereo matching algorithms to calculate three-dimensional information from two or more two-dimensional images. In the stereo matching process, errors may be introduced, affecting the accuracy of coupler positioning.

[0005] Therefore, it is necessary to develop a vision-based robot hook removal method to overcome the above problems. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a vision-based robot hook removal method. By scanning the coupler with a three-dimensional scanner, point cloud data carrying three-dimensional information can be directly obtained, and then a point cloud segmentation method based on deep learning is used to complete the positioning of the train coupler handle. During the hook removal process, the hook removal results are detected and optimized through target detection, so that the robot hook removal is more accurate and error-free.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] A vision-based robot hook removal method comprises the following steps:

[0009] S1, when the carriage stops running, the laser radar starts to collect point cloud data of the laser radar's surrounding environment, and calculates the position information of the coupler handle through the point cloud segmentation network;

[0010] S2, converting the position information of the coupler handle into the robot coordinate system through hand-eye calibration, and the robot reaches the specified position according to the coordinate information;

[0011] S3, the camera starts and collects the RGB image when the robot reaches the specified position, and determines whether the robot grabs the coupler handle through the target detection network;

[0012] S4, the robot performs the hook removal action according to the points on the hook removal path;

[0013] S5, after the unhooking is completed, the RGB image after the unhooking is completed is collected and the target detection network is used to determine whether the unhooking is successful. The carriage is unhooked, the unhooking action is completed, the robot returns to the initial position, and returns to step S1 to prepare for the next unhooking task.

[0014] A further improvement of the technical solution of the present invention is that in S1, the point cloud segmentation network adopts a point convolution network based on density and position adaptive convolution, and uses a labeled coupler point cloud data set to train the point cloud segmentation network. The position of the coupler handle is segmented by the trained point cloud segmentation network, and the centroid of the coupler handle point cloud is used as the grasping position.

[0015] A further improvement of the technical solution of the present invention is that: the point cloud segmentation network is composed of an encoding part and a decoding part, the encoding part is composed of a first encoding layer, a second encoding layer, a third encoding layer and a fourth encoding layer; the first encoding layer is composed of a common encoding layer and a encoding layer carrying context information in parallel, the second encoding layer, the third encoding layer and the fourth encoding layer are all common encoding layers; the decoding part is composed of four decoding layers and a fully connected layer.

[0016] A further improvement of the technical solution of the present invention is that: the first coding layer is composed of a sampling layer, a grouping layer and a feature extraction layer. In the sampling layer, a certain number of points are sampled by the farthest point. In the grouping layer, the sampled points are used as the center point, and the k points closest to the center point are divided into a group. The feature extraction layer encodes the center point into a higher dimensional point according to the relationship between the center point and each neighboring point.

[0017] A further improvement of the technical solution of the present invention is that: the feature extraction layer is a density-position adaptive convolution, which is composed of a ScoreNet and a weight library; the ScoreNet learns the position and density relationship between the center point and the neighborhood point through parallel MLPs and GRU, generates a series of adaptive coefficients, and the generated series of adaptive coefficients are multiplied with the weight matrix in the weight library to obtain a kernel function. After the input features are mapped one by one to the kernel function, the output features are obtained through an aggregation function.

[0018] A further improvement of the technical solution of the present invention is that the density relationship between the center point and the neighborhood point is calculated by adaptive kernel density estimation, the Gaussian function is used as the kernel function of the adaptive kernel density estimation, and the Euclidean distance between the two points is used as a measure to evaluate the correlation between the two points, which is formulated as follows:

[0019]

[0020] Among them, dis represents the distance between the center point and the neighborhood point; k represents the number of neighborhood points; h is the variable bandwidth, which represents the smoothness of the adaptive kernel density curve, and h is formulated as: Varies based on the different distribution of each point in the group.

[0021] A further improvement of the technical solution of the present invention is that in S2, the hand-eye calibration, i.e., the coordinate transformation between the robot and the laser radar coordinate system, fits the values ​​of a set of points in two different coordinate systems according to the principle of least squares method, and outputs the transformation matrix of the two coordinate systems.

[0022] A further improvement of the technical solution of the present invention is that in S3, whether the robot has successfully grasped is determined based on the positional relationship between the coupler handle and the clamp, and the position where the grasping failed is divided into three situations: biased to the left, biased to the right, and biased to the front. Each situation is distinguished by a target detection network, and the grasping position is adjusted based on the linear relationship between the size of the pixels occupied by the detection box in the image and the actual deviation until the grasping is successful.

[0023] A further improvement of the technical solution of the present invention is that the target detection network is trained pixel by pixel according to the improved yolov5s model, and the feature extraction capability of the model is enhanced by increasing the depth of the neck network and introducing a coordinate attention mechanism.

[0024] A further improvement of the technical solution of the present invention is that in S4, the hook-off action is an arc motion, and the length of the coupler handle is obtained through the point cloud segmentation network, and the handle is lifted to a certain angle with this length as the radius until the coupler is unhooked.

[0025] Due to the adoption of the above technical solution, the technical progress achieved by the present invention is:

[0026] In the vision-based robot hook removal method provided by the present invention, laser radar and RGB sensor are first used to collect target point cloud and image data respectively (the coupler is scanned with a three-dimensional scanner to directly obtain point cloud data carrying three-dimensional information), and the construction of point cloud and image data sets is completed; then a point cloud segmentation network and an image detection network are respectively built, and the constructed data sets are used for training; then the position of the robot grasping the coupler handle is determined in combination with the result of the point cloud segmentation, and the position of the target relative to the robot end effector is determined through hand-eye calibration; finally, the robot is path planned, the robot is controlled to complete the hook removal action, and the robot hook removal result is judged and optimized through the image detection algorithm; the robot hook removal is made more accurate and correct, which can improve the automation level of the dumper coal unloading system and ensure the life safety of the staff. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative labor.

[0028] Figure 1 is a flow chart of a vision-based robot hook-removing method provided in an embodiment of the present invention;

[0029] Figure 2 is a structural diagram of a point cloud segmentation network in an embodiment of the present invention;

[0030] Figure 3 It is a scorenet structure diagram in an embodiment of the present invention;

[0031] Figure 4 is a structural diagram of a weight library in an embodiment of the present invention;

[0032] Figure 5 It is a density-position adaptive convolution structure diagram in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or apparatus.

[0034] The present invention is further described in detail below with reference to the accompanying drawings and embodiments:

[0035] like Figure 1 As shown, a vision-based robot hook removal method includes the following steps:

[0036] S1, when the carriage stops running, the laser radar starts to collect point cloud data of the environment around the laser radar, and calculates the position information of the coupler handle through the point cloud segmentation network.

[0037] The point cloud segmentation network adopts a point convolution network based on density and position adaptive convolution. The point cloud dataset of the labeled coupler handle is used to train the point cloud segmentation network. The position of the coupler handle is segmented by the trained point cloud segmentation network, and the centroid of the coupler handle point cloud is used as the grasping position.

[0038] like Figure 2 As shown in the figure, the point cloud segmentation network consists of an encoding part and a decoding part. The encoding part consists of the first encoding layer, the second encoding layer, the third encoding layer and the fourth encoding layer. The first encoding layer consists of a common encoding layer and a contextual information encoding layer in parallel. The second encoding layer, the third encoding layer and the fourth encoding layer are all common encoding layers. The decoding part consists of four decoding layers and a fully connected layer. The difference between the common encoding layer and the contextual information encoding layer is that there is no GRU in the common encoding layer.

[0039] The first encoding layer consists of a sampling layer, a grouping layer, and a feature extraction layer. In the sampling layer, a certain number of points are sampled by the farthest point. In the grouping layer, the sampled points are used as the center point, and the k points closest to the center point are divided into a group. The feature extraction layer encodes the center point into a higher-dimensional point based on the relationship between the center point and each neighboring point.

[0040] like Figure 3 , 4 As shown in the figure, the feature extraction layer is a density-position adaptive convolution, which consists of ScoreNet and a weight library. ScoreNet learns the position and density relationship between the center point and the neighborhood point through parallel MLPs and GRU to generate a series of adaptive coefficients. The generated series of adaptive coefficients are multiplied with the weight matrix in the weight library to obtain the kernel function. After the input features are mapped one by one to the kernel function, the output features are obtained through the aggregation function.

[0041] like Figure 5 As shown in the figure, the density relationship between the center point and the neighborhood points is calculated by adaptive kernel density estimation. The Gaussian function is used as the kernel function of the adaptive kernel density estimation, and the Euclidean distance between the two points is used as a measure to evaluate the correlation between the two points. It can be formulated as:

[0042]

[0043] In the formula, dis represents the distance between the center point and the neighborhood point; k represents the number of neighborhood points; h is the variable bandwidth, which represents the smoothness of the adaptive kernel density curve. h can be formulated as: Varies based on the different distribution of each point in the group.

[0044] Specifically, the data set used by the point cloud segmentation network is a customized train coupler handle point cloud data set, which contains 2000 point cloud data. Each set of data is divided into two categories: coupler handle and background. The division ratio of training set and test set is 7:1. The loss function of network training is cross entropy loss and weight regularization. The number of training rounds is 100 rounds, and the learning rate is 0.05. At 0.6 and 0.8 of the total number of training rounds, the learning rate is adjusted to one tenth of the original. After the point cloud segmentation network training is completed, the trained model is evaluated on the test set. The evaluation indicators include accuracy and intersection ratio. After the lidar collects the data, the data is first filtered to filter out most of the noise points and background points, and then it is processed in blocks to make them have the same resolution. Finally, it is input into the point cloud segmentation network for segmentation, and the center of mass of this set of data is found on the segmented coupler handle as the robot's grasping position.

[0045] S2, the position information of the coupler handle is converted to the robot coordinate system through hand-eye calibration, and the robot reaches the specified position according to the coordinate information.

[0046] Specifically, hand-eye calibration is essentially to model the hand-eye calibration problem in the form of least squares method by overlapping one coordinate system with another coordinate system through a series of rotations and translations. The input of the model is a set of coordinate values ​​of points in the robot and lidar coordinate systems respectively. The robot is moved within the field of view of the lidar, and 20 sets of coordinate data are collected as input. The transformation matrix of hand-eye calibration is obtained according to the singular value decomposition method.

[0047] S3, the camera starts and collects the RGB image when the robot reaches the specified position, and determines whether the robot grabs the coupler handle through the target detection network.

[0048] The object detection network is trained pixel by pixel based on the improved YOLOv5s model. The feature extraction capability of the model is enhanced by increasing the depth of the neck network and introducing the coordinate attention mechanism.

[0049] Whether the robot has grasped successfully is determined based on the positional relationship between the coupler handle and the clamp. The positions where grasping fails are divided into three situations: biased to the left, biased to the right, and biased to the front. Each situation is distinguished through the target detection network, and the grasping position is adjusted according to the linear relationship between the size of the pixels occupied by the detection box in the image and the actual deviation until the grasping is successful.

[0050] Specifically, the dataset used by the target detection network is a customized train coupler handle image dataset, which contains 1,000 images, including five categories: the robot grasps the handle, fails to grasp and deviates to the left, fails to grasp and deviates to the right, successfully removes the hook, and fails to remove the hook. It is divided into training set and test set in a ratio of 7:3. The training loss function is the classification loss function, the positioning loss function, and the confidence loss function. The number of training rounds is 150 rounds, and the learning rate is 0.01. The cosine decay strategy is used to adjust the learning rate during the training process. After the network training is completed, the trained model is evaluated on the test set. The evaluation indicators include precision, recall, and average precision.

[0051] When grasping fails, the robot gripper needs to be adjusted forward or left and right. There is a linear relationship between the width of the detection frame and the actual adjustment distance. The robot grasping position is adjusted according to this linear relationship.

[0052] S4, the robot performs the hook removal action according to the points on the hook removal path.

[0053] The hook-off action is an arc motion. The length of the coupler handle is obtained through the point cloud segmentation network. With this length as the radius, it is lifted to a certain angle until the hook is unhooked.

[0054] Specifically, after the robot grabs the coupler handle, it moves in an arc with a radius of l from the grab position (x, y) to the highest point of the lifting rod, passing the middle point (x 1 ,y 1 ) is the lifting angle of 30°, the end point (x 2 ,y 2 ) is lifted at an angle of 70°, the path point coordinates are calculated as follows:

[0055] x 1 =x+l·sin30°

[0056] y 1 =y+l·(1-cos30°)

[0057] x 2 =x+l·sin70°

[0058] y 2 =y+l·(1-cos70°)

[0059] S5, after the hook is unhooked, the RGB image at this time (after the hook is unhooked) is collected and the target detection network is used to determine whether the hook is unhooked successfully. The carriage is unhooked, the unhooking action is completed, the robot returns to the initial position, and returns to step S1 to prepare for the next unhooking task.

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A vision-based robot hook removal method, characterized in that: The following steps are involved: S1, when the carriage stops running, the laser radar starts to collect point cloud data of the laser radar's surrounding environment, and calculates the position information of the coupler handle through the point cloud segmentation network; S2, converting the position information of the coupler handle into the robot coordinate system through hand-eye calibration, and the robot reaches the specified position according to the coordinate information; S3, the camera starts and collects the RGB image when the robot reaches the specified position, and determines whether the robot grabs the coupler handle through the target detection network; S4, the robot performs hook removal according to the points on the hook removal path; S5, after the unhooking is completed, the RGB image after the unhooking is completed is collected and the target detection network is used to determine whether the unhooking is successful. The carriage is unhooked, the unhooking action is completed, the robot returns to the initial position, and returns to step S1 to prepare for the next unhooking task.

2. A vision-based robot hook removal method according to claim 1, characterized in that: In S1, the point cloud segmentation network adopts a point convolution network based on density and position adaptive convolution. The annotated coupler point cloud dataset is used to train the point cloud segmentation network. The position of the coupler handle is segmented through the trained point cloud segmentation network, and the centroid of the coupler handle point cloud is used as the grasping position.

3. A vision-based robot hook removal method according to claim 2, characterized in that: The point cloud segmentation network is composed of an encoding part and a decoding part, the encoding part is composed of a first encoding layer, a second encoding layer, a third encoding layer and a fourth encoding layer; the first encoding layer is composed of a common encoding layer and a encoding layer carrying context information in parallel, and the second encoding layer, the third encoding layer and the fourth encoding layer are all common encoding layers; The decoding part consists of four decoding layers and one fully connected layer.

4. A vision-based robot hook removal method according to claim 3, characterized in that: The first encoding layer consists of a sampling layer, a grouping layer and a feature extraction layer. In the sampling layer, a certain number of points are sampled by the farthest point. In the grouping layer, the sampled points are used as the center point, and the k points closest to the center point are divided into a group. The feature extraction layer encodes the center point into a higher dimensional point based on the relationship between the center point and each neighboring point.

5. A vision-based robot hook removal method according to claim 4, characterized in that: The feature extraction layer is a density-position adaptive convolution, which consists of a ScoreNet and a weight library. The ScoreNet learns the position and density relationship between the center point and the neighborhood point through parallel MLPs and GRU to generate a series of adaptive coefficients. The generated series of adaptive coefficients are multiplied by the weight matrix in the weight library to obtain a kernel function. After the input features are mapped one by one to the kernel function, the output features are obtained through an aggregation function.

6. A vision-based robot hook removal method according to claim 5, characterized in that: The density relationship between the center point and the neighborhood points is calculated by adaptive kernel density estimation, using the Gaussian function as the kernel function of the adaptive kernel density estimation, and the Euclidean distance between the two points as a measure to evaluate the correlation between the two points, which is formulated as: Among them, dis represents the distance between the center point and the neighborhood point; k represents the number of neighborhood points; h is the variable bandwidth, which represents the smoothness of the adaptive kernel density curve, and h is formulated as: Varies based on the different distribution of each point in the group.

7. The vision-based robot hook removal method according to claim 1, characterized in that: In S2, the hand-eye calibration is the coordinate transformation between the robot and the laser radar coordinate system. According to the principle of least squares method, the values ​​of a set of points in two different coordinate systems are fitted to output the transformation matrix of the two coordinate systems.

8. The vision-based robot hook removal method according to claim 1, characterized in that: In S3, whether the robot has grasped successfully is determined based on the positional relationship between the coupler handle and the clamp. The position where the grasping fails is divided into three situations: left, right, and front. Each situation is distinguished through the target detection network, and the grasping position is adjusted according to the linear relationship between the size of the pixels occupied by the detection box in the image and the actual deviation until the grasping is successful.

9. A vision-based robot hook removal method according to claim 8, characterized in that: The object detection network is trained pixel by pixel based on the improved YOLOv5s model, and the feature extraction capability of the model is enhanced by increasing the depth of the neck network and introducing a coordinate attention mechanism.

10. The vision-based robot hook removal method according to claim 1, characterized in that: In S4, the hook-off action is an arc motion. The length of the coupler handle is obtained through the point cloud segmentation network, and the handle is lifted to a certain angle with this length as the radius until it is unhooked.

Images