A vision-based robotic hooking method
By combining 3D scanning and deep learning, high precision and safety of robot unhooking have been achieved, solving the problems of inaccurate positioning and danger of manual operation in existing technologies, and improving the automation level of the tippler coal unloading system.
Patent Information
- Application Number
- CN202411535385.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-31
- Publication Date
- 2025-11-28
- Estimated Expiration
- 2044-10-31
AI Technical Summary
Existing robotic unhooking methods suffer from insufficient positioning accuracy and pose safety hazards due to the dangers of manual unhooking operations.
A 3D scanner is used to acquire point cloud data, and a deep learning-based point cloud segmentation method is used to locate the coupler handle. The uncoupling process is optimized through target detection, and LiDAR and RGB camera are used together for precise grasping and detection.
This achieved high precision and safety in robot unhooking, improved the automation level of the tippler coal unloading system, and ensured the safety of the staff.
Smart Images
Figure CN119347758B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of train uncoupling, and in particular to a robot uncoupling method based on vision. BACKGROUND
[0002] With the rapid development of China's economy, the demand for coal in thermal power plants is increasing, and railway coal transportation is a common way for land-based thermal power plants to transport coal. After the coal train arrives at the power plant, a dumper is used to unload coal from each car. When the dumper system is working, the carriages on the heavy car line that need to be unloaded are uncoupled first, then the heavy car shunting machine pulls the uncoupled carriages into the dumper room, the dumper turns over to unload the cargo, and finally the next group of uncoupled carriages are pulled into the dumper room while the unloaded empty carriages are pushed to the empty car line, and the empty car shunting machine pushes them out of the car pulling platform.
[0003] In the dumper unloading system, most of the processes have been automated, but the carriage uncoupling operation before unloading is still mostly done manually. When manually uncoupling, the operation space is small, there is a risk of workers being caught in the equipment, and workers are used to uncoupling the train before it is stable. When workers are not focused or cannot adjust according to the actual situation, it can cause injury to personnel or damage to equipment.
[0004] Although robot uncoupling methods are disclosed in the prior art, they are all based on cameras taking photos and collecting two-dimensional RGB images, and then calculating three-dimensional information from two or more two-dimensional images through a stereo matching algorithm. In the stereo matching process, errors may be introduced, affecting the accuracy of the coupling positioning.
[0005] Therefore, there is a need to develop a robot uncoupling method based on vision to overcome the above problems. SUMMARY
[0006] The technical problem to be solved by the present application is to provide a robot uncoupling method based on vision, which uses a three-dimensional scanner to scan the coupler and directly obtains point cloud data carrying three-dimensional information, then uses a point cloud segmentation method based on deep learning to complete the positioning of the train coupler handle, and detects and optimizes the uncoupling result through target detection during uncoupling, making the robot uncoupling more accurate and accurate.
[0007] To solve the above technical problems, the technical solution adopted by the present application is:
[0008] A robot uncoupling method based on vision, comprising the following steps:
[0009] S1, when the carriage stops running, the laser radar starts, collects the point cloud data of the environment around the laser radar, and calculates the position information of the coupler handle through the point cloud segmentation network;
[0010] S2, the position information of the drawbar handle is converted to the robot coordinate system through hand-eye calibration, and the robot reaches the specified position according to the coordinate information;
[0011] S3, the camera starts and collects the RGB image when the robot reaches the specified position, and the target detection network is used to determine whether the robot grasps the drawbar handle;
[0012] S4, the robot performs the drawbar action according to the points of the drawbar path;
[0013] S5, after the drawbar is completed, the RGB image after the drawbar is completed is collected, and whether the drawbar is successful is determined through the target detection network, the car is uncoupled, the drawbar action is completed, the robot returns to the initial position, and returns to step S1 to prepare for the next drawbar task.
[0014] Further improvement of the technical scheme of the application is that in S1, the point cloud segmentation network adopts a point convolution network based on density and position adaptive convolution, the point cloud segmentation network is trained by using the labeled drawbar point cloud data set, the position of the drawbar handle is segmented by the trained point cloud segmentation network, and the center of mass of the drawbar handle point cloud is taken as the grasping position.
[0015] Further improvement of the technical scheme of the application is that the point cloud segmentation network is composed of an encoding part and a decoding part, the encoding part is composed of a first layer encoding layer, a second layer encoding layer, a third layer encoding layer and a fourth layer encoding layer; the first layer encoding layer is composed of one ordinary encoding layer and one encoding layer carrying context information in parallel, the second layer encoding layer, the third layer encoding layer and the fourth layer encoding layer are all ordinary encoding layers; the decoding part is composed of four decoding layers and one fully connected layer.
[0016] Further improvement of the technical scheme of the application is that the first layer encoding layer is composed of a sampling layer, a grouping layer and a feature extraction layer, a certain number of points are taken by farthest point sampling in the sampling layer, the center point is taken as the center point in the grouping layer, and the nearest k points around the center point are divided into a group, and the feature extraction layer encodes the center point and each neighborhood point into a higher dimensional point according to the relationship between the center point and each neighborhood point.
[0017] Further improvement of the technical scheme of the application is that the feature extraction layer is a density-position adaptive convolution, which is composed of ScoreNet and a weight bank; the ScoreNet learns the position and density relationship between the center point and the neighborhood point through parallel MLPs and GRUs, generates a series of adaptive coefficients, multiplies the generated series of adaptive coefficients with the weight matrix in the weight bank to obtain a kernel function, and inputs the feature and the kernel function one by one to obtain the output feature through the aggregation function.
[0018] Further improvement of the technical scheme of the present application is that the density relationship between the center point and the neighborhood point is calculated by adaptive kernel density estimation, a Gaussian function is used as the kernel function of adaptive kernel density estimation, and the Euclidean distance between the two points is used as the measure of the correlation between the two points, which is formulated as:
[0019]
[0020] where dis represents the distance between the center point and the neighborhood point, k represents the number of neighborhood points, h is a variable bandwidth, representing the smoothness of the adaptive kernel density curve, and h is formulated as: According to the different distribution of each point in the group.
[0021] Further improvement of the technical scheme of the present application is that in S2, the hand-eye calibration, i.e., the coordinate transformation between the robot and the laser radar coordinate system, according to the least square method, a group of points in two different coordinate systems are fitted, and the transformation matrix of the two coordinate systems is output.
[0022] Further improvement of the technical scheme of the present application is that in S3, according to the positional relationship between the car hook handle and the gripper, it is judged whether the robot is successfully grasped, the positions of the failed grasping are divided into three cases of left deviation, right deviation and front deviation, each case is distinguished by the target detection network, the grasping position is adjusted according to the linear relationship between the size of the detection frame in the image and the actual deviation until the grasping is successful.
[0023] Further improvement of the technical scheme of the present application is that the target detection network is trained pixel by pixel according to the improved yolov5s model, the depth of the neck network is increased and the coordinate attention mechanism is introduced to enhance the feature extraction capability of the model.
[0024] Further improvement of the technical scheme of the present application is that in S4, the hook pulling action is circular motion, the length of the car hook handle is obtained by the point cloud segmentation network, and the length is used as the radius to lift a certain angle until the hook is pulled off.
[0025] Due to the adoption of the above technical scheme, the present application has achieved the following technical progress:
[0026] The visual-based robot uncoupling method provided by the application comprises the following steps: collecting target point cloud and image data by using a laser radar and an RGB sensor respectively (scanning the coupler by using a three-dimensional scanner to directly obtain point cloud data carrying three-dimensional information), and completing construction of the point cloud and image data set; then, a point cloud segmentation network and an image detection network are built respectively, and the constructed data set is used for training; then, the pose of the robot for grabbing the coupler handle is determined in combination with the point cloud segmentation result, and the pose of the target relative to the end effector of the robot is determined through hand-eye calibration; finally, path planning is performed on the robot, the robot is controlled to complete the uncoupling action, the uncoupling result of the robot is judged through the image detection algorithm, and optimization is performed; so that the robot uncoupling is more accurate and accurate, the automation level of the car unloading system can be improved, and the life safety of the staff can be ensured. BRIEF DESCRIPTION OF DRAWINGS
[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.
[0028] Figure 1 It is a flow chart of the visual-based robot uncoupling method provided in the embodiment of the present application;
[0029] Figure 2 It is a structure diagram of the point cloud segmentation network in the embodiment of the present application;
[0030] Figure 3 It is a scorenet structure diagram in the embodiment of the present application;
[0031] Figure 4 It is a weight library structure diagram in the embodiment of the present application;
[0032] Figure 5 It is a density-position adaptive convolution structure diagram in the embodiment of the present application. DETAILED DESCRIPTION
[0033] It should be noted that the terms "include" and "have" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device containing a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.
[0034] The present application will be further described in detail below in combination with the drawings and embodiments:
[0035] As shown in Figure 1 A visual-based robot hook picking method, comprising the following steps:
[0036] S1, when the carriage stops running, the laser radar starts, collects the point cloud data of the environment around the laser radar, and calculates the position information of the hook handle through the point cloud segmentation network.
[0037] The point cloud segmentation network adopts a point convolution network based on density and position adaptive convolution, uses a labeled point cloud dataset of the hook handle to train the point cloud segmentation network, and segments the position of the hook handle through the trained point cloud segmentation network. The center of mass of the hook handle point cloud is taken as the grabbing position.
[0038] As shown in Figure 2 The point cloud segmentation network is composed of an encoding part and a decoding part, and the encoding part is composed of a first layer encoding layer, a second layer encoding layer, a third layer encoding layer and a fourth layer encoding layer. The first layer encoding layer is composed of a normal encoding layer and an encoding layer carrying context information in parallel, and the second layer encoding layer, the third layer encoding layer and the fourth layer encoding layer are all normal encoding layers. The decoding part is composed of four decoding layers and a fully connected layer. The difference between the normal encoding layer and the encoding layer carrying context information is that there is no GRU in the normal encoding layer.
[0039] The first layer encoding layer is composed of a sampling layer, a grouping layer and a feature extraction layer. A certain number of points are taken by farthest point sampling in the sampling layer. In the grouping layer, the taken points are taken as center points, and the nearest k points around the center points are divided into a group. The feature extraction layer encodes the center point and each neighborhood point into a higher-dimensional point according to the relationship between the center point and each neighborhood point.
[0040] As shown in Figure 3 , 4 The feature extraction layer is a density-position adaptive convolution, which is composed of ScoreNet and a weight bank. ScoreNet learns the position and density relationship between the center point and the neighborhood point through parallel MLPs and GRU, generates a series of adaptive coefficients, multiplies the generated series of adaptive coefficients with the weight matrix in the weight bank to obtain the kernel function, and inputs the feature and the kernel function one by one. After mapping, the output feature is obtained through the aggregation function.
[0041] As shown in Figure 5 The density relationship between the center point and the neighborhood point is calculated by adaptive kernel density estimation. The Gaussian function is taken as the kernel function of adaptive kernel density estimation, and the Euclidean distance between two points is taken as the measure of evaluating the correlation of two points, which can be formalized as:
[0042]
[0043] In the formula, dis represents the distance between the center point and the neighborhood point; k represents the number of neighborhood points; h is a variable bandwidth, representing the smoothness of the adaptive kernel density curve, which can be formulated as: It varies according to the different distribution of each point in the group.
[0044] Specifically, the point cloud segmentation network uses a self-defined train handle point cloud dataset, which contains 2000 point cloud data. Each group of data is divided into two categories: train handle and background. The division ratio of the training set and the test set is 7:1. The loss function for network training is cross-entropy loss and weight regularization. The number of training rounds is 100, and the learning rate is 0.05. At 0.6 and 0.8 of the total training rounds, the learning rate is adjusted to one-tenth of the original. After the point cloud segmentation network is trained, the trained model is evaluated on the test set. The evaluation indicators include accuracy and intersection over union. After the laser radar collects the data, the data is first filtered to filter out most of the noise points and background points, then block processing is performed to make it have the same resolution, and finally the point cloud segmentation network is input for segmentation. The center of mass of the segmented train handle is found as the grabbing position of the robot.
[0045] S2, through hand-eye calibration, the position information of the train handle is converted to the robot coordinate system. The robot reaches the specified position according to the coordinate information.
[0046] Specifically, hand-eye calibration is essentially to make one coordinate system coincide with another coordinate system through a series of rotations and translations. The hand-eye calibration problem is modeled as a least squares method. The input of the model is a set of point coordinates in the robot and laser radar coordinate systems. The robot moves within the field of view of the laser radar and collects 20 sets of coordinate data as input. The transformation matrix of the hand-eye calibration is obtained by singular value decomposition.
[0047] S3, the camera starts and collects the RGB image when the robot reaches the specified position. The target detection network is used to determine whether the robot has grabbed the train handle.
[0048] The target detection network is trained pixel by pixel based on the improved yolov5s model. By increasing the depth of the neck network and introducing the coordinate attention mechanism, the feature extraction capability of the model is enhanced.
[0049] According to the positional relationship between the train handle and the gripper, it is determined whether the robot has successfully grabbed. The positions where the grabbing fails are divided into three cases: left deviation, right deviation, and front deviation. Each case is distinguished by the target detection network. According to the linear relationship between the size of the detection box in the image and the actual deviation, the grabbing position is adjusted until the grabbing is successful.
[0050] Specifically, the data set used by the target detection network is a self-defined train handle picture data set, containing 1000 pictures, including five categories of robot grabbing handle, not grabbing and left, not grabbing and right, successfully unhooking and unsuccessfully unhooking, and being divided into a training set and a test set according to a ratio of 7:3. The loss function for training is a classification loss function, a positioning loss function and a confidence loss function. The number of training rounds is 150, the learning rate is 0.01, the cosine decay strategy is used to adjust the learning rate during training, and after the network training is completed, the trained model is evaluated on the test set. The evaluation indicators include precision, recall and average precision.
[0051] When the grabbing fails, the robot gripper needs to be adjusted forward or left and right. There is a linear relationship between the width of the detection box and the actual adjustment distance. According to this linear relationship, the robot grabbing position is adjusted.
[0052] S4, the robot performs an unhooking action according to the points of the unhooking path.
[0053] The unhooking action is a circular motion. The length of the train handle is obtained through the point cloud segmentation network, and the length is used as the radius to lift a certain angle until the hook is removed.
[0054] Specifically, after the robot grabs the train handle, the length l from the grabbing position (x, y) to the highest point of the lifting rod is used as the radius to make a circular motion. The intermediate point (x1, y1) is when the angle is 30°, and the end point (x2, y2) is when the angle is 70°. The path point coordinates are calculated as follows:
[0055] x1 = x + l sin 30°
[0056] y1 = y + l (1 - cos 30°)
[0057] x2 = x + l sin 70°
[0058] y2 = y + l (1 - cos 70°)
[0059] S5, after the unhooking is completed, the RGB image at this time (after the unhooking is completed) is collected, and whether the unhooking is successful is judged through the target detection network. The train carriage is unhooked, the unhooking action is completed, the robot returns to the initial position, and returns to step S1 to prepare to perform the next unhooking task.
[0060] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions recorded in the above embodiments can be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.
Claims
1. A vision-based robotic hook unhooking method, characterized by: The method comprises the following steps: S1, when the carriage stops running, the laser radar starts to collect point cloud data of the environment around the laser radar, and the position information of the drawbar handle is calculated through a point cloud segmentation network; The point cloud segmentation network is composed of an encoding part and a decoding part, and the encoding part is composed of a first layer encoding layer, a second layer encoding layer, a third layer encoding layer and a fourth layer encoding layer; The first layer encoding layer is composed of a sampling layer, a grouping layer and a feature extraction layer, a certain number of points are taken by farthest point sampling in the sampling layer, the taken points are taken as center points in the grouping layer, and the nearest k points around the center points are divided into a group, and the feature extraction layer encodes the center points and each neighborhood point into a higher-dimensional point according to the relationship between the center points and each neighborhood point; The feature extraction layer is a density-position adaptive convolution, which is composed of a ScoreNet and a weight bank; the ScoreNet learns the position and density relationship between the center point and the neighborhood point through parallel MLPs and GRUs, generates a series of adaptive coefficients, multiplies the generated series of adaptive coefficients with the weight matrix in the weight bank to obtain a kernel function, and inputs the feature and the kernel function one by one to obtain the output feature through an aggregation function; The density relationship between the center point and the neighborhood point is calculated by adaptive kernel density estimation, a Gaussian function is taken as the kernel function of adaptive kernel density estimation, and the Euclidean distance between two points is taken as the measure of evaluating the correlation between two points, which is formalized as: where dis represents the distance between the center point and the neighborhood points; k represents the number of neighborhood points; h is a variable bandwidth, which represents the smoothness of the adaptive kernel density curve, and is formulated as: varies according to the different distribution of each point in the group; S2, the position information of the drawbar handle is converted to the robot coordinate system through hand-eye calibration, and the robot reaches the specified position according to the coordinate information; S3, the camera starts and collects the RGB image when the robot reaches the specified position, and judges whether the robot holds the drawbar handle through the target detection network; S4, the robot performs the drawbar action according to the points of the drawbar path; S5, after the drawbar action is completed, the RGB image after the drawbar action is completed is collected, and whether the drawbar action is successful is judged through the target detection network, the carriage is uncoupled, the drawbar action is completed, the robot returns to the initial position, and returns to step S1 to prepare for the next drawbar task.
2. The vision-based robotic hook unhooking method of claim 1, wherein: In S1, the point cloud segmentation network adopts a point convolution network based on density and position adaptive convolution, the point cloud segmentation network is trained by using a labeled drawbar point cloud data set, the position of the drawbar handle is segmented by the trained point cloud segmentation network, and the center of mass of the drawbar handle point cloud is taken as the grabbing position.
3. The vision-based robotic unhooking method of claim 1, wherein: In S1, the first layer encoding layer is composed of a normal encoding layer and an encoding layer carrying context information in parallel, and the second layer encoding layer, the third layer encoding layer and the fourth layer encoding layer are all normal encoding layers; The decoding part is composed of four decoding layers and a fully connected layer.
4. The vision-based robotic unhooking method of claim 1, wherein: In S2, the hand-eye calibration is the coordinate transformation between the robot and the laser radar coordinate system, and a group of points in two different coordinate systems are fitted according to the least square method, and the transformation matrix of the two coordinate systems is output.
5. The vision-based robotic unhooking method of claim 1, wherein: In S3, whether the robot successfully grabs is determined according to the positional relationship between the coupler handle and the gripper. The positions where the grabbing fails are divided into three cases: left deviation, right deviation and front deviation. Each case is distinguished by a target detection network. The grabbing position is adjusted according to the linear relationship between the size of the detection frame in the image and the actual deviation until the grabbing is successful.
6. The vision-based robotic unhooking method of claim 5, wherein: The target detection network is trained pixel by pixel according to an improved yolov5s model. The feature extraction capability of the model is enhanced by increasing the depth of the neck network and introducing a coordinate attention mechanism.
7. The vision-based robotic unhooking method of claim 1, wherein: In S4, the decoupling action is a circular motion. The length of the coupler handle is obtained by a point cloud segmentation network. The length is used as a radius to lift a certain angle until the coupler is decoupled.