Industrial part grabbing method of mechanical arm based on U-Yolo6D algorithm

Through the combination of U-Yolo6D algorithm and image sensor, the three-dimensional object detection algorithm has been solved, and the three-dimensional object detection algorithm is low in accuracy and complexity in industrial production is achieved, more efficient robotic arm grabbing is achieved, and recognition accuracy and speed are improved. Enhanced system universality and generalization capabilities.

CN118906041AActive Publication Date: 2024-11-08FUDAN UNIVERSITY +1
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202310504094.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-06
Publication Date
2024-11-08
Estimated Expiration
2043-05-06

AI Technical Summary

Technical Problem

The existing three-dimensional object detection algorithm has low accuracy, high model complexity and slow computing speed in industrial production, and it is difficult to obtain a large amount of labeled data in some specific task areas, which limits the generalization ability and generalization of the robotic arm grab system. .

Method used

Using the robotic arm grabbing method based on the U-Yolo6D algorithm, by constructing the U-Yolo6D network model, image sensors are used to collect images and perform pose extraction, and feature maps of different sizes are generated by combining encoder and decoder, and features are fused through jump links. Figure, improve recognition accuracy and speed.

Benefits of technology

It improves the accuracy and speed of the robotic arm to grab industrial parts, enhances the generalization ability and versatility of the system, reduces the number of model parameters, and improves the success rate of grabbing in complex environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118906041A_ABST
    Figure CN118906041A_ABST
Patent Text Reader

Abstract

The invention provides a method for grabbing industrial parts by a mechanical arm based on a U-Yolo6D algorithm, which is characterized by comprising the following steps of: S1, obtaining a plurality of training images and corresponding annotation data as a training data set according to the industrial parts; s2, constructing a U-Yolo6D network model, training the U-Yolo6D network model through the training data set, and obtaining the trained U-Yolo6D network model as a pose extraction model; s3, acquiring an image containing the industrial part through an image sensor; s4, inputting the image into the pose extraction model to obtain a pose estimation result of the industrial part; and S5, according to the pose estimation result, the mechanical arm is controlled to grab the industrial part. In conclusion, the method can improve the accuracy and speed of grabbing the industrial parts by the mechanical arm.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the fields of image processing, visual guidance and robot arm grasping, and in particular to a method for a robot arm to grasp industrial parts based on a U-Yolo6D algorithm. Background Art

[0002] With the rise of smart factory production lines, the application of sorting robots is becoming more and more common. In industrial production, the degree of automation is getting higher and higher, and workpiece sorting is also developing in the direction of intelligence, customization, and efficiency. The technical core of the sorting robot is the robotic arm grasping system. This technology is mainly divided into two types. One is to teach the robotic arm at a fixed point, and the other is to help the robotic arm acquire and locate the target object through a machine vision system. In the technology of fixed-point teaching of the robotic arm, the grasping task is completed through a fixed path. This has relatively fixed requirements for the placement of the target object, and also has high requirements for the type and type of the workpiece. Generally, a fixed grasping device is required to execute, which is not very versatile and is not suitable for the expansion of the device. The technology of helping the robotic arm acquire and locate the target object through a machine vision system can more conveniently improve the robot's perception and control capabilities, and there is no need to make fixed restrictions on the types of grasping targets and grasping scenes, which greatly improves the generalization ability of the robotic arm grasping system.

[0003] The existing visual system mainly focuses on 3D target detection, which involves the later placement of the robot gripper. The 3D target detection technology is used to identify the position information of the object, which is passed to the robot arm, and the placement of the robot arm is adjusted to perform the grasping operation. The current mainstream 3D target detection algorithm is based on deep learning algorithms and requires large-scale labeled data. Some specific task areas cannot obtain large data sets in a timely manner, and the working conditions are extremely complex. In addition, the existing 3D target detection algorithm has low accuracy, high model complexity and slow model operation speed. Summary of the invention

[0004] The present invention is made to solve the above-mentioned problem, and aims to provide a method for a robotic arm to grasp industrial parts based on U-Yolo6D algorithm.

[0005] The invention provides a method for grabbing industrial parts by a robot arm based on a U-Yolo6D algorithm, which is used for grabbing industrial parts by a robot arm provided with an image sensor, and has the following characteristics, comprising the following steps: step S1, obtaining a plurality of training images and corresponding annotation data as a training data set according to the industrial parts; step S2, constructing a U-Yolo6D network model and training the U-Yolo6D network model through the training data set, and obtaining the trained U-Yolo6D network model as a pose extraction model; step S3, collecting images containing industrial parts through an image sensor; step S4, inputting the image into the pose extraction model, and obtaining a pose estimation result of the industrial parts; step S5, according to the pose estimation result, controlling the robot arm to grab the industrial parts, wherein in step S2, the training process is to input the training image into the U-Yolo6D network model, obtain a segmentation result and a detection result, and then update the network parameters of the U-Yolo6D network model according to the segmentation result and the detection result, and the U-Yolo6D network model is updated. The model includes multiple convolutional layers, encoders and decoders. The specific process of obtaining the segmentation result and the detection result is: downsampling the training image to obtain a scaled training image, continuously downsampling the scaled training image in the encoder to obtain a first feature map, a second feature map, and a third feature map in sequence, passing the third feature map through the convolution layer and the upsampling layer in sequence in the decoder, and then splicing the result of the convolution layer with the result of the second feature map to obtain a fourth feature map, passing the fourth feature map through the convolution layer and the upsampling layer in sequence, and then splicing it with the result of the convolution layer with the first feature map to obtain a fifth feature map, passing the scaled training image through four convolutional layers in sequence, and then splicing it with the result of the convolution layer and the upsampling layer with the fifth feature map to obtain a sixth feature map, passing the sixth feature map through the convolution layer to obtain the segmentation result, splicing the result of downsampling the third feature map, the result of downsampling the fourth feature map twice in a row, the result of downsampling the fifth feature map three times in a row, and the result of downsampling the sixth feature map four times in a row, and then passing the convolution layer to obtain the detection result.

[0006] In the method for grasping industrial parts by a robotic arm based on the U-Yolo6D algorithm provided by the present invention, it can also have the following characteristics: wherein, step S1 includes the following sub-steps: step S1-1, constructing a corresponding three-dimensional model according to the industrial part; step S1-2, collecting multi-view image data of the three-dimensional model as a training image through 3D software; step S1-3, constructing a minimum three-dimensional rectangular frame of the three-dimensional model, and then projecting the minimum three-dimensional rectangular frame onto the corresponding training image to obtain the annotation data corresponding to the training image, and using multiple training images and the corresponding annotation data as a training data set.

[0007] In the method for grasping industrial parts by a robotic arm based on the U-Yolo6D algorithm provided by the present invention, it can also have the following characteristics: wherein, step S2 includes the following sub-steps: step S2-1, selecting a training image from a training data set and inputting it into the U-Yolo6D network model to obtain segmentation results and detection results, the detection results including the classification of industrial parts in the training image, the overall confidence score and the coordinate position of the control point; step S2-2, calculating the total loss function of the U-Yolo6D network model according to the labeled data, the segmentation results and the detection results to obtain the total loss; step S2-3, updating the network parameters of the U-Yolo6D network model according to the total loss; step S2-4, repeating steps S2-1 to S2-3 until the training completion condition is met, and obtaining the trained U-Yolo6D network model as a pose extraction model.

[0008] In the method for grasping industrial parts by a robotic arm based on the U-Yolo6D algorithm provided by the present invention, the following features may also be provided: wherein, in step S2-2, the expression of the total loss function is: L = λ pt L pt +λ conf L conf +λ id L id +λ m L m , where L is the total loss, L pt is the coordinate loss corresponding to the coordinate position of the control point, L conf is the confidence loss corresponding to the overall confidence score, L id is the classification loss of the corresponding classification, L m is the segmentation loss corresponding to the segmentation result, λ pt , conf , id and λ m are the weight parameters corresponding to each loss.

[0009] In the method for grasping industrial parts by a robotic arm based on the U-Yolo6D algorithm provided by the present invention, it can also have the following features: wherein, step S4 includes the following sub-steps: step S4-1, inputting the image into a pose extraction model to obtain the position coordinates of the control points of the 2D projection of the center of mass, category and corners of the 3D bounding box of the industrial part; step S4-2, obtaining the pose estimation result according to the position coordinates through a perspective n-point pose estimation method.

[0010] In the method for grasping industrial parts by a robotic arm based on the U-Yolo6D algorithm provided by the present invention, it can also have the following characteristics: wherein, step S5 includes the following sub-steps: step S5-1, transforming the pose estimation result through the image sensor and the rotation matrix of the robotic arm to obtain the actual grasping pose; step S5-2, performing path planning and adjusting the posture of the gripper of the robotic arm according to the actual grasping pose, and then controlling the gripper to open; step S5-3, according to the result of path planning, controlling the end position of the gripper to move to the center point of the industrial part, and then controlling the gripper to close to achieve grasping of the industrial part.

[0011] Functions and Effects of the Invention

[0012] According to the method for grasping industrial parts by a robotic arm based on the U-Yolo6D algorithm involved in the present invention, the position of the industrial parts can be better located to generate auxiliary segmentation images through the segmentation results and segmentation losses during the training model, thereby eliminating the influence of other objects in the surrounding environment on the recognition, thereby helping the feature extraction network to better extract features. In addition, three layers of feature maps of different sizes are generated by the encoder and the decoder, and the high-resolution feature map is fused with the low-resolution feature map through the jump link, so that the recognition accuracy and speed can be further improved when the number of overall model parameters is reduced. Therefore, the method for grasping industrial parts by a robotic arm based on the U-Yolo6D algorithm of the present invention can improve the accuracy and speed of grasping industrial parts by the robotic arm. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a flow chart of a method for a robotic arm to grasp industrial parts based on the U-Yolo6D algorithm in an embodiment of the present invention;

[0014] Figure 2 is a schematic diagram of a chessboard rendered by 3ds Max in an embodiment of the present invention;

[0015] Figure 3 is a schematic diagram of a minimum three-dimensional rectangular frame in an embodiment of the present invention;

[0016] Figure 4 is a schematic diagram of a mask image in an embodiment of the present invention;

[0017] Figure 5 It is a schematic diagram of the structure of the U-Yolo6D network model in an embodiment of the present invention;

[0018] Figure 6 is a schematic diagram of the result of the Yolo6D network model generating a 3D bounding box for an industrial part in an embodiment of the present invention;

[0019] Figure 7It is a schematic diagram of the result of the U-Yolo6D network model generating a 3D bounding box for an industrial part in an embodiment of the present invention;

[0020] Figure 8 It is a schematic diagram of the number of iterations and the mAP index of the Yolo6D network model and the U-Yolo6D network model in an embodiment of the present invention;

[0021] Fig. 9 It is a schematic diagram of a robot arm actually grabbing industrial parts at a work site in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following embodiments and the accompanying drawings specifically illustrate the method of grasping industrial parts by a robotic arm based on the U-Yolo6D algorithm of the present invention.

[0023] Figure 1 It is a flow chart of a method for a robotic arm to grasp industrial parts based on the U-Yolo6D algorithm in an embodiment of the present invention.

[0024] like Figure 1 As shown, the method for grasping industrial parts by a robotic arm based on the U-Yolo6D algorithm of this embodiment is used to grasp industrial parts by a robotic arm provided with an image sensor. In this embodiment, the image sensor is a Fethch SCS40 depth camera arranged at the end of the robotic arm, and the industrial part is a bolt, and the method comprises the following steps:

[0025] Step S1, obtaining a plurality of training images and corresponding annotation data as a training data set according to industrial parts.

[0026] Wherein, step S1 includes the following sub-steps:

[0027] Step S1-1, constructing a corresponding three-dimensional model according to the industrial parts.

[0028] In this embodiment, the system unit conversion ratio is set in 3ds Max to 1 unit = 10 mm, so as to construct a first three-dimensional model of the industrial part, and then the position of the first three-dimensional model in the world coordinate system is adjusted to the origin of the world coordinate system, and the Y rotation angle is set to 30° and the Z rotation angle is set to 0°, so that the first three-dimensional model is flattened in the view, and then a plurality of three-dimensional models of different sizes are constructed based on the first three-dimensional model and placed around the first three-dimensional model, and the three-dimensional models do not overlap, so that all the three-dimensional models are the three-dimensional models corresponding to the industrial part.

[0029] Step S1-2, collecting multi-view image data of the three-dimensional model as training images through 3D software.

[0030] Step S1-2 of this embodiment includes the following sub-steps:

[0031] Step S1-2-1, create a target camera in 3ds Max, align the target camera with the model center of the three-dimensional model constructed in step S1-1, that is, the origin of the world coordinate system, and adjust the exposure parameters of the target camera.

[0032] Step S1-2-2, set the circular motion trajectory of the target camera, adjust the radius, height and rotation angle of the circular motion trajectory as the path constraint of the target camera, then convert the motion constraint into the circular motion trajectory, create a camera motion animation, and set the motion trajectory to 100 frames.

[0033] Step S1-2-3, select a background image as the background image of the camera view, and change the background to screen mode, and set the offset angle of the background image in the material editor so that it can better simulate the real scene in the camera view.

[0034] Step S1-2-4, set the active rendering time period to animation frames 0 to 99, each frame is a picture, adjust the appropriate rendering area, set the camera view as the acquisition target, and adjust the picture quality parameters. Finally, adjust the size of the output full-color picture to a 416*416 jpg format file. All jpg format files are multi-view image data of the three-dimensional model.

[0035] Step S1-3, constructing a minimum three-dimensional rectangular frame of the three-dimensional model, and then projecting the minimum three-dimensional rectangular frame onto the corresponding training image to obtain the annotation data corresponding to the training image, and using multiple training images and the corresponding annotation data as the training data set.

[0036] Step S1-3 of this embodiment includes the following sub-steps:

[0037] Step S1-3-1, create a cuboid in 3ds Max, set the material of the cuboid to checkerboard, and set the size of the checkerboard to 7*8, then render it through the 3ds Max renderer to make a standard checkerboard calibration plate, and obtain 18 checkerboard images of different angles and sizes based on the standard checkerboard calibration plate.

[0038] Figure 2 It is a schematic diagram of a chessboard rendered by 3ds Max in an embodiment of the present invention.

[0039] Step S1-3-2, input all the checkerboard images into MATLAB, and then calibrate the camera internal parameters, which include the focal length of the camera, the coordinates of the principal point of the optical axis projection in the image pixel coordinate system, the tilt parameter of the coordinate axis, the actual physical value represented by each pixel, the radial distortion coefficient and the tangential distortion coefficient of the camera.

[0040] Step S1-3-3, using the measurement tool of 3ds Max to measure the spatial coordinate values ​​of the eight vertices and the center point of the smallest three-dimensional rectangular box of the three-dimensional model in the world coordinate system.

[0041] Step S1-3-4, use MATLAB to calculate the translation matrix and rotation matrix of the target camera, obtain the external parameters of the target camera, and obtain the two-dimensional projection pixel coordinates of the vertex space coordinates of the minimum three-dimensional rectangular box of the three-dimensional model on the corresponding image by constructing the projection transformation relationship from the world coordinate system to the pixel coordinates. The labeling of the target minimum three-dimensional rectangular box is completed. The labeling data includes the coordinates of the 9 control points of the three-dimensional model, namely the 8 vertices and the center point, as well as the category information.

[0042] Figure 3 It is a schematic diagram of a minimum three-dimensional rectangular frame in an embodiment of the present invention.

[0043] like Figure 3 As shown, corresponding minimum three-dimensional rectangular frames are constructed for the three industrial parts of different sizes in the picture, and the coordinates of the eight vertices and the center point are obtained by marking the minimum three-dimensional rectangular frames through the horizontal axis and the vertical axis.

[0044] Step S2, constructing a U-Yolo6D network model and training the U-Yolo6D network model through a training data set, obtaining a trained U-Yolo6D network model as a pose extraction model. In this embodiment, the learning rate is set to 0.001, the number of iterations is 500, and the batch_sizes is 64, the ImageNet initial network is used for pre-training, the confidence of the regularization parameter is set to 0 to pre-train the network parameters of the U-Yolo6D network model, and then the λconf of the cell containing the target is set to 5 to train the U-Yolo6D network model, the clarity of the confidence function α is set to 2, the distance threshold is set to 30 pixels, and the stochastic gradient descent method is used for optimization.

[0045] In step S2, the training process is to input the training image into the U-Yolo6D network model, obtain the segmentation result and the detection result, and then update the network parameters of the U-Yolo6D network model according to the segmentation result and the detection result. Step S2 includes the following sub-steps:

[0046] Step S2-1, select a training image from the training data set and input it into the U-Yolo6D network model to obtain segmentation results and detection results, the detection results include the classification of industrial parts in the training image, the overall confidence score and the coordinate position of the control point. In this embodiment, the segmentation result is a mask image.

[0047] Figure 4Schematic diagram of a mask diagram in an embodiment of the present invention.

[0048] like Figure 4 As shown, in the mask image, the part containing industrial parts is represented by a white rectangle, and the part not containing industrial parts is represented by black.

[0049] Figure 5 It is a structural diagram of the U-Yolo6D network model in an embodiment of the present invention.

[0050] like Figure 5 As shown, the U-Yolo6D network model includes multiple convolutional layers, encoders and decoders. In this embodiment, the encoder consists of a repeated application of a 2x2 max pooling operation with a stride of 2 for downsampling. The specific process of obtaining the segmentation result and the detection result is as follows:

[0051] The training image is downsampled to obtain a scaled training image with a size of 208*208 and 32 feature channels. In the encoder, the scaled training image is continuously downsampled to obtain a first feature map with a size of 104*104 and 64 feature channels, a second feature map with a size of 52*52 and 128 feature channels, and a third feature map with a size of 26*26 and 256 feature channels.

[0052] In the decoder, the third feature map passes through the 3*3 convolution layer and the upsampling layer in sequence, and then is spliced ​​with the result of the second feature map through the 3*3 convolution layer to obtain a fourth feature map with a size of 52*52 and 128 feature channels. The fourth feature map passes through the 3*3 convolution layer and the upsampling layer in sequence, and then is spliced ​​with the result of the first feature map through the 3*3 convolution layer to obtain a fifth feature map with a size of 104*104 and 64 feature channels. The scaled training image passes through four 3*3 convolution layers in sequence, and then is spliced ​​with the result of the fifth feature map through the 3*3 convolution layer and the upsampling layer in sequence to obtain a sixth feature map with a size of 208*208 and 32 feature channels. The sixth feature map passes through a 3*3 convolution layer to obtain a segmentation result with a size of 208*208 and 3 feature channels.

[0053] The result of downsampling the third feature map once, the result of downsampling the fourth feature map twice in a row, the result of downsampling the fifth feature map three times in a row, and the result of downsampling the sixth feature map four times in a row are concatenated, and then a 3*3 convolutional layer is used to obtain a detection result with a size of 13*13 and 20 feature channels.

[0054] In this embodiment, the 6 layers of 3*3 convolutional layers that obtain the segmentation results are followed by a batch normalization layer and an activation function layer respectively. The detection result with a size of 13*13 and a feature channel number of 20 indicates that the training image is divided into 13*13 cells, each cell has a 20-dimensional vector, which represents the (x, y) coordinates of 9 control points, 1 class probability and 1 confidence value. Through the Anchor mechanism, each cell has 5 candidate objects, namely anchor boxes, and the anchor box with the size most similar to the current object is selected to obtain 9 control points.

[0055] Step S2-2, according to the labeled data, segmentation results and detection results, calculate the total loss function of the U-Yolo6D network model to obtain the total loss.

[0056] Among them, the expression of the total loss function is:

[0057] L=λ pt L pt +λ conf L conf +λ id L id +λ m L m ,

[0058] Where L is the total loss, L pt is the coordinate loss corresponding to the coordinate position of the control point, L conf is the confidence loss corresponding to the overall confidence score, L id is the classification loss of the corresponding classification, L m is the segmentation loss corresponding to the segmentation result, λ pt , conf , id and λ m are the weight parameters corresponding to each loss.

[0059] Step S2-3, updating the network parameters of the U-Yolo6D network model according to the total loss.

[0060] Step S2-4, repeat steps S2-1 to S2-3 until the training completion condition is met, and obtain the trained U-Yolo6D network model as the pose extraction model.

[0061] Step S3, collecting images containing industrial parts through an image sensor.

[0062] Step S4: input the image into a pose extraction model to obtain a pose estimation result of the industrial part.

[0063] Wherein, step S4 includes the following sub-steps:

[0064] Step S4-1, input the image into the pose extraction model to obtain the position coordinates of the control points of the 2D projection of the center of mass, category and corners of the 3D bounding box of the industrial part.

[0065] In this embodiment, after the image is input into the pose extraction model, the class probability of each grid unit after image segmentation and the score returned by the confidence function are obtained, the class probability and the score are multiplied together to obtain the class-specific confidence score of each object, and the units with lower confidence scores are pruned according to the confidence threshold to obtain the entire prediction result, that is, the center of mass of the industrial part, the category, and the position coordinates of the control points of the 2D projection of the corners of the 3D bounding box.

[0066] In this embodiment, for larger industrial parts in the image and industrial parts whose projections are located at the intersection of two cells, it is necessary to combine 3*3 cells in the vicinity of the industrial parts, and perform weighted averaging of the confidence scores of the industrial parts based on these 9 cells to obtain the final prediction result. The weights used in the weighted process are the confidence scores of the relevant cells.

[0067] Step S4-2, according to the position coordinates, obtain the pose estimation result by the perspective n-point pose estimation method. In this embodiment, the pose estimation result is 6-dimensional pose information (X, Y, Z, R x ,R y ,R z ), X, Y, and Z are the xyz-axis coordinate values ​​of the center point of the industrial part in the camera coordinate system, R x , R y , R z They are the rotation angles of industrial parts relative to the x-axis, y-axis, and z-axis respectively.

[0068] Step S5: Control the robot arm to grasp the industrial part according to the pose estimation result.

[0069] Wherein, step S5 includes the following sub-steps:

[0070] Step S5-1, transforming the pose estimation result through the image sensor and the rotation matrix of the robot arm to obtain the actual grasping pose.

[0071] In this embodiment, the rotation matrix is ​​obtained by the following method: a Fethch SCS40 depth camera and a gripper are set at the end of the robotic arm, the hand-eye calibration of the robotic arm is performed using the "eye on the hand" method, the robotic arm is moved to different positions, and the image of the calibration plate is captured to obtain a total of 12 sets of different posture information, and the posture of the robotic arm end in the camera coordinate system is calculated, and then the rotation matrix from the camera coordinate system to the robotic arm end coordinate system is obtained.

[0072] The calculation process of the actual grasping posture in this embodiment is: the world coordinates of the industrial parts to be grasped in the robot base coordinate system are calculated according to the transformation matrix. The actual grasping position of the robot is the center position where the gripper is closed and gathered together, not the end of the robot. Therefore, the distance in the z-axis direction of the coordinate system of the end of the robot is subtracted from the length of the gripper, and the values ​​of other coordinate axes and rotation angles remain unchanged to obtain the actual grasping posture.

[0073] Step S5-2, according to the actual grasping posture, perform path planning and adjust the posture of the gripper of the robot arm, and then control the gripper to open.

[0074] Step S5-3, according to the result of path planning, after controlling the end position of the gripper to move to the center point of the industrial part, the gripper is controlled to close to grasp the industrial part.

[0075] In this embodiment, a U-Yolo6D network model is constructed by the method for grasping industrial parts by a mechanical arm based on the U-Yolo6D algorithm of the present invention, a Yolo6D network model is constructed by the existing Yolo6D algorithm, 4800 images and corresponding annotation data are constructed as a data set by 3ds Max and industrial parts, and then divided into a training set, a test set and a validation set in a ratio of 6:2:2, the U-Yolo6D network model and the Yolo6D network model are trained by the training set and the test set, and then the average accuracy of the whole class of the two trained models under different pixel thresholds on the validation set is calculated, that is, the mAP index, the 10% pixel threshold indicates that the error between the anchor frame obtained by the model and the corresponding annotation data is within 10 pixels, the 5% pixel threshold indicates that the error between the anchor frame obtained by the model and the corresponding annotation data is within 5 pixels, and the 2% pixel threshold indicates that the error between the anchor frame obtained by the model and the corresponding annotation data is within 2 pixels. The mAP indicators of the two models at 10%, 5% and 2% pixel thresholds are shown in the following table:

[0076]

[0077] The first column is the model name, the second to fourth columns are the values ​​of the mAP index of the corresponding model at 10% pixel threshold, 5% pixel threshold and 2% pixel threshold respectively, and the fifth column is the average value of the mAP index of the corresponding model at all different pixel thresholds. For example, the cell in the third row and second column is the mAP index of the U-Yolo6D network model at a 10% pixel threshold of 99.89%. It can be seen that the method of grasping industrial parts by a robotic arm based on the U-Yolo6D algorithm of the present invention has a higher accuracy rate in identifying objects in images than the existing Yolo6D algorithm.

[0078] Figure 6It is a schematic diagram of the result of the Yolo6D network model generating a 3D bounding box for an industrial part in an embodiment of the present invention.

[0079] Figure 7 It is a schematic diagram of the result of the U-Yolo6D network model generating a 3D bounding box for an industrial part in an embodiment of the present invention.

[0080] like Figure 6 and Figure 7 As shown in the figure, compared with the Yolo6D network model, the U-Yolo6D network model can more accurately identify the position of industrial parts in the image, that is, the 3D bounding box, so as to obtain a pose estimation result that is more in line with the state of the industrial parts, thereby improving the success rate of grasping industrial parts.

[0081] In this embodiment, the same training data is used to train the Yolo6D network model and the U-Yolo6D network model. Different numbers of iterations are used in the training to obtain the corresponding trained models. The mAP indicators of the two trained models are calculated respectively through the test data.

[0082] Figure 8 It is a schematic diagram of the number of iterations and the mAP index of the Yolo6D network model and the U-Yolo6D network model in an embodiment of the present invention.

[0083] like Figure 8 As shown, the horizontal axis is the number of iterations of the model using training data, i.e. epoch, and the vertical axis is the value of the mAP index of the model trained under the test data. The solid line is the change curve of the U-Yolo6D network model, and the dotted line is the change curve of the Yolo6D network model. Under the same number of iterations, the U-Yolo6D network model has a better mAP index than the Yolo6D network model. It can be seen that the U-Yolo6D network model has a better recognition accuracy than the Yolo6D network model.

[0084] In this embodiment, the method for grabbing industrial parts by a robotic arm based on the U-Yolo6D algorithm of the present invention is used to test the robotic arm grabbing industrial parts in different positions multiple times. The test results are shown in the following table:

[0085] Number of crawls Average crawling time (seconds) Number of successful crawls Crawl success rate 100 times 7.57 91 91.0%

[0086] The first column is the total number of times the industrial parts were grabbed in the test, the second column is the average grabbing time of the robot arm grabbing the industrial parts in the test, the third column is the number of successful grabs in the test, and the fourth example is the probability of successful grabs in the test. For example, the cell in the fourth column of the second row indicates that the success rate of grabbing industrial parts when the robot arm grabbed 100 times in the test is 91.0%. It can be seen that the method of grabbing industrial parts by a robot arm based on the U-Yolo6D algorithm of the present invention can more accurately realize the grabbing of industrial parts in complex environments.

[0087] Fig. 9 It is a schematic diagram of a robot arm actually grabbing industrial parts at a work site in an embodiment of the present invention.

[0088] like Fig. 9 As shown, (a) to (j) are pictures of the robot arm's movements taken in chronological order during the actual process of the robot arm grasping industrial parts at the work site. (a) to (e) show the process in which the robot arm opens its gripper and moves in a point-to-point motion until the end of the gripper in the closed state moves to the center point of the industrial part, and then closes the gripper to grasp the industrial part. (f) to (j) show the process in which the robot arm starts to move after the gripper grasps the industrial part, thereby grasping the industrial part to the specified position.

[0089] Functions and Effects of the Embodiments

[0090] According to the method for grasping industrial parts by a robotic arm based on the U-Yolo6D algorithm involved in this embodiment, firstly, according to the characteristics of the industrial parts, only the minimum 3D bounding box of the industrial parts needs to be identified without identifying the surface texture information of the object, thereby reducing the number of model parameters when building the model. Secondly, when training the model, the position of the industrial parts can be better located through the segmentation results and segmentation losses to generate auxiliary segmentation images, thereby eliminating the influence of other objects in the surrounding environment on the recognition, thereby helping the feature extraction network to better extract features. Finally, three layers of feature maps of different sizes are generated through the encoder and decoder, and the high-resolution feature map is fused with the low-resolution feature map through jump links, so that the recognition accuracy and speed can be further improved while the number of overall model parameters is reduced. In short, this method can improve the accuracy and speed of grasping industrial parts by the robotic arm.

[0091] The above-mentioned embodiments are preferred examples of the present invention and are not intended to limit the protection scope of the present invention.

Claims

1. A method for grasping industrial parts by a robotic arm based on the U-Yolo6D algorithm, which is used to grasp industrial parts by a robotic arm equipped with an image sensor, characterized in that: The following steps are involved: Step S1, obtaining a plurality of training images and corresponding annotation data as a training data set according to the industrial parts; Step S2, constructing a U-Yolo6D network model and training the U-Yolo6D network model through the training data set to obtain the trained U-Yolo6D network model as a pose extraction model; Step S3, collecting an image containing the industrial part by the image sensor; Step S4, inputting the image into the pose extraction model to obtain a pose estimation result of the industrial part; Step S5, controlling the robot arm to grasp the industrial part according to the pose estimation result, Wherein, in the step S2, the training process is to input the training image into the U-Yolo6D network model to obtain a segmentation result and a detection result, and then update the network parameters of the U-Yolo6D network model according to the segmentation result and the detection result. The U-Yolo6D network model includes multiple convolutional layers, encoders and decoders. The specific process of obtaining the segmentation result and the detection result is as follows: Downsampling the training image to obtain a scaled training image, In the encoder, the scaled training image is continuously downsampled to obtain a first feature map, a second feature map, and a third feature map in sequence, In the decoder, the third feature map is sequentially passed through a convolution layer and an upsampling layer, and then concatenated with the result of the second feature map passing through the convolution layer to obtain a fourth feature map. The fourth feature map is sequentially passed through a convolution layer and an upsampling layer, and then concatenated with the result of the first feature map passing through the convolution layer to obtain a fifth feature map. The scaled training image is sequentially passed through four convolutional layers and then concatenated with the result of the fifth feature map sequentially passing through the convolutional layer and the upsampling layer to obtain a sixth feature map. The segmentation result is obtained by performing a convolution on the sixth feature map. The result of downsampling the third feature map, the result of downsampling the fourth feature map twice in a row, the result of downsampling the fifth feature map three times in a row, and the result of downsampling the sixth feature map four times in a row are concatenated, and then the detection result is obtained through a convolutional layer.

2. The method for grabbing industrial parts by a robotic arm based on the U-Yolo6D algorithm according to claim 1, Features: Wherein, the step S1 includes the following sub-steps: Step S1-1, constructing a corresponding three-dimensional model according to the industrial part; Step S1-2, collecting multi-view image data of the three-dimensional model as the training image through 3D software; Step S1-3, constructing a minimum three-dimensional rectangular frame of the three-dimensional model, and then projecting the minimum three-dimensional rectangular frame onto the corresponding training image to obtain the annotation data corresponding to the training image, and using multiple training images and the corresponding annotation data as the training data set.

3. The method for grabbing industrial parts by a robotic arm based on the U-Yolo6D algorithm according to claim 1, Features: Wherein, the step S2 includes the following sub-steps: Step S2-1, selecting a training image from the training data set and inputting it into the U-Yolo6D network model to obtain the segmentation result and the detection result, wherein the detection result includes the classification of the industrial parts in the training image, the overall confidence score and the coordinate position of the control point; Step S2-2, calculating the total loss function of the U-Yolo6D network model according to the labeled data, the segmentation result and the detection result to obtain the total loss; Step S2-3, updating the network parameters of the U-Yolo6D network model according to the total loss; Step S2-4, repeatedly execute step S2-1 to step S2-3 until the training completion condition is met, and obtain the trained U-Yolo6D network model as the pose extraction model.

4. The method for grabbing industrial parts by a robotic arm based on the U-Yolo6D algorithm according to claim 3 is characterized in that: in, In step S2-2, the expression of the total loss function is: L=λ pt L pt +λ conf L conf +λ id L id +λ m L m , Where L is the total loss, L pt is the coordinate loss corresponding to the coordinate position of the control point, L conf is the confidence loss corresponding to the overall confidence score, L id is the classification loss corresponding to the classification, L m is the segmentation loss corresponding to the segmentation result, λ pt , conf , id and λ m are the weight parameters corresponding to each loss.

5. The method for grabbing industrial parts by a robotic arm based on the U-Yolo6D algorithm according to claim 1, Features: Wherein, the step S4 includes the following sub-steps: Step S4-1, inputting the image into the pose extraction model to obtain the position coordinates of the control points of the 2D projection of the centroid, category and corners of the 3D bounding box of the industrial part; Step S4-2, obtaining the pose estimation result according to the position coordinates through a perspective n-point pose estimation method.

6. The method for grabbing industrial parts by a robotic arm based on the U-Yolo6D algorithm according to claim 1, Features: Wherein, the step S5 includes the following sub-steps: Step S5-1, transforming the pose estimation result via the image sensor and the rotation matrix of the robotic arm to obtain an actual grasping pose; Step S5-2, performing path planning and adjusting the posture of the gripper of the robotic arm according to the actual gripping posture, and then controlling the gripper to open; Step S5-3, according to the result of the path planning, after controlling the end position of the gripper to move to the center point of the industrial part, the gripper is controlled to close to grasp the industrial part.

Citation Information

Patent Citations

  • Industrial mechanical arm visual control method based on depth convolution neural network

    CN106874914A

  • Robot grabbing method and system based on 6D pose estimation

    CN115641322A

  • Manipulator grabbing method based on deep learning target detection and image segmentation

    CN115816460A

  • Systems and methods for picking objects using 3-d geometry and segmentation

    US20220410381A1

  • Autonomous mobile grabbing method for mechanical arm based on visual-haptic fusion under complex illumination condition

    US20230042756A1