A mechanical arm posture detection method based on point cloud and image fusion

Abstract

The present application relates to mechanical arm posture detection technology, specifically a kind of mechanical arm posture detection method based on point cloud and image fusion.The present application solves the problems of slow detection speed, low accuracy of detection results and poor robustness of detection results of existing mechanical arm posture detection technology.A kind of mechanical arm posture detection method based on point cloud and image fusion, the method is realized by using the following steps: step one: obtain point cloud data by laser radar;Step two: obtain point cloud two-dimensional projection view after two-dimensional projection of point cloud data;Step three: target segmentation is carried out on point cloud two-dimensional projection view by MSNet network;Step four: the position information of mechanical arm in point cloud two-dimensional projection view is clustered;Step five: the minimum envelope rectangle is drawn to the clustering result, and the center point coordinates and angle of minimum envelope rectangle are adjusted, so that the intersection and union ratio of minimum envelope rectangle and mechanical arm target position reaches maximum.The present application is suitable for mechanical arm posture detection in various scenes.

Description

Technical Field

[0001] The present invention relates to a robot arm posture detection technology, and in particular to a robot arm posture detection method based on point cloud and image fusion. Background Art

[0002] With the rapid advancement of science and technology and the rapid development of the manufacturing industry, robotic arms are increasingly being used in the manufacturing industry. In practical applications, to determine whether a robotic arm is operating normally, it is necessary to detect its posture. However, due to the limitations of existing robotic arm posture detection technologies, they generally suffer from slow detection speed, low accuracy, and poor robustness. Therefore, it is necessary to develop a robotic arm posture detection method based on point cloud and image fusion to address these issues. Summary of the Invention

[0003] In order to solve the problems of slow detection speed, low detection accuracy and poor robustness of existing robotic arm posture detection technologies, the present invention provides a robotic arm posture detection method based on point cloud and image fusion.

[0004] The present invention is achieved by adopting the following technical solutions:

[0005] A robotic arm posture detection method based on point cloud and image fusion is implemented using the following steps:

[0006] Step 1: Obtain point cloud data through lidar;

[0007] Step 2: Jointly calibrate the LiDAR and monocular camera, project the point cloud data into two dimensions to obtain a two-dimensional projection view of the point cloud, and then record the depth information of each pixel in the two-dimensional projection view of the point cloud;

[0008] Step 3: Use the MSNet network to segment the target in the point cloud 2D projection view, find the target area of ​​the robot arm in the point cloud 2D projection view, and then perform point cloud inverse calculation on the target area and the depth information of each pixel in the point cloud 2D projection view to find the position information of the robot arm in the point cloud 2D projection view;

[0009] Step 4: Cluster the position information of the robotic arm in the two-dimensional projection view of the point cloud;

[0010] Step 5: Draw a minimum envelope rectangle for the clustering results, and adjust the center point coordinates and angle of the minimum envelope rectangle so that the intersection ratio of the minimum envelope rectangle and the target position of the robotic arm reaches the maximum, and then use the minimum envelope rectangle as the posture detection result of the robotic arm.

[0011] In the step 1, after acquiring the point cloud data, the point cloud data is preprocessed using a graphics computing algorithm library; the preprocessing step specifically includes bilateral filtering and redundant point cloud removal.

[0012] In step 2, the joint calibration steps are as follows:

[0013] First, the three-dimensional point cloud data of the calibration plate is obtained by the laser radar, and the two-dimensional image data of the calibration plate is obtained by the monocular camera, thereby obtaining the three-dimensional coordinate information and two-dimensional coordinate information of multiple points on the calibration plate;

[0014] Then, according to the three-dimensional coordinate information and two-dimensional coordinate information of multiple points on the calibration plate, the coordinate transformation relationship matrix between the lidar and the monocular camera is calculated using the EPnP algorithm.

[0015] In step 2, the two-dimensional projection step is specifically as follows:

[0016] First, the point cloud data acquired by the lidar in step 1 is converted to the imaging physical coordinate system of the monocular camera according to the coordinate transformation relationship matrix between the lidar and the monocular camera;

[0017] Then, the converted point cloud data is projected into the pixel coordinate system of the monocular camera according to the intrinsic parameter matrix of the monocular camera.

[0018] The expression of the intrinsic parameter matrix of the monocular camera is:

[0019]

[0020] Where: f x and f y represents the focal length of the monocular camera; x0 and y0 represent the principal point offset of the monocular camera.

[0021] In step 3, the target segmentation step is specifically as follows:

[0022] First, we create sample data of the robot arm's position information projection map, and use this data to train the MSNet network.

[0023] Then, the two-dimensional projection view of the point cloud is input into the trained MSNet network, and the feature extractor of the MSNet network performs multi-scale feature extraction on the two-dimensional projection view of the point cloud to obtain feature maps of different scales;

[0024] Then, the SAM module and CBAM module of the MSNet network are used to fuse the feature maps of different scales together, and the size of the fused feature map is changed to the same size as the input point cloud 2D projection view through upsampling;

[0025] Then, a 1×1×1 convolutional layer is used to perform pixel-level prediction on the feature map, thereby outputting a high-precision target segmentation map.

[0026] The feature extractor of the MSNet network includes the Conv2-2 convolution layer of the VGG16 network, the Conv3-3 convolution layer of the VGG16 network, the Conv4-3 convolution layer of the VGG16 network, and the Conv5-3 convolution layer of the VGG16 network; the SAM module includes the SAM1 module, the SAM2 module, and the SAM3 module, and the number of channels of the convolution kernel is 256, 128, and 64 respectively; the SAM1 module, the SAM2 module, and the SAM3 module are each composed of two serial The proposed method consists of a 4-branch structure, and the convolution kernel sizes of the 4-branch structure are 1×1, 3×3, 5×5, and 7×7. In order to reduce the computational complexity, a 1×1 convolution layer is added before the 3×3 convolution layer, the 5×5 convolution layer, and the 7×7 convolution layer to reduce the number of channels by half, and the number of output channels of each branch is set to be equal. The 1×1 convolution layer is used to retain the input feature scale, and the 3×3 convolution layer, the 5×5 convolution layer, and the 7×7 convolution layer are used to expand the receptive field and extract multi-scale features. The number of the CBAM modules is two.

[0027] In step 4, the clustering step is specifically as follows:

[0028] First, by setting the height range, the 3D point cloud data points belonging to the ground and ceiling in the location information are filtered out;

[0029] Then, the Euclidean clustering algorithm is used to remove noise points and distant points in the location information.

[0030] In step five, a control device is provided on the robotic arm, and the position and angle of the robotic arm can be adjusted by the control device, thereby adjusting the coordinates and angle of the center point of the minimum envelope rectangle; the target position of the robotic arm is a manually calibrated rectangle.

[0031] Compared with the existing robotic arm posture detection technology, the robotic arm posture detection method based on point cloud and image fusion described in the present invention realizes efficient and accurate robotic arm posture detection by fusing point cloud and image and applying deep learning, thereby not only effectively improving the detection speed, but also achieving high accuracy and strong robustness of detection results.

[0032] The present invention effectively solves the problems of slow detection speed, low detection accuracy and poor robustness of detection results in existing robotic arm posture detection technologies, and is suitable for robotic arm posture detection in various scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flow chart of the present invention.

[0034] Figure 2 It is a schematic diagram of the MSNet network in the present invention.

[0035] Figure 3 It is a schematic diagram of the SAM module in the present invention. DETAILED DESCRIPTION

[0036] A robotic arm posture detection method based on point cloud and image fusion is implemented using the following steps:

[0037] Step 1: Obtain point cloud data through lidar;

[0038] Step 2: Jointly calibrate the LiDAR and monocular camera, project the point cloud data into two dimensions to obtain a two-dimensional projection view of the point cloud, and then record the depth information of each pixel in the two-dimensional projection view of the point cloud;

[0039] Step 3: Use the MSNet network to segment the target in the point cloud 2D projection view, find the target area of ​​the robot arm in the point cloud 2D projection view, and then perform point cloud inverse calculation on the target area and the depth information of each pixel in the point cloud 2D projection view to find the position information of the robot arm in the point cloud 2D projection view;

[0040] Step 4: Cluster the position information of the robotic arm in the two-dimensional projection view of the point cloud;

[0041] Step 5: Draw a minimum envelope rectangle for the clustering results, and adjust the center point coordinates and angle of the minimum envelope rectangle so that the intersection ratio of the minimum envelope rectangle and the target position of the robotic arm reaches the maximum, and then use the minimum envelope rectangle as the posture detection result of the robotic arm.

[0042] In the step 1, after acquiring the point cloud data, the point cloud data is preprocessed using a graphics computing algorithm library; the preprocessing step specifically includes bilateral filtering and redundant point cloud removal.

[0043] In step 2, the joint calibration steps are as follows:

[0044] First, the three-dimensional point cloud data of the calibration plate is obtained by the laser radar, and the two-dimensional image data of the calibration plate is obtained by the monocular camera, thereby obtaining the three-dimensional coordinate information and two-dimensional coordinate information of multiple points on the calibration plate;

[0045] Then, according to the three-dimensional coordinate information and two-dimensional coordinate information of multiple points on the calibration plate, the coordinate transformation relationship matrix between the lidar and the monocular camera is calculated using the EPnP algorithm.

[0046] In step 2, the two-dimensional projection step is specifically as follows:

[0047] First, the point cloud data acquired by the lidar in step 1 is converted to the imaging physical coordinate system of the monocular camera according to the coordinate transformation relationship matrix between the lidar and the monocular camera;

[0048] Then, the converted point cloud data is projected into the pixel coordinate system of the monocular camera according to the intrinsic parameter matrix of the monocular camera.

[0049] The expression of the intrinsic parameter matrix of the monocular camera is:

[0050]

[0051] Where: f x and f y represents the focal length of the monocular camera; x0 and y0 represent the principal point offset of the monocular camera.

[0052] In step 3, the target segmentation step is specifically as follows:

[0053] First, we create sample data of the robot arm's position information projection map, and use this data to train the MSNet network.

[0054] Then, the two-dimensional projection view of the point cloud is input into the trained MSNet network, and the feature extractor of the MSNet network performs multi-scale feature extraction on the two-dimensional projection view of the point cloud to obtain feature maps of different scales;

[0055] Then, the SAM module and CBAM module of the MSNet network are used to fuse the feature maps of different scales together, and the size of the fused feature map is changed to the same size as the input point cloud 2D projection view through upsampling;

[0056] Then, a 1×1×1 convolutional layer is used to perform pixel-level prediction on the feature map, thereby outputting a high-precision target segmentation map.

[0057] The feature extractor of the MSNet network comprises a Conv2-2 convolutional layer of a VGG16 network, a Conv3-3 convolutional layer of the VGG16 network, a Conv4-3 convolutional layer of the VGG16 network, and a Conv5-3 convolutional layer of the VGG16 network; the SAM module comprises a SAM1 module, a SAM2 module, and a SAM3 module, and the channel numbers of the convolution kernels of the SAM1 module, the SAM2 module, and the SAM3 module are 256, 128, and 64 respectively; the SAM1 module, the SAM2 module, and the SAM3 module each comprise two 4-branch structures connected in series, and the convolution kernel sizes of the 4-branch structures are 1*1, 3*3, 5*5, and 7*7; in order to reduce the calculation complexity, a 1*1 convolutional layer is added before the 3*3 convolutional layer, the 5*5 convolutional layer, and the 7*7 convolutional layer, the channel number is reduced by half, and the output channel numbers of each branch are set to be equal; the 1*1 convolutional layer is used to retain the input feature scale, and the 3*3 convolutional layer, the 5*5 convolutional layer, and the 7*7 convolutional layer are used to expand the receptive field and extract multi-scale features; the number of the CBAM modules is two.

[0058] In the fourth step, the clustering step is specifically as follows:

[0059] Firstly, by setting a height range, three-dimensional point cloud data points belonging to the ground and the ceiling in the position information are filtered out.

[0060] Then, by using the Euclidean clustering algorithm, noise points and points far away in the position information are removed.

[0061] In the fifth step, a control device is arranged on the mechanical arm, the position and the angle of the mechanical arm can be adjusted through the control device, and thus the center point coordinates and the angle of the minimum envelope rectangle are adjusted; the target position of the mechanical arm is an artificially calibrated rectangle.

[0062] Although the specific embodiments of the present application are described above, those skilled in the art should understand that these are only illustrative, the protection scope of the present application is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present application, and these changes and modifications all fall within the protection scope of the present application.

Claims

1. A method for detecting the posture of a robotic arm based on point cloud and image fusion, characterized by: This method is implemented using the following steps: Step 1: Obtain point cloud data through lidar; Step 2: Jointly calibrate the LiDAR and monocular camera, project the point cloud data into two dimensions to obtain a two-dimensional projection view of the point cloud, and then record the depth information of each pixel in the two-dimensional projection view of the point cloud; Step 3: Use the MSNet network to segment the target in the point cloud 2D projection view, find the target area of ​​the robot arm in the point cloud 2D projection view, and then perform point cloud inverse calculation on the target area and the depth information of each pixel in the point cloud 2D projection view to find the position information of the robot arm in the point cloud 2D projection view; Step 4: Cluster the position information of the robotic arm in the two-dimensional projection view of the point cloud; Step 5: Draw a minimum envelope rectangle for the clustering results, and adjust the center point coordinates and angle of the minimum envelope rectangle so that the intersection-over-union ratio of the minimum envelope rectangle and the target position of the robotic arm is maximized. The minimum envelope rectangle is then used as the posture detection result of the robotic arm. In step 2, the joint calibration steps are as follows: First, the three-dimensional point cloud data of the calibration plate is obtained by the laser radar, and the two-dimensional image data of the calibration plate is obtained by the monocular camera, thereby obtaining the three-dimensional coordinate information and two-dimensional coordinate information of multiple points on the calibration plate; Then, based on the three-dimensional coordinate information and two-dimensional coordinate information of multiple points on the calibration plate, the EPnP algorithm is used to calculate the coordinate transformation relationship matrix between the lidar and the monocular camera; In step 2, the two-dimensional projection step is specifically as follows: First, the point cloud data acquired by the lidar in step 1 is converted to the imaging physical coordinate system of the monocular camera according to the coordinate transformation relationship matrix between the lidar and the monocular camera; Then, the transformed point cloud data is projected into the pixel coordinate system of the monocular camera according to the intrinsic parameter matrix of the monocular camera; In step 3, the target segmentation step is specifically as follows: First, we create sample data of the robot arm's position information projection map, and use this data to train the MSNet network. Then, the two-dimensional projection view of the point cloud is input into the trained MSNet network, and the feature extractor of the MSNet network performs multi-scale feature extraction on the two-dimensional projection view of the point cloud to obtain feature maps of different scales; Then, the SAM module and CBAM module of the MSNet network are used to fuse the feature maps of different scales together, and the size of the fused feature map is changed to the same size as the input point cloud 2D projection view through upsampling; Then, a 1×1×1 convolutional layer is used to perform pixel-level prediction on the feature map, thereby outputting a high-precision target segmentation map.

2. The method for detecting the posture of a robotic arm based on point cloud and image fusion according to claim 1, characterized in that: In the step 1, after acquiring the point cloud data, the point cloud data is preprocessed using a graphics computing algorithm library; the preprocessing step specifically includes bilateral filtering and redundant point cloud removal.

3. The method for detecting the posture of a robotic arm based on point cloud and image fusion according to claim 1, characterized in that: The expression of the intrinsic parameter matrix of the monocular camera is: ； Where: and Indicates the focal length of the monocular camera; and Indicates the principal point offset of the monocular camera.

4. The method for detecting the posture of a robotic arm based on point cloud and image fusion according to claim 1, wherein: The feature extractor of the MSNet network includes the Conv2-2 convolution layer of the VGG16 network, the Conv3-3 convolution layer of the VGG16 network, the Conv4-3 convolution layer of the VGG16 network, and the Conv5-3 convolution layer of the VGG16 network; the SAM module includes the SAM1 module, the SAM2 module, and the SAM3 module, and the number of channels of the convolution kernels are 256, 128, and 64 respectively; the SAM1 module, the SAM2 module, and the SAM3 module Each block consists of two 4-branch structures connected in series. The convolution kernel sizes of the 4-branch structures are 1×1, 3×3, 5×5, and 7×7. To reduce computational complexity, a 1×1 convolution layer is added before the 3×3 convolution layer, 5×5 convolution layer, and 7×7 convolution layer to reduce the number of channels by half, and the number of output channels of each branch is set to be equal. The 1×1 convolution layer is used to preserve the input feature scale, while the 3×3 convolution layer, 5×5 convolution layer, and 7×7 convolution layer are used to expand the receptive field and extract multi-scale features. The number of the CBAM modules is two.

5. The method for detecting the posture of a robotic arm based on point cloud and image fusion according to claim 1, wherein: In step 4, the clustering step is specifically as follows: First, by setting the height range, the 3D point cloud data points belonging to the ground and ceiling in the location information are filtered out; Then, the Euclidean clustering algorithm is used to remove noise points and distant points in the location information.

6. The method for detecting the posture of a robotic arm based on point cloud and image fusion according to claim 1, wherein: In step five, a control device is provided on the robotic arm, and the position and angle of the robotic arm are adjusted by the control device, thereby adjusting the coordinates and angle of the center point of the minimum envelope rectangle; the target position of the robotic arm is a manually calibrated rectangle.

Images