A target positioning detection method based on laser radar and camera information fusion

Abstract

The application relates to the field of image processing and discloses a target positioning detection method based on laser radar and camera information fusion, which comprises the following steps: S1, preprocessing 3D point cloud data of a detection scene to obtain a visual scene point cloud graph; S2, segmenting the visual scene point cloud graph by using a point cloud segmentation algorithm to obtain a 3D target object region point cloud graph; S3, selecting a template target object point cloud graph from the 3D target object region point cloud graph; S4, collecting 3D point cloud data and 2D images of the detection scene in real time, processing 3D point cloud data to be detected according to the operations of steps S1 and S2 to obtain a 3D target object region point cloud graph to be detected; using a neural network to detect a target object from the 2D images to generate a target object detection result graph; S5, fusing the target object detection result graph and the 3D target object region point cloud graph to be detected to obtain a target object region fusion point cloud graph to be detected; and S6, registering the target object region fusion point cloud graph to be detected and the template target object point cloud graph to obtain pose information of the target object in the target object region fusion point cloud graph to be detected.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically to a target localization and detection method based on the fusion of LiDAR and camera information. Background Technology

[0002] In unmanned slag pot hoisting systems, target detection technology is employed to accurately perceive the actual position of the slag pot, thereby achieving automated hoisting and reducing manual assistance, especially avoiding human intervention in high-risk work environments. In steel companies, steel production is a key technical indicator, encompassing a series of processes including slag formation, slag removal, refining, and steel tapping, involving a large number of steel slag transfer operations. Steel slag transfer is a crucial link in steel production, and currently, the vast majority of steel slag transportation relies on manual operation and monitoring. This is problematic because the high-temperature environment and slag splashes pose significant safety risks; furthermore, manual operation, relying primarily on visual observation, is prone to misjudgments, leading to unnecessary crane starts and stops, and some operations require multiple attempts, resulting in extremely low efficiency. To improve steel company production efficiency and enhance worker safety, it is best to reduce or eliminate the involvement of ground support and monitoring personnel. Therefore, in intelligent steel slag handling processes, the ability to automatically confirm the status of the hooks and identify the slag pot position, achieving intelligent steel slag transfer, is a future development trend with strong practical significance and economic value. Therefore, there is an urgent need for a method that can automatically and accurately identify the position of the slag container and hook, shorten the data acquisition cycle, and efficiently and safely complete the detection and positioning of the hook, thereby realizing unmanned operation of steel slag transfer. Summary of the Invention

[0003] In view of the problems and shortcomings of the existing technology, the purpose of this invention is to provide a target localization and detection method based on the fusion of lidar and camera information.

[0004] To achieve the objectives of this invention, the technical solution adopted is as follows:

[0005] A target localization and detection method based on the fusion of lidar and camera information, characterized by the following steps:

[0006] S1. Preprocess the acquired 3D point cloud data of the detection scene to eliminate point cloud noise, filter out non-target objects in the scene, and obtain a visualized scene point cloud map.

[0007] S2. The point cloud image of the visualized scene obtained in step S1 is segmented using the Euclidean clustering point cloud segmentation algorithm based on the hairline angle to separate the target object from the non-target scene objects and obtain a 3D target object region point cloud image.

[0008] S3. Select a point cloud map containing complete geometric and semantic information features of the target object from the 3D target object region point cloud map obtained in step S2, and use it as a template target object point cloud map.

[0009] S4. Real-time acquisition of 3D point cloud data and 2D images of the detection scene to obtain 3D point cloud data and 2D images to be detected. Process the 3D point cloud data to be detected according to the operations of steps S1 and S2 to obtain a point cloud map of the 3D target object region to be detected. Use a deep convolutional neural network to detect the target object in the 2D image to be detected and generate a target object detection result map. The target object detection result map includes the bounding box of the target object, the target object category, and the confidence level.

[0010] S5. The target object detection result map is fused with the point cloud map of the 3D target object region to be detected, and the point cloud map of the 3D target object region to be detected is colored to obtain a fused point cloud map of the target object region to be detected with accurate coordinate information.

[0011] S6. A global registration algorithm based on local features is used to register the fused point cloud map of the target object region to be inspected with the point cloud map of the template target object. Based on the registration information, the pose information of the target object in the fused point cloud map of the target object region to be inspected is obtained.

[0012] According to the target localization and detection method based on the fusion of lidar and camera information described above, preferably, the step of using the Euclidean clustering point cloud segmentation algorithm based on the hairline angle to perform point cloud segmentation processing on the visualized scene point cloud map obtained in step S1 is as follows:

[0013] S21. Calculate the average distance and root mean square error between point clouds within a radius of r in the visualized scene point cloud map;

[0014] S22. Calculate the mean angle between normals between neighboring points within different radius ranges r of the visualized scene point cloud map;

[0015] S23. Utilize the differences in the angle of the point cloud normals in abruptly overlapping and contact areas, determine whether the point cloud is a boundary point of the target object's point cloud based on the average value of the neighborhood normals within different radius ranges calculated. Then, based on the boundary points of the target object's point cloud, segment the target object and non-target objects in the visualized scene point cloud map.

[0016] According to the target localization and detection method based on the fusion of lidar and camera information described above, preferably, step S21 is specifically performed as follows:

[0017] S211. For any point vq in the visualized scene point cloud map VQ... i vq iFor any ∈VQ, iterate through its k nearest neighbors to obtain a set of k points, denoted as {vq}. i1 vq i2 , ..., vq ik}, calculate {vq} according to Equation 1 i1 vq i2 , ..., vq ik Each point in the set and q i average distance d i ;

[0018]

[0019] In Equation 1, d i Represents {vq i1 ,vq i2 ,…,vq ik Each point in the set is related to vq i average distance,

[0020] S212. Calculate the average distance d of all points in the visualized scene point cloud map VQ according to the operation described in step S21. i Then, Equations 2 and 3 are used to calculate the mean distance d and the root mean square error σ of the point cloud in neighborhoods with different radii r.

[0021]

[0022]

[0023] Where N represents the point cloud size of the visualized scene point cloud map VQ.

[0024] According to the target localization and detection method based on the fusion of lidar and camera information described above, preferably, step S22 is specifically performed as follows:

[0025] S221. Calculate the mean angle between the normals of neighboring points within different radius ranges r:

[0026] For any point vq in the visualized scene point cloud map VQ i vq i ∈VQ, with vq i Starting from the search point, all neighboring points within a radius of r are searched, resulting in a set of k points, denoted as {vq}. i1 ,vq i2 ,…,vq ik Then, the least squares method is used to fit the search starting point vq according to Equations 4 and 5. i The least squares plane,

[0027]

[0028]

[0029] Where q(n,d) represents the average distance in the neighborhood with a search radius of r, n represents the unit normal vector of the least squares plane, and q represents the centroid value. xi vq represents the i-th point i The centroid value in the x-direction, q yi vq represents the i-th point i The centroid value in the y-direction, q zi vq represents the i-th point i The centroid value in the z-direction;

[0030] S222, Point vq i The normal value is denoted as v i Search point vq i The normal vector values ​​of the k nearest neighbors are denoted as {v i1 ,v i2 ,…,v ik The points vq are calculated using equations 6 and 7 respectively. i The angle α between the normals of adjacent points within the radius r. i and the mean of the included angle of the normals

[0031]

[0032]

[0033] Where, α i Indicates the angle between the normals. Let represent the mean of the included angles of the normals, g represent the number of neighboring points within the radius r, and N represent the number of included angles of the normals of neighboring points within the radius r.

[0034] According to the target localization and detection method based on the fusion of lidar and camera information described above, preferably, the specific steps in step S6 of registering the fused point cloud map of the target object region with the template target object point cloud map using a global registration algorithm based on local features are as follows:

[0035] S61. The SIFT algorithm is used to extract key points from the point cloud map of the template target object and the fused point cloud map of the target object region to be inspected, respectively. The FPFH is used to describe the extracted key points to obtain key point features.

[0036] S62. Use a global registration algorithm to pair the key points in the fused point cloud map of the target object region to be inspected with the key points in the template target object point cloud map to obtain key point pairs.

[0037] S63. Interchangeability test, three-point test and spatial position test are used in sequence to eliminate incorrect matches in key point pairs, and an error function is set to verify the key point pairs to obtain matching key point pairs; based on the matching key point pair information, the pose information of the target object in the fused point cloud map of the target object region is obtained.

[0038] According to the target localization and detection method based on the fusion of lidar and camera information described above, preferably, in step S63, interchangeability consistency test, three-point test, and spatial position test are used sequentially to eliminate erroneous matches in key point pairs, and an error function is set to verify the key point pairs to obtain matching key point pairs. The specific operation is as follows:

[0039] S631. For any key point p in the fused point cloud map P of the target object region to be inspected. i The keypoint features extracted using FPFH are represented as F(p i If the keypoint FPFH feature set of the fused point cloud map P of the target object region to be inspected is defined as: F(P) = {F(p)} i ):p i ∈P}; for any key point q in the point cloud map Q of the template target object. i The keypoint features extracted using FPFH are represented as F(q) i If the keypoint FPFH feature set of the target object point cloud Q can be defined as: F(Q) = {F(q)} i ):q i Let C1 be the set of correspondences between F(P) and F(Q) ∈Q};

[0040] S632, Interchangeability Consistency Test: For critical point pairs (p i ,q i ), if and only if F(p) i ) is F(q) i The nearest neighbor of q in F(P), and only if F(q) i ) is F(p i When the keypoint pair (p) is the nearest neighbor in F(Q), the keypoint pair (p) is the nearest neighbor in F(Q). i ,q i ) is retained in the result point pair C 11 middle;

[0041] S633, Three-point test: Compare C with the result point 11 Three keypoint pairs (p1, q1), (p2, q2), and (p3, q3) are randomly selected. The results are then examined to see if (p1, p2, p3) and (q1, q2, q3) conform to the verification relationship shown in Equation 8. Keypoint pairs that satisfy the verification relationship are stored in the result point pair C. 111 middle;

[0042]

[0043] Where τ = 0.9;

[0044] S634. Spatial Position Test: Calculate the point pair C according to Equation 9. 111 All key points in the middle to all key points q i The centroid of the point cloud, Centriod, is calculated according to Equation 10. 111 All keypoints in the middle, each keypoint q i The positional relationship between C(q) and the centroid of the point cloud Centriod i ), and calculate all key points q i C(q) i The average value of ) is used to calculate the value of each key point q. i C(q) i ) and C(q) i Compare the average values ​​and remove C(q). i ) greater than C(q) i Key points of the average;

[0045]

[0046]

[0047] S635. Error Function Verification: Calculate the result point pair C after processing in step S54 using the error function. 111 The error between each keypoint pair and the error between the two keypoints is calculated to obtain the error of each keypoint pair; then the result point pair C is obtained. 111 The mean error μ and variance error δ of all keypoint pairs 2 If μ < δ 2 Then the verification is complete, and the result is for point C. 111 The remaining keypoint pairs are the matching keypoint pairs; the calculation formula of the error function E(T,L) is shown in Equation 11.

[0048]

[0049] Where λ represents a random number, and its range is: λ∈[0,1], T i Let T represent the rigid body transformation matrix of the i-th keypoint pair. i+1 Let p represent the rigid body transformation matrix of the (i+1)th keypoint pair, p represent the keypoint set in the fused point cloud map of the target object region to be inspected, q represent the keypoint set in the template target object point cloud map, and ψ(l p,q () indicates a key-point pair relationship.

[0050] According to the target localization and detection method based on the fusion of LiDAR and camera information described above, preferably, the preprocessing in step S1 is at least one of the following: direct-pass filtering algorithm processing, distance-based radius filtering algorithm processing, and voxelized mesh filtering algorithm processing.

[0051] According to the target localization and detection method based on the fusion of lidar and camera information described above, preferably, the target is a slag pot and a hook for suspending the slag pot.

[0052] According to the target localization and detection method based on the fusion of LiDAR and camera information described above, preferably, the specific operation of fusing the target object detection result map with the point cloud map of the 3D target object region in step S5 is as follows:

[0053] S51. Calibrate the intrinsic parameter matrix of the camera device for acquiring 2D images to be detected;

[0054] S52. Based on the bounding box information in the target object detection result map in step S4, select the contour edge corner points of the target object from the target object detection result map and record the position coordinates of the contour edge corner points; obtain the edge corner points of the target object from the point cloud map of the 3D target object region to be detected and record their position coordinates.

[0055] S53. Select the contour edge corner points of the target object from the target object detection result map and obtain the edge corner points of the target object from the point cloud map of the 3D target object region to be detected, and perform corresponding point iterative calculation to obtain the external parameter matrix containing residuals and reprojection errors.

[0056] S54. Calculate the camera pixel coordinates of the camera device based on the position coordinates of the point cloud in the point cloud map of the 3D target object region to be detected, the intrinsic parameter matrix obtained in step S51, and the extrinsic parameter matrix obtained in step S53. Then, assign the RGB information of the target object detection result map to the point cloud in the point cloud map of the 3D target object region to be detected, complete the point cloud coloring, and obtain the fused point cloud map of the target object region to be detected with accurate coordinate information.

[0057] According to the above-mentioned target positioning and detection method based on the fusion of lidar and camera information, preferably, a laser scanning system is used to acquire 3D point cloud data of the detection scene. The laser scanning system includes a 2D linear laser scanner and an intelligent gimbal device, wherein the 2D linear laser scanner is mounted on the intelligent gimbal.

[0058] According to the target positioning and detection method based on the fusion of lidar and camera information described above, preferably, a camera device is used to acquire 2D images of the detection scene, wherein the camera device is an infrared digital high-definition high-speed intelligent PTZ camera.

[0059] Compared with the prior art, the positive and beneficial effects achieved by the present invention are as follows:

[0060] (1) The present invention uses a point cloud segmentation algorithm based on hairline angle to segment point clouds. This point cloud segmentation algorithm can remove outliers and redundant points, and can also meet the segmentation requirements of the object to be measured. Moreover, it can achieve self-adjustment of point cloud search distance during the segmentation process, effectively separating target objects from non-target scenes. The point cloud segmentation accuracy is high and the segmentation speed is fast, providing a good foundation for subsequent point cloud registration. At the same time, it overcomes the defects of traditional Euclidean clustering segmentation algorithm, which is prone to undersegmentation and oversegmentation.

[0061] (2) The present invention uses a global registration algorithm based on local features to pair the key points in the fused point cloud map of the target object region to be inspected with the key points in the point cloud map of the template target object. This algorithm not only has low computational cost, but can also accurately identify and calculate the pose matrix of the target object in complex on-site working scenarios. Compared with the coarse registration SAC algorithm, the global registration algorithm based on local features of the present invention does not require initialization and has a lower computational cost, solving the problems of the SAC algorithm having many iterations and slow speed. Compared with the fine registration ICP algorithm, the global registration algorithm based on local features of the present invention has higher accuracy and reliability.

[0062] (3) The present invention first uses a deep convolutional neural network to detect the target object in the 2D image to be detected, generates the target object detection result map, and then fuses the target object detection result map with the point cloud map of the 3D target object region to be detected, so that the target localization is more accurate, greatly improves the accuracy of the target detection result, and has stronger robustness.

[0063] (4) When screening and matching key point pairs, the present invention introduces an error function for verification. The setting of the error function can ensure that the fused point cloud map of the target object region to be inspected and the point cloud map of the template target object are automatically verified, which greatly improves the speed of the registration algorithm.

[0064] (5) Before performing key point matching, the present invention first constructs a template target object point cloud map, which can establish an accurate model library for subsequent target object recognition and positioning.

[0065] (6) This invention deeply integrates 3D point cloud and 2D image data, which can achieve real-time accurate detection, improve positioning accuracy, classify accurately, and improve work efficiency; at the same time, it can accurately detect the position and relative posture of slag hopper and hook during steel slag transfer, realize intelligent perception and unmanned target of steel slag transfer, reduce the probability of safety accidents, further reduce the cost of steel slag loading, unloading and transfer, accurately detect the intelligent decision of steel slag and hook target grabbing and unhooking, and build an automated system that is both effective and practical. Attached Figure Description

[0066] Figure 1 This is a step diagram of the Euclidean clustering point cloud segmentation algorithm based on hairline angle of the present invention;

[0067] Figure 2 The image shows the effect of point cloud segmentation using the Euclidean clustering point cloud segmentation algorithm based on hairline angle according to the present invention.

[0068] Figure 3 This is a schematic diagram illustrating the use of the deep convolutional neural network YOLOv3 to detect target objects in a 2D image.

[0069] Figure 4 This is a registration effect diagram of the global registration algorithm based on local features of this invention. Detailed Implementation

[0070] The present invention will be further described in detail below with reference to specific embodiments, but this does not limit the scope of the present invention.

[0071] Example 1:

[0072] A target localization and detection method based on the fusion of lidar and camera information includes the following steps:

[0073] S1. A laser scanning system installed on the vehicle is used to collect 3D point cloud data of the detection scene. The laser scanning system includes an intelligent gimbal device and a 2D linear laser scanner installed on the intelligent gimbal device. The acquired 3D point cloud data of the detection scene is preprocessed to eliminate point cloud noise and filter out non-target objects in the scene to obtain a visualized scene point cloud map.

[0074] The preprocessing involves at least one of the following algorithms: direct-pass filtering, distance-based radius filtering, and voxelized mesh filtering. One algorithm can be selected based on the specific circumstances, or a combination of algorithms can be used. The preferred preprocessing operation of this invention is as follows:

[0075] (1) Use a pass-through filtering algorithm to remove redundant points: Since the data is processed using a Cartesian coordinate system, it is necessary to convert the spherical coordinate system to a Cartesian coordinate system. Because the laser scanner is not installed perpendicular to the z-axis, the influence of the laser installation tilt angle needs to be considered. The specific conversion formula is as follows:

[0076]

[0077]

[0078] In the laser coordinate system, the xoy plane is defined as the plane where the gimbal is located, and the z-axis is the direction perpendicular to the rotating gimbal. Therefore, r... θ is defined as the distance between the object and the origin, the angle between the object and the z-axis, and the angle between the object's xoy projection and the positive x-axis. Finally, a pass-through filtering algorithm is applied in the x-axis and z-axis coordinate directions to filter out the left fence point cloud and the ground noise point cloud in the target scene.

[0079] (2) A radius filtering algorithm based on distance is used to filter and remove isolated noise point clouds and suspended point clouds;

[0080] (3) Although the above two steps filter out various interference noises, point cloud segmentation is time-consuming. Therefore, we consider downsampling the dense point cloud to obtain a sparse point cloud to improve the point cloud processing speed. Since it is necessary to preserve the characteristics of the original point cloud, we apply the voxelized mesh filtering method, calculate the centroid of the point cloud using the point cloud normal vector and distance value, and set the voxel side length to 1.0 cm to approximate the point cloud data of the neighborhood to achieve the purpose of data simplification.

[0081] S2. A Euclidean clustering point cloud segmentation algorithm based on hairline angle is adopted (algorithm steps are shown in the figure). Figure 1 As shown, the point cloud image of the visualized scene obtained in step S1 is processed by point cloud segmentation to separate the target object from non-target objects, resulting in a 3D point cloud image of the target object region. The steps for processing the point cloud image of the visualized scene obtained in step S1 using the Euclidean clustering point cloud segmentation algorithm based on hairline angle are as follows:

[0082] S21. Calculate the average distance and standard deviation between point clouds within a radius of r in the visualized scene point cloud map. The specific steps are as follows:

[0083] S211. For any point vq in the visualized scene point cloud map VQ... i vq i For any ∈VQ, iterate through its k nearest neighbors to obtain a set of k points, denoted as {vq}. i1 ,vq i2 ,…,vq ik}, calculate {vq} according to Equation 1 i1 ,vq i2 ,…,vq ik Each point in the set and q i average distance d i ;

[0084]

[0085] In Equation 1, d i Represents {vq i1 ,vq i2 ,…,vq ik Each point in the set is related to vq i average distance,

[0086] S212. Calculate the average distance d of all points in the visualized scene point cloud map VQ according to the operation described in step S21. i Then, Equations 2 and 3 are used to calculate the mean distance d and the root mean square error σ of the point cloud in neighborhoods with different radii r.

[0087]

[0088]

[0089] Where N represents the point cloud size of the visualized scene point cloud map VQ.

[0090] Preferably, the mean distance, standard deviation, and running time of the point cloud under different r-neighborhoods are shown in Table 1.

[0091] Table 1. Mean distance, standard deviation, and running time of point clouds under different r-neighborhoods.

[0092]

[0093] S22. Calculate the mean normal angle between neighboring points within different radius ranges r of the visualized scene point cloud map. The calculation of the normal angle in the point cloud uses principal component analysis. The specific operation is as follows:

[0094] S221. Calculate the mean angle between the normals of neighboring points within different radius ranges r:

[0095] For any point vq in the visualized scene point cloud map VQ i vq i ∈VQ, with vq i Starting from the search point, all neighboring points within a radius of r are searched, resulting in a set of k points, denoted as {vq}. i1 ,vq i2 ,…,vq ik Then, the least squares method is used to fit the search starting point vq according to Equations 4 and 5. i The least squares plane,

[0096]

[0097]

[0098] Where q(n,d) represents the average distance in the neighborhood with a search radius of r, n represents the unit normal vector of the least squares plane, and q represents the centroid value. xi vq represents the i-th point i The centroid value in the x-direction, q yi vq represents the i-th point iThe centroid value in the y-direction, q zi vq represents the i-th point i The centroid value in the z-direction;

[0099] S222, Point vq i The normal value is denoted as v i Search point vq i The normal vector values ​​of the k nearest neighbors are denoted as {v i1 ,v i2 ,…,v ik The points vq are calculated using equations 6 and 7 respectively. i The angle α between the normals of adjacent points within the radius r. i and the mean of the included angle of the normals

[0100]

[0101]

[0102] Where, α i Indicates the angle between the normals. Let represent the mean of the included angles of the normals, g represent the number of neighboring points within the radius r, and N represent the number of included angles of the normals of neighboring points within the radius r.

[0103] The mean values ​​of the normal angles for different numbers of nearest neighbor points r are shown in Table 2.

[0104] Table 2. Mean normal angles for different numbers of nearest neighbor points r.

[0105]

[0106] S23. Utilize the differences in the angle of the point cloud normals in abruptly overlapping and contact areas to determine whether the point cloud is a boundary point of the target object's point cloud based on the calculated mean of the neighborhood normal angles within different radius ranges (compare the calculated mean of the neighborhood normal angles with the Averagenormalangle of the corresponding Knumber in Table 2; if the calculated mean of the neighborhood normal angles falls within the Averagenormalangle range, it is determined to be a boundary point). Then, based on the boundary points of the target object's point cloud, segment the target object and non-target objects in the visualized scene point cloud map.

[0107] Furthermore, to verify the effectiveness of the proposed Euclidean clustering point cloud segmentation algorithm based on hairline angle, the same bolt data image was processed using both the traditional Euclidean clustering algorithm and the proposed Euclidean clustering point cloud segmentation algorithm based on hairline angle. The segmentation effects of the two algorithms on the target point cloud were compared, and the results are as follows: Figure 2 As shown.

[0108] Depend on Figure 2 It can be seen that the Euclidean clustering point cloud segmentation algorithm based on the included normal angle of this invention can identify three more bolt objects than the traditional Euclidean clustering method, achieving an accuracy of 81.8%, which is 12.7% higher than the accuracy of the traditional Euclidean clustering method. Therefore, this invention first uses a filtering algorithm for preprocessing, and then uses the Euclidean clustering algorithm based on the included normal angle to segment the point cloud, which can both remove outliers and redundant points and meet the segmentation requirements of the objects under test; on the other hand, the algorithm of this invention has significantly improved speed and robustness compared with the original Euclidean clustering method.

[0109] S3. Select a point cloud map containing complete geometric and semantic information features of the target object from the 3D target object region point cloud map obtained in step S2, and use it as a template target object point cloud map.

[0110] S4. A laser scanning system is used to acquire 3D point cloud data of the detection scene in real time. A camera device (infrared digital high-definition high-speed intelligent PTZ camera) is used to acquire left and right view 2D image pairs of the detection scene, obtaining the 3D point cloud data and the 2D image to be detected. The 3D point cloud data to be detected is processed according to steps S1 and S2 to obtain a point cloud map of the 3D target object region to be detected. A deep convolutional neural network (YOLOv3) is used to detect the target object in the 2D image (a schematic diagram of using YOLOv3 to detect the target object in the 2D image is shown below). Figure 3 As shown in the figure, a target object detection result map is generated, which includes the bounding box of the target object, the target object category and the confidence level.

[0111] S5. The target object detection result map is fused with the point cloud map of the 3D target object region to be detected, and the point cloud map of the 3D target object region to be detected is colored to obtain a fused point cloud map of the target object region to be detected with accurate coordinate information.

[0112] The specific operation for fusing the target object detection result map with the point cloud map of the 3D target object region to be detected is as follows:

[0113] S51. Calibrate the intrinsic matrix of the camera device that acquires the 2D image to be detected, and convert the acquired radar point cloud containing the target object in text format into PCD format.

[0114] S52. Based on the bounding box information in the target object detection result image in step S4, select the contour edge corner points of the target object from the target object detection result image, record the position coordinates of the contour edge corner points, and store them in data / corner_photo.txt; obtain the edge corner points of the target object from the point cloud image of the 3D target object region to be detected, record their position coordinates, and store them in data / corner_lidar.txt.

[0115] S53. Read the data in data / corner_photo.txt and data / corner_lidar.txt, select the contour edge corner points of the target object from the target object detection result map and obtain the edge corner points of the target object from the point cloud map of the 3D target object region to be detected, and perform corresponding point iterative calculation to obtain the extrinsic parameter matrix containing residuals and reprojection errors.

[0116] S54. Calculate the camera pixel coordinates of the camera device based on the position coordinates of the point cloud in the point cloud map of the 3D target object region to be detected, the intrinsic parameter matrix obtained in step S51, and the extrinsic parameter matrix obtained in step S53. Then, assign the RGB information of the target object detection result map to the point cloud in the point cloud map of the 3D target object region to be detected to complete the point cloud coloring and obtain a fused point cloud map of the target object region to be detected with accurate coordinate information.

[0117] S6. A global registration algorithm based on local features is used to register the fused point cloud image of the target object region to be detected with the point cloud image of the template target object. Based on the registration information, the pose information of the target object in the fused point cloud image of the target object region to be detected is obtained. The specific steps are as follows:

[0118] S61. The SIFT (Scale Invariant Feature Transform) algorithm is used to extract key points from the point cloud map of the template target object and the fused point cloud map of the target object region to be inspected, respectively. The FPFH (Fast Point Feature Histogram) algorithm is used to describe the features of the extracted key points to obtain key point features.

[0119] S62. Use a global registration algorithm to pair the key points in the fused point cloud map of the target object region to be inspected with the key points in the template target object point cloud map to obtain key point pairs.

[0120] S63. Interchangeability test, three-point test and spatial position test are used in sequence to eliminate incorrect matches in key point pairs, and an error function is set to verify the key point pairs to obtain matching key point pairs; based on the matching key point pair information, the pose information of the target object in the fused point cloud map of the target object region is obtained.

[0121] In step S63, interchangeability consistency testing, three-point testing, and spatial location testing are used to eliminate incorrect matches in keypoint pairs. An error function is then set to verify the keypoint pairs to obtain matching keypoint pairs. The specific operation is as follows:

[0122] S631. For any key point p in the fused point cloud map P of the target object region to be inspected. i The keypoint features extracted using FPFH are represented as F(p i If the keypoint FPFH feature set of the fused point cloud map P of the target object region to be inspected is defined as: F(P) = {F(p)} i ):p i ∈P}; for any key point q in the point cloud map Q of the template target object. i The keypoint features extracted using FPFH are represented as F(q) i If the keypoint FPFH feature set of the target object point cloud Q can be defined as: F(Q) = {F(q)} i ):q i Let C1 be the set of correspondences between F(P) and F(Q) ∈Q};

[0123] S632, Interchangeability Consistency Test: For critical point pairs (p i ,q i ), if and only if F(p) i ) is F(q) i The nearest neighbor of q in F(P), and only if F(q) i ) is F(p i When the keypoint pair (p) is the nearest neighbor in F(Q), the keypoint pair (p) is the nearest neighbor in F(Q). i ,q i ) is retained in the result point pair C 11 middle;

[0124] S633, Three-point test: Compare C with the result point 11 Three keypoint pairs (p1, q1), (p2, q2), and (p3, q3) are randomly selected. The results are then examined to see if (p1, p2, p3) and (q1, q2, q3) conform to the verification relationship shown in Equation 8. Keypoint pairs that satisfy the verification relationship are stored in the result point pair C. 111 middle;

[0125]

[0126] Where τ = 0.9;

[0127] S634. Spatial Position Test: Calculate the point pair C according to Equation 9. 111 All key points in the middle to all key points q i The centroid of the point cloud, Centriod, is calculated according to Equation 10.111 All keypoints in the middle, each keypoint q i The positional relationship between C(q) and the centroid of the point cloud Centriod i ), and calculate all key points q i C(q) i The average value of ) is used to calculate the value of each key point q. i C(q) i ) and C(q) i Compare the average values ​​and remove C(q). i (greater than C)q i Key points of the average;

[0128]

[0129]

[0130] S635. Error Function Verification: Calculate the result point pair C after processing in step S54 using the error function. 111 The error between each keypoint pair and the error between the two keypoints is calculated to obtain the error of each keypoint pair; then the result point pair C is obtained. 111 The mean error μ and variance error δ of all keypoint pairs 2 If μ < δ 2 Then the verification is complete, and the result is for point C. 111 The remaining keypoint pairs are the matching keypoint pairs; the calculation formula of the error function E(T,L) is shown in Equation 11.

[0131]

[0132] Where λ represents a random number, and its range is: λ∈[0,1], T i Let T represent the rigid body transformation matrix of the i-th keypoint pair. i+1 Let p represent the rigid body transformation matrix of the (i+1)th keypoint pair, p represent the keypoint set in the fused point cloud map of the target object region to be inspected, q represent the keypoint set in the template target object point cloud map, and ψ(l p,q () indicates a key-point pair relationship.

[0133] The derivation of the error function E(T,L) is as follows:

[0134] Key points in the fused point cloud map P of the target object region to be inspected are selected in sequence, and the index of the corresponding point is found in the template target object point cloud map Q using the nearest neighbor search algorithm (FLANN). This index is used when estimating the normal of the key points in the future. In order to find the template target object point cloud map Q and the rigid body transformation matrix T that is aligned with the fused point cloud map P of the target object region to be inspected, the target formula E(T) is set as shown in 12.

[0135] E(T)=∑ (p,q)∈K ρ(||p-Tq||) Equation 12

[0136] Here, ρ(·) is an estimator. It is crucial to use a robust error function because many terms in Equation 12 are contributed by spurious constraints. This is achieved by setting an estimator ρ (calculated as shown in Equation 13) that can perform verification automatically, without incurring additional computational costs.

[0137]

[0138] We assume L = {l p,q The corresponding process is linear, and the following joint objective is optimized on T and L:

[0139] E(T,L)=∑ (p,q)∈K l p,q ||p-Tq|| 2 +∑ (p,q)∈K ψ(l p,q Formula 14

[0140] The point-to-point relationship is:

[0141]

[0142] To minimize the partial derivative of E(T,L) with respect to the formula, we obtain:

[0143]

[0144] The calculated result is:

[0145]

[0146] Calculate l in Equation 14 using the result of Equation 17. p,q Then, the value of E(T) can be calculated. However, when L is fixed, this solution cannot be extended to the registration of multiple surfaces. Therefore, we propose a more flexible method, defining the rotation variable w and transformation vector t in the transformation matrix as a vector with six parameters: ξ=(ω,t)=(α,β,γ,a,b,c). The transformation matrix can then be linearly represented as:

[0147]

[0148] In formula 18 This is the pose estimation result from the previous iteration. Equation 7 is the optimization objective of the sum of squared errors of ξ, which is transformed into a linear relationship using the Newton-Raphson iteration method. Let's solve for ξ, where the residual is r, and J r It is its Jacobian determinant. By mapping ξ to The transformation matrix T is then calculated.

[0149] To verify the registration effect of the global registration algorithm based on local features of this invention, the detection target objects (slag pot and hook) were registered using the global registration algorithm based on local features of this invention and the existing SAC+ICP algorithm respectively (see steps S1-S5 for specific operations before registration). The comparison results of the registration effects are shown in Table 3 and... Figure 4 As shown.

[0150] Table 3 Target object pose deviations under different registration algorithms

[0151]

[0152] From Table 3 and Figure 4 It can be seen that the global registration algorithm based on local features of the present invention can accurately identify and calculate the pose of target objects such as slag pots and hooks in complex field working scenarios, and has higher accuracy and reliability compared with the SAC+ICP algorithm.

[0153] The above description is only a preferred embodiment of the present invention, but is not limited to the above examples. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A target localization and detection method based on the fusion of lidar and camera information, characterized in that, Includes the following steps: S1. Preprocess the acquired 3D point cloud data of the detection scene to eliminate point cloud noise, filter out non-target objects in the scene, and obtain a visualized scene point cloud map. S2. The Euclidean clustering point cloud segmentation algorithm based on the angle of normal is used to perform point cloud segmentation on the visualized scene point cloud map obtained in step S1, and the target object is separated from the non-target scene to obtain a 3D target object region point cloud map. S3. Select a point cloud map containing complete geometric and semantic information features of the target object from the 3D target object region point cloud map obtained in step S2, and use it as a template target object point cloud map. S4. Real-time acquisition of 3D point cloud data and 2D images of the detection scene to obtain 3D point cloud data and 2D images to be detected. Process the 3D point cloud data to be detected according to the operations of steps S1 and S2 to obtain a point cloud map of the 3D target object region to be detected. Use a deep convolutional neural network to detect the target object in the 2D image to be detected and generate a target object detection result map. The target object detection result map includes the bounding box of the target object, the target object category, and the confidence level. S5. The target object detection result map is fused with the point cloud map of the 3D target object region to be detected, and the point cloud map of the 3D target object region to be detected is colored to obtain a fused point cloud map of the target object region to be detected with accurate coordinate information. S6. A global registration algorithm based on local features is used to register the fused point cloud map of the target object region to be inspected with the point cloud map of the template target object. Based on the registration information, the pose information of the target object in the fused point cloud map of the target object region to be inspected is obtained.

2. The target localization and detection method based on lidar and camera information fusion according to claim 1, characterized in that, The steps for segmenting the visualized scene point cloud image obtained in step S1 using the Euclidean clustering point cloud segmentation algorithm based on the included normal angle are as follows: S21. Calculate the average distance and root mean square error between point clouds within a radius of r in the visualized scene point cloud map; S22. Calculate the mean angle between normals between neighboring points within different radius ranges r of the visualized scene point cloud map; S23. Determine whether the point cloud is the boundary point of the target object point cloud based on the mean of the neighborhood normal angles under different radius ranges calculated. Then, based on the boundary points of the target object point cloud, segment the target object and non-target objects in the visualized scene point cloud map.

3. The target localization and detection method based on lidar and camera information fusion according to claim 2, characterized in that, The specific operation of step S21 is as follows: S211. For any point vq in the visualized scene point cloud map VQ... i vq i For any ∈VQ, iterate through its k nearest neighbors to obtain a set of k points, denoted as {vq}. i1 ,vq i2 ,…,vq ik }, calculate {vq} according to Equation 1 i1 ,vq i2 ,…,vq ik Each point in the set is related to q i average distance d i ; In Equation 1, d i Represents {vq i1 ,vq i2 ,…,vq ik Each point in the set is related to vq i average distance, S212. Calculate the average distance d of all points in the visualized scene point cloud map VQ according to the operation described in step S21. i Then, Equations 2 and 3 are used to calculate the mean distance d and the root mean square error σ of the point cloud in neighborhoods with different radii r. Where N represents the point cloud size of the visualized scene point cloud map VQ.

4. The target localization and detection method based on lidar and camera information fusion according to claim 3, characterized in that, The specific operation of step S22 is as follows: S221. For any point vq in the visualized scene point cloud map VQ... i vq i ∈VQ, with vq i Starting from the search point, all neighboring points within a radius of r are searched, resulting in a set of k points, denoted as {vq}. i1 ,vq i2 ,…,vq ik }; S222, Point vq i The normal value is denoted as v i Search point vq i The normal vector values ​​of the k nearest neighbors are denoted as {v i1 ,v i2 ,…,v ik The points vq are calculated using equations 6 and 7 respectively. i The angle α between the normals of adjacent points within the radius r. i and the mean of the included angle of the normals Where, α i Indicates the angle between the normals. Let represent the mean of the included angles of the normals, g represent the number of neighboring points within the radius r, and N represent the number of included angles of the normals of neighboring points within the radius r.

5. The target localization and detection method based on the fusion of lidar and camera information according to claim 4, characterized in that, The specific steps in step S6, which use a global registration algorithm based on local features to register the fused point cloud image of the target object region with the template target object point cloud image, are as follows: S61. The SIFT algorithm is used to extract key points from the point cloud map of the template target object and the fused point cloud map of the target object region to be inspected, respectively. The FPFH is used to describe the extracted key points to obtain key point features. S62. Use a global registration algorithm to pair the key points in the fused point cloud map of the target object region to be inspected with the key points in the template target object point cloud map to obtain key point pairs. S63. Interchangeability test, three-point test and spatial position test are used in sequence to eliminate incorrect matches in key point pairs, and an error function is set to verify the key point pairs to obtain matching key point pairs; based on the matching key point pair information, the pose information of the target object in the fused point cloud map of the target object region is obtained.

6. The target localization and detection method based on lidar and camera information fusion according to claim 5, characterized in that, In step S63, interchangeability consistency test, three-point test, and spatial location test are used sequentially to eliminate incorrect matches in the key point pairs. An error function is then set to verify the key point pairs to obtain matching key point pairs. The specific operation is as follows: S631. For any key point p in the fused point cloud map P of the target object region to be inspected. i The keypoint features extracted using FPFH are represented as F(p i If the keypoint FPFH feature set of the fused point cloud map P of the target object region to be inspected can be defined as: F(P) = {F(p)} i ):p i ∈P}; for any key point q in the point cloud map Q of the template target object. i The keypoint features extracted using FPFH are represented as F(q) i If the keypoint FPFH feature set of the target object point cloud Q can be defined as: F(Q) = {F(q)} i ):q i Let C1 be the set of correspondences between F(P) and F(Q) ∈Q}; S632, Interchangeability Consistency Test: For critical point pairs (p i ,q i ), if and only if F(p) i ) is F(q) i The nearest neighbor of q in F(P), and only if F(q) i ) is F(p i When the keypoint pair (p) is the nearest neighbor in F(Q), the keypoint pair (p) is the nearest neighbor in F(Q). i ,q i ) is retained in the result point pair C 11 middle; S633, Three-point test: Compare C with the result point 11 Three keypoint pairs (p1, q1), (p2, q2), and (p3, q3) are randomly selected. The results are then examined to see if (p1, p2, p3) and (q1, q2, q3) conform to the verification relationship shown in Equation 8. Keypoint pairs that satisfy the verification relationship are stored in the result point pair C. 111 middle; Where τ = 0.9; S634. Spatial Position Test: Calculate the point pair C according to Equation 9. 111 All key points in the middle to all key points q i The centroid of the point cloud, Centriod, is calculated according to Equation 10. 111 All keypoints in the middle, each keypoint q i The positional relationship between C(q) and the centroid of the point cloud Centriod i ), and calculate all key points q i C(q) i The average value of ) is used to calculate the value of each key point q. i C(q) i ) and C(q) i Compare the average values ​​and remove C(q). i ) greater than C(q) i Key points of the average; S635. Error Function Verification: Calculate the result point pair C after processing in step S54 using the error function. 111 The error between each keypoint pair and the error between the two keypoints is calculated to obtain the error of each keypoint pair; then the result point pair C is obtained. 111 The mean error μ and variance error δ of all keypoint pairs 2 If μ < δ 2 Then the verification is complete, and the result is for point C. 111 The remaining keypoint pairs are the matching keypoint pairs; the calculation formula of the error function E(T,L) is shown in Equation 11. Where λ represents a random number, and its range is: λ∈[0,1], T i Let T represent the rigid body transformation matrix of the i-th keypoint pair. i+1 Let p represent the rigid body transformation matrix of the (i+1)th keypoint pair, p represent the keypoint set in the fused point cloud map of the target object region to be inspected, q represent the keypoint set in the template target object point cloud map, and ψ(l p,q () indicates a key-point pair relationship.

7. The target localization and detection method based on lidar and camera information fusion according to claim 6, wherein the preprocessing in step S1 is at least one of the following: direct-pass filtering algorithm processing, distance-based radius filtering algorithm processing, and voxelized mesh filtering algorithm processing.

8. The target localization and detection method based on lidar and camera information fusion according to claim 7, characterized in that, The deep convolutional neural network mentioned in step S4 is YOLOv3.

9. The target positioning and detection method based on lidar and camera information fusion according to claim 8, wherein the target is a slag pot and a hook for suspending the slag pot.

10. The target positioning and detection method based on lidar and camera information fusion according to claim 9, wherein a laser scanning system is used to acquire 3D point cloud data of the detection scene, the laser scanning system comprising a 2D linear laser scanner and an intelligent gimbal device, the 2D linear laser scanner being mounted on the intelligent gimbal.

Images