Multi-sensor online three-dimensional detection method and device for large assembly scene

Abstract

The invention discloses a multi-sensor online three-dimensional detection method and device for a large assembly scene, and the method comprises the steps: enabling a plurality of non-repeated scanning laser radars to be connected with an embedded platform, and carrying out the scanning from different angles, and preprocessing the point cloud data at different angles to realize coordinate system unification and splicing fusion of the multi-sensor non-repetitive scanning point cloud data, collecting the point cloud data of the whole large assembly scene and making a data set. A large-scale assembly scene detection model is constructed for processing a target detection task in a large-scale assembly scene, and a trained optimal model is obtained through training, testing and offline visualization verification. And on the basis of the trained model, real-time large-scale assembly scene point cloud data is subjected to rapid reasoning, and high-precision real-time online detection of a mobile robot and the like in a large-scale assembly scene is realized. And manpower resources are liberated, and the assembling efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of online three-dimensional target detection in large-scale industrial manufacturing and assembly scenes, and in particular, relates to a multi-sensor online three-dimensional detection method and device for large-scale assembly scenes. Background Art

[0002] The existing manufacturing and assembly model mainly relies on a large amount of manual operation, single-machine monitoring and multi-robot production lines. However, with the increase in product complexity, large-scale assembly scenarios face more complex component assembly challenges, such as positioning, alignment and installation. Manual operation and inspection efficiency are low, and multiple mobile robots completing multiple tasks at the same time brings serious collaborative interference problems. The parallelism and suddenness of their behaviors and the conflicts in their positions will cause conflicts and confrontations between robots. The complex working environment will also bring safety risks to the staff.

[0003] In this context, it is particularly important to conduct accurate online detection of complex components, workers, mobile robots and other targets in large-scale assembly scenarios, which is directly related to the production and operation efficiency of the entire manufacturing and assembly model and the quality level of the products. In recent years, deep learning technology has gradually attracted widespread attention in the field of three-dimensional target detection. Some researchers have proposed traditional visual detection methods based on single sensor data in many scenarios. However, these methods still find it difficult to meet the strict engineering requirements of high-precision three-dimensional online target detection in large-scale manufacturing and assembly scenarios. Therefore, there is an urgent need to conduct in-depth research on a non-repetitive scanning multi-sensor online detection method and device for large-scale assembly scenarios. At present, there are several difficulties in the specific implementation process: 1. Large-scale manufacturing and assembly environments are usually complex and changeable, especially in the assembly scenarios of large and complex parts, where there are problems such as large space range and many obstacles. Robots need to have extremely high adaptability and flexibility to operate in such environments, which is difficult for traditional fixed equipment and single sensors to handle.

[0004] 2. The sparsity of point cloud data is a major challenge faced by point-based 3D object detection methods. In large-scale manufacturing and assembly scenarios, especially in large-scale or long-distance scanning, point cloud data may be very sparse, resulting in insufficient information in local areas. This sparsity affects feature extraction and accurate object detection.

[0005] 3. In large-scale manufacturing and assembly environments, multiple mobile robots may move at high speeds and change directions frequently, which makes online real-time detection and tracking more complicated. The detection method needs to have high real-time and high responsiveness. At the same time, robots may partially block other robots or be blocked by objects. This occlusion will cause part of the target information to be lost, making it difficult for the detection algorithm to accurately identify and locate. Summary of the invention

[0006] In response to the above technical problems, the present invention provides a multi-sensor online three-dimensional detection method and device for large-scale assembly scenes, which aims to optimize the technical problems faced by the online three-dimensional target detection process in large-scale complex component manufacturing and assembly scenes, such as limited scene coverage and detection range, large amount of point cloud data and sparsity, complex online real-time detection and tracking caused by multi-robot movement, and occlusion.

[0007] The technical solution adopted by the present invention to solve the technical problem is: A multi-sensor online three-dimensional detection method for a large assembly scene, the method comprising the following steps: S100: Collect 3D point cloud data in large assembly scenes based on non-repetitive scanning LiDAR; S200: Based on the embedded platform, the coordinate system of the 3D point cloud data is converted and the point cloud is spliced ​​and fused, and then the fused point cloud data is annotated to complete the production of the 3D target detection data set for large assembly scenes; S300: Construct a multi-sensor 3D point cloud online target detection model, which includes a point cloud encoding module, a centroid prediction and aggregation module, a non-parametric encoding module, and a point cloud memory module. The point cloud encoding module is used to extract geometric and semantic features from the point cloud; the centroid prediction and aggregation module is used to improve the positioning and feature representation of the target by predicting the target centroid and aggregating the adjacent points of the centroid; the non-parametric encoding module is used to flexibly extract global features from the input point cloud in the classification head; the point cloud memory module is used to perform similarity matching through the features of the point cloud, output the prediction of the point cloud category, and store and retrieve the point cloud features; after the classification prediction is completed, the bounding box is regressed to generate the anchor box, and the final detection result is obtained after 3D-NMS post-processing and filtering; S400: Input the multi-sensor large-scale assembly scene point cloud data training set into the model, set the loss function according to the model, back-propagate the model to update the parameters, adjust the parameters until the model converges to the best effect, and obtain the trained point cloud online target detection model after training. Send the training results to the PC for offline visualization, and finally deploy the trained point cloud online target detection model to the platform to complete real-time online detection.

[0008] Preferably, S100 includes: S110: Four Livox-Avia non-repetitive scanning laser radars are fixedly installed on the ceiling beams of the large assembly scene. The laser radars are numbered L0, L1, L2, and L3. The installation positions and angles are adjusted to ensure that the four laser radars can scan the entire large assembly scene. S120: Use four non-repetitive scanning LiDARs L0, L1, L2, and L3 to perform non-repetitive scanning of large manufacturing and assembly scenes, and simultaneously capture point cloud data of assembly scenes from multiple angles and positions, ensuring that each perspective has sufficient overlapping area and field of view coverage to obtain point cloud data separately. , , , .

[0009] Preferably, S200 includes: S210: Modify the multi_topic parameter in the lidar_rviz.launch file in the Livox non-repetitive scanning lidar driver file, so that the point cloud data received by the four lidars are divided into four data streams for output. At this time, the coordinate systems of the four point cloud data are not unified. Select the world coordinate system as the standard coordinate system to align and splice the point cloud data; S220: Create a workspace in the embedded platform and create a new ROS node: First initialize the ROS node, create a ROS node handle, and create publishers and subscribers at the same time. The subscriber subscribes to the point cloud topic data scanned by four laser radars, and the publisher publishes the aligned and spliced ​​data. Then, the subscribed point cloud data is rotated and translated through the callback function. The point cloud data is first aligned. Based on one of the laser radar point cloud data, the other three point cloud data are rotated and translated to align with the point cloud data of the main laser radar. After alignment, the four point cloud data are spliced ​​and fused to obtain a fused point cloud data; finally, the fused point cloud data is rotated and translated to achieve coordinate conversion to the standard coordinate system. After compiling the code, the ROS system is called to start the ROS node to obtain the final fused point cloud data of the four laser radars in the standard coordinate system. ; S230: Record the rosbag file, convert it into a single-frame pcd file or bin file, and then sample it to obtain the fused point cloud data; label 60% of the sampled point cloud data, and the labeled data set is recorded as Klabel. The remaining 40% of the point cloud data samples are not labeled and are recorded as Kunlabel. S240: Divide the labeled data set Klabel and the unlabeled data set Kunlabel respectively: divide 70% of the data in the labeled data set Klabel as a training set, divide the remaining 30% of the labeled data set Klabel as a validation set, and use the unlabeled data set Kunlabel as a test set.

[0010] Preferably, the point cloud encoding module includes a downsampling and a combined abstraction layer, the combined abstraction layer includes a sampling layer, a grouping layer and a point network layer, the point cloud encoding module in S300 is used to extract geometric and semantic features from the point cloud, and the point cloud encoding module includes the following steps: S311: Input the point cloud data after multi-sensor fusion, and use the world coordinate system as the standard coordinate system to mathematically represent the point cloud, specifically as a vector representation of size N×4, with the content (x, y, z, i), where (x, y, z) are the three-dimensional position coordinates of the point cloud data, i is the reflection intensity of the point cloud data, and N is the number of point clouds; S312: The sampling layer performs iterative farthest point sampling on the input point cloud set after the coordinate system is converted. The grouping layer combines the input point cloud set with the coordinates of the centroid obtained by sampling to obtain an output size of N ‘ ×K ×(d + C) point set groups, each of which corresponds to a local area, N ‘ represents the number of local regions, K is the number of points in the neighborhood of the centroid point, and K is different between groups; S313: N' local area points of N'×K×(d+C) points are transformed through the point network layer so that their coordinates are converted into a local frame relative to the centroid point. By using relative coordinates and point features, the point-to-point relationship in the local area is captured. The output data volume is N'×(d +C'). Each local area in the output is abstracted by the local features of its centroid and the encoded centroid neighborhood. These features represent the location, local geometric information and semantic information of the point.

[0011] Preferably, the centroid prediction and aggregation module in S300 is used to improve the positioning and feature representation of the target by predicting the target centroid and aggregating the points adjacent to the centroid, including the following steps: S321: selectively downsample the point cloud after the point cloud encoding module, and first use two MLP layers to attach to the encoding layer to further estimate the semantic category of each point; S322: pass the semantic category output by S321 through a foreground-background classification branch, predict each point, output the confidence that the point belongs to the foreground, retain the points with the top k scene scores, and feed them to the centroid of the next encoding layer; S323: Use contextual clues around the bounding box to predict the center of mass and predict the offset to the instance center ; In addition, a regularization term is added to minimize the uncertainty of the centroid prediction, specifically: ; in, is the coordinate of the ith point, is the weight of the i-th point, which is determined by the confidence of the foreground point, is the number of points involved in centroid prediction; S324: For the displaced centroid points, the Point-Net++ module is used to learn the potential representation of each instance, transform the adjacent points into the local canonical coordinate system, and then aggregate the point features through the shared MLP and symmetric functions, and feed the aggregated centroid point features into the proposal generation head to generate potential target area candidate boxes.

[0012] Preferably, the non-parametric encoding module in S300 is used to flexibly extract global features in the input point cloud in the classification head, including the following steps: S331: After the network generates a 3D candidate box, a non-parametric encoding module is first used to obtain the global features of the detected object; for the input point cloud, the non-parametric encoding module uses a trigonometric function to embed the point into a one-dimensional vector and encode its position; S332: After encoding, perform feature dimension expansion: use FPS to downsample the number of point clouds, and for each center point obtained after downsampling, use k-NN to find k neighborhood points of each center point and the corresponding features; splice the center point features and feature dimensions to achieve feature dimension expansion; S333: normalizing the coordinates using the mean and standard deviation, encoding the relative position using trigonometric functions, extracting geometric information using the obtained relative geometric weights, and performing local feature aggregation using maximum pooling and average pooling; S334: Repeat S332 and S333 three times. After completing four geometric feature aggregations, apply maximum pooling and average pooling again to obtain the global features of the point cloud.

[0013] Preferably, the point cloud memory module in S300 is used to perform similarity matching through point cloud features, output a prediction of the point cloud category, and store and retrieve point cloud features, including the following steps: S341: Use a non-parametric encoding module to build a point cloud memory library. The point cloud memory library includes feature memory and label memory. The global features of N training set point clouds are obtained through the non-parametric encoding module. At the same time, the corresponding classification labels are converted into one-dimensional vector codes, and then they are spliced ​​along the sample dimension and cached into two matrices. S342: Use the constructed point cloud memory library to perform two matrix multiplications to complete the classification: calculate the global features of the point cloud through the non-parametric encoding module, and calculate the cosine similarity between the feature memory, weight the one-dimensional vector label in the label memory and the cosine similarity, and perform similarity matching between the global features obtained from the test point cloud and the feature memory stored in the training set to classify the target of the test point cloud; S343: storing point cloud features, and retrieving point cloud features according to retrieval tags, wherein the retrieval tags include geometric positions, feature similarities, point categories or semantic information.

[0014] Preferably, the loss function in S400 includes the downsampling strategy loss L s , centroid prediction loss L c , classification loss L cls and box generation loss L b , specifically: ; Among them, the downsampling strategy loss Ls includes the class-aware sampling loss and weighted cross entropy loss , specifically: ; Where C represents the number of categories, is a one-hot label, represents the predicted logarithm; ; ; In the formula, Indicate point The ground truth offset to the center point, is an indicator function that determines whether the point is used to estimate the instance center. It is used to predict the number of points at the center of the instance, using the surrounding representative points in the large context for centroid prediction; ; in, represents the total number of samples, represents the total number of categories, represents the true label of the i-th sample in the c-th category, Represents the probability that the model predicts the i-th sample as category c; Box generation loss L b Decomposed into position, size, angle-bin, angle-res, angle part, specifically: .

[0015] Preferably, in S400, the finally trained point cloud online target detection model is deployed to the platform to complete real-time online detection, specifically: Deploy the 3D point cloud target detection model to the Jetson AGX Orin developer kit, receive the real-time point cloud topic data of four non-repetitive scanning laser radars after coordinate system conversion and splicing under the ROS system, process the data and convert it into a point cloud data input format suitable for the 3D detection model, parse the configuration file under ROS to create the model and load the model parameters through the trained model weights. After loading, input each frame of data after real-time multi-lidar coordinate system conversion and splicing for model reasoning. The model responds in real time and prints out the target prediction results of each frame of point cloud data; Convert the predicted detection box to 3D corner points, clear the markers in marker_array and marker_array_text, create a marker according to the predicted box and add it to marker_array and marker_array_text, clean up redundant markers, publish marker_array and marker_array_text, realize real-time visualization of the prediction results in the RVIZ platform, and demonstrate the online detection effect of non-repetitive scanning multi-sensor.

[0016] A multi-sensor online 3D detection device for large assembly scenes, including a non-repetitive scanning laser radar, an embedded platform and a PC terminal. Non-repetitive scanning LiDAR is used to collect 3D point cloud data in large assembly scenes; The embedded platform is used to transform the coordinate system of 3D point cloud data and to merge and stitch point clouds, and then to annotate the fused point cloud data to complete the production of 3D target detection data sets for large assembly scenes; a multi-sensor 3D point cloud online target detection model is constructed. The multi-sensor 3D point cloud online detection model includes a point cloud encoding module, a centroid prediction and aggregation module, a non-parametric encoding module, and a point cloud memory module. The point cloud encoding module is used to extract geometric and semantic features from the point cloud; the centroid prediction and aggregation module is used to improve the positioning and feature representation of the target by predicting the target centroid and aggregating the adjacent points of the centroid; the non-parametric encoding module is used to flexibly extract the global Features; The point cloud memory module is used to perform similarity matching through the features of the point cloud, output the prediction of the point cloud category, and store and retrieve the point cloud features; After the classification prediction is completed, the bounding box regression is performed to generate the anchor box, and the final detection result is obtained after 3D-NMS post-processing and filtering; The multi-sensor large-scale assembly scene point cloud data training set is input into the model, the loss function is set according to the model, the model is back-propagated to update the parameters, and the parameters are adjusted to achieve the best model convergence effect. After the training is completed, the trained point cloud online target detection model is obtained, and the training results are sent to the PC for offline visualization. The finally trained point cloud online target detection model is deployed to the platform to complete real-time online detection.

[0017] The above-mentioned multi-sensor online 3D detection method and device for large-scale assembly scenes use multiple laser radars to collect point cloud data from different perspectives, comprehensively capture various angles and details of large-scale assembly scenes, large and complex components, and mobile robots, etc., input the model for training and online reasoning, and achieve more refined and accurate online detection, liberating human resources and improving assembly efficiency. It is not only suitable for 3D online detection of large-scale assembly scenes, but also has the potential value of being applied to 3D online detection of any large-scale industrial manufacturing and assembly scenes. Its online detection performance has the characteristics of high precision, fast real-time response, and multi-sensor perception, which promotes the high-quality development of intelligent manufacturing. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flow chart of a multi-sensor online three-dimensional detection method for a large assembly scene in one embodiment of the present invention; Figure 2 This is a flow chart of a multi-sensor point cloud data fusion algorithm for a large assembly scene in one embodiment of the present invention; Figure 3 It is a schematic diagram of a multi-sensor three-dimensional point cloud target detection model for a large assembly scene in one embodiment of the present invention; Figure 4 This is a flowchart of non-repetitive scanning multi-sensor online three-dimensional detection of large assembly scenes in one embodiment of the present invention; Figure 5 Schematic diagram of a multi-sensor detection device for a large assembly scene in one embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0020] In one embodiment, Figure 1 As shown, a multi-sensor online three-dimensional detection method for a large assembly scene, the method comprising the following steps: S100: Collect 3D point cloud data in large assembly scenes based on non-repetitive scanning LiDAR; S200: Based on the embedded platform, the coordinate system of the 3D point cloud data is converted and the point cloud is spliced ​​and fused, and then the fused point cloud data is annotated to complete the production of the 3D target detection data set for large assembly scenes; S300: Construct a multi-sensor 3D point cloud online target detection model, which includes a point cloud encoding module, a centroid prediction and aggregation module, a non-parametric encoding module, and a point cloud memory module. The point cloud encoding module is used to extract geometric and semantic features from the point cloud; the centroid prediction and aggregation module is used to improve the positioning and feature representation of the target by predicting the target centroid and aggregating the adjacent points of the centroid; the non-parametric encoding module is used to flexibly extract global features from the input point cloud in the classification head; the point cloud memory module is used to perform similarity matching through the features of the point cloud, output the prediction of the point cloud category, and store and retrieve the point cloud features; after the classification prediction is completed, the bounding box is regressed to generate the anchor box, and the final detection result is obtained after 3D-NMS post-processing and filtering; S400: Input the multi-sensor large-scale assembly scene point cloud data training set into the model, set the loss function according to the model, back-propagate the model to update the parameters, adjust the parameters until the model converges to the best effect, and obtain the trained point cloud online target detection model after training. Send the training results to the PC for offline visualization, and finally deploy the trained point cloud online target detection model to the platform to complete real-time online detection.

[0021] The above-mentioned multi-sensor online 3D detection method for large-scale assembly scenes uses multiple laser radars to collect point cloud data from different perspectives, comprehensively captures all angles and details of large-scale assembly scenes, large and complex components, and mobile robots, etc., and inputs the model for training and online reasoning to achieve more refined and accurate online detection, liberating human resources and improving assembly efficiency. It is not only suitable for 3D online detection of large-scale assembly scenes, but also has the potential value of being applied to 3D online detection of any large-scale industrial manufacturing and assembly scenes. Its online detection performance has the characteristics of high precision, fast real-time response, and multi-sensor perception, which promotes the high-quality development of intelligent manufacturing.

[0022] In one embodiment, S100 includes: S110: Four Livox-Avia non-repetitive scanning laser radars are fixedly installed on the ceiling beams of the large assembly scene. The laser radars are numbered L0, L1, L2, and L3. The installation positions and angles are adjusted to ensure that the four laser radars can scan the entire large assembly scene. S120: Use four non-repetitive scanning LiDARs L0, L1, L2, and L3 to perform non-repetitive scanning of large manufacturing and assembly scenes, and simultaneously capture point cloud data of assembly scenes from multiple angles and positions, ensuring that each perspective has sufficient overlapping area and field of view coverage to obtain point cloud data separately. , , , .

[0023] Specifically, the embedded platform is built based on the NVIDIA Jetson A GX Orin developer kit. Four lidars are connected to the NVIDIA Jetson AGX Orin developer kit, multiple lidars are powered and transmitted, and a laptop is used to remotely communicate and control the Jetson AGX Orin developer kit.

[0024] In one embodiment, Figure 2 and Figure 4 As shown, S200 includes: S210: Modify the multi_topic parameter in the lidar_rviz.launch file in the Livox non-repetitive scanning lidar driver file, so that the point cloud data received by the four lidars are divided into four data streams for output. At this time, the coordinate systems of the four point cloud data are not unified. Select the world coordinate system as the standard coordinate system to align and splice the point cloud data; S220: Create a workspace in the Ubuntu system of the Jetson AGX Orin developer kit and create a new ROS node: First initialize the ROS node, create a ROS node handle, and create publishers and subscribers at the same time. The subscriber subscribes to the point cloud topic data scanned by four lidars, and the publisher publishes the aligned and spliced ​​data. Then, the subscribed point cloud data is rotated and translated through the callback function. The point cloud data is first aligned. Based on one of the lidar point cloud data, the other three point cloud data are rotated and translated to align with the point cloud data of the main lidar. After alignment, the four point cloud data are spliced ​​and fused to obtain a fused point cloud data; finally, the fused point cloud data is rotated and translated to achieve coordinate conversion to the standard coordinate system. After compiling the code, the ROS system is called to start the ROS node to obtain the fused point cloud data of the four lidars in the standard coordinate system. ; S230: Record the rosbag file, convert it into a single-frame pcd file or bin file, and then sample it to obtain the fused point cloud data; label 60% of the sampled point cloud data, and the labeled data set is recorded as Klabel. The remaining 40% of the point cloud data samples are not labeled and are recorded as Kunlabel. S240: Divide the labeled data set Klabel and the unlabeled data set Kunlabel respectively: divide 70% of the data in the labeled data set Klabel as a training set, divide the remaining 30% of the labeled data set Klabel as a validation set, and use the unlabeled data set Kunlabel as a test set.

[0025] In one embodiment, Figure 3 As shown, the point cloud encoding module includes a downsampling and a combined abstraction layer, the combined abstraction layer includes a sampling layer, a grouping layer and a point network layer, and the point cloud encoding module in S300 is used to extract geometric and semantic features from the point cloud. The point cloud encoding module includes the following steps: S311: Input the point cloud data after multi-sensor fusion, and use the world coordinate system as the standard coordinate system to mathematically represent the point cloud, specifically as a vector representation of size N×4, with the content (x, y, z, i), where (x, y, z) are the three-dimensional position coordinates of the point cloud data, i is the reflection intensity of the point cloud data, and N is the number of point clouds; S312: The sampling layer performs iterative farthest point sampling on the input point cloud set after the coordinate system is converted. The grouping layer combines the input point cloud set with the coordinates of the centroid obtained by sampling to obtain an output size of N ‘ ×K ×(d + C) point set groups, each of which corresponds to a local area, N ‘ represents the number of local regions, K is the number of points in the neighborhood of the centroid point, and K is different between groups; S313: N' local area points of N'×K×(d+C) points are transformed through the point network layer so that their coordinates are converted into a local frame relative to the centroid point. By using relative coordinates and point features, the point-to-point relationship in the local area is captured. The output data volume is N'×(d +C'). Each local area in the output is abstracted by the local features of its centroid and the encoded centroid neighborhood. These features represent the location, local geometric information and semantic information of the point.

[0026] Specifically, iterative farthest point sampling provides better coverage of the entire point set compared to random sampling given the same number of centroids.

[0027] In one embodiment, Figure 3 As shown, the centroid prediction and aggregation module in S300 is used to improve the positioning and feature representation of the target by predicting the target centroid and aggregating the adjacent points of the centroid, including the following steps: S321: selectively downsample the point cloud after the point cloud encoding module, and first use two MLP layers to attach to the encoding layer to further estimate the semantic category of each point; S322: pass the semantic category output by S321 through a foreground-background classification branch, predict each point, output the confidence that the point belongs to the foreground, retain the points with the top k scene scores, and feed them to the centroid of the next encoding layer; S323: Use contextual clues around the bounding box to predict the center of mass and predict the offset to the instance center ; In addition, a regularization term is added to minimize the uncertainty of the centroid prediction, specifically: ; in, is the coordinate of the ith point, is the weight of the i-th point, which is determined by the confidence of the foreground point, is the number of points involved in centroid prediction; S324: For the displaced centroid points, the Point-Net++ module is used to learn the potential representation of each instance, transform the adjacent points into the local canonical coordinate system, and then aggregate the point features through the shared MLP and symmetric functions, and feed the aggregated centroid point features into the proposal generation head to generate potential target area candidate boxes.

[0028] Specifically, during the training process, S321 uses class-aware sampling loss for supervision; during the training process, S322 uses a soft point mask to assign different weights to points in the bounding box according to their spatial positions, thereby implicitly incorporating geometric priors into network training, and uses weighted cross entropy loss for supervision, multiplying the soft point mask with the loss term of the foreground point to assign a higher probability to points close to the center. During the inference process, the bounding box is no longer required. If the model is well trained, only the top k points with the highest scores after downsampling need to be retained. During the training process, S322 is supervised by the centroid prediction loss.

[0029] In one embodiment, Figure 3 As shown, the non-parametric encoding module in S300 is used to flexibly extract global features in the input point cloud in the classification head, including the following steps: S331: After the network generates a 3D candidate box, a non-parametric encoding module is first used to obtain the global features of the detected object; for the input point cloud, the non-parametric encoding module uses a trigonometric function to embed the point into a one-dimensional vector and encode its position; S332: After encoding, perform feature dimension expansion: use FPS to downsample the number of point clouds, and for each center point obtained after downsampling, use k-NN to find k neighborhood points of each center point and the corresponding features; splice the center point features and feature dimensions to achieve feature dimension expansion; S333: normalizing the coordinates using the mean and standard deviation, encoding the relative position using trigonometric functions, extracting geometric information using the obtained relative geometric weights, and performing local feature aggregation using maximum pooling and average pooling; S334: Repeat S332 and S333 three times. After completing four geometric feature aggregations, apply maximum pooling and average pooling again to obtain the global features of the point cloud.

[0030] In one embodiment, Figure 3 As shown, the point cloud memory module in S300 is used to perform similarity matching through point cloud features, output point cloud category prediction, and store and retrieve point cloud features, including the following steps: S341: Use a non-parametric encoding module to build a point cloud memory library. The point cloud memory library includes feature memory and label memory. The global features of N training set point clouds are obtained through the non-parametric encoding module. At the same time, the corresponding classification labels are converted into one-dimensional vector codes, and then they are spliced ​​along the sample dimension and cached into two matrices. S342: Use the constructed point cloud memory library to perform two matrix multiplications to complete the classification: calculate the global features of the point cloud through the non-parametric encoding module, and calculate the cosine similarity between the feature memory, weight the one-dimensional vector label in the label memory and the cosine similarity, and perform similarity matching between the global features obtained from the test point cloud and the feature memory stored in the training set to classify the target of the test point cloud; S343: storing point cloud features, and retrieving point cloud features according to retrieval tags, wherein the retrieval tags include geometric positions, feature similarities, point categories or semantic information.

[0031] Specifically, In one embodiment, the loss function in S400 includes the downsampling strategy loss L s , centroid prediction loss L c , classification loss L cls and box generation loss L b , specifically: ; Among them, the downsampling strategy loss Ls includes the class-aware sampling loss and weighted cross entropy loss , specifically: ; Where C represents the number of categories, is a one-hot label, represents the predicted logarithm; ; ; In the formula, Indicate point The ground truth offset to the center point, is an indicator function that determines whether the point is used to estimate the instance center. It is used to predict the number of points at the center of the instance, using the surrounding representative points in the large context for centroid prediction; ; in, represents the total number of samples, represents the total number of categories, represents the true label of the i-th sample in the c-th category, Represents the probability that the model predicts the i-th sample as category c; Box generation loss L b Decomposed into position, size, angle-bin, angle-res, angle part, specifically: .

[0032] Specifically, the training set of the large assembly scene point cloud dataset is input into the above model, and the model parameters are optimized through the loss function. After the training is completed, the performance of the hyperparameters is evaluated using the validation set, and the hyperparameters such as the network learning rate, batch size, and feature dimension are adjusted according to the results to find the optimal local hyperparameter solution, thereby obtaining the network model with the best performance under the current dataset.

[0033] In one embodiment, Figure 4 As shown, in S400, the finally trained point cloud online target detection model is deployed to the platform to complete real-time online detection, specifically: Deploy the 3D point cloud target detection model to the Jetson AGX Orin developer kit, receive the real-time point cloud topic data of four non-repetitive scanning laser radars after coordinate system conversion and splicing under the ROS system, process the data and convert it into a point cloud data input format suitable for the 3D detection model, parse the configuration file under ROS to create the model and load the model parameters through the trained model weights. After loading, input each frame of data after real-time multi-lidar coordinate system conversion and splicing for model reasoning. The model responds in real time and prints out the target prediction results of each frame of point cloud data; Convert the predicted detection box to 3D corner points, clear the markers in marker_array and marker_array_text, create a marker according to the predicted box and add it to marker_array and marker_array_text, clean up redundant markers, publish marker_array and marker_array_text, realize real-time visualization of the prediction results in the RVIZ platform, and demonstrate the online detection effect of non-repetitive scanning multi-sensor.

[0034] In one embodiment, after S400, the method further includes: The test set of the large assembly scene dataset is input into the trained 3D point cloud online object detection model for testing. After the model inference is completed, the test results are obtained. ; Download and install the open3d platform in the standard coordinate system, and use the test result file obtained The original point cloud files of the dataset are visualized offline based on the open3d platform to verify the accuracy of the detection results and the generalization ability of the model.

[0035] The above-mentioned multi-sensor online three-dimensional detection method for a large assembly scene is connected to the Jetson AGX Orin developer kit through multiple non-repetitive scanning laser radars, scans from different angles, and pre-processes the point cloud data at different angles to realize the coordinate system unity and splicing fusion of the multi-sensor non-repetitive scanning point cloud data, collects the point cloud data of the entire large assembly scene and produces a data set. A large-scale assembly scene detection model is constructed, which can handle the target detection task in the large-scale assembly scene, and the trained best model is obtained through training, testing and offline visual verification. Finally, the model is deployed on the Jetson AGX Orin developer kit platform, and by docking the model with the ROS system, the real-time point cloud data of the large-scale assembly scene is obtained and fast reasoning is performed, so as to realize high-precision real-time online detection of mobile robots in large-scale assembly scenes. The method and device provide comprehensive and accurate point cloud data of large-scale assembly scenes and fast real-time online detection capabilities, improve product quality and production efficiency in the manufacturing and assembly process, and further ensure the stability of the entire system and the safety of the working environment, providing strong support for the high-quality and intelligent manufacturing and assembly of large and complex parts.

[0036] Through non-repetitive scanning methods and the construction of multi-sensor devices, more comprehensive and detailed positions and spatial point cloud data of large and complex components, mobile robots and workers in large assembly scenes can be obtained, and higher-precision detection can be achieved, helping multi-robot systems to carry out more effective path planning and task allocation, reduce conflicts and interference between robots, and improve overall assembly efficiency; real-time online detection improves real-time response capabilities, allowing mobile robots and workers to quickly adapt to dynamically changing assembly environments and improve operational flexibility and efficiency; at the same time, accurate detection can avoid potential obstacles, reduce the possibility of accidents, and ensure the safety of the working environment.

[0037] like Figure 5 As shown, a multi-sensor online three-dimensional detection device for large assembly scenes includes a non-repetitive scanning laser radar, an embedded platform and a PC terminal. Non-repetitive scanning LiDAR is used to collect 3D point cloud data in large assembly scenes; The embedded platform is used to transform the coordinate system of 3D point cloud data and to merge and stitch point clouds, and then to annotate the fused point cloud data to complete the production of 3D target detection data sets for large assembly scenes; a multi-sensor 3D point cloud online target detection model is constructed. The multi-sensor 3D point cloud online detection model includes a point cloud encoding module, a centroid prediction and aggregation module, a non-parametric encoding module, and a point cloud memory module. The point cloud encoding module is used to extract geometric and semantic features from the point cloud; the centroid prediction and aggregation module is used to improve the positioning and feature representation of the target by predicting the target centroid and aggregating the adjacent points of the centroid; the non-parametric encoding module is used to flexibly extract the global Features; The point cloud memory module is used to perform similarity matching through the features of the point cloud, output the prediction of the point cloud category, and store and retrieve the point cloud features; After the classification prediction is completed, the bounding box regression is performed to generate the anchor box, and the final detection result is obtained after 3D-NMS post-processing and filtering; The multi-sensor large-scale assembly scene point cloud data training set is input into the model, the loss function is set according to the model, the model is back-propagated to update the parameters, and the parameters are adjusted to achieve the best model convergence effect. After the training is completed, the trained point cloud online target detection model is obtained, and the training results are sent to the PC for offline visualization. The finally trained point cloud online target detection model is deployed to the platform to complete real-time online detection.

[0038] For the refinement of the multi-sensor online three-dimensional detection device based on a large-scale assembly scenario, please refer to the refinement of the multi-sensor online three-dimensional detection method for a large-scale assembly scenario, which will not be repeated here.

[0039] High-precision online detection of complex components, workers, mobile robots, etc. in large-scale assembly scenarios is the key to ensuring the efficiency and quality of the manufacturing and assembly processes. The density of point cloud data in large-scale assembly scenarios and the reasoning performance of the assembly scene target detection algorithm model determine the accuracy of online detection of large-scale assembly scenarios. Therefore, it is necessary to construct a multi-sensor online three-dimensional detection method and device for large-scale assembly scenarios. Traditional online detection usually relies on manual and single-machine detection, which has problems such as low efficiency, limited detection range, and slow real-time response speed when facing large-scale assembly scenarios with large scene scales, many task processes, high precision requirements, and complex operation scenes. It is impossible to provide comprehensive and detailed information for real-time high-precision online detection, and it is difficult to guarantee the task requirements of complex component manufacturing and assembly in large-scale scenarios. Therefore, the beneficial effects of the present invention are as follows: 1. It can realize high-precision online detection of complex parts, workers, mobile robots and other targets in large-scale assembly scenarios, liberating human resources and improving assembly efficiency; 2. In large-scale assembly scenarios, single sensor data cannot meet the full coverage of large-scale and complex working environments. Multiple lidars are used to collect point cloud data from different perspectives, comprehensively capture all angles and details of large-scale assembly scenarios, large and complex components, and mobile robots, and input into the model for training and online reasoning to achieve more sophisticated and accurate online detection; 3. In the process of online inspection of large assembly scenes, mobile robots are running at high speeds and changing directions frequently, so the real-time capability of rapid response is particularly important. However, ordinary sensors have slow response and large data volumes, which leads to slow model processing. Using the non-repetitive scanning method, the laser radar traverses the entire horizontal angle range in each frame scan, but does not repeat the same scanning angle, avoiding repeated data collection and processing. It can more accurately capture the real-time state of the scene, quickly generate real-time scene data, and improve the efficiency, accuracy and real-time performance of online inspection. 4. For online target detection in large assembly scenarios, complex and parameter-intensive models cannot meet the actual engineering requirements of online real-time detection. An instance-aware downsampling method is used to retain more effective foreground points of the detection target to achieve accurate detection of the target. At the same time, a non-parametric network is added to reduce the amount of parameter calculation, reduce the computational complexity, significantly speed up the reasoning process, and improve the model efficiency and real-time capability of rapid response; 5. Point cloud data collection in large assembly scenes is sparse, especially in non-repetitive scanning over a large range or long distance, which leads to inaccurate detection of distant targets and complex components by the model. Multi-sensor point cloud data processing technology is introduced to perform coordinate system conversion and splicing operations on point cloud data from multiple sensors to achieve the fusion of multi-sensor point cloud data, effectively solve the sparsity problem of point cloud data, optimize the quality of point cloud data, and provide high-quality input point cloud data for subsequent model training and online reasoning.

[0040] The present invention is not only suitable for three-dimensional online detection of large-scale assembly scenes, but also has the potential value of being applied to three-dimensional online detection of any large-scale industrial manufacturing and assembly scenes. Its online detection performance is characterized by high precision, fast real-time response, and multi-sensor perception, which promotes the high-quality development of intelligent manufacturing.

[0041] The above is a detailed introduction to the multi-sensor online three-dimensional detection method and device for a large assembly scene provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A multi-sensor online three-dimensional detection method for large assembly scenes, characterized in that: The method comprises the following steps: S100: Collect 3D point cloud data in large assembly scenes based on non-repetitive scanning LiDAR; S200: Based on the embedded platform, the coordinate system of the 3D point cloud data is converted and the point cloud is spliced ​​and fused, and then the fused point cloud data is annotated to complete the production of the 3D target detection data set for large assembly scenes; S300: Construct a multi-sensor 3D point cloud online target detection model, which includes a point cloud encoding module, a centroid prediction and aggregation module, a non-parametric encoding module, and a point cloud memory module. The point cloud encoding module is used to extract geometric and semantic features from the point cloud; the centroid prediction and aggregation module is used to improve the positioning and feature representation of the target by predicting the target centroid and aggregating the adjacent points of the centroid; the non-parametric encoding module is used to flexibly extract global features from the input point cloud in the classification head; the point cloud memory module is used to perform similarity matching through the features of the point cloud, output the prediction of the point cloud category, and store and retrieve the point cloud features; after the classification prediction is completed, the bounding box is regressed to generate the anchor box, and the final detection result is obtained after 3D-NMS post-processing and filtering; S400: Input the multi-sensor large-scale assembly scene point cloud data training set into the model, set the loss function according to the model, back-propagate the model to update the parameters, adjust the parameters until the model converges to the best effect, and obtain the trained point cloud online target detection model after training. Send the training results to the PC for offline visualization, and finally deploy the trained point cloud online target detection model to the platform to complete real-time online detection.

2. The method according to claim 1, characterized in that S100 includes: S110: Four Livox-Avia non-repetitive scanning laser radars are fixedly installed on the ceiling beams of the large assembly scene. The laser radars are numbered L0, L1, L2, and L3. The installation positions and angles are adjusted to ensure that the four laser radars can scan the entire large assembly scene. S120: Use four non-repetitive scanning LiDARs L0, L1, L2, and L3 to perform non-repetitive scanning of large manufacturing and assembly scenes, and simultaneously capture point cloud data of assembly scenes from multiple angles and positions, ensuring that each perspective has sufficient overlapping area and field of view coverage to obtain point cloud data separately. , , , .

3. The method according to claim 2, characterized in that S200 includes: S210: Modify the multi_topic parameter in the lidar_rviz.launch file in the Livox non-repetitive scanning lidar driver file, so that the point cloud data received by the four lidars are divided into four data streams for output. At this time, the coordinate systems of the four point cloud data are not unified. Select the world coordinate system as the standard coordinate system to align and splice the point cloud data; S220: Create a workspace in the embedded platform and create a new ROS node: First initialize the ROS node, create a ROS node handle, and create publishers and subscribers at the same time. The subscriber subscribes to the point cloud topic data scanned by four laser radars, and the publisher publishes the aligned and spliced ​​data. Then, the subscribed point cloud data is rotated and translated through the callback function. The point cloud data is first aligned. Based on one of the laser radar point cloud data, the other three point cloud data are rotated and translated to align with the point cloud data of the main laser radar. After alignment, the four point cloud data are spliced ​​and fused to obtain a fused point cloud data; finally, the fused point cloud data is rotated and translated to achieve coordinate conversion to the standard coordinate system. After compiling the code, the ROS system is called to start the ROS node to obtain the final fused point cloud data of the four laser radars in the standard coordinate system. ; S230: Record the rosbag file, convert it into a single-frame pcd file or bin file, and then sample it to obtain the fused point cloud data; label 60% of the sampled point cloud data, and the labeled data set is recorded as Klabel. The remaining 40% of the point cloud data samples are not labeled and are recorded as Kunlabel. S240: Divide the labeled data set Klabel and the unlabeled data set Kunlabel respectively: divide 70% of the data in the labeled data set Klabel as a training set, divide the remaining 30% of the labeled data set Klabel as a validation set, and use the unlabeled data set Kunlabel as a test set.

4. The method according to claim 3, characterized in that The point cloud coding module includes downsampling and combined abstraction layers. The combined abstraction layer includes a sampling layer, a grouping layer, and a point network layer. The point cloud coding module in S300 is used to extract geometric and semantic features from the point cloud. The point cloud coding module includes the following steps: S311: Input the point cloud data after multi-sensor fusion, and use the world coordinate system as the standard coordinate system to mathematically represent the point cloud, specifically as a vector representation of size N×4, with the content (x, y, z, i), where (x, y, z) are the three-dimensional position coordinates of the point cloud data, i is the reflection intensity of the point cloud data, and N is the number of point clouds; S312: The sampling layer performs iterative farthest point sampling on the input point cloud set after the coordinate system is converted. The grouping layer combines the input point cloud set with the coordinates of the centroid obtained by sampling to obtain an output size of N ‘ ×K ×(d + C) point set groups, each of which corresponds to a local area, N ‘ represents the number of local regions, K is the number of points in the neighborhood of the centroid point, and K is different between groups; S313: N' local area points of N'×K×(d+C) points are transformed through the point network layer so that their coordinates are converted into a local frame relative to the centroid point. By using relative coordinates and point features, the point-to-point relationship in the local area is captured. The output data volume is N'×(d +C'). Each local area in the output is abstracted by the local features of its centroid and the encoded centroid neighborhood. These features represent the location, local geometric information and semantic information of the point.

5. The method according to claim 4, characterized in that The centroid prediction and aggregation module in S300 is used to improve the positioning and feature representation of the target by predicting the target centroid and aggregating the points adjacent to the centroid, including the following steps: S321: selectively downsample the point cloud after the point cloud encoding module, and first use two MLP layers to attach to the encoding layer to further estimate the semantic category of each point; S322: pass the semantic category output by S321 through a foreground-background classification branch, predict each point, output the confidence that the point belongs to the foreground, retain the points with the top k scene scores, and feed them to the centroid of the next encoding layer; S323: Use contextual clues around the bounding box to predict the center of mass and predict the offset to the instance center ; In addition, a regularization term is added to minimize the uncertainty of the centroid prediction, specifically: ； in, is the coordinate of the ith point, is the weight of the i-th point, which is determined by the confidence of the foreground point, is the number of points involved in centroid prediction; S324: For the displaced centroid points, the Point-Net++ module is used to learn the potential representation of each instance, transform the adjacent points into the local canonical coordinate system, and then aggregate the point features through the shared MLP and symmetric functions, and feed the aggregated centroid point features into the proposal generation head to generate potential target area candidate boxes.

6. The method according to claim 5, characterized in that The non-parametric encoding module in S300 is used to flexibly extract global features from the input point cloud in the classification head, including the following steps: S331: After the network generates a 3D candidate box, a non-parametric encoding module is first used to obtain the global features of the detected object; for the input point cloud, the non-parametric encoding module uses a trigonometric function to embed the point into a one-dimensional vector and encode its position; S332: After encoding, perform feature dimension expansion: use FPS to downsample the number of point clouds, and for each center point obtained after downsampling, use k-NN to find k neighborhood points of each center point and the corresponding features; splice the center point features and feature dimensions to achieve feature dimension expansion; S333: normalizing the coordinates using the mean and standard deviation, encoding the relative position using trigonometric functions, extracting geometric information using the obtained relative geometric weights, and performing local feature aggregation using maximum pooling and average pooling; S334: Repeat S332 and S333 three times. After completing four geometric feature aggregations, apply maximum pooling and average pooling again to obtain the global features of the point cloud.

7. The method according to claim 6, characterized in that The point cloud memory module in S300 is used to perform similarity matching through point cloud features, output point cloud category predictions, and store and retrieve point cloud features, including the following steps: S341: Use a non-parametric encoding module to build a point cloud memory library. The point cloud memory library includes feature memory and label memory. The global features of N training set point clouds are obtained through the non-parametric encoding module. At the same time, the corresponding classification labels are converted into one-dimensional vector codes, and then they are spliced ​​along the sample dimension and cached into two matrices. S342: Use the constructed point cloud memory library to perform two matrix multiplications to complete the classification: calculate the global features of the point cloud through the non-parametric encoding module, and calculate the cosine similarity between the feature memory, weight the one-dimensional vector label in the label memory and the cosine similarity, and perform similarity matching between the global features obtained from the test point cloud and the feature memory stored in the training set to classify the target of the test point cloud; S343: storing point cloud features, and retrieving point cloud features according to retrieval tags, wherein the retrieval tags include geometric positions, feature similarities, point categories or semantic information.

8. The method according to claim 7, characterized in that The loss function in S400 includes the downsampling strategy loss L s , centroid prediction loss L c , classification loss L cls and box generation loss L b , specifically: ； Among them, the downsampling strategy loss Ls includes the class-aware sampling loss and weighted cross entropy loss , specifically: ； Where C represents the number of categories, is a one-hot label, represents the predicted logarithm; ； ； In the formula, Indicate point The ground truth offset to the center point, is an indicator function that determines whether the point is used to estimate the instance center. It is used to predict the number of points at the center of the instance, using the surrounding representative points in the large context for centroid prediction; ； in, represents the total number of samples, represents the total number of categories, represents the true label of the i-th sample in the c-th category, Represents the probability that the model predicts the i-th sample as category c; Box generation loss L b Decomposed into position, size, angle-bin, angle-res, angle part, specifically: 。 9. The method according to claim 8, characterized in that In S400, the finally trained point cloud online target detection model is deployed to the platform to complete real-time online detection, specifically: Deploy the 3D point cloud target detection model to the Jetson AGX Orin developer kit, receive the real-time point cloud topic data of four non-repetitive scanning laser radars after coordinate system conversion and splicing under the ROS system, process the data and convert it into a point cloud data input format suitable for the 3D detection model, parse the configuration file under ROS to create the model and load the model parameters through the trained model weights. After loading, input each frame of data after real-time multi-lidar coordinate system conversion and splicing for model reasoning. The model responds in real time and prints out the target prediction results of each frame of point cloud data; Convert the predicted detection box to 3D corner points, clear the markers in marker_array and marker_array_text, create a marker according to the predicted box and add it to marker_array and marker_array_text, clean up redundant markers, publish marker_array and marker_array_text, realize real-time visualization of the prediction results in the RVIZ platform, and demonstrate the online detection effect of non-repetitive scanning multi-sensor.

10. A multi-sensor online three-dimensional detection device for large-scale assembly scenes, characterized in that: Including non-repetitive scanning laser radar, embedded platform and PC terminal, Non-repetitive scanning LiDAR is used to collect 3D point cloud data in large assembly scenes; The embedded platform is used to transform the coordinate system of 3D point cloud data and to merge and stitch point clouds, and then to annotate the fused point cloud data to complete the production of 3D target detection data sets for large assembly scenes; a multi-sensor 3D point cloud online target detection model is constructed. The multi-sensor 3D point cloud online detection model includes a point cloud encoding module, a centroid prediction and aggregation module, a non-parametric encoding module, and a point cloud memory module. The point cloud encoding module is used to extract geometric and semantic features from the point cloud; the centroid prediction and aggregation module is used to improve the positioning and feature representation of the target by predicting the target centroid and aggregating the adjacent points of the centroid; the non-parametric encoding module is used to flexibly extract the global Features; The point cloud memory module is used to perform similarity matching through the features of the point cloud, output the prediction of the point cloud category, and store and retrieve the point cloud features; After the classification prediction is completed, the bounding box regression is performed to generate the anchor box, and the final detection result is obtained after 3D-NMS post-processing and filtering; The multi-sensor large-scale assembly scene point cloud data training set is input into the model, the loss function is set according to the model, the model is back-propagated to update the parameters, and the parameters are adjusted to achieve the best model convergence effect. After the training is completed, the trained point cloud online target detection model is obtained, and the training results are sent to the PC for offline visualization. The finally trained point cloud online target detection model is deployed to the platform to complete real-time online detection.

Images