A workpiece point cloud instance segmentation method for industrial scenes

Abstract

This invention discloses a workpiece point cloud instance segmentation method for industrial scenarios, relating to the fields of 3D machine vision and deep learning technology, including: Step 1, image acquisition and data conversion to obtain workpiece point cloud data; Step 2, point cloud preprocessing to filter out invalid points and workbench planes in the point cloud data; Step 3, point cloud annotation to obtain annotation files by annotating the preprocessed point cloud data; the annotation files and point cloud data constitute the point cloud instance segmentation dataset required for model training; Step 4, data augmentation; Step 5, point cloud instance segmentation model construction; Step 6, point cloud instance segmentation model training; Step 7, point cloud instance segmentation model prediction, using the trained point cloud instance segmentation model for inference, inputting data from the validation set, and outputting and displaying the predicted instance categories and point cloud composition.

Description

Technical Field

[0001] This invention relates to the fields of 3D machine vision and deep learning technology, and in particular to a method for segmenting workpiece point cloud instances for industrial scenarios. Background Technology

[0002] In recent years, with the advancement of technology and social development, industrial robotic arms have been widely used in various industrial production fields. However, industrial robotic arms lack the ability to perceive their surrounding environment and can only rely on pre-set programs to repeatedly complete tasks. For common industrial production processes such as stacking, palletizing, and tray placement, relying on manual assistance to complete the preliminary processing and placement steps can no longer meet the needs of intelligent manufacturing. To adapt to the current trend of highly intelligent and information-based manufacturing, it is necessary to use sensor technology and deep learning methods to compensate for the lack of perception in industrial robotic arms, enabling them to independently complete more complex industrial production processes. With the rapid development of 3D data acquisition technology, raw 3D data can be easily acquired using sensors such as 3D scanners, LiDAR, and RGB-D cameras. Compared to 2D images, 3D data can provide richer feature information, such as geometric shape and scale. If an intelligent perception module can be built by preprocessing and feature extraction of raw 3D data, the industrial robotic arm can gain a certain degree of perception of its surrounding environment, thereby assisting it in completing industrial production processes.

[0003] However, if traditional point cloud instance segmentation methods are used as the core inference module, these methods mainly rely on geometric features such as the distance and normal vector between points in the point cloud to compare the features of a point with the features of other points in its neighborhood to determine whether the point and its neighbors belong to the same set. This results in drawbacks such as heavy dependence on feature selection and prior knowledge, low segmentation accuracy, and low efficiency, making it difficult to meet the accuracy and efficiency requirements of industrial production.

[0004] In recent years, with the rapid development of deep learning and 2D object detection and instance segmentation, 3D object detection, point cloud semantic segmentation, and point cloud instance segmentation have also become research hotspots, providing new solutions for the construction of intelligent perception modules in industrial scenarios. Compared with traditional point cloud instance segmentation methods, deep learning methods have the following advantages: 1. Deep learning methods are data-driven, allowing for pre-training on large-scale datasets to obtain generalized models, which can then be fine-tuned on small-scale, scenario-specific datasets to obtain models tailored to specific scenarios, thus enabling stronger generalization capabilities; 2. Deep learning methods extract high-dimensional abstract features through multi-layer nonlinear computational units, enabling the extraction of higher-level information such as point cloud color, density, geometric information, and semantics, resulting in more precise boundary handling of target instances and achieving higher accuracy segmentation; 3. Deep learning methods typically use GPUs for accelerated computation, resulting in faster inference speeds for the models.

[0005] Li Zongmin et al., in their Chinese invention patent application "A Transformer-Based Point Cloud Instance Segmentation Method" (application number CN202111319758.8), implemented point cloud instance segmentation using deep learning and computer vision algorithms. This method includes steps such as acquiring point cloud scene information via LiDAR, downsampling the original point cloud data, dividing the sampled point cloud data into regions, processing the point cloud data of each region using a self-attention structure, and extracting the point cloud features of each region using an MLP to obtain the Q, K, and V values ​​for each region. This method can fully exploit the relationship between local and global features in each region to achieve the goal of segmenting each instance. However, the Transformer-based model has high computational complexity and long inference time, making it unsuitable for workpiece point cloud instance segmentation in industrial scenarios.

[0006] In summary, the application of deep learning-based point cloud instance segmentation methods to industrial scenarios still faces the following challenges:

[0007] 1. Currently, most point cloud instance segmentation backbone networks use 3D U-Net networks that incorporate submanifold sparse convolution. This network performs well in feature extraction for large-scale point clouds, but its feature extraction performance is poor for small objects such as workpieces in industrial scenarios, failing to meet the requirements for high-precision segmentation of workpieces.

[0008] 2. Point cloud instance segmentation methods may struggle to handle situations where the workpiece surface has reflections or shadows. When reflections or shadows appear on the workpiece surface, they can affect the data acquisition from the 3D camera, resulting in incomplete point cloud data and inaccurate segmentation results. Therefore, more robust processing logic needs to be added to the algorithm.

[0009] 3. Deep learning-based point cloud instance segmentation methods have high hardware requirements, requiring more computing resources and time to process point clouds and predict instance segmentation results. However, industrial scenarios have high real-time requirements, so it is necessary to design a lighter network model to improve inference speed.

[0010] Therefore, those skilled in the art are dedicated to developing a new workpiece point cloud instance segmentation method for industrial scenarios to solve the aforementioned problems in the existing technology. Summary of the Invention

[0011] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is how to achieve better feature extraction in workpiece point cloud instance segmentation, including overcoming the influence of reflection or shadow on the workpiece surface on the segmentation effect, and how to achieve a better trade-off between speed and accuracy in workpiece point cloud instance segmentation.

[0012] To achieve the above objectives, this invention provides a method for segmenting workpiece point cloud instances in industrial scenarios based on MinkowskiNet and multi-level feature fusion. First, a ResNet-34 backbone network is constructed using Minkowski sparse convolution provided by MinkowskiNet to extract point cloud features. Then, center offset vector prediction is used as an auxiliary task, passing through a semantic category prediction branch, a center offset vector prediction branch, and a clustering module to output preliminary segmentation prediction instances. Next, a ResNet-18 network is constructed using Minkowski sparse convolution for feature re-extraction and cross-level feature fusion. Finally, based on the fused features, a mask prediction branch and a mask scoring branch are used to generate the final predicted instance result.

[0013] This invention provides a method for segmenting workpiece point cloud instances in industrial scenarios, comprising the following steps:

[0014] Step 1: Perform image acquisition and data conversion to obtain point cloud data of the workpiece;

[0015] Step 2: Point cloud preprocessing, filtering out invalid points and workbench planes in the point cloud data, and saving the remaining point cloud data as the point cloud representation of the workpiece;

[0016] Step 3: Point cloud annotation. Instance annotation is performed on the preprocessed point cloud data to obtain an annotation file; the annotation file and the point cloud data constitute the point cloud instance segmentation dataset required for model training.

[0017] Step 4: Data augmentation. Perform data augmentation on the point cloud instance segmentation dataset.

[0018] Step 5: Point cloud instance segmentation model construction, including the construction of backbone network module, semantic category and offset vector prediction module, clustering module, feature re-extraction and fusion module, mask prediction and mask scoring module;

[0019] Step 6: Point cloud instance segmentation model training. The data-augmented point cloud instance segmentation dataset is divided into a training set and a validation set in an 8:2 ratio. The training set is then input into the point cloud instance segmentation model for training.

[0020] Step 7: Point cloud instance segmentation model prediction. Use the trained point cloud instance segmentation model for inference, input the data in the validation set, and output and display the predicted instance category and point cloud composition.

[0021] Furthermore, step 5 includes the following sub-steps:

[0022] Step 5.1: The backbone network module extracts multi-scale point cloud features through MinkowskiNet, wherein MinkowskiNet introduces Minkowski tensors and Minkowski convolutions to construct ResNet-34 as the point cloud feature extraction network;

[0023] Step 5.2: For the point cloud features extracted in Step 5.1, the semantic category and offset vector prediction module predicts the semantic category of each point in the point cloud instance segmentation dataset and the offset vector from each point to the geometric center of the instance to which the point belongs using a multi-layer MLP. The semantic category and the offset vector are represented by N×N... classes The point cloud instance segmentation dataset is represented by an N×3 dimensional vector, where N is the number of points in the dataset. classes The number of categories of the workpiece;

[0024] Step 5.3: The clustering module moves each point in the point cloud instance segmentation dataset to the geometric center of the instance to which the point belongs based on the offset vector, obtains all points within a sphere with radius r centered on the geometric center of the instance, and clusters the points with the same semantic category as the geometric center of the instance into a cluster to obtain several preliminary segmentation prediction instances.

[0025] Step 5.4: The feature re-extraction and fusion module uses Minkowski sparse convolution to construct ResNet-18 as the feature re-extraction network. The ResNet-18 is used to extract instance-level features of several preliminary segmentation prediction instances, and then fused with the point cloud features extracted by ResNet-34 to obtain fused features.

[0026] Step 5.5: The mask prediction and mask scoring module predicts an instance-level mask based on the fusion features through two layers of MLP to filter out background points in the preliminary segmentation prediction instance; at the same time, it generates a confidence score for the final segmentation prediction instance through a mask scoring branch.

[0027] Furthermore, the mask scoring branch in step 5.5 is also based on the fused features. It calculates the IoU between the mask and the ground truth using a global average pooling layer and an MLP layer, outputting the confidence score of the final segmentation prediction instance. The output result is expressed in K×N... classes The vector representation is given by dimensionality, where K is the number of the final segmentation prediction instances in the point cloud instance segmentation dataset.

[0028] Further, in step 6, the total training loss of the point cloud instance segmentation model includes: semantic branch loss, center offset loss, mask branch loss, and mask scoring loss, specifically expressed as follows:

[0029] Loss = L semantic +L offset +L mask +L score

[0030] Among them, L semantic For the semantic branch loss, cross-entropy loss is used; L offset For the center offset loss, use L1 regression loss; L mask For the mask branch loss, a binary cross-entropy loss is used; L score The L2 regression loss is used for the mask scoring loss.

[0031] Furthermore, in step 6, the hyperparameters of the point cloud instance segmentation model are set as follows: the number of points in the training set is randomly cropped to a maximum of 250k, the Adam optimizer is used for 50 rounds of training, the batch size is set to 4, the learning rate is initialized to 0.001, and the learning rate is adjusted using the cosine annealing algorithm.

[0032] Furthermore, in step 1, an RGB-D camera is used to acquire RGB images and depth images of the workpiece, and the RGB images and depth images can be converted into point cloud data using the Open3D library.

[0033] Further, in step 2, the `remove_statistical_outlier()` function from the Open3D library is used to filter out outliers, i.e., invalid points, from the point cloud data using statistical methods. Then, the `voxel_down_sample()` and `uniform_down_sample()` functions are used to denoise and downsample the point cloud data. The `segment_plane()` function is used to fit the workbench plane to obtain all interior points representing the workbench plane. All interior points representing the workbench plane are removed from the denoised point cloud data, and the remaining point cloud data is saved as the point cloud representation of the workpiece and saved in .pcd format.

[0034] Furthermore, in step 3, the point cloud data is annotated using the open-source annotation tool CloudCompare to obtain the annotation file; the annotation file is a txt annotation file, and each line stores the coordinate information, color information and semantic category information of the points in the point cloud data.

[0035] Furthermore, the format of the txt annotation file is the same as the annotation format of the S3DIS dataset.

[0036] Furthermore, in step 4, the data augmentation methods include random rotation, random translation, random scaling, random noise, random mirroring, and color transformation.

[0037] The workpiece point cloud instance segmentation method provided by this invention for industrial scenarios has at least the following technical effects:

[0038] 1. The technical solution provided by this invention uses MinkowskiNet to build the backbone network. Compared with other sparse convolution modules and traditional convolution modules, the Minkowski sparse convolution has the characteristics of high computational efficiency, large receptive field and strong feature extraction capability. It is suitable for processing sparse point cloud data and can improve the point cloud segmentation effect of workpieces.

[0039] 2. The technical solution provided by the present invention can avoid or reduce the reflection of light on some workpiece surfaces by reasonably arranging the relative positions of the light source, camera and workpiece, so that the collected data contains more key information. At the same time, through a series of preprocessing processes such as point cloud filtering, the integrity of the point cloud can be improved, the number of noise points in the point cloud can be reduced, and the point cloud segmentation effect of the model on the workpiece can be further improved.

[0040] 3. The technical solution provided by this invention uses Minkowski sparse convolution to construct lightweight ResNet-34 and ResNet-18 networks as feature extraction networks, which can improve the inference speed of the model, reduce storage space, improve generalization ability, and is easier to optimize and debug, so as to meet the real-time requirements of industrial scenarios as much as possible.

[0041] 4. The technical solution provided by this invention, through the improvements described above, enables the model to possess strong robustness and generalization ability, and improves the accuracy of workpiece point cloud instance segmentation. Compared with existing workpiece point cloud instance segmentation methods, the technical solution provided by this invention can quickly achieve point cloud instance segmentation for various workpieces in industrial scenarios, achieving a better balance between workpiece point cloud instance segmentation speed and accuracy.

[0042] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating a preferred embodiment of the workpiece point cloud instance segmentation method of the present invention;

[0044] Figure 2 yes Figure 1 The data acquisition, data preprocessing process, and point cloud instance segmentation model structure diagram of the embodiment shown are illustrated.

[0045] Figure 3 yes Figure 1 The diagram shows the architecture of ResNet-18, a lightweight feature re-extraction network in the embodiment shown. Detailed Implementation

[0046] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0047] In embodiments of this invention, a MinkowskiNet backbone network is proposed to extract point cloud features and form a multi-level feature map representation, enabling feature fusion between hierarchical features to enhance the feature representation. Addressing the sparsity of point clouds, a Minkowski sparse convolution model, capable of effectively handling large-scale sparse data, is used to construct the model, which can also be efficiently parallelized on a GPU. Embodiments of this invention also introduce a staged prediction mode, dividing the point cloud instance segmentation task into a coarse prediction stage and a fine prediction stage. In the coarse prediction stage, a center offset vector prediction task is used as an auxiliary task, its loss is calculated, and additional supervision signals are introduced, thereby learning better feature representations and improving model performance. In the fine prediction stage, cross-level feature fusion improves the feature representation capability, thus enhancing the model's prediction accuracy. The technical solution provided by these embodiments effectively addresses the problem of insufficient accuracy in workpiece point cloud instance segmentation methods in industrial scenarios, improving model performance through multi-faceted model improvements while effectively balancing the two key task indicators of inference speed and prediction accuracy.

[0048] Example 1

[0049] This invention provides a method for segmenting workpiece point cloud instances in industrial scenarios, comprising the following steps: Figure 1 As shown:

[0050] Step 1: Perform image acquisition and data conversion to obtain point cloud data of the workpiece;

[0051] Step 2: Point cloud preprocessing. Filter out invalid points and workbench planes in the point cloud data, and save the remaining point cloud data as the point cloud representation of the workpiece.

[0052] Step 3: Point cloud annotation. Instance annotation is performed on the preprocessed point cloud data to obtain annotation files. The annotation files and point cloud data constitute the point cloud instance segmentation dataset required for model training.

[0053] Step 4: Data augmentation. Perform data augmentation on the point cloud instance segmentation dataset.

[0054] Step 5: Point cloud instance segmentation model construction, including the construction of backbone network module, semantic category and offset vector prediction module, clustering module, feature re-extraction and fusion module, mask prediction and mask scoring module;

[0055] Step 6: Point cloud instance segmentation model training. Divide the data-augmented point cloud instance segmentation dataset into a training set and a validation set in an 8:2 ratio. Input the training set into the point cloud instance segmentation model for training.

[0056] Step 7: Point cloud instance segmentation model prediction. Use the trained point cloud instance segmentation model for inference, input the data in the validation set, and output and display the predicted instance categories and point cloud composition.

[0057] Example 2

[0058] Building upon Example 1, step 5 first extracts multi-scale point cloud features using MinkowskiNet, then performs multi-scale feature fusion to enhance feature representation. Finally, based on the fused multi-scale features, the semantic category and center offset vector are predicted via a semantic prediction branch and a center offset vector prediction branch, respectively. A preliminary segmentation prediction instance is obtained through a clustering module. Based on the preliminary predicted segmentation instance, features are further extracted and fused with the features extracted from the backbone network to obtain a fused feature vector. The final prediction instance result is then generated via a mask prediction branch and a mask scoring branch.

[0059] Specifically, step 5 includes the following sub-steps, such as Figure 2 As shown:

[0060] Step 5.1: The backbone network module extracts multi-scale point cloud features through MinkowskiNet. MinkowskiNet introduces Minkowski tensors and Minkowski convolutions to construct ResNet-34 as the point cloud feature extraction network. MinkowskiNet is a deep learning framework for constructing sparse convolutions, which can be used to process sparse high-dimensional data. Compared with submanifold sparse convolutions, the Minkowski sparse convolutions it provides have higher flexibility, better receptive field, and higher computational efficiency, and can extract point cloud features better.

[0061] Step 5.2: For the point cloud features extracted in Step 5.1, the semantic category and offset vector prediction module uses a multilayer perceptron (MLP) to predict the semantic category of each point in the point cloud instance segmentation dataset and the offset vector from each point to the geometric center of the instance to which that point belongs. The semantic category and offset vector are represented by N×N... classes The data is represented by an N×3 dimensional vector, where N is the number of points in the point cloud instance segmentation dataset. classes This represents the number of workpiece categories;

[0062] Step 5.3: The clustering module moves each point in the point cloud instance segmentation dataset to the geometric center of the instance to which the point belongs based on the offset vector, obtains all points within a sphere with radius r centered on the geometric center of the instance, and clusters the points with the same semantic category as the geometric center of the instance into a cluster. Following the above algorithm process, several preliminary segmentation prediction instances are obtained.

[0063] Step 5.4, Feature Re-extraction and Fusion Module: A ResNet-18 network is constructed using Minkowski sparse convolution as the feature re-extraction network. ResNet-18 extracts instance-level features from several preliminary segmentation prediction instances, and these features are fused with the point cloud features extracted by ResNet-34 to obtain fused features. The fused features have stronger expressive power (e.g., ...). Figure 3 (As shown); ResNet-18 is shallower than ResNet-34, which can reduce the number of model parameters, improve model training speed, and reduce memory consumption;

[0064] Step 5.5: Since the preliminary segmentation prediction instances contain foreground and background points, a mask prediction branch needs to be constructed to predict the instance mask in each preliminary segmentation prediction instance. The mask prediction and mask scoring module predicts instance-level masks based on fused features through two layers of MLP, filtering out background points in the preliminary segmentation prediction instances; at the same time, a mask scoring branch is used to generate the confidence score of the final segmentation prediction instance.

[0065] Specifically, the mask scoring branch in step 5.5 is also based on fused features. It calculates the IoU (Intersection over Union) between the mask and the ground truth through a global average pooling layer and an MLP layer, and outputs the confidence score of the final segmented predicted instance. The output result is expressed in K×N. classes The vector representation is given by dimensionality, where K is the number of final segmentation prediction instances in the point cloud instance segmentation dataset.

[0066] Example 3

[0067] Based on Examples 1 and 2, in step 6, the total training loss of the point cloud instance segmentation model includes: semantic branch loss, center offset loss, mask branch loss, and mask scoring loss, specifically expressed as follows:

[0068] Loss = L semantic +L offset +L mask +L score

[0069] Among them, L semantic For semantic branching loss, cross-entropy loss is used; L offset For the center offset loss, use L1 regression loss; L mask For the masked branch loss, binary cross-entropy loss is used; L score For masked scoring loss, L2 regression loss is used.

[0070] Specifically, in step 6, the hyperparameters of the point cloud instance segmentation model are set as follows: the number of points in the training set is randomly cropped to a maximum of 250k, the Adam optimizer is used for 50 rounds of training, the batch size is set to 4, the learning rate is initialized to 0.001, and the learning rate is adjusted using the cosine annealing algorithm.

[0071] Example 4

[0072] Based on the above embodiments, in step 1, an RGB-D camera is used to acquire RGB images and depth images of the workpiece, and the RGB images and depth images can be converted into point cloud data using the Open3D library.

[0073] Specifically, firstly, in a well-lit environment, a material frame containing various types of workpieces is placed on the workbench. A high-quality RGB image and depth image are captured using an industrial RGB-D camera located directly above the workbench. Then, combining the camera's intrinsic and extrinsic parameters, the RGB and depth images are converted into RGBD images using the `create_from_color_and_depth()` function in the Open3D library. Finally, the `create_from_rgbd_image()` function is used to convert the RGBD image into a point cloud, with each point represented by (x, y, z, r, g, b), where x, y, and z are coordinate information, and r, g, and b are color information.

[0074] In step 2, invalid points in the point cloud are first filtered out using methods such as voxel filtering, Gaussian filtering, statistical filtering, and uniform downsampling from the Open3D library. Then, the workbench plane is fitted using the RANSAC (Random Sample Consensus) algorithm and the plane is filtered out. The remaining point cloud data is saved as the point cloud representation of the workpiece, reducing the number of noise points in the point cloud and improving the overall quality of the point cloud.

[0075] Specifically, the `remove_statistical_outlier()` function from the Open3D library is used to filter out outliers (i.e., invalid points) from the point cloud data using statistical methods. Then, the `voxel_down_sample()` and `uniform_down_sample()` functions are used to denoise and downsample the point cloud data. The `segment_plane()` function is used to fit the workbench plane to obtain all interior points representing the workbench plane. All interior points representing the workbench plane are removed from the denoised point cloud data, and the remaining point cloud data is saved as the point cloud representation of the workpiece and saved in .pcd format.

[0076] In step 3, a point cloud annotation tool is used to annotate the preprocessed point cloud data. The annotations include the category of the workpiece and the point clouds it contains. The resulting annotation files and point cloud data are used to construct the point cloud instance segmentation dataset required for model training.

[0077] Specifically, the point cloud data was annotated using the open-source annotation tool CloudCompare to obtain annotation files. The annotation files are txt files, with each line storing the coordinates, color, and semantic category information of the points in the point cloud data. The format of the txt annotation files is the same as that of the S3DIS dataset.

[0078] In step 4, since manually labeling the dataset is inefficient and the collected and labeled point cloud data is insufficient for training the network model, data augmentation is performed on the created workpiece point cloud instance segmentation dataset to enhance its data diversity and improve the model's generalization ability and robustness. To enhance the diversity of point cloud data, data augmentation is performed on the created point cloud instance segmentation dataset, including methods such as random rotation, random scaling, random mirroring, random noise, and color transformation.

[0079] Specifically, the functions include: random rotation: randomly rotating the point cloud around a random axis; random translation: randomly translating the point cloud along the x, y, and z axes; random scaling: randomly generating a scaling factor and multiplying all point cloud coordinates by this factor to change the position and density of each point in the point cloud; random noise: generating noise values ​​within a certain range and adding these noise values ​​to all point cloud coordinates to obtain new coordinate values; random mirroring: randomly selecting the x, y, or z axis and mirroring the point cloud according to the selected axis; and color transformation: randomly generating RGB values ​​and replacing the original RGB values ​​of the points with these values ​​to change the color of the point cloud.

[0080] Example 5

[0081] Based on the above embodiments, in step 7, the trained point cloud instance segmentation model is used for inference. The workpiece point cloud data from the validation set is input, and the predicted instance categories and point cloud composition are output and displayed. The model's mAP and FPS on the created workpiece point cloud instance segmentation dataset are calculated. The segmented validation set is input into the point cloud instance segmentation model trained in step 6. After model prediction, the semantic categories and instance labels in the point cloud are output, and different colors are used to represent different instances. The point cloud is then drawn using the Open3D library.

[0082] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for segmenting workpiece point cloud instances in industrial scenarios, characterized in that, The method includes the following steps: Step 1: Perform image acquisition and data conversion to obtain point cloud data of the workpiece; Step 2: Point cloud preprocessing, filtering out invalid points and workbench planes in the point cloud data, and saving the remaining point cloud data as the point cloud representation of the workpiece; Step 3: Point cloud annotation. Instance annotation is performed on the preprocessed point cloud data to obtain an annotation file; the annotation file and the point cloud data constitute the point cloud instance segmentation dataset required for model training. Step 4: Data augmentation. Perform data augmentation on the point cloud instance segmentation dataset. Step 5: Point cloud instance segmentation model construction, including the construction of backbone network module, semantic category and offset vector prediction module, clustering module, feature re-extraction and fusion module, mask prediction and mask scoring module; Step 6: Point cloud instance segmentation model training. The data-augmented point cloud instance segmentation dataset is divided into a training set and a validation set in an 8:2 ratio. The training set is then input into the point cloud instance segmentation model for training. Step 7: Point cloud instance segmentation model prediction. Use the trained point cloud instance segmentation model for inference, input the data in the validation set, and output and display the predicted instance category and point cloud composition. Step 5 includes the following sub-steps: Step 5.1: The backbone network module extracts multi-scale point cloud features through MinkowskiNet, wherein MinkowskiNet introduces Minkowski tensors and Minkowski convolutions to construct ResNet-34 as the point cloud feature extraction network; Step 5.2: For the point cloud features extracted in Step 5.1, the semantic category and offset vector prediction module predicts the semantic category of each point in the point cloud instance segmentation dataset and the offset vector from each point to the geometric center of the instance to which that point belongs using a multi-layer MLP. The semantic category and the offset vector are respectively... peacekeeping Represented by a 3D vector. The number of points in the segmented dataset of the point cloud instance. The number of categories of the workpiece; Step 5.3: The clustering module moves each point in the point cloud instance segmentation dataset to the geometric center of the instance to which the point belongs based on the offset vector, obtains all points within a sphere with radius r centered on the geometric center of the instance, and clusters the points with the same semantic category as the geometric center of the instance into a cluster to obtain several preliminary segmentation prediction instances. Step 5.4: The feature re-extraction and fusion module uses Minkowski sparse convolution to construct ResNet-18 as the feature re-extraction network. The ResNet-18 is used to extract instance-level features of several preliminary segmentation prediction instances, and then fused with the point cloud features extracted by ResNet-34 to obtain fused features. Step 5.5: The mask prediction and mask scoring module predicts an instance-level mask based on the fusion features through two layers of MLP to filter out background points in the preliminary segmentation prediction instance; at the same time, it generates a confidence score for the final segmentation prediction instance through a mask scoring branch.

2. The workpiece point cloud instance segmentation method for industrial scenarios as described in claim 1, characterized in that, The mask scoring branch in step 5.5 is also based on the fused features. It calculates the IoU between the mask and the ground truth using a global average pooling layer and an MLP layer, outputting the confidence score of the final segmentation prediction instance. The output result is used... dimensional vector representation, where, The number of final segmentation prediction instances in the point cloud instance segmentation dataset.

3. The workpiece point cloud instance segmentation method for industrial scenarios as described in claim 1, characterized in that, In step 6, the total training loss of the point cloud instance segmentation model is... It includes: semantic branch loss, center offset loss, mask branch loss, and mask scoring loss, specifically represented as follows: in, For the semantic branch loss, cross-entropy loss is used; For the loss of the center offset, use Regression loss; For the mask branch loss, use binary cross-entropy loss; For the mask scoring loss, use Regression loss.

4. The workpiece point cloud instance segmentation method for industrial scenarios as described in claim 1, characterized in that, In step 6, the hyperparameters of the point cloud instance segmentation model are set as follows: the number of points in the training set is randomly cropped to a maximum of 250k, the Adam optimizer is used for 50 rounds of training, the batch size is set to 4, the learning rate is initialized to 0.001, and the learning rate is adjusted using the cosine annealing algorithm.

5. The workpiece point cloud instance segmentation method for industrial scenarios as described in claim 1, characterized in that, In step 1, an RGB-D camera is used to acquire RGB and depth images of the workpiece, and the RGB and depth images can be converted into point cloud data using the Open3D library.

6. The workpiece point cloud instance segmentation method for industrial scenarios as described in claim 5, characterized in that, In step 2, the `remove_statistical_outlier()` function from the Open3D library is used to filter out outliers (i.e., invalid points) from the point cloud data using statistical methods. Then, the `voxel_down_sample()` and `uniform_down_sample()` functions are used to denoise and downsample the point cloud data. The `segment_plane()` function is used to fit the workbench plane to obtain all interior points representing the workbench plane. All interior points representing the workbench plane are removed from the denoised point cloud data, and the remaining point cloud data is saved as the point cloud representation of the workpiece and saved in .pcd format.

7. The workpiece point cloud instance segmentation method for industrial scenarios as described in claim 1, characterized in that, In step 3, the point cloud data is annotated using the open-source annotation tool CloudCompare to obtain the annotation file; the annotation file is a txt annotation file, and each line stores the coordinate information, color information and semantic category information of the points in the point cloud data.

8. The workpiece point cloud instance segmentation method for industrial scenarios as described in claim 7, characterized in that, The format of the txt annotation file is the same as the annotation format of the S3DIS dataset.

9. The workpiece point cloud instance segmentation method for industrial scenarios as described in claim 1, characterized in that, In step 4, the data augmentation methods include random rotation, random translation, random scaling, random noise, random mirroring, and color transformation.

Images