Intelligent segmentation method for ceramic sanitary ware point cloud working plane

Abstract

The invention discloses an intelligent segmentation method for a ceramic sanitary ware point cloud working plane, and the method comprises the following steps: 1, obtaining ceramic sanitary ware point cloud data through a three-dimensional camera, and taking the data as input; the point cloud data comprises three-dimensional position information of the sanitary appliance surface and is the basis of subsequent processing and segmentation; 2, the segmentation model aggregates structural information through a Point Embedding module at a first layer of a coding stage; the method is high in feature learning capability and adaptability, the model can automatically learn feature representation in the point cloud, a feature extractor does not need to be designed manually, and the model can automatically adjust parameters in the training process for various ceramic sanitary ware point clouds, so that the training efficiency is improved. The method does not need to manually adjust parameters or design a special algorithm to adapt to different ceramic sanitary ware point cloud data like a traditional method.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional visual processing and robot automation, and specifically is an intelligent segmentation method for a point cloud working surface of ceramic sanitary ware. Background Art

[0002] With the development of automation technology, robots are gradually being used to replace manual labor in ceramic sanitary ware grinding and processing production lines, which can avoid physical harm to workers caused by harsh environments and high-intensity work, and can also obtain higher-precision products, improve production efficiency, and reduce production costs. However, before grinding, the robot needs to be taught manually. For small-batch customized products, the teaching time is too long and the benefits are low.

[0003] In order to segment the working surfaces with different grinding processes, guide the ceramic sanitary ware robot to automatically generate grinding trajectories and use appropriate grinding tools, and reduce the tedious manual teaching process, an intelligent segmentation method for ceramic sanitary ware point cloud working surfaces is designed. Summary of the invention

[0004] In view of the above situation, in order to overcome the defects of the prior art, the present invention provides an intelligent segmentation method for the point cloud working surface of ceramic sanitary ware, which effectively solves the problems raised by the background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: an intelligent segmentation method for a point cloud work surface of ceramic sanitary ware, comprising the following steps:

[0006] Step 1: The point cloud data of ceramic sanitary ware obtained by the 3D camera is used as input; these point cloud data contain the 3D position information of the sanitary ware surface, which is the basis for subsequent processing and segmentation;

[0007] Step 2: The segmentation model aggregates structural information through a Point Embedding module in the first layer of the encoding stage;

[0008] This module is mainly an adaptive convolution module, which automatically adjusts the convolution kernel size and weight according to the position information between point clouds to better capture the point cloud structure information at different scales;

[0009] Assuming point cloud P, the input and output feature maps of P in the convolutional layer can be represented as S and F, then for each point p i The point convolution can be defined as:

[0010] f i =M({K(p i , p j )|p j ∈N i})#(1)

[0011] Among them, N i Represents all neighborhood points, K is a kind of i and its neighboring point p j The function of the positional relationship output convolution weight, M is an aggregation function;

[0012] Step 3: The Point transformer module is built based on multi-head self-attention;

[0013] The self-attention operator at the core of the Transformer network is essentially a set operator, which is invariant to the arrangement and quantity of the input data, and has inherent output invariance to the input ceramic sanitary ware point cloud; Transformer also has the characteristics of parallel computing and maximum short path, which enables the model to simultaneously establish the most relevant relationship between each point and all other points, further improving the efficiency of the model; this module integrates multi-head attention, residual connection, regularization and feedforward network; the idea of ​​Transformer is applied to point clouds. Since the input of point cloud data itself contains three-dimensional position coordinates, it is naturally suitable for position. Instead of using operations such as sine and cosine commonly used in NLP or image processing to construct position encoding, the position encoding is designed as a learnable parameter δ based on the three-dimensional coordinate information, forming an attention operation applied to the point cloud;

[0014] δ=θ(h i -h j )#(2)

[0015] In the above formula, θ is a function composed of two linear layers and a ReLU nonlinear MLP, h i and h j Represents the three-dimensional coordinates of two points in the point cloud;

[0016] Step 4: Downsample is a downsampling module that reduces the number of point clouds that the deep network needs to calculate by farthest pointsampling, and retains the feature information of the point cloud to the greatest extent;

[0017] The steps of the farthest point sampling method are as follows:

[0018] 1) Randomly select a point in the point cloud as the selected sampling point;

[0019] 2) Calculate the distance d between each point in the unsampled point set and the selected sampling point set, and add the point with the largest distance to the selected sampling point set;

[0020] 3) Update the distance d and iterate until the target number of sampling points is reached;

[0021] Step 5: Upsample is an upsampling module; for the points that have been feature extracted, linear interpolation is used to propagate this feature to the points lost in the downsampling process, thereby predicting the category label of the point cloud set;

[0022] Step 6: Use the mean interaction ratio mIoU (mean Intersection over Union) as the evaluation index for the segmentation of the sanitary ware operation surface;

[0023] IoU is to calculate the interaction ratio of two point cloud sets, that is, the overlap ratio of the real point cloud set and the predicted point cloud set, while mIoU is to calculate the IoU of each work surface separately and then take the average value;

[0024]

[0025] Among them, TP (True Positive) means that the prediction is a positive sample and the prediction result is correct, FP (False Positive) means that the prediction is a positive sample but the prediction result is wrong, and FN (False Negative) means that the prediction is a negative sample and the prediction result is wrong. ij Expressed as predicting category i as category j, the expression of mIoU is as follows:

[0026]

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] 1. The model has strong feature learning ability and strong adaptability. It can automatically learn the feature representation in the point cloud without manually designing the feature extractor. For a wide variety of ceramic sanitary ware point clouds, the model can automatically adjust the parameters during the training process. There is no need to manually adjust the parameters or design a special algorithm to adapt to different ceramic sanitary ware point cloud data like traditional methods.

[0029] 2. End-to-end training: The model can learn the target function directly from the original ceramic sanitary ware point cloud through end-to-end training, without the need to design multiple stages of segmentation operations as in traditional methods, thus simplifying the entire segmentation process;

[0030] 3. Compared with traditional methods, this method has stronger generalization ability and adaptability in the segmentation of ceramic sanitary ware point cloud work surface, with higher segmentation accuracy, which meets the needs of visual guidance of polishing robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0032] In the attached picture:

[0033] Figure 1 It is the overall framework flow chart of the present invention;

[0034] Figure 2 Provides a network framework for segmenting ceramic sanitary ware point cloud work surfaces;

[0035] Figure 3 The point cloud model of ceramic sanitary ware is obtained by using a 3D camera;

[0036] Figure 4 This is the effect diagram after segmenting the above original point cloud using the trained model; DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0038] The present invention relates to an intelligent segmentation method for a ceramic sanitary ware point cloud work surface, comprising the following steps:

[0039] Step 1: The point cloud data of ceramic sanitary ware obtained by the 3D camera is used as input; these point cloud data contain the 3D position information of the sanitary ware surface, which is the basis for subsequent processing and segmentation;

[0040] Step 2: The segmentation model aggregates structural information through a Point Embedding module in the first layer of the encoding stage;

[0041] This module is mainly an adaptive convolution module, which automatically adjusts the convolution kernel size and weight according to the position information between point clouds to better capture the point cloud structure information at different scales;

[0042] Assuming point cloud P, the input and output feature maps of P in the convolutional layer can be represented as S and F, then for each point p i The point convolution can be defined as:

[0043] f i =M({K(p i , p j )|p j ∈N i})#(1)

[0044] Among them, N i Represents all neighborhood points, K is a kind of i and its neighboring point p j The function of the positional relationship output convolution weight, M is an aggregation function;

[0045] Step 3: The Point transformer module is built based on multi-head self-attention;

[0046] The self-attention operator at the core of the Transformer network is essentially a set operator, which is invariant to the arrangement and quantity of the input data, and has inherent output invariance to the input ceramic sanitary ware point cloud; Transformer also has the characteristics of parallel computing and maximum short path, which enables the model to simultaneously establish the most relevant relationship between each point and all other points, further improving the efficiency of the model; this module integrates multi-head attention, residual connection, regularization and feedforward network; the idea of ​​Transformer is applied to point clouds. Since the input of point cloud data itself contains three-dimensional position coordinates, it is naturally suitable for position. Instead of using operations such as sine and cosine commonly used in NLP or image processing to construct position encoding, the position encoding is designed as a learnable parameter δ based on the three-dimensional coordinate information, forming an attention operation applied to the point cloud;

[0047] δ=θ(h i -h j )#(2)

[0048] In the above formula, θ is a function composed of two linear layers and a ReLU nonlinear MLP, h i and h j Represents the three-dimensional coordinates of two points in the point cloud;

[0049] Step 4: Downsample is a downsampling module that reduces the number of point clouds that the deep network needs to calculate by farthest pointsampling, and retains the feature information of the point cloud to the greatest extent;

[0050] The steps of the farthest point sampling method are as follows:

[0051] 1) Randomly select a point in the point cloud as the selected sampling point;

[0052] 2) Calculate the distance d between each point in the unsampled point set and the selected sampling point set, and add the point with the largest distance to the selected sampling point set;

[0053] 3) Update the distance d and iterate until the target number of sampling points is reached;

[0054] Step 5: Upsample is an upsampling module; for the points that have been feature extracted, linear interpolation is used to propagate this feature to the points lost in the downsampling process, thereby predicting the category label of the point cloud set;

[0055] Step 6: Use the mean interaction ratio mIoU (mean Intersection over Union) as the evaluation index for the segmentation of the sanitary ware operation surface;

[0056] IoU is to calculate the interaction ratio of two point cloud sets, that is, the overlap ratio of the real point cloud set and the predicted point cloud set, while mIoU is to calculate the IoU of each work surface separately and then take the average value;

[0057]

[0058] Among them, TP (True Positive) means that the prediction is a positive sample and the prediction result is correct, FP (False Positive) means that the prediction is a positive sample but the prediction result is wrong, and FN (False Negative) means that the prediction is a negative sample and the prediction result is wrong. ij Expressed as predicting category i as category j, the expression of mIoU is as follows:

[0059]

[0060] like Figure 1 As shown, the overall system of the present invention consists of two parts: offline and online:

[0061] 1) The offline part mainly consists of data collection, data set construction, segmentation network construction and network training. Point cloud data is mainly obtained by scanning physical sanitary ware with a 3D camera. The scanned data is manually labeled to construct a data set, and data enhancement is used to expand the samples. In view of the characteristics of the point cloud work surface of ceramic sanitary ware, the segmentation network is improved based on transformer;

[0062] 2) In the online part, the segmentation network model pre-trained in the offline part is used to segment the working surface of the ceramic sanitary ware point cloud obtained by the 3D camera in real time, and the segmentation results are analyzed to generate guidance information to guide the robot to perform polishing;

[0063] like Figure 2 As shown, the framework of the segmentation model:

[0064] 1) The main framework of the model can be divided into two parts: encoding and decoding. In the encoding stage, the point set is downsampled, the sampling frequency is [1, 4, 4, 4], the number of input point sets is N, and the cardinality of the point set in the encoding stage changes to [N, N / 4, N / 16, N / 64, N / 256];

[0065] 2) For semantic segmentation, upsampling is used in the decoding stage to segment the original point cloud based on the feature points obtained from downsampling, and the point set changes to [N / 256, N / 64, N / 16, N / 4, N]. For classification, global averaging is performed after downsampling to obtain feature points, and the cardinality becomes 1;

[0066] 3) In the first layer of the encoding stage, a Point Embedding module aggregates the structural information, followed by a Point Transformer module to establish the most relevant relationship between the center point and the neighboring points and ensure the output invariance; the downsampling layer in the encoding stage consists of a Downsample module and a Point Transformer module; in the decoding stage, in order to reduce the amount of calculation, the upsampling layer does not use the Transformer module;

[0067] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0068] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent segmentation method for ceramic sanitary ware point cloud work surface, comprising the following steps: Step 1: The point cloud data of ceramic sanitary ware obtained by the 3D camera is used as input; these point cloud data contain the 3D position information of the sanitary ware surface, which is the basis for subsequent processing and segmentation; Step 2: The segmentation model aggregates structural information through a Point Embedding module in the first layer of the encoding stage; This module is mainly an adaptive convolution module, which automatically adjusts the convolution kernel size and weight according to the position information between point clouds to better capture the point cloud structure information at different scales; Assuming point cloud P, the input and output feature maps of P in the convolutional layer can be represented as S and F, then for each point p i The point convolution can be defined as: f i =M({K(p i ,p j )|p j ∈N i })#(1) Among them, N i Represents all neighborhood points, K is a kind of i and its neighboring point p j The function of the positional relationship output convolution weight, M is an aggregation function; Step 3: The Point transformer module is built based on multi-head self-attention; The self-attention operator at the core of the Transformer network is essentially a set operator, which is invariant to the arrangement and quantity of the input data, and has inherent output invariance to the input ceramic sanitary ware point cloud; Transformer also has the characteristics of parallel computing and maximum short path, which enables the model to simultaneously establish the most relevant relationship between each point and all other points, further improving the efficiency of the model; this module integrates multi-head attention, residual connection, regularization and feedforward network; the idea of ​​Transformer is applied to point clouds. Since the input of point cloud data itself contains three-dimensional position coordinates, it is naturally suitable for position. Instead of using operations such as sine and cosine commonly used in NLP or image processing to construct position encoding, the position encoding is designed as a learnable parameter δ based on the three-dimensional coordinate information, forming an attention operation applied to the point cloud; δ=θ(h i -h j )#(2) In the above formula, θ is a function composed of two linear layers and a ReLU nonlinear MLP, h i and h j Represents the three-dimensional coordinates of two points in the point cloud; Step 4: Downsample is a downsampling module that reduces the number of point clouds that the deep network needs to calculate through farthest point sampling, and retains the feature information of the point cloud to the greatest extent; The steps of the farthest point sampling method are as follows: 1) Randomly select a point in the point cloud as the selected sampling point; 2) Calculate the distance d between each point in the unsampled point set and the selected sampling point set, and add the point with the largest distance to the selected sampling point set; 3) Update the distance d and iterate until the target number of sampling points is reached; Step 5: Upsample is an upsampling module; for the points that have been feature extracted, linear interpolation is used to propagate this feature to the points lost in the downsampling process, thereby predicting the category label of the point cloud set; Step 6: Use the mean interaction ratio mIoU (mean Intersection over Union) as the evaluation index for the segmentation of the sanitary ware operation surface; IoU is to calculate the interaction ratio of two point cloud sets, that is, the overlap ratio of the real point cloud set and the predicted point cloud set, while mIoU is to calculate the IoU of each work surface separately and then take the average value; Among them, TP (True Positive) means that the prediction is a positive sample and the prediction result is correct, FP (False Positive) means that the prediction is a positive sample but the prediction result is wrong, and FN (False Negative) means that the prediction is a negative sample and the prediction result is wrong. ij Expressed as predicting category i as category j, the expression of mIoU is as follows:

Images