A weakly supervised point cloud segmentation method and system for robot assembly

Abstract

The present invention discloses a weakly supervised point cloud segmentation method and system for robot assembly, including: inputting a point cloud, converting an original image into a standardized input, building a model for the image, and providing coordinate and RGB image inputs for the model; encoding based on the features of the input standardized input; designing a central attention mechanism based on the Transformer algorithm, learning point cloud features by extracting global features of neighboring points and sharing them among various neighborhoods; classifying the point cloud based on the result learned by the central attention mechanism, and outputting a final result. It realizes point cloud segmentation at a low manual annotation cost, thereby making training more efficient and the cost lower.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent manufacturing, and in particular, to a weakly supervised point cloud segmentation method and system for robot assembly. Background Art

[0002] In industrial manufacturing, the ability to successfully grasp an object is an essential skill for a robot. With the development of Industry 5.0, robots already have multi-degree-of-freedom, precise perception, and control capabilities, and can perform tasks autonomously or in cooperation with workers on the production line. Currently, vision-based deep learning and reinforcement learning can achieve certain results in robot object grasping.

[0003] However, during the process of implementing the technical solutions of the present invention by the inventors of this application, it is found that the above technologies have at least the following technical problems:

[0004] In modeling the assembly posture of a robot, researchers use a vision method based on point cloud to learn the robot posture, and point cloud segmentation plays an important role in the understanding of the three-dimensional posture of the robot. However, in the scenario of an intelligent factory, the cost of manually annotating the assembly posture of a robot is high and time-consuming. How to solve this problem is a challenge. Summary of the Invention

[0005] The embodiments of the present application provide a weakly supervised point cloud segmentation method and system for robot assembly, which solve the technical problem that in the scenario of an intelligent factory in the prior art, the cost of manually annotating the assembly posture of a robot is high and time-consuming, and achieve point cloud segmentation with low manual annotation cost, thereby making the training more efficient and the cost lower.

[0006] The embodiments of the present application provide a weakly supervised point cloud segmentation method for robot assembly, including:

[0007] S1, input the point cloud, convert the original image into a standardized input, build a model for the image, and provide coordinate and RGB image inputs for the model;

[0008] S2, encode based on the features of the input standardized input;

[0009] S3, based on the Transformer algorithm, design a central attention mechanism, learn point cloud features by extracting global features of neighboring points and sharing them among various neighborhoods, and obtain two embedded global features and a position encoding module;

[0010] S4, classify the point cloud based on the result of the central attention mechanism learning, and output the final result.

[0011] Further, in step S3, it includes:

[0012] S31. Based on the conversion of the original image into a standardized form, coordinates and features are obtained. After inputting the standardized coordinates and features into a multi-layer perceptron, after passing through one layer of MLP, features are obtained. and features , , The following formula:

[0013] ;

[0014] ;

[0015] Among them, F d， P d is the initial feature of the input;

[0016] S32. The central attention mechanism learns on and .

[0017] Furthermore, the central attention mechanism further includes:

[0018] For each center point , its feature is passed through a linear layer , the has a dimension of 1; at the same time, the k neighbor point coordinates of this center point are obtained using the KNN algorithm, , as well as the features corresponding to the k neighbor points. Then, the global feature is extracted by integrating the central weight and the neighbor point features through the first embedding, as shown in the following formula:

[0019] ;

[0020] Among them, is the feature at the coordinate (i, j), is the linear layer of the first embedding, is the feature at point i, P ij is P i 's neighbor point coordinates, and N and C are two neighbor points of K.

[0021] Furthermore, the central attention mechanism further includes:

[0022] Based on the global feature after the first embedding, the second embedding is obtained, as shown in the following formula:

[0023] ;

[0024] Among them, is the global feature after the second embedding, is the global feature after the first embedding.

[0025] Further, in step S4, it also includes:

[0026] The input for classifying the point cloud is the output coordinates of the last central learning module and features , and using a fully connected layer and a ReLU activation function, the classification information Y of the point cloud image is output d , as shown in the following formula:

[0027] ;

[0028] where, is the fully connected layer, is the activation function, is the learning vector.

[0029] A weakly supervised point cloud segmentation system for robot assembly includes,

[0030] A data preprocessing module for inputting point clouds, converting the original pictures into standardized inputs, building a model for the pictures, and providing coordinate and RGB image inputs for the model;

[0031] A downsampling module for encoding based on the features of the input standardized inputs;

[0032] A central learning module, based on the Transformer algorithm, for designing a central attention mechanism, learning point cloud features by extracting global features of neighboring points and sharing them among various neighborhoods, and obtaining two embedded global features and a position encoding module;

[0033] A classification module for classifying the point cloud based on the results of the central attention mechanism learning and outputting the final result.

[0034] Further, in the central learning module, it includes:

[0035] Based on the conversion of the original picture into standardization, coordinates and features are obtained, the input standardized coordinates and features are input, and in the multi-layer perceptron, after passing through one layer of MLP, features and are obtained, where , as shown in the following formula:

[0036] ;

[0037] ;

[0038] where, Fd , P d is the initial feature of the input;

[0039] Then, the central attention mechanism learns on and .

[0040] Furthermore, in the central learning module, it includes:

[0041] For each center point , its feature passes through a linear layer , the dimension is 1; meanwhile, the k neighbor point coordinates of this center point are obtained using the KNN algorithm , as well as the features corresponding to the k neighbor points, and then the global feature is extracted by integrating the center weight and the neighbor point features through the first embedding, as shown in the following formula:

[0042] ;

[0043] Among them, is the feature at the coordinate (i, j), is the linear layer of the first embedding, is the feature at point i, P ij is P i 's neighbor point coordinates, N and C are two neighbor points of K.

[0044] Furthermore, in the central learning module, it includes:

[0045] Based on the global feature after the first embedding, the second embedding is obtained, as shown in the following formula:

[0046] ;

[0047] Among them, is the global feature after the second embedding, is the global feature after the first embedding.

[0048] Furthermore, in the classification module, it includes:

[0049] The input for classifying the point cloud is the output coordinates and features of the last central learning module. Using a fully connected layer and a ReLU activation function, the classification information Y d of the point cloud image is output, as shown in the following formula:

[0050] ;

[0051] Among them, is a fully connected layer, is an activation function, is the learning vector.

[0052] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0053] The present invention aims to solve the above problems by designing a center-based attention mechanism and a Transformer architecture, enhancing the feature representation of unlabeled points through two embedding processes and a position encoding module, thereby improving the performance of point cloud segmentation in a weakly supervised setting, achieving point cloud segmentation with low manual annotation cost, making the training more efficient and the cost lower. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flowchart of a weakly supervised point cloud segmentation method for robot assembly;

[0055] Figure 2 is a schematic diagram of the sampling module;

[0056] Figure 3 is a schematic diagram of the center learning module;

[0057] Figure 4 is the center attention mechanism;

[0058] Figure 5 is a system diagram of a weakly supervised point cloud segmentation for robot assembly. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0059] The invention is a weakly supervised point cloud segmentation method and system for robot assembly. The core technology is to design a center learning module based on Transformer, extract the global features of the center points, and then share them among each neighborhood. By sharing the weights of the center points, the local features are enhanced, and at the same time, the global features are retained, better completing the point cloud feature learning in the case of sparse annotation. At the same time, position encoding is also introduced into the attention mechanism to supplement the geometric features according to the positions of the center points and their neighbor points. The invention designs a downsampling module, introducing the farthest point sampling method and pooling operation to better learn the features of different center points.

[0060] In order to better understand the above technical solutions, the above technical solutions will be described in detail below in conjunction with the accompanying drawings of the specification and specific embodiments.

[0061] See Figure 1 , a weakly supervised point cloud segmentation method for robot assembly, including,

[0062] S1. Input the point cloud, convert the original image into a standardized input, build a model for the image, and provide coordinates and RGB image input for the model;

[0063] Specifically, convert the original image into a standardized input. Format the three-dimensional coordinates of the object into , where is the coordinate of the th point, is the number of points. Format the RGB features of the object into , where is the RGB feature of the th point, is the number of points.

[0064] S2. Encode based on the features of the input standardized input;

[0065] See Figure 2 , specifically, the input is the standardized coordinates and features . First, this module uses the farthest point sampling algorithm (a well-known method) to sample N points. Then, use the KNN algorithm to cluster the sampled points and select K nearest neighbors for each point . Then, through a layer of MLP (multi-layer perceptron) and a max pooling layer to aggregate the features of these k neighbors as the output of this point of this module and , as shown in Equation 1:

[0066] (1);

[0067] where are the K nearest neighbor features corresponding to each point, is the multi-layer perceptron, is the max pooling layer.

[0068] S3. Based on the Transformer algorithm, design a central attention mechanism to learn the point cloud features by extracting the global features of neighboring points and sharing them among different neighborhoods, and obtain two embedded global features and a position encoding module;

[0069] Specifically, based on the Transformer algorithm, design a central attention mechanism to better learn the point cloud features by extracting the global features of neighboring points and then sharing them among different neighborhoods. The process of this module is as Figure 3 shown, and the process of the central attention mechanism is as Figure 4 shown.

[0070] First, the inputs of this module are and , and after passing through a layer of MLP, the features and are obtained. , The following is the formula:

[0071] ;

[0072] ;

[0073] where F d , P d are the initial features of the input;

[0074] Secondly, the central attention mechanism learns on and as follows.

[0075] The central attention mechanism is: for each center point , the present invention passes its feature through a linear layer , the dimension is 1; meanwhile, the k neighbor point coordinates of this center point are obtained by using the KNN algorithm, , and the features corresponding to the k neighbor points, and then the global features are extracted by integrating the central weight and the neighbor point features through the first embedding, as shown in Formula 3:

[0076] (3);

[0077] P ij is the neighbor point coordinate of P i , and N and C are two neighbor points of K.

[0078] where is the feature at the coordinate (i, j), is the linear layer of the first embedding, is the feature at point i.

[0079] The representation of the unlabeled neighboring points in the weakly supervised point cloud is improved by the global features, but the key local features of the neighborhood are often lacking. Based on the global features after the first embedding, the second embedding is obtained, as shown in Formula 4:

[0080] (4);

[0081] where is the global feature after the second embedding, is the global feature after the first embedding.

[0082] Through the above global features the central point feature is shared with the corresponding neighborhood. This method regards the global feature of the central point as the weight of its neighborhood, and ensures that the central point feature is effectively propagated to each point in its corresponding neighborhood through matrix multiplication.

[0083] To construct the attention weight covering the global feature and the central point neighborhood, this paper combines the above two embeddings with positional encoding, as shown in Equation (5):

[0084] (5);

[0085] where, is matrix dot product, the global feature is transformed through (linear layer), and is integrated using the learnable parameters , and . This integration enhances the local features through the central point sharing method while retaining the global features. In addition, the positional encoding is introduced into the attention weight to supplement the geometric features according to the point position, and the calculation methods are shown in Equations (6), (7), and (8):

[0086] (6);

[0087] (7);

[0088] (8);

[0089] where, is the elevation angle, is the azimuth angle, is the coordinate difference between the neighbor point and the central point, is the Euclidean distance, is the MLP (Multi-Layer Perceptron), is the bias parameter, is the learnable parameter.

[0090] After introducing the positional encoding, the output feature of each central point is finally obtained, as shown in Equation (9):

[0091] (9);

[0092] where, is the Softmax function, is the transpose of, is the linear layer, represents the matrix dot product.

[0093] After learning the features of the center point and its neighbors through the central attention mechanism and then using the MLP (Multi-Layer Perceptron) to learn the vector to obtain the output of this module as shown in Equation 10:

[0094] (10);

[0095] S4. Classify the point cloud based on the results of the central attention mechanism learning and output the final result.

[0096] Specifically, the input for classification is the output of the last central learning module and using a fully connected layer and a ReLU activation function to output the classification information Y of the point cloud image d as shown in Equation 11:

[0097] (11);

[0098] where, is the fully connected layer, is the activation function, is the learning vector.

[0099] The local features are enhanced by the method of sharing weights at the center point, while the global features are retained, better completing the point cloud feature learning in the case of sparse annotation. At the same time, position encoding is also introduced into the attention mechanism to supplement geometric features according to the positions of the center point and its neighbor points.

[0100] A weakly supervised point cloud segmentation system for robot assembly, including,

[0101] The data preprocessing module 01 is used to input the point cloud, convert the original image into a standardized input, build a model for the image, and provide coordinates and RGB image input for the model;

[0102] Specifically, this module converts the original image into a standardized input. Format the three-dimensional coordinates of the object into where is the coordinate of the th point, is the number of points. Format the RGB features of the object into where is the The RGB features of the points, is the number of points.

[0103] The downsampling module 02 is used to encode based on the features of the input normalized input;

[0104] See Figure 2 , specifically, the input of the downsampling module is the normalized coordinates and features . First, this module uses the farthest point sampling algorithm (a well-known method) to sample N points. Then, the KNN algorithm is used to cluster the sampled points, and K nearest neighbors are selected for each point . Then, through a layer of MLP (Multi-Layer Perceptron) and a max pooling layer to aggregate the features of these k neighbors as the output of this module for this point and , as shown in Equation 1:

[0105] (1);

[0106] Among them, are the K nearest neighbor features corresponding to each point, is the Multi-Layer Perceptron, is the max pooling layer.

[0107] The center learning module 03, based on the Transformer algorithm, is used to design a center attention mechanism to learn point cloud features by extracting the global features of neighboring points and sharing them among different neighborhoods, obtaining two embedded global features and a position encoding module;

[0108] Specifically, based on the Transformer algorithm, a center attention mechanism is designed to better learn point cloud features by extracting the global features of neighboring points and then sharing them among different neighborhoods. The process of this module is as Figure 3 shown, and the process of the center attention mechanism is as Figure 4 shown.

[0109] First, the input of this module is and . After passing through a layer of MLP, the features and are obtained, , as follows:

[0110] ;

[0111] ;

[0112] Among them, F d， Pd is the initial feature of the input;

[0113] Secondly, the central attention mechanism learns on and as follows.

[0114] The central attention mechanism is: for each center point , the present invention transforms its feature through a linear layer , where the dimension is 1; meanwhile, the coordinates of k neighbor points of this center point are obtained by using the KNN algorithm , as well as the features corresponding to the k neighbor points. Then, the global feature is extracted by integrating the central weight and the neighbor point features through the first embedding, as shown in Equation (3):

[0115] (3);

[0116] Although the representation of unlabeled neighboring points in the weakly supervised point cloud is improved by the global feature, the key local features of the neighborhood are often lacking. Therefore, based on the global feature after the first embedding, the second embedding is obtained, as shown in Equation (4):

[0117] (4);

[0118] By sharing the center point feature with the corresponding neighborhood through the above global feature , this method regards the global feature of the center point as the weight of its neighborhood, and ensures that the center point feature is effectively propagated to each point in its corresponding neighborhood through matrix multiplication.

[0119] In order to construct the attention weight covering the global feature and the center point neighborhood, this paper combines the above two embeddings with the position encoding, as shown in Equation (5):

[0120] (5);

[0121] where, is matrix dot multiplication, the global feature is transformed through (linear layer), and is integrated using the learnable parameters , and . This integration enhances the local features while retaining the global features through the method of center point sharing. In addition, the position encoding It is introduced into the attention weight to supplement geometric features according to the point position, and the calculation methods are shown in Formulas 6, 7, and 8:

[0122] (6);

[0123] (7);

[0124] (8);

[0125] where, is the elevation angle, is the azimuth angle, is the coordinate difference between the neighbor point and the center point, is the Euclidean distance, is the MLP (Multi-Layer Perceptron), is the bias parameter, is the learnable parameter.

[0126] After introducing the position encoding, the output feature of each center point is finally obtained, as shown in Formula 9:

[0127] (9);

[0128] where, is the Softmax function, is the transpose of, is the linear layer, represents matrix dot multiplication.

[0129] After learning the features of the center point and its neighbors through the central attention mechanism again, the MLP (Multi-Layer Perceptron) is used to learn the vector to obtain the output of this module, as shown in Formula 10:

[0130] (10);

[0131] The classification module 04 classifies the point cloud based on the results learned by the central attention mechanism and outputs the final result.

[0132] Specifically, the input of the classification module 04 is the output of the last central learning module, and a fully connected layer and a ReLU activation function are used to output the classification information Y d of the point cloud image, as shown in Formula 11:

[0133] (11);

[0134] Among them, is a fully connected layer, is an activation function, is a learning vector.

[0135] The method of sharing weights through the center point enhances local features while retaining global features, better completing the learning of point cloud features in the case of sparse annotation. At the same time, position encoding is also introduced into the attention mechanism to supplement geometric features according to the positions of the center point and its neighboring points.

[0136] By designing a center-based attention mechanism and a Transformer architecture to solve the above problems, the feature representation of unlabeled points is enhanced through two embedding processes and a position encoding module, thereby improving the performance of point cloud segmentation in a weakly supervised setting.

[0137] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0138] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for realizing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0139] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured product including an instruction device, and the instruction device realizes the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0140] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus, so that a series of operational steps are performed on the computer or other programmable apparatus to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one process or a plurality of processes and / or blocks Figure 1 one process or a plurality of processes and / or blocks Figure 1 or steps for implementing the functions specified in one block or a plurality of blocks.

[0141] Although the preferred embodiments of the present invention have been described, additional changes and modifications can be made by those skilled in the art once they learn of the basic inventive concept. Therefore, the appended claims are intended to be construed to cover the preferred embodiments as well as all changes and modifications falling within the scope of the present invention.

[0142] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these modifications and variations.

Claims

1. A weakly supervised point cloud segmentation method for robot assembly, characterized in that, including S1, input point cloud, convert the original image into a standardized input, build a model for the image, and provide coordinate and RGB image inputs for the model; S2, encode based on the features of the input standardized input; S3. Based on the Transformer algorithm, design a central attention mechanism. By extracting the global features of neighboring points and sharing them among various neighborhoods, learn the point cloud features to obtain two embedded global feature and position encoding modules; based on the conversion of the original image into a standardized form, obtain coordinates and features. Input the standardized coordinates and features. In the multi-layer perceptron, after passing through one layer of MLP, obtain feature P i and feature F i , where F i , P i The following formula: F i = MLP(F d ); P i = MLP(P d ); Among them, F d , P d is the initial feature of the input; The central attention mechanism learns about P i and F i ; For each center point p i , its feature f i is passed through a linear layer g1 with dimension 1; meanwhile, the K nearest neighbor point coordinates P i of the center point p ij ={p ij : i ∈ N, j ∈ K} ∈ R N×K×3 are obtained using the KNN algorithm, as well as the features F ij ={f ij : i ∈ N, j ∈ K} ∈ R N×K×C corresponding to the K nearest neighbor points. Then, the global feature is extracted by integrating the center weight and the neighboring point features through the first embedding, as shown in the following formula: where σ is the Softmax function, and f ij is the feature at coordinate (i, j), g1 is the linear layer of the first embedding, and f i is the feature at point i, p ij is p i 's neighbor point coordinates, N and C are two neighbor points of K; based on the global feature e1 after the first embedding, the second embedding is obtained as shown in the following formula: e2 = f i × e1; wherein, e2 is the global feature after the second embedding, and e1 is the global feature after the first embedding; S4, classify the point cloud based on the result of learning by the central attention mechanism, and output the final result.

2. The weakly supervised point cloud segmentation method for robot assembly according to claim 1, wherein, In step S4, it further includes: The input for classifying the point cloud is the output coordinates P of the last central learning module d and the feature F d , using a fully connected layer and a ReLU activation function, the classification information Y of the point cloud image is output d , as shown in the following formula: Y d = ReLU(Fullyconnect(F′ d )); Among them, Fullyconnect is the fully connected layer, ReLU is the activation function, and F′ d is the learning vector.

3. A weakly supervised point cloud segmentation system for robot assembly, characterized in that, including a data preprocessing module, used for inputting point cloud, converting the original image into a standardized input, building a model for the image, and providing coordinate and RGB image inputs for the model; a downsampling module, used for encoding based on the features of the input standardized input; The central learning module, based on the Transformer algorithm, is used to design a central attention mechanism. By extracting the global features of neighboring points and sharing them among various neighborhoods, it learns the point cloud features and obtains two embedded global features and a position encoding module; based on the conversion of the original image into a standardized form, coordinates and features are obtained. After inputting the standardized coordinates and features, in the multi-layer perceptron, after passing through one layer of MLP, features P i and feature F i are obtained, where F i , P i are as follows: F i = MLP(F d ); P i = MLP(P d ); Among them, F d , P d is the initial feature of the input; The central attention mechanism learns about P i and F i for learning; For each center point p i , its feature f i passes through a linear layer g1 with a dimension of 1; meanwhile, the K nearest neighbor point coordinates P i of the center point p are obtained using the KNN algorithm ij ={p ij : i ∈ N, j ∈ K} ∈ R N×K×3 , and the features F ij corresponding to the K nearest neighbor points ij ={f N×K×C : i ∈ N, j ∈ K} ∈ R. Then, the global feature is extracted by integrating the center weight and the neighboring point features through the first embedding, as shown in the following formula: where σ is the Softmax function, and f ij is the feature at coordinates (i, j), g1 is the linear layer of the first embedding, and f i is the feature at point i, p ij is p i coordinates of the neighbor points of, N and C are two neighbor points of K; based on the global feature e1 after the first embedding, the second embedding is obtained as shown in the following formula: e2 = f i × e1; wherein, e2 is the global feature after the second embedding, and e1 is the global feature after the first embedding; a classification module, which classifies the point cloud based on the result of learning by the central attention mechanism and outputs the final result.

4. The weakly supervised point cloud segmentation system for robot assembly according to claim 3, wherein, In the classification module, it includes: The input for classifying the point cloud is the output coordinates P of the last central learning module d and the feature F d , and using a fully connected layer and a ReLU activation function, the classification information Y of the point cloud image is output d , as shown in the following formula: Y d = ReLU(Fullyconnect(F′ d )); Among them, Fullyconnect is the fully connected layer, ReLU is the activation function, and F′ d is the learning vector.

Images