Single-hand saliency map and few double-hand annotation guided double-hand saliency analysis method

Abstract

The present application belongs to the field of point cloud processing, and relates to a single-hand saliency map and a few double-hand annotation guided double-hand saliency analysis method. The present application first generates a double-hand saliency map by using a single-hand grasping saliency map and single-double-hand grasping correlation. First, the point cloud of an object is input into a single-hand saliency map prediction model to obtain a single-hand saliency map, which is used as an initial value of a double-hand saliency map. Second, a double-hand grasping contact point dataset is constructed, and a corresponding relationship between left and right hand grasping annotation points and saliency is established. Based on the corresponding relationship and the initial value, a double-hand grasping saliency map learning network named BSPN is designed to generate an object double-hand saliency map. An iterative training strategy is further proposed to update the initial value of the double-hand saliency map. Finally, the point cloud is input into a contact point prediction network trained with the assistance of BSPN to obtain double-hand grasping contact points. The present application is evaluated by using an existing grasping saliency dataset, and a human-like double-hand grasping gesture is successfully generated.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of point cloud processing, and particularly relates to a single-hand saliency map and a few-hand annotation guided double-hand saliency analysis method. BACKGROUND

[0002] Grasp saliency reflects the tendency of human to select the position when grasping the surface of a three-dimensional object. For a point on the surface of an object, the higher the grasp saliency value is, the higher the probability that the human contacts the point when grasping. The grasp saliency value is a quantification of human grasping experience and is one of the important geometric features in the field of computer vision.

[0003] Based on human grasping experience, analyzing the grasp saliency of an object is an important content of computer vision research and has important application value in the field of robots. Grasp saliency analysis of a three-dimensional model refers to taking an object as the core, taking grasping function as the guide, obtaining the saliency region and specific contact point coordinates of the surface of the three-dimensional object that can serve the grasping task based on a geometric feature analysis method, so as to guide the robot to obtain a high-quality grasping result. At present, the grasp saliency analysis method mainly focuses on single-hand grasping tasks and uses a deep neural network to realize grasp saliency analysis of a three-dimensional point cloud model. The network is composed of an encoder and a decoder, and the specific design is determined by the actual task. The encoder is generally composed of multiple layers of perceptrons, inputs the three-dimensional coordinates of the point cloud, and outputs the high-dimensional geometric features of each point in the point cloud related to the task. The decoder is generally composed of fully connected layers, inputs the high-dimensional geometric features extracted by the previous stage encoder, and outputs the grasp saliency value of each point. In the training process, the accuracy of the grasp saliency value prediction is supervised by the true value, which is a supervised learning task.

[0004] The current grasp saliency analysis method has the following problems: (1) the saliency database guided by human grasping experience is collected in the case of single-hand grasping, and lacks true value data of double-hand grasping saliency; due to the lack of true value data, the existing supervised grasp saliency analysis method can only predict single-hand saliency and cannot calculate the grasp saliency value for double-hand grasping tasks. (2) the existing method can only generate a grasp saliency map to reflect the general grasping region; however, the robot grasping needs specific contact points, and the saliency map cannot directly guide the actual robot grasping task. SUMMARY

[0005] The present application mainly aims at improving the above-mentioned deficiencies of the grasp saliency analysis method, and proposes a single-hand saliency map and a small amount of double-hand annotation guided double-hand saliency analysis method, which takes the coordinate matrix of the model and the single-hand saliency map as input, takes the single-hand saliency map as the initial value of the double-hand saliency map, uses the double-hand grasp annotation to obtain the corresponding relationship of the left and right hand grasp regions, and constructs the corresponding relationship into a corresponding vector in the Euclidean space, adjusts the initial value to the double-hand saliency map based on the corresponding vector, and uses an iterative updating strategy during training, so that the initial value can change according to the double-hand grasp annotation during the training process.

[0006] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0007] The single-hand saliency map and the small amount of double-hand annotation guided double-hand saliency analysis method comprises the following steps:

[0008] Step S1: based on a small amount of user-labeled double-hand grasp annotation, constructing left and right hand contact position labels and corresponding vector true values of left and right hand contact points;

[0009] Step S2: inputting the three-dimensional point cloud into a pre-trained single-hand saliency value prediction module to obtain a single-hand grasp saliency map of the model, and taking the single-hand grasp saliency map as a double-hand saliency initial value;

[0010] Further, the step S2 is specifically:

[0011] Step S21: using an existing single-hand grasp saliency database to pre-train the single-hand grasp saliency value prediction module, and setting hyperparameters to enable the pre-training module to obtain reasonable single-hand saliency map results;

[0012] Step S22: based on a K-nearest neighbor search method, constructing a local neighborhood for each point in the input point cloud, inputting the local neighborhood into the pre-trained single-hand saliency map prediction network, and outputting the single-hand saliency value of each point. Save the output single-hand saliency map as the initial value of the double-hand saliency value prediction module.

[0013] Step S3: inputting the three-dimensional point cloud and the double-hand saliency initial value into the double-hand saliency map prediction module to obtain a saliency corresponding vector and a double-hand saliency value adjustment score, and obtaining a predicted double-hand saliency map based on the adjustment score and the initial value;

[0014] Further, the step S3 is specifically:

[0015] Step S31: inputting the three-dimensional point cloud and the initial value into the double-hand saliency map prediction module, and using two encoders to extract spatial position features and saliency value adjustment features, respectively;

[0016] Step S32: inputting the spatial position features and the saliency value adjustment features into the decoder, respectively, and outputting the saliency value corresponding vector and the saliency value adjustment score, respectively;

[0017] Step S33: add the saliency adjustment score to the initial value as the output two-hand grasp saliency value;

[0018] Step S34: train the two-hand saliency map prediction module using the two-hand corresponding adjustment loss, and constrain the predicted saliency value corresponding vector and saliency adjustment score;

[0019] Step S4: use the iterative update initial value strategy for the two-hand saliency map prediction module to update the initial value used in the training process;

[0020] Further, the step S4 is specifically:

[0021] Step S41: when the number of training iterations exceeds the set threshold parameter, save the current two-hand saliency map prediction result;

[0022] Step S42: set the newly saved two-hand saliency map as the initial value, and continue to train the two-hand grasp saliency value prediction module;

[0023] Step S5: input the three-dimensional point cloud into the contact point prediction module, use the left and right hand contact position label as the true value, and train the two-hand saliency map prediction as an auxiliary task to output the contact position label of each point;

[0024] Further, the step S5 is specifically:

[0025] Step S51: input the three-dimensional point cloud into the encoder of the contact point prediction module to extract global features and local features of each point;

[0026] Step S52: concatenate the local features with the global features, and input the concatenated features into the decoder to output the contact point label of each point;

[0027] Step S53: during the training process of the contact point prediction module, train the two-hand saliency value prediction as an auxiliary task together;

[0028] Step S6: based on the predicted two-hand grasp contact points, use the existing ContactGrasp optimization algorithm to generate the two-hand grasp pose of the robot hand.

[0029] Further, the step S6 is specifically:

[0030] Step S61: input the mesh vertices of the three-dimensional model into the contact point prediction module to obtain the contact point label of each point;

[0031] Step S62: set the left-hand contact points as attractive points and other points as repulsive points, and then input the mesh of the three-dimensional model and the contact point label into ContactGrasp to obtain the left-hand grasp pose;

[0032] Step S63: set the right hand contact point as an attractive point, set other points as repulsive points, and input the grid and contact point label of the three-dimensional model into ContactGrasp to obtain a right hand grasping posture;

[0033] Compared with the prior art, the present application has the following beneficial effects:

[0034] (1) Unlike previous grasp saliency analysis methods that can only be trained based on existing data sets, in the absence of a two-handed grasp saliency data set, the present application proposes a method that uses the correlation between single-handed and two-handed grasps, uses existing single-handed grasp saliency data and a small amount of two-handed grasp position annotation, and does not require two-handed grasp saliency true values to predict two-handed grasp saliency maps and contact points. This is the first method to perform grasp saliency analysis in the case of two-handed grasping. (2) The present application designs a two-handed correspondence adjustment loss function, which realizes the adjustment from a single-handed saliency map to a two-handed saliency map based on the correspondence between the left and right hands, and ensures the rationality of the two-handed grasp saliency map. (3) In the training process of the two-handed grasp saliency map prediction module of the present application, an iterative update initial value strategy is used, so that the initial value can change according to the two-handed grasp position annotation. This not only allows the modification of unreasonable initial values, but also makes the two-handed saliency map more consistent with human grasping habits. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is the overall structure designed by the present application.

[0036] Figure 2 is the two-handed grasp saliency map prediction module designed by the present application.

[0037] Figure 3 is the two-handed grasp contact point prediction module designed by the present application.

[0038] Figure 4 is the encoder and decoder structure of the two-handed grasp saliency map prediction module designed by the present application.

[0039] Figure 5 is the grasp generation process designed by the present application. DETAILED DESCRIPTION

[0040] The technical solutions of the present application will be further described below in combination with specific embodiments and drawings.

[0041] The two-handed saliency analysis method guided by the single-handed saliency map and a small amount of two-handed annotation is as follows:

[0042] Step S1: For a point cloud model P containing N points in the single-handed grasp saliency data set, P ∈ R N× 3, based on a small amount of user-annotated two-handed grasp annotation, a left and right hand contact position label set is constructed and the corresponding vector ground truth of the left and right hand contact points

[0043] Further, the step S1 is specifically:

[0044] Step S11: Based on a small amount of user-labeled double-hand grasp annotations, label annotations are made on the point cloud using point cloud labeling software. Wherein the label of the left-hand grasp contact position is l i = 1, and the label of the right-hand grasp contact position is l i = 2.

[0045] Step S12: Based on the left and right hand contact position label set L, the corresponding vector ground truth V gt is constructed for the points belonging to the left and right hand grasp positions. For the left-hand contact point m points in the right-hand annotation region are randomly selected as its corresponding points, and the set of corresponding vectors of the point is: For the right-hand grasp point m points in the left-hand annotation region are randomly selected as its corresponding points, and the set of corresponding vectors of the point is:

[0046] Step S2: input the three-dimensional point cloud P into the pre-trained single-hand saliency prediction module SSPN to obtain the single-hand grasp saliency map S of the model, and take it as the double-hand saliency initial value;

[0047] Further, the step S2 is specifically:

[0048] Step S21: pre-train the single-hand grasp saliency prediction module SSPN using an existing single-hand grasp saliency database, and set hyperparameters so that SSPN can obtain reasonable single-hand saliency map results;

[0049] Step S22: for any point x i in the input point cloud P, use the K-nearest neighbor search method to construct the local neighborhood P i of the point x i in P, input the local neighborhood P i into the pre-trained SSPN, and output the single-hand saliency value s i of each point. Save the output single-hand saliency map S as the initial value of the double-hand saliency prediction module.

[0050] Step S3: as shown in Figure 2 , input the three-dimensional point cloud P and the double-hand saliency initial value S into the double-hand saliency prediction module to obtain the saliency corresponding vector and the double-hand saliency adjustment score, and then obtain the predicted double-hand saliency map based on the adjustment score and the initial value;

[0051] Further, the step S3 is specifically:

[0052] Step S31: input the three-dimensional point cloud P and the initial value S into the two-hand saliency map prediction module BSPN, and use two encoders Enc1 and Enc2 to extract spatial position features F s ∈R N×d and saliency value adjustment features F t ∈R N×d . As shown in the figure, the encoder is composed of a 3-layer multi-layer perception mechanism. Wherein N is the number of points in the point cloud, and d is the embedding dimension of the point cloud depth feature. Figure 4

[0053] Step S32: input the spatial position features F s and the saliency value adjustment features F t into two decoders Dec1 and Dec2 respectively, and output the saliency value corresponding vector V∈R N× 3 and the saliency value adjustment score ΔS∈R N×1 respectively. As shown in the figure, the decoder is composed of a 4-layer multi-layer perception mechanism. Figure 4

[0054] Step S33: add the saliency value adjustment score ΔS to the initial value S to obtain the two-hand grasp saliency value B=ΔS+S.

[0055] Step S34: train BSPN using the two-hand corresponding adjustment loss L to constrain the predicted saliency value corresponding vector V and the saliency value adjustment score ΔS. The two-hand corresponding adjustment loss L is composed of two parts: the first part is the corresponding vector prediction loss: wherein The second part is the saliency value adjustment loss wherein The two-hand corresponding adjustment loss L is finally represented as: L=L C +L s .

[0056] Step S4: as shown in the figure, use the iterative update initial value strategy for BSPN to update the initial value S used in the training process. Figure 2

[0057] Further, the step S4 is specifically:

[0058] Step S41: when the number of training iterations of BSPN exceeds k, the network is trained for the t-th time, and the two-hand saliency map prediction result B t-1 of the last training is saved.

[0059] Step S42: set the newly saved two-hand saliency map B t-1 as the initial value S of the t-th training​​​t =B t-1 Continue training the saliency prediction module for bi-handed grasping.

[0060] Step S5: As Figure 3 As shown, P is input to the contact point prediction module, the left and right hand contact position labels L are used as the ground truth, and BSPN is used as an auxiliary task for training, and the contact position label of each point is output.

[0061] Furthermore, step S5 specifically includes:

[0062] Step S51: Input P into the encoder Enc of the contact point prediction module to extract global features. and local features of each point Where d2 and d3 are the embedding dimensions of the global and local features of the point cloud, respectively.

[0063] Step S52: Transfer local features F local Concatenate with global feature F: F′=concat(F local ,F), characteristics after cascading Input to decoder Dec, output contact point label for each point.

[0064] Step S53: Train the contact point prediction module and minimize the cross-entropy loss function:

[0065]

[0066] Train BPSN as an auxiliary task and minimize the joint loss function:

[0067]

[0068] Where w is a hyperparameter manually set during training.

[0069] Step S6: As Figure 5 As shown, based on the predicted gripping contact points of both hands, the existing ContactGrasp optimization algorithm is used to generate the gripping posture of the robotic hand.

[0070] Furthermore, step S6 specifically includes:

[0071] Step S61: Transfer the spatial coordinates V of the mesh vertices of the 3D model. mesh Input the contact point prediction module to obtain the contact point label for each vertex. Where K is the number of grid vertices.

[0072] Step S62: The point of the left hand contact point is set as an attractive point, other points are set as repulsive points, and the grid of the three-dimensional model and the attractive point label are input into ContactGrasp to obtain the left hand grasping posture.

[0073] Step S63: The point of the right hand contact point is set as an attractive point, other points are set as repulsive points, and the grid of the three-dimensional model and the attractive point label are input into ContactGrasp to obtain the right hand grasping posture.

[0074] The above steps S3 and S5 are the expansion models of the encoder and the decoder based on PointNet, which are only the preferred embodiments of the present application. Any equivalent changes and modifications made within the scope of the patent application of the present application shall be within the scope of the present application.

Claims

1. A method for biphasic saliency analysis guided by single-handed saliency maps and a small number of biphasic annotations, characterized by the following steps: include: Step S1: Based on the user-annotated two-handed grasping annotations, construct the contact position labels of the left and right hands, as well as the corresponding vector ground values ​​of the contact points of the left and right hands; Step S2: Input the 3D point cloud into the pre-trained single-hand saliency prediction module to obtain the single-hand grasping saliency map of the model, and use it as the initial saliency value of both hands; Step S3: Input the 3D point cloud and the initial saliency values ​​of both hands into the saliency map prediction module to obtain the saliency correspondence vector and the adjusted scores of the saliency values ​​of both hands. Based on the adjusted scores and the initial values, the predicted saliency map of both hands is obtained. Step S4: Use an iterative initial value update strategy to update the initial values ​​used during training for the two-handed saliency map prediction module; Step S5: Input the 3D point cloud into the contact point prediction module, use the left and right hand contact position labels as ground truth, and use the two-hand saliency map prediction as an auxiliary task for training, and output the contact position label of each point. Step S6: Based on the predicted gripping contact points, generate the gripping posture of the robotic hand using the existing ContactGrasp optimization algorithm.

2. The biphasic saliency analysis method guided by single-hand saliency maps and a small number of biphasic annotations according to claim 1, characterized in that, Step S2 specifically involves: Step S21: Use the existing single-hand grasping saliency database to pre-train the single-hand grasping saliency prediction module, and set the hyperparameters so that the pre-trained module can obtain reasonable single-hand saliency map results. Step S22: Construct a local neighborhood for each point in the input point cloud based on the K-nearest neighbor search method, input the local neighborhood into the pre-trained one-handed saliency map prediction network, output the one-handed saliency value of each point, and save the output one-handed saliency map as the initial value of the two-handed saliency value prediction module.

3. The biphasic saliency analysis method guided by single-hand saliency maps and a small number of biphasic annotations according to claim 1 or 2, characterized in that, Step S3 specifically involves: Step S31: Input the 3D point cloud and initial values ​​into the two-handed saliency map prediction module, and use two encoders to extract spatial location features and saliency adjustment features respectively; Step S32: Input the spatial location features and saliency adjustment features into the decoder respectively, and output the saliency value corresponding vector and saliency value adjustment score respectively; Step S33: Add the saliency value adjustment score to the initial value and use it as the output saliency value from both hands; Step S34: Train the bi-handed saliency map prediction module using bi-handed correspondence adjustment loss, and constrain the predicted saliency value correspondence vector and saliency value adjustment score.

4. The biphasic saliency analysis method guided by single-hand saliency maps and a small number of biphasic annotations according to claim 1 or 2, characterized in that, Step S4 specifically involves: Step S41: When the number of training iterations exceeds the set threshold parameter, save the current hand saliency map prediction results; Step S42: Set the newly saved saliency map of both hands as the initial value and continue training the saliency prediction module for both hands grasping.

5. The biphasic saliency analysis method guided by single-hand saliency maps and a small number of biphasic annotations according to claim 3, characterized in that, Step S4 specifically involves: Step S41: When the number of training iterations exceeds the set threshold parameter, save the current hand saliency map prediction results; Step S42: Set the newly saved saliency map of both hands as the initial value and continue training the saliency prediction module for both hands grasping.

6. The biphasic saliency analysis method guided by single-hand saliency maps and a small number of biphasic annotations according to claim 1, 2, or 5, is characterized in that, Step S5 specifically involves: Step S51: Input the 3D point cloud into the encoder of the contact point prediction module to extract global features and local features of each point; Step S52: Concatenate the local features with the global features, input the concatenated features into the decoder, and output the contact point label for each point; Step S53: During the training process of the contact point prediction module, the prediction of salience values ​​of both hands is trained as an auxiliary task.

7. The biphasic saliency analysis method guided by single-hand saliency maps and a small number of biphasic annotations according to claim 3, characterized in that, Step S5 specifically involves: Step S51: Input the 3D point cloud into the encoder of the contact point prediction module to extract global features and local features of each point; Step S52: Concatenate the local features with the global features, input the concatenated features into the decoder, and output the contact point label for each point; Step S53: During the training process of the contact point prediction module, the prediction of salience values ​​of both hands is trained as an auxiliary task.

8. The biphasic saliency analysis method guided by single-hand saliency maps and a small number of biphasic annotations according to claim 4, characterized in that, Step S5 specifically involves: Step S51: Input the 3D point cloud into the encoder of the contact point prediction module to extract global features and local features of each point; Step S52: Concatenate the local features with the global features, input the concatenated features into the decoder, and output the contact point label for each point; Step S53: During the training process of the contact point prediction module, the prediction of salience values ​​of both hands is trained as an auxiliary task.

9. The biphasic saliency analysis method guided by single-hand saliency maps and a small number of biphasic annotations according to claim 1, 2, 5, 7, or 8, is characterized in that, Step S6 specifically involves: Step S61: Input the mesh vertices of the 3D model into the contact point prediction module to obtain the contact point label for each point; Step S62: Set the left hand contact point as the attraction point and the other points as the repulsion points. Then input the mesh and contact point labels of the 3D model into ContactGrasp to obtain the left hand grasping posture. Step S63: Set the right-hand contact point as the attraction point and the other points as the repulsion points. Then input the mesh and contact point labels of the 3D model into ContactGrasp to obtain the right-hand grasping posture.

10. The biphasic saliency analysis method guided by single-hand saliency maps and a small number of biphasic annotations according to claim 3, characterized in that, Step S6 specifically involves: Step S61: Input the mesh vertices of the 3D model into the contact point prediction module to obtain the contact point label for each point; Step S62: Set the left hand contact point as the attraction point and the other points as the repulsion points. Then input the mesh and contact point labels of the 3D model into ContactGrasp to obtain the left hand grasping posture. Step S63: Set the right-hand contact point as the attraction point and the other points as the repulsion points. Then input the mesh and contact point labels of the 3D model into ContactGrasp to obtain the right-hand grasping posture.

Images