Grabbing stability analysis method based on contact domain force feedback

Abstract

A grabbing stability analysis method based on contact domain force feedback comprises the steps that a two-finger gripper is used for conducting simulated grabbing on all sampled grabbing poses in a simulation environment, a grabbing margin matrix is generated to serve as a training set, and a neural network obtained through construction is trained; and in the online stage, through the trained neural network, carrying out stability level prediction on the grabbing margin matrix obtained through real-time calculation. According to the method, a large amount of contact domain force feedback data is generated in a simulation environment, a grabbing margin matrix is constructed in combination with object mass center information, and a neural network model is trained to predict the grabbing stability level.

Description

Technical Field

[0001] The present invention relates to a technology in the field of robot control, specifically a method for analyzing the grasping stability based on contact domain force feedback. Background Art

[0002] The existing research on grasping strategies based on simulation datasets and deep learning models uses force feedback data in a relatively simple way. Usually, only the magnitude of the force or the number of contact points is considered, and the force and pose information are not fully combined to accurately model the grasping stability. In addition, the multi-dimensional data representation and computing power during the grasping process also limit the expansion of practical applications. Summary of the Invention

[0003] Aiming at the above deficiencies of the existing technology, the present invention proposes a method for analyzing the grasping stability based on contact domain force feedback. By generating a large amount of contact domain force feedback data in a simulation environment, combining the centroid information of the object to construct a grasping margin matrix, and training a neural network model to predict the grasping stability level.

[0004] The present invention is realized through the following technical solutions:

[0005] The present invention relates to a method for analyzing the grasping stability based on contact domain force feedback. By using a two-finger gripper to simulate grasping all the sampled grasping poses in a simulation environment and generating a grasping margin matrix as a training set, the constructed neural network is trained; during the online stage, the trained neural network is used to predict the stability level of the grasping margin matrix calculated in real time.

[0006] The neural network described above includes: first and second convolutional modules with batch normalization layers, ReLU activation layers, and max pooling layers, a third convolutional module, a first fully connected layer using ReLU activation and Dropout, and a second connected layer, where: the first convolutional module passes through a 3D convolutional layer with 32 kernels, the kernel size is (1, 3, 3), followed by batch normalization, ReLU activation, and max pooling with a size of (1, 2, 2), and the output feature shape is (M, 32, 2, 4, 2); the second convolutional module passes through a 3D convolutional layer with 64 kernels, the kernel size is (1, 3, 3), and after also going through batch normalization, ReLU activation, and pooling, the output feature shape is (M, 64, 2, 2, 1); the third convolutional module passes through a 3D convolutional layer with 128 kernels, the kernel size is (2, 3, 3), and finally, after compressing the shape of the feature map to (M, 128, 1, 1, 1), the output is flattened into a one-dimensional vector of (M, 128), and through a fully connected layer, the features are mapped to 256 dimensions, and ReLU activation and Dropout with a probability of 0.5 are used for regularization, and finally, through another fully connected layer, the logits for predicting stability are output, with a shape of (M, 9), which is used for the classification prediction of grasping stability.

[0007] The grasping margin matrix described above is generated in the following way: randomly initialize the grasping scenario in the simulation environment and perform grasping pose sampling and simulation to obtain the force feedback signal, the centroid position of the grasped object, and the grasping stability level. Based on the force feedback signal and the centroid position information, calculate the force margin of each sensing unit and the moment margin of the clamping surface to generate the grasping margin matrix.

[0008] The training set described above is created by cloning to create multiple simulation grasping scenarios and repeatedly generating the corresponding grasping margin matrix and grasping stability level.

[0009] The grasping stability level Where: [] represents the integer operation, and ef is the force applied to the grasped object simultaneously in the x, y, and z coordinate directions, with the unit of Newton.

[0010] The grasping margin matrix described above includes the force margin of each sensing unit in the three-dimensional coordinate axis directions and the moment margin of each clamping surface.

[0011] The present invention relates to a system for implementing the above method, including: a grasping margin matrix calculation module, an Isaac Sim simulation environment, and a grasping stability level prediction neural network. Among them: the grasping margin matrix calculation module calculates the force margin and torque margin of the force sensing unit according to the force feedback information and the centroid of the grasped object, and obtains the grasping margin matrix; the Isaac Sim simulation environment simulates the two-finger gripper and the force sensing unit, performs simulated grasping according to the sampled grasping poses, obtains the grasping stability level, and calculates the grasping margin matrix according to the force feedback information and the centroid of the grasped object, obtaining a training set including the grasping margin matrix and the grasping stability level; the grasping stability level prediction neural network predicts the grasping stability level according to the grasping margin matrix. Technical effects

[0012] The present invention initializes the grasping scenario using the Issac Sim simulation environment, constructs the grasping margin matrix through the contact domain force feedback signal and the centroid position information. Combining stability detection, while quickly generating a large-scale grasping data set using the cloning method, the grasping data set is automatically generated through the simulation environment, significantly improving the data preparation efficiency; innovatively proposes a method for calculating the grasping margin matrix, combining force feedback and centroid information to accurately characterize the grasping stability; based on the efficient prediction mechanism of the neural network model, improving the real-time performance and accuracy of grasping stability analysis; supporting the stability prediction of the actual grasping scenario, enhancing the practicality and robustness of the robot grasping technology. Description of the drawings

[0013] Figure 1 It is the flow chart of the present invention;

[0014] Figure 2 It is the schematic diagram of the visualization result of the force detection data of the array force sensing unit included in the present invention;

[0015] Figure 3 It is the schematic diagram of the force margin calculation method of the present invention;

[0016] Figure 4 It is the schematic diagram of the torque margin calculation method of the present invention;

[0017] Figure 5 It is the Isaac Sim clone scenario diagram of the present invention;

[0018] Figure 6 It is the relationship diagram between the components of the grasping margin matrix and the grasping stability level of the present invention. Detailed implementation manners

[0019] As Figure 1 shown, this embodiment designs a grasping stability analysis method based on contact domain force feedback, including:

[0020] Step S1: Start the Issac Sim simulation environment, import the URDF model of the two-finger gripper into the Issac Sim environment, and set 8 rows and 5 columns of sensing units on its left and right clamping surfaces. Each sensing unit can feedback signals of the magnitude, position, and direction of the force. Randomly initialize the grasping scene P at the origin position.

[0021] Step S2: Densely sample the potential grasping positions and grasping directions of all objects to be grasped in the scene, and combine the two into the grasping pose g 1 , specifically: Obtain the maximum and minimum values of the object to be grasped in the x, y, and z coordinate axes, and perform voxel sampling within this range. For each sampling point, perform equi-arc sampling on the three Euler angles of the grasping direction.

[0022] Step S3: Use the two-finger gripper to simulate grasping for all sampled grasping poses, specifically including

[0023] 3.1 In the Issac Sim environment, transform the two-finger gripper to the grasping pose sampled in Step S2, simulate the closing of the two-finger gripper, and stop closing when the left and right clamping surfaces of the two-finger gripper are in contact with the object and cannot continue to close;

[0024] 3.2 Delete the grasps that cannot be executed normally, and perform stability detection on the grasps that can be executed normally: After the two-finger gripper is closed, slowly move it up 1 meter, and detect whether the grasped object falls. If it falls, the stability level sf of this grasp is 0. If it does not fall, apply a gradually increasing force ef, in newtons, to the grasped object in the x, y, and z coordinate directions simultaneously until the object falls, and the stability level of the grasp where: [] is the integer operation.

[0025] 3.3 Obtain all the grasping poses g in the world coordinate system 2 , the force feedback signal F when the corresponding grasp is closed, the visualization result is as Figure 2 shown, the centroid position pos of the grasped object and the stability level sf.

[0026] Step S4: As Figure 3 and Figure 4 shown, calculate the grasping margin matrix CM composed of the margins [ax, ay, az, mx, my, mz] of the sensing units according to the force feedback signal F and the centroid position pos of the grasped object obtained in Step S3, specifically including:

[0027] 4.1 Obtain the force fa of any sensing unit and its direction vector da = [da_x, da_y, da_z];

[0028] 4.2 Calculate the angles between the force fa and the three coordinate axes x, y, and z respectively: where: θx, θy, and θz are the angles between the force fa and the three coordinate axes x, y, and z respectively, and tan -1 () is the arcsine function;

[0029] 4.3 Calculate the force margins in the directions of the three coordinate axes x, y, and z respectively, specifically: where: ax, ay, and az are the force margins in the directions of the three coordinate axes x, y, and z respectively, K is the force-moment magnitude harmonic coefficient, taking 10, is the inner angle of the friction cone.

[0030] 4.4 Construct the spatial contact surface S between the clamping surface and the object based on the force feedback information F;

[0031] 4.5 Project the spatial contact surface S onto the YoZ plane, XoZ plane, and XoY plane respectively, and calculate the moment margins where: x min , x max are the x-coordinates of the two minimum and maximum sensing units along the x-axis on the XoZ projection plane and XoY projection plane respectively, y min , y max are the y-coordinates of the two minimum and maximum sensing units along the y-axis on the XoY projection plane and YoZ projection plane respectively, z min , z max are the z-coordinates of the two minimum and maximum sensing units along the z-axis on the XoZ projection plane and YoZ projection plane respectively, e x , e y , e z are the unit vectors in the directions of the three coordinate axes x, y, and z respectively, are the force directions corresponding to the sensing units at the (i, j) positions on the YoZ projection plane, XoZ projection plane, and XoY projection plane respectively, O x (i, j), O y (i, j), O z (i, j) are the vectors from the center of mass of the object to the (i, j) positions on the YoZ projection plane, XoZ projection plane, and XoY projection plane respectively, and δ is the side length of the sensing unit.

[0032] Step S5: Use the cloning method of Issac Sim to create N grasping scenarios, and execute the content of steps S1 - S4 for each scenario. As Figure 5 shown, collect all the data and create a dataset D: Collect the grasping margin matrix CM and the stability level sf corresponding to all normal grasping poses g 2 , and save each CM and the corresponding sf as a piece of data in the dataset D.

[0033] Step S6: Construct a neural network SAN and train it using the dataset D created in Step S5. Input a five-dimensional grasping margin matrix with the shape of (M, 2, 8, 5, 6) into the neural network SAN, where M is the batch size, and M = 8 is taken. The 6 in the last dimension represents the margin information of the sensing unit, which is rearranged as (M, 6, 2, 8, 5) at the beginning of the network. Finally, the neural network outputs stability prediction logits with the shape of (M, 9). The index of the maximum value in the second dimension is the predicted stability level. Calculate the loss together with the grasping stability label in the dataset through the cross-entropy loss function, and use the Adam optimizer to iteratively update the network parameters.

[0034] Step S7: Obtain the force feedback signal F’ and the centroid position pos’ of the grasped object when the grasp is closed in the actual situation. Calculate the grasping margin matrix CM’ by the method in Step S4, and then use the neural network SAN trained in Step S6 for prediction to obtain the final stability level sf’.

[0035] In the actual process, nine grasped objects are selected, and one hundred grasping scenarios are randomly generated. Ten thousand grasping poses are randomly sampled for each grasping scenario. Steps S1 - S4 are executed for each grasping pose in the Isaac Sim simulation environment to obtain a dataset containing the grasping margin matrix and the grasping stability level. Analyze the relationship between each component of the grasping margin matrix and the grasping stability level respectively. When the grasping stability is small, the influence of gravity is large, and they are combined. The final result is as Figure 6 shown. There is an obvious positive proportional trend between each component of the grasping margin matrix and the grasping stability level, indicating that the grasping margin matrix can accurately model the grasping stability.

[0036] As shown in Table 1, compared with the existing technology that only uses force data for stability analysis in the present invention.

[0037] Table 1

[0038] For example, when using the force closure criterion to judge the grasping stability, the grasping margin matrix and the neural network SAN proposed by this method greatly improve the prediction accuracy of the grasping stability. At the same time, for different other grasping stability prediction methods, the force data and the grasping margin matrix are used for prediction respectively, and the prediction accuracy obtained by using the grasping margin matrix is higher than that obtained by using the force data.

[0039] The above specific implementation can be locally adjusted in different ways by those skilled in the art without departing from the principles and purposes of the present invention. The protection scope of the present invention is subject to the claims and is not limited by the above specific implementation. All implementation solutions within its scope are subject to the present invention.

Claims

1. A grasping stability analysis method based on contact domain force feedback, characterized in that: The constructed neural network is trained by using a two-finger gripper to simulate grasping of all sampled grasping postures in a simulation environment and generating a grasping margin matrix as a training set; in the online stage, the trained neural network is used to predict the stability level of the grasping margin matrix calculated in real time; The training set is obtained by creating multiple simulated grasping scenarios through a cloning method and repeatedly generating corresponding grasping margin matrices and grasping stability levels; The gripping margin matrix includes the force margin of each sensing unit in the direction of the three-dimensional coordinate axis and the torque margin of each clamping surface.

2. The grasping stability analysis method based on contact domain force feedback according to claim 1 is characterized in that: The neural network comprises: a first and a second convolutional module with a batch normalization layer, a ReLU activation layer and a maximum pooling layer, a third convolutional module, a first fully connected layer using ReLU activation and Dropout, and a second connection layer, wherein: the first convolutional module passes through a 32-core 3D convolutional layer with a convolution kernel size of (1,3,3), followed by batch normalization, ReLU activation and (1,2,2) maximum pooling, and the output feature shape is (M,32,2,4,2); the second convolutional module passes through a 64-core 3D convolutional layer with a convolution kernel size of (1,3,3), also after batch normalization, ReLU activation and (1,2,2) maximum pooling, and the output feature shape is (M,32,2,4,2); After eLU activation and pooling, the output feature shape is (M, 64, 2, 2, 1); the third convolution module passes through a 128-core 3D convolution layer with a convolution kernel size of (2, 3, 3), and finally compresses the shape of the feature map to (M, 128, 1, 1, 1), and then flattens the output to a one-dimensional vector of (M, 128). The feature is mapped to 256 dimensions through a fully connected layer, and ReLU activation and Dropout are used with a probability of 0.5 for regularization. Finally, another fully connected layer outputs the logits of stability prediction with a shape of (M, 9) for classification prediction of grasping stability.

3. The grasping stability analysis method based on contact domain force feedback according to claim 1 is characterized in that: The grasping margin matrix is ​​generated in the following way: randomly initializing the grasping scene in the simulation environment and performing grasping posture sampling and simulation, obtaining the force feedback signal, the center of mass position of the grasped object and the grasping stability level, and based on the force feedback signal and the center of mass position information, calculating the force margin of each sensor unit and the torque margin of the clamping surface to generate the grasping margin matrix.

4. The grasping stability analysis method based on contact domain force feedback according to claim 1 is characterized in that: Described grip stability level Where: [] is a rounding operation, ef is the force applied to the grasped object simultaneously along the three coordinate directions of xyz, in Newton.

5. The grasping stability analysis method based on contact domain force feedback according to any one of claims 1 to 4, characterized in that: include: Step S1, start the Issac Sim simulation environment, import the two-finger gripper URDF model into the Issac Sim environment, and set 8 rows and 5 columns of sensor units on its left and right gripping surfaces. Each sensor unit can feedback the magnitude, position and direction signals of the force, and randomly initialize the grasping scene P at the origin position; Step S2: densely sample the potential grasping positions and grasping directions of all objects to be grasped in the scene, and combine the two into a grasping posture g1, specifically: obtain the maximum and minimum values ​​of the object to be grasped in the three coordinate axes of xyz, and perform voxel sampling within this range, and perform equal arc sampling on the three Euler angles from each sampling point to the grasping direction; Step S3: using a two-finger gripper to simulate grasping of all sampled grasping postures, specifically including: 3.1 In the Issac Sim environment, transform the two-finger gripper to the grasping posture sampled in step S2, simulate the closing of the two-finger gripper, and stop closing when the left and right gripping surfaces of the two-finger gripper are in contact with the object and cannot continue to close; 3.2 Delete the grasps that cannot be performed normally, and perform stability test on the grasps that are performed normally: after the two-finger gripper is closed, slowly move it up 1 meter to detect whether the grasped object falls. If it falls, the grasping stability level sf is 0. If it does not fall, gradually increase the force ef in the three coordinate directions of xyz to the grasped object at the same time, in Newtons, until the object falls. The grasping stability level Among them: [] is the rounding operation; 3.3 Obtain all grasping postures g2 in the world coordinate system, the corresponding force feedback signal F when the grasp is closed, and visualize the center of mass position pos and stability level sf of the grasped object; Step S4: Calculate the grasping margin matrix CM composed of the margins [ax, ay, az, mx, my, mz] of the sensor units according to the force feedback signal F obtained in step S3 and the center of mass position pos of the grasped object, specifically including: 4.1 Obtain the force fa of any sensing unit and its direction vector da = [da_x, da_y, da_z]; 4.2 Calculate the angles between the force fa and the three coordinate axes xyz respectively: Among them: θx, θy, θz are the angles between force fa and the three coordinate axes xyz, tan -1 () is the inverse sine function; 4.3 Calculate the force margin in the three coordinate axes xyz respectively, specifically: Among them: ax, ay, az are the force margins in the directions of the three coordinate axes xyz, K is the force-torque harmonic coefficient, is the internal angle of the friction cone; 4.4 constructing a spatial contact surface S between the clamping surface and the object according to the force feedback information F; 4.5 Project the spatial contact surface S onto the YoZ plane, XoZ plane and XoY plane respectively, and calculate the torque margins accordingly Where: x min , x max are the x-coordinates of the minimum and maximum two sensor units along the x-axis on the XoZ projection plane and the XoY projection plane, respectively. min ,y max are the y coordinates of the minimum and maximum two sensor units along the y axis on the XoY projection plane and the YoZ projection plane, respectively. min , z max are the z coordinates of the minimum and maximum two sensor units along the z axis on the XoZ projection plane and the YoZ projection plane, respectively. x , e y , e z are the unit vectors in the directions of the three coordinate axes xyz, are the force directions of the sensor unit corresponding to the position (i, j) on the YoZ projection plane, XoZ projection plane, and XoY projection plane, respectively. x (i,j),O y (i,j),O z (i, j) are the vectors pointing from the object's center of mass to the position (i, j) on the YoZ projection plane, XoZ projection plane, and XoY projection plane, respectively, and δ is the side length of the sensor unit; Step S5, using the cloning method of IssacSim, create N grasping scenes, each of which executes the contents of steps S1-S4, collects all data, and creates a data set D: collects the grasping margin matrix CM and stability level sf corresponding to all normal grasping postures g2, and saves each CM and corresponding sf as a piece of data in the data set D; Step S6, construct a neural network SAN, and use the data set D created in step S5 for training: input a five-dimensional grasping margin matrix of shape (M, 2, 8, 5, 6) into the neural network SAN, where M is the batch size, and the last dimension 6 represents the margin information of the sensor unit. It is rearranged to (M, 6, 2, 8, 5) at the beginning of the network. Finally, the neural network outputs stability prediction logits of shape (M, 9). The index of the maximum value in the second dimension is the predicted stability level. The loss is calculated together with the grasping stability label in the data set through the cross entropy loss function, and the network parameters are iteratively updated using the Adam optimizer; Step S7, obtain the force feedback signal F' and the center of mass position pos' of the grasped object when the grasping is closed in actual situation, calculate the grasping margin matrix CM' by the method of step S4, and then use the neural network SAN trained in step S6 to make predictions to obtain the final stability level sf'.

6. A grasping stability analysis system for implementing the method described in any one of claims 1 to 5, characterized in that: include: A grasping margin matrix calculation module, an IsaacSim simulation environment, and a grasping stability level prediction neural network, wherein: the grasping margin matrix calculation module calculates the force margin and torque margin of the force sensing unit according to the force feedback information and the center of mass of the grasped object to obtain the grasping margin matrix; the IsaacSim simulation environment simulates the two-finger gripper and the force sensing unit, simulates grasping according to the grasping posture generated by sampling, obtains the grasping stability level, and calculates the grasping margin matrix according to the force feedback information and the center of mass of the grasped object to obtain a training set including the grasping margin matrix and the grasping stability level; the grasping stability level prediction neural network predicts the grasping stability level according to the grasping margin matrix.

Images