Robot calibration method based on feature differentiation learning neural network model

Abstract

The invention provides a robot calibration method based on a feature differentiation learning neural network model. According to the method, a robot operation reference object is calibrated through a credible feature discrimination convolutional neural network, a feature differentiation learning strategy and a three-dimensional frame prediction network based on a view cone-link dynamic graph. According to the invention, in combination with multi-modal visual information data, a credible feature discrimination convolutional neural network, a feature differentiation learning strategy and a three-dimensional frame prediction network based on a view cone-link dynamic graph are constructed, so that the problem that an existing robot calibration method is easily influenced by polymorphic reference substances and random noise is solved; the robot calibration precision in a large-scale production environment is improved.

Description

Technical Field

[0001] The invention relates to a robot calibration method based on a feature differentiation learning neural network model, and belongs to the technical field of industrial robot calibration. Background Art

[0002] In a large-scale production environment, industrial robots need to calibrate reference objects according to the task, determine the relative position between the robot and the reference object in the working space, and lay the foundation for the subsequent stable operation of the end effector. Due to the different tasks of the robot, the type, shape and position of the reference object are different. At present, the robot reference object calibration method mainly uses algorithms such as matrix operations or feature point extraction to solve the transformation matrix or transformation relationship between each coordinate system, so as to achieve accurate calibration and positioning of the robot vision system. However, the existing methods face many challenges in a large-scale production environment. Among them, the diverse morphology of reference objects and random noise have an adverse effect on their feature extraction, resulting in limited reference object calibration accuracy. Therefore, it is urgent to study a new robot vision calibration method suitable for multi-morphological reference objects under high noise, so as to provide robots with more comprehensive and accurate scene perception capabilities, thereby improving the robot vision calibration accuracy in complex production environments. Summary of the invention

[0003] The technical problem to be solved by the present invention is that the existing robot calibration method is easily affected by polymorphic reference objects and random noise.

[0004] In order to solve the above technical problems, the technical solution of the present invention is to disclose a robot calibration method based on a feature differentiation learning neural network model, which is characterized by comprising the following steps:

[0005] Step 1: After building the robot binocular vision information acquisition device, collect multimodal visual information and construct training sets and test sets for subsequent calibration model training and testing;

[0006] Step 2: Construct a credible feature discrimination convolutional neural network, which includes a feature extraction module, an evidence modeling module, an evidence fusion module and an output module, wherein:

[0007] The feature extraction module is used to extract the features of the work reference object in the input image;

[0008] Based on the evidence theory, the evidence modeling module converts the features extracted by the feature extraction module and their coefficients into supporting evidence for detection, and divides its associated features into credible features or uncredible features according to the weight of the evidence;

[0009] The evidence fusion module is based on evidence theory and uses the evidence weights of credible features and uncredible features to calculate the support of all credible features for the detection results and the support of all uncredible features for the complement of the detection results.

[0010] The output module converts the extracted features into the predicted reference object category and location information;

[0011] Step 3: Based on the credible feature discriminant convolutional neural network in step 2, a credible loss function is constructed, and an uncertainty penalty factor is introduced to increase the learning cost of low-uncertainty detection errors and reduce the learning cost of high-uncertainty recognition, so as to guide the network to focus on credible feature learning and ignore untrustworthy feature learning during training;

[0012] Step 4: Train the credible feature discrimination convolutional neural network:

[0013] The two-dimensional image data training set obtained in step 1 is combined with the credible feature discrimination convolutional neural network constructed in step 2 and the feature differentiation learning strategy established in step 3 to train the credible feature discrimination convolutional neural network and save the optimal network model;

[0014] Step 5: Detect the border of the work reference object in the 2D image:

[0015] Using the trusted feature discriminant convolutional neural network trained in step 4, output the 2D bounding box of the work reference object in the 2D image dataset and save the label file. Then, project the 2D bounding box area into the 3D point cloud obtained in step 1 to form a frustum candidate area. After binary serialization and deserialization of the frustum candidate area, save the label of the 2D bounding box.

[0016] Step 6: Build a 3D bounding box prediction network based on the cone-link dynamic graph:

[0017] A linked dynamic graph network is used to extract the correlation features between points based on the view frustum point cloud candidate area, and the probability of each point belonging to the reference object is output. Based on the obtained reference object probability, the T-Net network and PointNet network are used to predict the 3D bounding box of the reference object.

[0018] Step 7: Train a 3D bounding box prediction network based on the cone-link dynamic graph:

[0019] The point cloud of the view cone point cloud candidate area is used to train the 3D bounding box prediction network based on the view cone-link dynamic graph, and the optimal training model is saved.

[0020] Step 8: Test set calibration:

[0021] Using the trusted feature discrimination convolutional neural network trained in step 4 and the 3D bounding box prediction network based on the cone-link dynamic graph in step 7, the test calibration of the work reference object is performed on the 3D point cloud data test set and the 2D image data test set in step 1.

[0022] Preferably, in step 1, the robot binocular vision information acquisition device includes an imaging module, a support module for supporting the imaging module, and a lighting module for providing lighting for the imaging module, wherein the imaging module includes a main view three-dimensional camera, a main view two-dimensional camera and a lens.

[0023] Preferably, in step 2, the evidence modeling module uses average pooling to convert the multi-dimensional feature map of the deformable convolution layer into a one-dimensional feature map, and based on the evidence theory, combines the eigenvalues ​​and their weights and biases to convert the features into evidence, and then divides the feature categories according to the evidence weights. The specific calculation process is as follows:

[0024]

[0025] In the formula, w ij Represents the weight of evidence after feature transformation in row i and column j of the feature graph. represents the eigenvalue of the i-th row and j-th column in the feature map, K is the total number of features in the feature map, and β ij is the weight value of the feature of the i-th row and j-th column in the one-dimensional standard convolution, λ ij is the bias of the feature of the i-th row and j-th column in the one-dimensional standard convolution, μ ij is the average activation value of the feature map, RF is a credible feature, and UF is an untrustworthy feature.

[0026] Preferably, in step 2, the evidence fusion module obtains the support of all credible features for the detection results and the support of all uncredible features for the complement of the detection results according to the basic probability distribution function and evidence combination rules in the evidence theory, and the calculation process is as follows:

[0027]

[0028] In the formula, Represents the weight of evidence after the conversion of the credible feature in the i-th row and j-th column of the feature graph, represents the weight of evidence after the untrustworthy feature in the nth row and mth column of the feature graph is transformed, R is the detection result, is the complement of the test result, m(·) is the basic probability distribution function, s is the support, Rules for combining evidence.

[0029] Preferably, step 4 is specifically expressed as the following formula:

[0030] Loss T=P U (Loss C +Loss CIoU )

[0031]

[0032] In the formula, p a is the predicted probability of category a, y a is the true value of category a, ρ 2 (b,b gt ) is the Euclidean distance between the center of the predicted box and the center of the real box, w gt and h gt is the width and height of the real box, w and h are the width and height of the predicted box, c is the diagonal distance of the minimum closure area, αυ is the CIoU adjustment factor, P U is the uncertainty penalty factor, U R is the uncertainty of the test result R, Bel(·) is the trust function, and Pl(·) is the likelihood function.

[0033] Preferably, in step 6, the center point of the target object is first identified through a T-Net network, and then the identified center point of the object is converted into a coordinate origin by multiplying the coordinate transformation matrix, and then the three-dimensional bounding box of the reference object is obtained through the PointNet network.

[0034] In view of the data uncertainty caused by the diverse shapes and random noise of robot working reference objects, the method disclosed in the present invention first uses a credible feature discrimination convolutional neural network to identify effective reference object features and analyze the credibility of the features; secondly, a feature differentiation learning strategy is used to assign different learning cost weights to reference object recognition errors with different credibility, enhance the error feedback of high-credibility feature learning, and improve the detection accuracy of robot working reference objects in two-dimensional images; finally, a three-dimensional bounding box prediction network based on a cone-link dynamic graph is used to project the two-dimensional detection bounding box of the robot working reference object onto a three-dimensional point cloud, and output the three-dimensional bounding box information of the robot working reference object.

[0035] The present invention adopts a robot calibration method based on a feature differentiation learning neural network model to calibrate the robot's working reference objects. Combined with multimodal visual information data, a credible feature discrimination convolutional neural network, a feature differentiation learning strategy and a three-dimensional bounding box prediction network based on a cone-link dynamic graph are constructed to solve the problem that the existing robot calibration method is easily affected by multi-modal reference objects and random noise, so as to improve the robot calibration accuracy in a large-scale production environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 The credible feature discrimination convolutional neural network of the present invention is illustrated;

[0037] Figure 2 The link dynamic graph network of the present invention is illustrated;

[0038] Figure 3 The three-dimensional bounding box output network of the present invention is illustrated. DETAILED DESCRIPTION

[0039] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0040] like Figure 1 As shown, a robot calibration method based on a feature differentiation learning neural network model disclosed in an embodiment of the present invention includes the following steps:

[0041] Step 1: Build a robot binocular vision information acquisition device. The robot vision information acquisition device used in the present invention mainly includes: an imaging module (including a main view 3D camera, a main view 2D camera and a lens), a support module (camera bracket), and a lighting module (strip light source).

[0042] Step 2: Collect multimodal visual information. The collection cycle is pre-set. After the main view 3D camera and the main view 2D camera in step 1 receive the action command, they synchronously collect the robot's forward visual information, save the corresponding 3D point cloud data and 2D image data, and randomly divide the training set and test set into a ratio of 8:2 for subsequent calibration model training and testing.

[0043] Step 3: Construct a convolutional neural network for credible feature discrimination. The features extracted by deformable convolution are divided into credible features and uncredible features using the evidence modeling module and the evidence fusion module, and the features and their correlation coefficients are analyzed to obtain the uncertainty of the network characterization of the reference object under high noise. The credible feature discrimination convolutional neural network mainly consists of a feature extraction module, an evidence modeling module, an evidence fusion module, and an output module.

[0044] The feature extraction module consists of 5 standard convolutional layers and 4 deformable convolutional layers, which are used to extract the features of the work reference objects in the input image.

[0045] Based on evidence theory, the evidence modeling module converts the features extracted by the deformable convolution layer and its coefficients into supporting evidence for detection, and divides its associated features into credible features or uncredible features according to the weight of the evidence. Specifically, average pooling is used to convert the multi-dimensional feature map of the deformable convolution layer into a one-dimensional feature map, and based on evidence theory, the feature is converted into evidence by combining the eigenvalues, their weights and biases, and then the feature categories are divided according to the weight of the evidence. The specific calculation process is as follows:

[0046]

[0047]

[0048] In the formula, w ij Represents the weight of evidence after feature transformation in row i and column j of the feature graph. represents the eigenvalue of the i-th row and j-th column in the feature map, K is the total number of features in the feature map, and β ij is the weight value of the feature of the i-th row and j-th column in the one-dimensional standard convolution, λ ij is the bias of the feature of the i-th row and j-th column in the one-dimensional standard convolution, μ ij is the average activation value of the feature map, RF is a credible feature, and UF is an untrustworthy feature.

[0049] The evidence fusion module is based on evidence theory and uses the evidence weights of credible features and uncredible features to calculate the support of all credible features for the detection results and the support of all uncredible features for the complement of the detection results. Specifically, according to the basic probability distribution function and evidence combination rules in evidence theory, the support of all credible features for the detection results and the support of all uncredible features for the complement of the detection results are obtained. The calculation process is as follows:

[0050]

[0051]

[0052] In the formula, Represents the weight of evidence after the conversion of the credible feature in the i-th row and j-th column of the feature graph, represents the weight of evidence after the untrustworthy feature in the nth row and mth column of the feature graph is transformed, R is the detection result, is the complement of the test result, m(·) is the basic probability distribution function, s is the support, Rules for combining evidence.

[0053] The output module converts the extracted features into predicted reference object category and location information.

[0054] Step 4: Design a feature differentiation learning strategy. Based on the credible feature discrimination convolutional neural network in step 3, a credible loss function is constructed, and an uncertainty penalty factor is introduced to increase the learning cost of low-uncertainty detection errors and reduce the learning cost of high-uncertainty recognition, so as to guide the network to focus on credible feature learning and ignore untrustworthy feature learning during training, as shown in the following formula:

[0055] Loss T =P U (Loss C +Loss CIoU ) (10)

[0056]

[0057] In the formula, p a is the predicted probability of category a, y a is the true value of category a, ρ 2 (b,b gt ) is the Euclidean distance between the center of the predicted box and the center of the real box, w gt and h gt is the width and height of the real frame, w and h are the width and height of the prediction box, c is the diagonal distance of the minimum closure area, αυ is the CIoU adjustment factor, P U is the uncertainty penalty factor, U R is the uncertainty of the test result R, Bel(·) is the trust function, and Pl(·) is the likelihood function.

[0058] Step 5: Train the credible feature discrimination convolutional neural network. Combine the two-dimensional image data training set in step 2 with the credible feature discrimination convolutional neural network in step 3 and the feature differentiation learning strategy in step 4 to train the credible feature discrimination convolutional neural network and save the optimal network model.

[0059] Step 6: Detect the bounding box of the work reference object in the 2D image. Use the trusted feature discrimination convolutional neural network trained in step 5 to output the 2D bounding box of the work reference object in the 2D image dataset and save the label file. Then project the 2D bounding box area into the 3D point cloud obtained in step 2 to form a cone candidate area. After binary serialization and deserialization of the cone candidate area, save the label of the 2D bounding box.

[0060] Step 7: Construct a 3D bounding box prediction network based on the cone-linked dynamic graph. The linked dynamic graph network (LDGCNN) is used to extract the correlation features between points based on the candidate area of ​​the cone point cloud, and output the probability of each point belonging to the reference object. The overall structure of the LDGCNN network includes four layers of feature extraction layers, each of which is composed of an edge convolution feature extraction module. The edge convolution learns the neighborhood features between points. The number of channels for feature extraction in each layer is (64, 64, 64, 128), and the convolution kernel is 1×1. After four layers of feature extraction, the features learned in each layer are spliced ​​and input into a fully connected layer. Then, the global features of the point cloud are obtained through maximum pooling, and spliced ​​with each layer of features obtained in the feature extraction part. The spliced ​​global features are aggregated through four layers of multi-layer perceptrons, and finally the probability of each point belonging to the reference object is obtained. Compared with using PointNet to extract point features, using a linked dynamic graph network to extract point cloud features can provide rich target location information and help the model locate reference objects.

[0061] According to the obtained reference object probability, the three-dimensional bounding box of the reference object is predicted. The network first uses a T-Net network to identify the center point of the target object, and then converts the identified center point of the object into the coordinate origin by multiplying it by the coordinate transformation matrix. The coordinate transformation is performed to facilitate the identification of the orientation of the reference object, and then the three-dimensional bounding box of the reference object is obtained through the PointNet bounding box recognition network. The T-Net network structure is composed of a multi-layer perceptron with a weight sharing of (128, 256, 512) channels plus maximum pooling to obtain global features, and then the global features are input into the multi-layer perceptron (MLP) to obtain the residual of the center point. Finally, the coordinate transformation is performed to convert the coordinate origin to the identified center point. The coordinate transformation method is to subtract the residual from all point clouds to obtain the point cloud in the coordinate system with the center of the reference object as the origin.

[0062] In the embodiment of the present invention, a link dynamic graph network is used, such as Figure 2 As shown in , extract the correlation features between points in the candidate area of ​​the view cone point cloud in step 6, and output the probability that each point belongs to the reference object, and then use the T-Net network and PointNet network, as shown in Figure 3 As shown, the 3D bounding box of the reference object is output.

[0063] Step 8: Train the 3D bounding box prediction network based on the cone-link dynamic graph. Use the point cloud of the cone point cloud candidate area to train the 3D bounding box prediction network based on the cone-link dynamic graph, and save the optimal training model.

[0064] Step 9: Test set calibration. Using the trusted feature discrimination convolutional neural network trained in step 5 and the 3D bounding box prediction network based on the cone-link dynamic graph in step 8, perform test calibration of the work reference object on the 3D point cloud data test set and the 2D image data test set in step 2.

Claims

1. A robot calibration method based on a feature differentiation learning neural network model, characterized in that: The following steps are involved: Step 1: After building the robot binocular vision information acquisition device, collect multimodal visual information and construct training sets and test sets for subsequent calibration model training and testing; Step 2: Construct a credible feature discrimination convolutional neural network, which includes a feature extraction module, an evidence modeling module, an evidence fusion module and an output module, wherein: The feature extraction module is used to extract the features of the work reference object in the input image; Based on the evidence theory, the evidence modeling module converts the features extracted by the feature extraction module and their coefficients into supporting evidence for detection, and divides its associated features into credible features or uncredible features according to the weight of the evidence; The evidence fusion module is based on evidence theory and uses the evidence weights of credible features and uncredible features to calculate the support of all credible features for the detection results and the support of all uncredible features for the complement of the detection results. The output module converts the extracted features into the predicted reference object category and location information; Step 3: Based on the credible feature discriminant convolutional neural network in step 2, a credible loss function is constructed, and an uncertainty penalty factor is introduced to increase the learning cost of low-uncertainty detection errors and reduce the learning cost of high-uncertainty recognition, so as to guide the network to focus on credible feature learning and ignore untrustworthy feature learning during training; Step 4: Train the credible feature discrimination convolutional neural network: The two-dimensional image data training set obtained in step 1 is combined with the credible feature discrimination convolutional neural network constructed in step 2 and the feature differentiation learning strategy established in step 3 to train the credible feature discrimination convolutional neural network and save the optimal network model; Step 5: Detect the border of the work reference object in the 2D image: Using the trusted feature discriminant convolutional neural network trained in step 4, output the 2D bounding box of the work reference object in the 2D image dataset and save the label file. Then, project the 2D bounding box area into the 3D point cloud obtained in step 1 to form a frustum candidate area. After binary serialization and deserialization of the frustum candidate area, save the label of the 2D bounding box. Step 6: Build a 3D bounding box prediction network based on the cone-link dynamic graph: A linked dynamic graph network is used to extract the correlation features between points based on the view frustum point cloud candidate area, and the probability of each point belonging to the reference object is output. Based on the obtained reference object probability, the T-Net network and PointNet network are used to predict the 3D bounding box of the reference object. Step 7: Train a 3D bounding box prediction network based on the cone-link dynamic graph: The point cloud of the view cone point cloud candidate area is used to train the 3D bounding box prediction network based on the view cone-link dynamic graph, and the optimal training model is saved. Step 8: Test set calibration: Using the trusted feature discrimination convolutional neural network trained in step 4 and the 3D bounding box prediction network based on the cone-link dynamic graph in step 7, the test calibration of the work reference object is performed on the 3D point cloud data test set and the 2D image data test set in step 1.

2. A robot calibration method based on feature differentiation learning neural network model as claimed in claim 1, characterized in that: In step 1, the robot binocular vision information acquisition device includes an imaging module, a support module for supporting the imaging module, and a lighting module for providing lighting for the imaging module, wherein the imaging module includes a main view three-dimensional camera, a main view two-dimensional camera and a lens.

3. A robot calibration method based on feature differentiation learning neural network model as claimed in claim 1, characterized in that: In step 2, the evidence modeling module uses average pooling to transform the multi-dimensional feature map of the deformable convolutional layer into a one-dimensional feature map, and based on the evidence theory, combines the eigenvalues ​​and their weights and biases to transform the features into evidence, and then divides the feature categories according to the evidence weights. The specific calculation process is as follows: In the formula, w ij Represents the weight of evidence after feature transformation in row i and column j of the feature graph. represents the eigenvalue of the i-th row and j-th column in the feature map, K is the total number of features in the feature map, and β ij is the weight value of the feature of the i-th row and j-th column in the one-dimensional standard convolution, λ ij is the bias of the feature of the i-th row and j-th column in the one-dimensional standard convolution, μ ij is the average activation value of the feature map, RF is a credible feature, and UF is an unreliable feature.

4. A robot calibration method based on feature differentiation learning neural network model as claimed in claim 3, characterized in that: In step 2, the evidence fusion module obtains the support of all credible features for the detection results and the support of all uncredible features for the complement of the detection results according to the basic probability distribution function and evidence combination rules in the evidence theory, and the calculation process is as follows: In the formula, Represents the weight of evidence after the conversion of the credible feature in the i-th row and j-th column of the feature graph, represents the weight of evidence after the untrustworthy feature in the nth row and mth column of the feature graph is transformed, R is the detection result, is the complement of the detection result, m(·) is the basic probability distribution function, s is the support, and (m⊕m)(R) is the evidence combination rule.

5. A robot calibration method based on feature differentiation learning neural network model as claimed in claim 4, characterized in that: Step 4 is specifically expressed as follows: Loss T =P U (Loss C +Loss CIoU ) In the formula, p a is the predicted probability of category a, y a is the true value of category a, ρ 2 (b,b gt ) is the Euclidean distance between the center of the predicted box and the center of the real box, w gt and h gt is the width and height of the real box, w and h are the width and height of the predicted box, c is the diagonal distance of the minimum closure area, αυ is the CIoU adjustment factor, P U is the uncertainty penalty factor, U R is the uncertainty of the test result R, Bel(·) is the trust function, and Pl(·) is the likelihood function.

6. A robot calibration method based on feature differentiation learning neural network model as claimed in claim 1, characterized in that: In step 6, a T-Net network is first used to identify the center point of the target object, and then the identified center point of the object is converted into the coordinate origin by multiplying the coordinate transformation matrix, and then the three-dimensional bounding box of the reference object is obtained through the PointNet network.

Images