A robot calibration method based on feature differentiation learning neural network model

Abstract

The application provides a robot calibration method based on a feature difference learning neural network model, which calibrates a robot work reference object through a trusted feature discrimination convolutional neural network, a feature difference learning strategy and a three-dimensional bounding box prediction network based on a view cone-link dynamic graph. The application combines multi-modal visual information data to construct a trusted feature discrimination convolutional neural network, a feature difference learning strategy and a three-dimensional bounding box prediction network based on a view cone-link dynamic graph, solves the problem that existing robot calibration methods are easily affected by multi-form reference objects and random noise, and improves the robot calibration precision in a large-scale production environment.

Description

TECHNICAL FIELD

[0001] The application relates to a robot calibration method based on a feature difference learning neural network model and belongs to the technical field of industrial robot calibration. BACKGROUND

[0002] Industrial robots need to calibrate reference objects according to work tasks in a large-scale production environment to determine the relative positions between the robots and the reference objects in a work space, thereby laying a foundation for stable work of an end effector. Due to different robot work tasks, the types, shapes and positions of the reference objects are not the same. At present, a robot reference object calibration method mainly uses matrix operation or feature point extraction algorithm to solve a conversion matrix or a transformation relationship between coordinate systems, so as to realize accurate calibration and positioning of a robot vision system. However, the existing method faces many challenges in a large-scale production environment, in which the shapes of the reference objects are various and random noise has an adverse effect on feature extraction of the reference objects, so that the calibration accuracy of the reference objects is limited. Therefore, it is urgent to research a new robot vision calibration method suitable for high-noise multi-shape reference objects, so as to provide more comprehensive and accurate scene perception ability for the robot, thereby improving the robot vision calibration accuracy in a complex production environment. SUMMARY

[0003] The technical problem to be solved by the application is that the existing robot calibration method is easily affected by multi-shape reference objects and random noise.

[0004] In order to solve the above technical problem, the technical scheme of the application is to disclose a robot calibration method based on a feature difference learning neural network model, characterized by comprising the following steps:

[0005] Step 1: After a robot binocular vision information acquisition device is built, multi-modal vision information is acquired, a training set and a test set are constructed, and the training set and the test set are used for subsequent calibration model training and testing;

[0006] Step 2: a trusted feature discrimination convolutional neural network is constructed, the trusted feature discrimination convolutional neural network comprises a feature extraction module, an evidence modeling module, an evidence fusion module and an output module, wherein:

[0007] The feature extraction module is used for extracting features of work reference objects in an input image;

[0008] The evidence modeling module converts the features and their coefficients extracted by the feature extraction module into detected support evidence based on evidence theory, and divides the associated features into trusted features or untrusted features according to the weights of the evidence;

[0009] The evidence fusion module calculates the support degree of all trusted features to the detection result and the support degree of all untrusted features to the complement of the detection result based on the evidence theory and the evidence weights of trusted features and untrusted features;

[0010] The output module converts the extracted features into predicted reference object categories and position information;

[0011] Step 3: According to the trusted feature discrimination convolutional neural network of step 2, a trusted loss function is constructed, an uncertainty penalty factor is introduced, the learning cost of low uncertainty detection error is increased, and the learning cost of high uncertainty recognition is reduced, so as to guide the network to pay attention to trusted feature learning and ignore untrusted feature learning during the training process;

[0012] Step 4: Training the trusted feature discrimination convolutional neural network:

[0013] The two-dimensional image data training set obtained in step 1 is combined with the trusted feature discrimination convolutional neural network constructed in step 2 and the feature differentiation learning strategy established in step 3 to train the trusted feature discrimination convolutional neural network, and the optimal network model is saved;

[0014] Step 5: Detecting the bounding box of the work reference object in the two-dimensional image:

[0015] The trained trusted feature discrimination convolutional neural network in step 4 is used to output the two-dimensional bounding box of the work reference object in the two-dimensional image data set, and the label file is saved, and then the two-dimensional bounding box region is projected into the three-dimensional point cloud obtained in step 1 to form a view candidate region, and the view candidate region is serialized and deserialized to save the label of the two-dimensional bounding box;

[0016] Step 6: Constructing a three-dimensional bounding box prediction network based on a view-link dynamic graph:

[0017] The link dynamic graph network is used to extract the correlation features between points based on the view point cloud candidate region, and output the probability that each point belongs to the reference object, and according to the obtained reference object probability, the T-Net network and the PointNet network are used to predict the three-dimensional bounding box of the reference object;

[0018] Step 7: Training the three-dimensional bounding box prediction network based on the view-link dynamic graph:

[0019] The three-dimensional bounding box prediction network based on the view-link dynamic graph is trained using the point cloud of the view point cloud candidate region, and the optimal training model is saved.

[0020] Step 8: Test set calibration:

[0021] The trained credible feature discriminant convolutional neural network of step 4 and the three-dimensional bounding box prediction network based on the cone-link dynamic graph in step 7 are used to test the calibration of the job reference on the three-dimensional point cloud data test set and the two-dimensional image data test set in step 1.

[0022] Preferably, in step 1, the robot binocular vision information acquisition device comprises an imaging module, a support module for supporting the imaging module, and an illumination module for providing illumination for the imaging module, wherein the imaging module comprises a front-view three-dimensional camera, a front-view two-dimensional camera, and a lens.

[0023] Preferably, in step 2, the evidence modeling module converts the multi-dimensional feature map of the deformable convolutional layer into a one-dimensional feature map using average pooling, and converts the features into evidence based on evidence theory, combining feature values and their weights and biases, and then divides the feature categories according to the evidence weights. The specific calculation process is as follows:

[0024]

[0025] In the formula, w ij represents the converted evidence weight of the feature in the i-th row and j-th column of the feature map, represents the feature value in the i-th row and j-th column of the feature map, K is the total number of features in the feature map, β ij is the weight value of the feature in the i-th row and j-th column in one-dimensional standard convolution, λ ij is the bias of the feature in the i-th row and j-th column in one-dimensional standard convolution, μ ij is the average activation value of the feature map, RF is a credible feature, and UF is an untrustworthy feature.

[0026] Preferably, in step 2, the evidence fusion module obtains the support degree of all credible features for the detection result and the support degree of the complement of all untrustworthy features for the detection result according to the basic probability assignment function and the evidence combination rule in the evidence theory, and the calculation process is as follows:

[0027]

[0028] In the formula, represents the converted evidence weight of the credible feature in the i-th row and j-th column of the feature map, represents the converted evidence weight of the untrustworthy feature in the n-th row and m-th column of the feature map, R is the detection result, is the complement of the detection result, m(·) is the basic probability assignment function, s is the support degree, is the evidence combination rule.

[0029] Preferably, step 4 is specifically represented as follows:

[0030] Loss T=P U (Loss C +Loss CIoU )

[0031]

[0032] In the formula, p a Let y be the predicted probability of class a. a ρ represents the true value of category a. 2 (b,b gt ) represents the Euclidean distance between the center of the predicted bounding box and the center of the ground truth bounding box, w gt and h gt Let w and h be the width and height of the ground truth bounding box, respectively; let c be the diagonal distance of the minimum closure region; let αυ be the CIoU adjustment factor; and let P be the distance to the predicted bounding box. U U is the uncertainty penalty factor. R Let R be the uncertainty of the detection result, Bel(·) be the confidence function, and Pl(·) be the likelihood function.

[0033] Preferably, in step 6, the center point of the target object is first identified by a T-Net network, then the identified center point of the object is transformed into the origin of the coordinate system by multiplying by a coordinate transformation matrix, and finally the three-dimensional bounding box of the reference object is obtained by a PointNet network.

[0034] To address the uncertainty in data caused by the diverse shapes of robot operation reference objects and random noise, the method disclosed in this invention first utilizes a reliable feature discriminative convolutional neural network to identify valid reference object features and analyze the reliability of these features. Secondly, it employs a feature differentiation learning strategy to assign different learning cost weights to the recognition errors of reference objects with different reliability levels, enhancing error feedback for high-reliability feature learning and improving the detection accuracy of robot operation reference objects in 2D images. Finally, it utilizes a 3D bounding box prediction network based on a view frustum-linked dynamic graph to project the 2D detected bounding boxes of robot operation reference objects onto a 3D point cloud and output the 3D bounding box information of the robot operation reference objects.

[0035] This invention employs a robot calibration method based on a feature-differentiated learning neural network model for robot operation reference object calibration. By combining multimodal visual information data, a reliable feature discrimination convolutional neural network, a feature-differentiated learning strategy, and a 3D bounding box prediction network based on a view frustum-linked dynamic graph are constructed. This solves the problem that existing robot calibration methods are easily affected by multi-morphological reference objects and random noise, thereby improving the robot calibration accuracy in large-scale production environments. Attached Figure Description

[0036] Figure 1 This illustrates the reliable feature discrimination convolutional neural network of the present invention;

[0037] Figure 2 A link dynamic graph network of the present application is illustrated;

[0038] Figure 3 A three-dimensional bounding box output network of the present application is illustrated. DETAILED DESCRIPTION

[0039] The present application will be further described below in connection with specific embodiments. It should be understood that these embodiments are only used to illustrate the present application and not used to limit the scope of the present application. Furthermore, it should be understood that after reading the content taught by the present application, those skilled in the art can make various modifications or changes to the present application, and these equivalent forms also fall within the scope defined by the appended claims of the present application.

[0040] As Figure 1 shown, the robot calibration method based on feature differentiation learning neural network model disclosed by the embodiment of the present application comprises the following steps:

[0041] Step 1: Build a robot binocular vision information acquisition device. The robot vision information acquisition device used in the present application mainly includes: an imaging module (including a main view three-dimensional camera, a main view two-dimensional camera and a lens) and a support module (a camera support), an illumination module (a bar light source).

[0042] Step 2: Collect multi-modal visual information. The acquisition cycle is set in advance, after the main view three-dimensional camera and the main view two-dimensional camera of step 1 receive the action instruction, the robot front view visual information is synchronously collected, and the corresponding three-dimensional point cloud data and two-dimensional image data are saved, the training set and the test set are randomly divided according to the ratio of 8:2, which are used for subsequent calibration model training and testing.

[0043] Step 3: Construct a trusted feature discrimination convolutional neural network. The features extracted by the deformable convolution are divided into trusted features and untrusted features by using the evidence modeling module and the evidence fusion module, and the features and their correlation coefficients are analyzed to obtain the uncertainty of the network representing the reference object under high noise. The trusted feature discrimination convolutional neural network mainly consists of a feature extraction module, an evidence modeling module, an evidence fusion module and an output module.

[0044] The feature extraction module is composed of 5 standard convolutional layers and 4 deformable convolutional layers, which are used to extract the features of the reference object in the input image.

[0045] The evidence modeling module converts the features and their coefficients extracted by the deformable convolution layer into detection support evidence based on evidence theory, and divides the associated features into reliable features or unreliable features according to the weight of the evidence. Specifically, the multi-dimensional feature map of the deformable convolution layer is converted into a one-dimensional feature map by using average pooling, and based on evidence theory, the features are converted into evidence in combination with the feature values and their weights and biases, and then the feature categories are divided according to the evidence weight. The specific calculation process is as follows:

[0046]

[0047]

[0048] wherein w ij represents the converted evidence weight of the feature in the i-th row and the j-th column of the feature map, represents the feature value in the i-th row and the j-th column of the feature map, K is the total number of features in the feature map, β ij is the weight value of the feature in the i-th row and the j-th column in the one-dimensional standard convolution, λ ij is the bias of the feature in the i-th row and the j-th column in the one-dimensional standard convolution, μ ij is the average activation value of the feature map, RF is the reliable feature, and UF is the unreliable feature.

[0049] The evidence fusion module calculates the support degree of all reliable features to the detection result and the support degree of all unreliable features to the complement of the detection result based on evidence theory using the evidence weight of the reliable features and the evidence weight of the unreliable features. Specifically, according to the basic probability assignment function and the evidence combination rule in the evidence theory, the support degree of all reliable features to the detection result and the support degree of all unreliable features to the complement of the detection result are obtained, and the specific calculation process is as follows:

[0050]

[0051]

[0052] wherein, represents the converted evidence weight of the reliable feature in the i-th row and the j-th column of the feature map, represents the converted evidence weight of the unreliable feature in the n-th row and the m-th column of the feature map, R is the detection result, is the complement of the detection result, m(·) is the basic probability assignment function, s is the support degree, is the evidence combination rule.

[0053] The output module converts the extracted features into prediction reference object categories and position information.

[0054] Step 4: Design feature differentiation learning strategy. According to the trusted feature discrimination convolutional neural network in step 3, a trusted loss function is constructed, an uncertainty penalty factor is introduced, the learning cost of low uncertainty detection error is increased, and the learning cost of high uncertainty recognition is reduced to guide the network to focus on trusted feature learning and ignore untrusted feature learning in the training process. The specific formula is as follows:

[0055] Loss T =P U (Loss C +Loss CIoU ) (10)

[0056]

[0057] In the formula, p a is the prediction probability of a class, y a is the true value of a class, ρ 2 (b,b gt ) is the Euclidean distance between the center of the predicted box and the center of the true box, w gt and h gt are the width and height of the true box, w and h are the width and height of the predicted box, c is the diagonal distance of the minimum closed region, αυ is the CIoU adjustment factor, P U is the uncertainty penalty factor, U R is the uncertainty when the detection result R, Bel(·) is the belief function, and Pl(·) is the likelihood function.

[0058] Step 5: Train the trusted feature discrimination convolutional neural network. Train the trusted feature discrimination convolutional neural network by combining the two-dimensional image data training set in step 2, the trusted feature discrimination convolutional neural network in step 3, and the feature differentiation learning strategy in step 4, and save the optimal network model.

[0059] Step 6: Detect the bounding box of the job reference object in the two-dimensional image. Use the trained trusted feature discrimination convolutional neural network in step 5 to output the two-dimensional bounding box of the job reference object in the two-dimensional image data set, save the label file, and then project the two-dimensional bounding box region into the three-dimensional point cloud obtained in step 2 to form a view candidate region. After binary serialization and deserialization, save the label of the two-dimensional bounding box.

[0060] Step 7: Construct a three-dimensional bounding box prediction network based on a view cone-linked dynamic graph. A linked dynamic graph network (LDGCNN) is used to extract the correlation features between points in the view cone point cloud candidate region and output the probability that each point belongs to the reference object. The overall structure of the LDGCNN network includes four layers of feature extraction, each layer consisting of an edge convolution feature extraction module. The edge convolution learns the neighborhood features between points, and the channel numbers of each layer of feature extraction are (64, 64, 64, 128), respectively. After four layers of feature extraction, the learned features of each layer are concatenated and input into a fully connected layer, then the global features of the point cloud are obtained through maximum pooling, and are concatenated with the features of each layer obtained in the feature extraction part. The concatenated global features are aggregated through four layers of multilayer perceptron to obtain the probability that each point belongs to the reference object. Compared with using PointNet to extract point features, using the linked dynamic graph network to extract point cloud features can provide rich target position information, which helps the model to locate the reference object.

[0061] According to the obtained reference object probability, the three-dimensional bounding box of the reference object is predicted. The network first identifies the center point of the target object through a T-Net network, then converts the identified object center point into the coordinate origin by multiplying the coordinate transformation matrix. The coordinate transformation is to facilitate the identification of the orientation of the reference object, and then the three-dimensional bounding box of the reference object is obtained through the PointNet bounding box identification network. The T-Net network structure is obtained by a weight-shared multilayer perceptron with a channel number of (128, 256, 512) and maximum pooling to obtain global features, then the global features are input into a multilayer perceptron (MLP) to obtain the residual of the center point, and finally the coordinate origin is converted to the identified center point through coordinate transformation. The coordinate transformation is to subtract the residual from all point clouds to obtain the point cloud in the coordinate system with the reference object center as the origin.

[0062] In the embodiment of the present application, a linked dynamic graph network is used to extract the correlation features between points in the view cone point cloud candidate region in step 6 and output the probability that each point belongs to the reference object, as shown in Figure 2 , and then a T-Net network and a PointNet network are used to output the three-dimensional bounding box of the reference object, as shown in Figure 3 .

[0063] Step 8: Train the three-dimensional bounding box prediction network based on the view cone-linked dynamic graph. The three-dimensional bounding box prediction network based on the view cone-linked dynamic graph is trained using the point cloud of the view cone point cloud candidate region, and the optimal trained model is saved.

[0064] Step 9: Test set calibration. Using the trained reliable feature discriminative convolutional neural network in step 5, and the frustum-linking dynamic graph based 3D bounding box prediction network in step 8, the test calibration of the working reference is performed on the 3D point cloud data test set and the 2D image data test set in step 2.

Claims

1. A robot calibration method based on feature differentiation learning of a neural network model, characterized by, The method comprises the following steps: Step 1: After the robot binocular vision information acquisition device is built, multi-modal visual information is collected, a training set and a test set are constructed, and are used for subsequent calibration model training and testing; Step 2: a trusted feature discriminant convolutional neural network is constructed, the trusted feature discriminant convolutional neural network comprises a feature extraction module, an evidence modeling module, an evidence fusion module and an output module, wherein: The feature extraction module is used for extracting the work reference object features in the input image; The evidence modeling module converts the features and their coefficients extracted by the feature extraction module into detection support evidence based on evidence theory, and divides the associated features of the evidence into trusted features or untrusted features according to the weight of the evidence; The evidence fusion module calculates the support degree of all trusted features to the detection result and the support degree of all untrusted features to the complement of the detection result based on evidence theory, using the evidence weight of the trusted features and the evidence weight of the untrusted features; The output module converts the extracted features into predicted reference object category and position information; Step 3: according to the trusted feature discriminant convolutional neural network of step 2, a trusted loss function is constructed, an uncertainty penalty factor is introduced, the learning cost of low-uncertainty detection errors is increased, and the learning cost of high-uncertainty recognition is reduced, to guide the network to focus on trusted feature learning and ignore untrusted feature learning during the training process; Step 4: training the trusted feature discriminant convolutional neural network: The two-dimensional image data training set obtained in step 1 is combined with the trusted feature discriminant convolutional neural network constructed in step 2 and the feature differentiation learning strategy established in step 3 to train the trusted feature discriminant convolutional neural network, and the optimal network model is saved; Step 5: detecting the bounding box of the work reference object in the two-dimensional image: The two-dimensional bounding box of the work reference object in the two-dimensional image data set is output by using the trusted feature discriminant convolutional neural network trained in step 4, and a label file is saved, and then the two-dimensional bounding box region is projected into the three-dimensional point cloud obtained in step 1 to form a view candidate region, and the label of the two-dimensional bounding box is saved after the view candidate region is serialized and deserialized; Step 6: constructing a three-dimensional bounding box prediction network based on a view-link dynamic graph: A link dynamic graph network is used to extract the correlation features between points based on the view point cloud candidate region, and output the probability that each point belongs to the reference object, and the T-Net network and the PointNet network are used to predict the three-dimensional bounding box of the reference object according to the obtained reference object probability; Step 7: training the three-dimensional bounding box prediction network based on the view-link dynamic graph: The three-dimensional bounding box prediction network based on the view-link dynamic graph is trained by using the point cloud of the view point cloud candidate region, and the optimal training model is saved; Step 8: test set calibration: The trusted feature discriminant convolutional neural network trained in step 4 and the three-dimensional bounding box prediction network based on the view-link dynamic graph in step 7 are used to test and calibrate the work reference object on the three-dimensional point cloud data test set and the two-dimensional image data test set in step 1. 2.The robot calibration method based on feature difference learning neural network model according to claim 1, wherein, In step 1, the robot binocular vision information acquisition device comprises an imaging module, a support module for supporting the imaging module, and an illumination module for providing illumination for the imaging module, wherein the imaging module comprises a main-view three-dimensional camera, a main-view two-dimensional camera and a lens. 3.The robot calibration method based on feature difference learning neural network model according to claim 1, wherein, In step 2, the evidence modeling module converts the multi-dimensional feature map of the deformable convolution layer into a one-dimensional feature map by using average pooling, and converts the feature into evidence based on the evidence theory, in combination with the feature value and its weight and bias, and then divides the feature categories according to the evidence weight, and the specific calculation process is as follows: where w ij represents the transformed evidence weight of the feature in the i-th row and j-th column of the feature map, represents the feature value in the i-th row and j-th column of the feature map, K is the total number of features in the feature map, β ij is the weight value of the feature in the i-th row and j-th column in one-dimensional standard convolution, λ ij is the bias of the feature in the i-th row and j-th column in one-dimensional standard convolution, μ is the average activation value of the feature map, RF is a reliable feature, and UF is an unreliable feature. 4.The robot calibration method based on feature difference learning neural network model according to claim 3, wherein, In step 2, the evidence fusion module obtains the support degree of all reliable features to the detection result and the support degree of the complement of all unreliable features to the detection result according to the basic probability assignment function and the evidence combination rule in the evidence theory, and the specific calculation process is as follows: wherein, represents the transformed belief weight of the i-th row and j-th column of the feature map, represents the transformed belief weight of the n-th row and m-th column of the feature map, R is the detection result, is the complement of the detection result, m(·) is the basic probability assignment function, s is the support degree, and (m⊕m)(R) is the evidence combination rule.

5. The robot calibration method based on feature differentiation learning neural network model according to claim 4, characterized in that, Step 4 is specifically represented by the following formula: Loss T = P U (Loss C + Loss CIoU ) In the formula, p a Let y be the predicted probability of class a. a ρ represents the true value of category a. 2 (b,b gt ) represents the Euclidean distance between the center of the predicted bounding box and the center of the ground truth bounding box, w gt and h gt Let w and h be the width and height of the ground truth bounding box, respectively; let c be the diagonal distance of the minimum closure region; let αυ be the CIoU adjustment factor; and let P be the distance to the predicted bounding box. U U is the uncertainty penalty factor. R Let R be the uncertainty of the detection result, Bel(·) be the confidence function, and Pl(·) be the likelihood function.

6. The robot calibration method based on feature-differentiated learning neural network model according to claim 1, wherein, In step 6, first, a T-Net network is used to identify the center point of the target object, then the identified object center point is converted into the coordinate origin by multiplying the coordinate transformation matrix, and then the three-dimensional bounding box of the reference object is obtained through the PointNet network.

Images