System and method for validation of vision pose estimation models for random bin picking system

Abstract

A method for validation of a pose estimation model is provided. The method includes selecting critical features associated with an industrial part. Images of the industrial part are captured. A six-degree of freedom (6DoF) pose of the industrial part is predicted, which includes a 6DoF pose for each critical feature associated with a computer-aided design (CAD) model of the industrial part. The 6DoF pose of each critical feature is converted from the CAD model of the industrial part to predicted 2D coordinates in the image space. The predicted 2D coordinates are projected on the images of the industrial part. An accuracy of the pose prediction model is evaluated based on comparing a distance between each critical feature identified in the images of the industrial part and the corresponding predicted 2D coordinates in the images of the industrial part with a position accuracy distance threshold.

Description

SYSTEM AND METHOD FOR VALIDATION OF VISION POSE ESTIMATION MODELS FOR RANDOM BIN PICKING SYSTEMFIELD

[0001] The present disclosure relates to a random bin picking system. In particular, the present disclosure relates to verification of a six degrees of freedom (6D0F) pose estimation accuracy of a vision-based pose estimation perception system in robotic arm-based random bin picking applications.BACKGROUND

[0002] Commonly, deep learning-based neural network models that implement six degrees of freedom (6DoF) pose estimation, use training data that solely consists of synthetic generated images due to the labeling challenges associated with real -world captured images. Validation of these models is performed using a synthetically generated dataset. A real-world validation dataset is challenging and time consuming to obtain. Due to domain gap and lacking labeled data, the performance of a pose estimation model on real industrial parts during random bin picking operations is undetermined.SUMMARY

[0003] A first aspect of the present disclosure provides a method for validation of a vision-based pose estimation model applied to real -world images, the method comprising: selecting a set of critical features associated with an industrial part; capturing, using a vision system, one or more images of the industrial part present placed in a bin; predicting, using a pose prediction model, a six-degree of freedom (6DoF) pose of the industrial part within the bin, wherein predicting the 6DoF pose of the industrial part within the bin comprises predicting a 6DoF pose for each critical feature of the set of critical features associated with a computer-aided design (CAD) model of the industrial part; converting an assigned 6DoF pose of each critical feature of the set of critical features from the CAD model of the industrial part to reference 2D coordinates in an image space based on calibrated camera intrinsics and the predicted 6DoF pose of the industrial part; projecting the predicted 2D coordinates on the one or more images of the industrial part; and evaluating an accuracy of the pose prediction model based on comparing a distance between each critical feature of the set of critical features identified in the one or more images of the industrial part and the correspondingpredicted 2D coordinates in the one or more images of the industrial part with a position accuracy distance threshold.

[0004] According to an implementation of the first aspect, converting the assigned 6D0F pose of each critical feature of the set of critical features to predicted two-dimensional (2D) coordinates in the image space is based on a calibrated camera intrinsics and the predicted 6D0F pose of the part.

[0005] According to an implementation of the first aspect, the one or more images of the industrial part along with the predicted 2D coordinates are displayed in a graphical user interface (GUI).

[0006] According to an implementation of the first aspect, the method further comprises: determining that the pose prediction model is accurate based on determining that the distance between each critical feature of the set of critical features and the corresponding predicted 2D coordinates in the one or more images of the industrial part is less than the position accuracy distance threshold; and converting the predicted 6DoF pose of each critical feature of the set of critical features of the industrial part from a camera coordinate frame to a robot base frame based on the predicted 6DoF pose of the industrial part and a homogeneous transformation matrix; and providing the converted predicted 6DoF pose of each critical feature of the set of critical features to a robotic arm, wherein the robotic arm is configured to move a tool-tip mechanically coupled to the robotic arm to approach at least one critical feature of the set of critical features of the industrial part.

[0007] According to an implementation of the first aspect, the method further comprises: discarding the predicted 6DoF pose; and determining that the pose prediction model is inaccurate based on determining that the distance between the set of critical features and the predicted 2D coordinates in the one or more images of the industrial part is greater than the position accuracy distance threshold.

[0008] According to an implementation of the first aspect, the position accuracy distance threshold is empirically determined based on the industrial part and a gripper design.

[0009] A second aspect of the present disclosure provides a method for validation of a vision-based pose estimation model applied to real -world images, the method comprising: selecting a set of critical features associated with an industrial part; capturing, using a vision system, one or more images of the industrial part present placed in a bin; predicting, using a pose prediction model, a 6DoF pose of the industrial part within the bin, wherein predicting the 6DoF pose of the industrial part within the bin comprises predicting a 6DoF pose for each critical feature of a set of critical features associated with a computer-aided design (CAD)model of the industrial part; converting the predicted 6D0F pose of each critical feature of the set of critical features of the industrial part from a camera coordinate frame to a robot base frame based on the predicted 6D0F pose of the industrial part and a homogeneous transformation matrix; providing the converted predicted 6D0F pose of each critical feature to a robotic arm, wherein the robotic arm is configured to move a tool-tip mechanically coupled to the robotic arm to approach the industrial part on at least one critical feature of the set of critical features; and evaluating an accuracy of the pose estimation model based on a distance measured between a position of the tool-tip of the robotic arm with respect to a predicted 6D0F pose of the at least one critical feature and a corresponding reference position of the at least one critical feature on the industrial part.

[0010] According to an implementation of the second aspect, the method further comprises: determining that the pose prediction model is accurate based on determining that the distance between the tool-tip of the robotic arm and the corresponding reference position of the at least one critical feature on the industrial part is less than a position accuracy distance threshold.

[0011] According to an implementation of the second aspect, the method further comprises: discarding the predicted 6DoF pose; and determining that the pose prediction model is inaccurate based on determining that the distance between the tool-tip of the robotic arm and the corresponding reference position of the at least one critical feature on the industrial part is greater than a position accuracy distance threshold.

[0012] According to an implementation of the second aspect, the position accuracy distance threshold is empirically determined based on the industrial part and a gripper design.

[0013] A third aspect of the present disclosure provides a method for validation of a vision-based pose estimation model applied to real -world images, the method comprising: selecting a set of critical features associated with an industrial part; capturing, using a vision system, one or more images of the industrial part present placed in a bin; predicting, using a pose prediction model, a six-degree -of-freedom (6DoF) pose of the industrial part within the bin, wherein predicting the 6DoF pose of the industrial part within the bin comprises predicting a 6DoF pose for each critical feature of the set of critical features associated with a computer-aided design (CAD) model of the industrial part; converting an assigned 6DoF pose of each critical feature of the set of critical features from the CAD model of the industrial part to predicted two-dimensional (2D) coordinates in an image space based on the predicted 6DoF pose of the industrial part; projecting the predicted 2D coordinates on the one or more images of the industrial part; determining that a distance between each critical feature of theset of critical features identified in the one or more images of the industrial part and the corresponding predicted 2D coordinates in the one or more images of the industrial part is less than a position accuracy distance threshold; converting the predicted 6D0F pose of each critical feature of the set of critical features of the industrial part from a camera coordinate frame to a robot base frame based on the predicted 6D0F pose of the industrial part and a homogeneous transformation matrix; providing the converted predicted 6D0F pose of each critical feature to a robotic arm, wherein the robotic arm is configured to move a tool-tip mechanically coupled to the robotic arm to approach at least one critical feature of the set of critical features of the industrial part; and evaluating an accuracy of the pose estimation model based on a distance measured between a position of the tool-tip of the robotic arm with respect to a predicted 6D0F pose of the at least one critical feature and a corresponding reference position of the at least one critical feature on the industrial part.

[0014] According to an implementation of the third aspect, the method further comprises: determining that the pose prediction model is accurate based on determining that the distance between the tool-tip of the robotic arm and the corresponding reference position of the at least one critical feature on the industrial part is less than a position accuracy distance threshold.

[0015] According to an implementation of the third aspect, the one or more images of the industrial part along with the predicted 2D coordinates are displayed in a graphical user interface (GUI).BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Subject matter of the present disclosure will be described in even greater detail below based on the exemplary figures. All features described and / or illustrated herein can be used alone or combined in different combinations. The features and advantages of various embodiments will become apparent by reading the following detailed description with reference to the attached drawings, which illustrate the following:

[0017] FIG. 1 illustrates a simplified diagram for a random bin picking system, according to one or more examples of the present disclosure;

[0018] FIG. 2 illustrates exemplary diagrams related to selecting of reference checkpoints in CAD mesh model for accuracy evaluation of the pose prediction;

[0019] FIG. 3 illustrates exemplary diagrams related to a transformation matrix associated with a random bin picking system, according to one or more examples of the present disclosure;

[0020] FIG. 4 illustrates exemplary diagrams related to a transformation matrix associated with a random bin picking system, according to one or more examples of the present disclosure;

[0021] FIG. 5 illustrates a robotic arm performing verification of a pose estimation model associated with a random bin picking system, according to one or more examples of the present disclosure;

[0022] FIG. 6 illustrates verifying the pose estimation model associated with the random bin picking system by back-projecting critical feature vertices, according to one or more examples of the present disclosure;

[0023] FIG. 7 illustrates predicting critical feature vertices of an industrial part, according to one or more examples of the present disclosure;

[0024] FIG. 8 illustrates a simplified block diagram of one or more devices or systems within the exemplary environment of FIG. 1 , according to one or more examples of the present disclosure;

[0025] FIG. 9 illustrates a process performed by a controller as part of a random bin picking system, according to one or more examples of the present disclosure; and

[0026] FIG. 10 illustrates another process performed by a controller as part of a random bin picking system, according to one or more examples of the present disclosure; and

[0027] FIG. 11 illustrates another process performed by a controller as part of a random bin picking system, according to one or more examples of the present disclosure.DETAILED DESCRIPTION

[0028] Examples of the presented application will now be described more fully hereinafter with reference to the accompanying FIGS., in which some, but not all, examples of the application are shown. Indeed, the application may be exemplified in different forms and should not be construed as limited to the examples set forth herein; rather, these examples are provided so that the application will satisfy applicable legal requirements.Where possible, any terms expressed in the singular form herein are meant to also include the plural form and vice versa, unless explicitly stated otherwise. Also, as used herein, the term “a” and / or “an” shall mean “one or more” even though the phrase “one or more” is also used herein. Furthermore, when it is said herein that something is “based on” something else, itmay be based on one or more other things as well. In other words, unless expressly indicated otherwise, as used herein “based on” means “based at least in part on” or “based at least partially on.”

[0029] High speed automated assembly lines involve robots that are responsible for bringing together different industrial parts to build an object. Machine tending robots of highspeed assembly lines may include tool-tips that may be configured to pick up industrial parts from different bins and place the parts picked up on a conveyer belt for further processing. In some embodiments, further processing of industrial parts may include other robot arms picking up the part for assembly, blow off, wash, deburring, gauging, inspection, etc. High speed automated assembly lines are generally used in assembling automobiles, airplanes, printed circuit boards, and the like. For the high speed automated assembly lines to function smoothly, the industrial parts that are randomly positioned in a bin may be required to be correctly picked up. In order for the industrial parts to be picked up in the correct manner, a perception system may be required to identify the industrial parts that are randomly placed in the bin and a pose estimation model may precisely predict a 6D0F pose in space of the industrial parts in order to correctly grasp the industrial parts using the robot arm and correctly place the industrial parts on a convey belt or perform subsequent processing. This disclosure provides methods for evaluation and verification the pose prediction accuracy when a vision-based pose prediction model is used to predict the 6D0F poses for industrial parts randomly distributed in a bin based on real images captured by cameras.

[0030] The pose prediction model can be designed with feature-based computer vision techniques, point cloud or depth based registration or pose search techniques, and learningbased Al techniques. Commonly, validation of computer-vision based pose prediction models is performed using a synthetically generated dataset. A real-world validation dataset is challenging and time consuming to obtain because real-world data includes noise and other disturbances that may need to be ignored. Due to domain gap and lacking labeled data, the performance of a pose estimation model on real industrial parts during random bin picking operations is undetermined.

[0031] Embodiments of the present disclosure describe a process to verify the performance of a vision-based pose predictor to predict a 6DoF pose for an industrial part in robotic arm-based random bin picking applications. The pose estimation verification is an important step to efficiently validate the performance of the pose prediction model in real world circumstances before being fully deployed. With accurate pose estimation, roboticmachinery may be used to automatically pick up the parts in the proper manner so that they can be efficiently used in further applications such as assembly.

[0032] FIG. 1 illustrates a simplified diagram for a random bin picking system, according to one or more examples of the present disclosure. System 100 includes a controller 104, a perception system 108, a robotic arm 102, and a memory 106. The perception system 108 performs an evaluation of pose estimation models. The pose estimation models may be designed using conventional computer vision techniques, point cloud registration techniques, and may consists of one or more deep learning artificial neural networks. In this process, the perception system 108 is used to determine the accuracy of a vision-based pose estimation model 110 (also known as pose estimation model 110) that predicts a 6-degree-of-freedom (6DoF) pose of identified industrial parts that are randomly placed in a bin. In some embodiments, once the accuracy of the vision-based pose estimation model 110 is validated, this information may be used to efficiently pick up the industrial part using a robotic arm for further assembly processing. In some embodiments, the perception system 108 may also include an object detection model to identify industrial parts that are randomly placed in the bin. Once industrial parts are identified, a 6DoF pose of the identified industrial part is predicted using a vision-based pose estimation model 110.

[0033] The perception system 108 includes imaging devices 114 that captures a set of images that are related to real-world random bin scenes. In some embodiments, a bin may include a collection of industrial parts that are randomly distributed throughout the bin. The imaging devices 114 may be used to photograph the bin. In some cases, the imaging devices 114 may include a monochrome camera and / or a color camera for taking pictures of the bins that are filled with randomly distributed parts. In some other cases, the imaging devices 114 may include multiple cameras arranged at different angles to capture pictures of the industrial part in the bin from different angles. The multiple cameras of the imaging devices 114 may also be arranged at a different altitude with respect to the bin, so as to capture the depth of the industrial parts randomly placed in the bin. In some embodiments, once the different images are captured by the imaging devices 114, the images are provided to an object detection model. The object detection model, implemented using a neural network, may be used to identify individual industrial parts from the industrial parts randomly distributed in the bin. The industrial objects that are identified in the bin are then provided to the vision-based pose estimation model 110. In some embodiments, the vision-based pose estimation model 110 and the object detection model 116 are incorporated in the vision-based pose estimation model 110. In such cases, once the different images are captured by the imaging devices 114,the image information is provided to a pose estimation model 110. An object detection model of the vision-based pose estimation model 110 may be used to determine individual industrial parts from the industrial parts randomly distributed in the bin and the vision-based pose estimation model 110 may perform a 6DoF pose estimation process on the detected individual industrial parts. In some embodiments, the 6DoF pose estimation process may be refined using deep learning methods, iteratively closet point registration, and / or multi-view geometry algorithms. The output of the vision-based pose estimation model 110 may be in the form of a six degrees of freedom (6DoF) pose of the industrial part.

[0034] The vision-based pose estimation model 110 and object detection model 116 may be stored in computing accelerator 112. In some embodiments, the vision-based pose estimation model 110 is applied by the controller 104 to the images of the bin full of parts captured by the imaging devices 114. The pose estimation model 110 is used to predict a 6DoF pose for an industrial part in the collection of industrial parts that are randomly distributed in the bin. In such cases, the vision-based pose estimation model 110 may take the images of the bin captured by the imaging devices 114 and process the images to predict the pose of parts that are randomly distributed in the bin.

[0035] The vision-based pose estimation model 110 may be developed with traditional computer vision techniques and may be trained or learned using Al-based models. A computer-aided design (CAD) mesh model of the identified industrial part may be used for vision-based pose prediction models and may be stored in memory 106. For computing acceleration, the Al-based neural network models may be stored in the memory on the accelerated computing device, graphics processing unit (GPU) 112.

[0036] In some embodiments, the critical feature vertices of the industrial part are selected in the CAD model of the industrial part because they are visible, easily distinguishable, and easily perceived by human eyes. Upon selecting the critical feature vertices, also defined as checkpoints, the coordinates and 6DoF pose of such vertices in a model coordinate frame are determined. The coordinates of the 6DoF pose of the critical feature vertices in the model coordinate frame may be determined relative to the origin of the CAD mesh model. The coordinates of the critical feature vertices may be saved along with the CAD mesh model.

[0037] In some embodiments, the vision-based pose estimation model 110 determines a 6DoF pose of an identified industrial part randomly distributed in a bin based on images of the identified industrial part captured by the imaging devices 114. The 6DoF pose of the identified industrial part may be determined in a camera coordinate frame. For example, eachcamera of the imaging devices 114 may be assigned a camera coordinate frame that is centered on the respective camera. In some embodiments, a 6DoF pose of the identified industrial part may be determined with respect to each different camera coordinate frame, based on the image of the identified industrial part captured by the respective camera. Using the determined 6DoF poses of the identified individual industrial part for each camera in the imaging devices 114 in the camera coordinate frame and the 6DoF of the critical feature vertices of the identified industrial part in the model coordinate frame, a set of two dimensional (2D) pixel coordinates of the critical feature vertices may be computed in an image space using calibrated camera intrinsics. Additionally, given a position of the cameras of the imaging devices 114, with respect to the robotic arm, a position of the camera in a coordinate system of the robot, also known as a robot base frame, is determined. In some cases, a robot base frame is a unifying coordinate frame. The 6DoF poses of the identified industrial part in the different camera coordinate frames may be converted to a common robot base frame for further processing. Given the position of the camera in the robot base frame, the predicted 6DoF pose of the part, and the 6DoF pose of the critical feature vertices in the model coordinate frame, the 6DoF pose of each critical feature vertex of the identified industrial part may be computed in the robot base frame. The 6DoF pose of a tool center point (TCP) may also be generated with respect to the robot base 102.

[0038] The vision-based pose estimation model 110 predicts the poses of those parts in the captured images of the bin from the imaging devices 114. The vision-based pose estimation model 110 may contain subsequent pose refiners.

[0039] Once the pose of the industrial part is predicted, using the vision-based pose estimation model 110, the accuracy of the vision-based pose estimation model 110 may be verified. In order to verify the accuracy of the predicted pose of the vision-based pose estimation model 110, the controller 104 sends the 6DoF pose of each critical feature vertex converted to the robot base frame to the robotic arm 102. A pre-defined program in the robotic arm 102 may be used to move a tool-tip of the robotic arm 102 to physically approach each critical feature vertex of the identified industrial part in the bin. An operator and / or associated engineer may examine the stopping position of a tool-tip of the robotic arm 102 to see whether the robotic arm 102 closely approaches the critical feature vertices of the industrial part as defined in the CAD mesh model 116 associated with the part. In some embodiments, when physically approaching the critical feature vertices of the identified industrial part in the bin, the 6DoF pose for each critical feature vertex may be transformed to a 6DoF pose in the robot base frame using a transformation matrix. The operator and / orengineer may make the comparison by determining a distance between the position of the tool center point and / or tool tip based on the predicted critical feature vertices on the industrial part 6D0F and their respective critical feature vertices as saved along with the CAD mesh model in memory 106. In some cases, a distance may be calculated using controller 104. In such embodiments, the closer the predicted position of the critical feature vertices on the identified industrial part align with critical feature vertices saved along with the CAD mesh model 118, the higher the indicated pose estimation accuracy of the pose estimation model 110. Because the critical feature vertices are selected as they are easily distinguishable by human eyes, the operator and / or engineer may compare the position of the tool tip based on the predicted position of the critical feature vertices projected on the identified industrial part with the critical features of the industrial part.

[0040] A positioning accuracy distance threshold may be empirically determined depending on the industrial part, gripper design, and the application. The positioning accuracy distance threshold measured may be used to determine whether the pose estimation model 110 is accurate or inaccurate. In case the distance calculated between a position of the tool center point of the robotic arm 102 and the coordinates of the respective critical feature vertices derived from the CAD mesh model of the identified industrial part is less than the positioning accuracy distance threshold, the computing accelerator 112 determines that the pose estimation model is accurate. In case the distance calculated between a position of the tool center point of the robotic arm 102 and the coordinates of the respective critical feature vertices derived from the CAD mesh model of the identified industrial part is greater than the positioning accuracy distance threshold, the computing accelerator 112 determines that the pose estimation model is inaccurate. In some cases, a positioning accuracy distance threshold of 3 mm indicates a good picking accuracy among the random bin picking applications.

[0041] In some other cases, in order to verify the accuracy of the predicted pose of the pose estimation model 110, an additional verification step may be performed, before the 6DoF pose is fed to the robotic arm 102. As part of the additional verification, using the calibrated camera intrinsics and the predicted 6DoF pose of the industrial part in the camera coordinate frame, the controller 104 may back-project selected critical feature vertices of the CAD mesh model of the identified industrial part onto the images of the identified industrial parts captured by the imaging devices 114. Once the 6DoF critical feature vertices are identified and back-projected on the image of the identified industrial part captured by the imaging devices 114, an engineer and / or operator may perform a visual check of how well the back-projected checkpoints to the image space align with the critical features presentedon the identified industrial part in the 2D images for each camera of the imaging devices 114. The critical feature vertices shown in the 2D images are originated from the critical feature vertices saved in the CAD model of the industrial part. In some embodiments, the back- projecting of the 6D0F pose of the critical feature vertices selected from the CAD model of the industrial part is performed by transforming the 6D0F pose of the critical feature vertices from the model coordinate frame to pixels in the 2D image space using the calibrated camera intrinsics and predicted 6D0F pose of the part in the camera coordinate frame. In some cases, the pixels indicating the predicted critical feature vertices of the individual industrial part with a predicted 6D0F pose may be superimposed on an image of the industrial part or a CAD model of the industrial part. In performing the visual check, the engineer and / or operator checks whether the critical feature vertices that are back-projected on the images of the industrial part captured by the imaging devices 114 match the critical features on images of the identified industrial part in the 2D images vertices of the CAD mesh model 118 of the industrial part by comparing the pixels. In some cases, if the operator and / or engineer determines that the 6DoF pose of the critical feature vertices of the identified industrial part does not match the critical feature vertices from the CAD mesh model 118 that are back- projected on the image of the identified industrial part, the engineer and / or operator may discard the predicted pose of the industrial part without having the robot perform a physical approach with the tool center point on the industrial part in the bin. In some embodiments, a graphical user interface (GUI) is used to display the images captured by the imaging devices 114. The GUI may also be used to display the critical feature vertices that are back-projected on the images of the industrial part captured by the imaging devices 114.

[0042] In some embodiments, a combination of the tool tip approach of the robotic arm 102 and a comparison of the 2D pixel coordinates of the critical feature vertices and the predicted critical feature vertices may be used to determine the accuracy of the vision-based pose estimation model 110. For example, before approaching the critical feature vertices on real industrial parts in the bin, a check may be performed in an image space to determine how well the critical feature vertices of the identified industrial part align with the back-projection of the critical feature vertices from the CAD model of the identified industrial part. The back- projection of the critical feature vertices of the identified industrial part may be formed based on the predicted 6D pose of the identified industrial part. If the alignment is off, the physical approach of the tool-tip of the robot arm 102 may be skipped and the pose prediction error may be debugged. This offers a quick pose prediction check without moving the robot arm

[0043] FIG. 2 illustrates exemplary diagrams related to validation of a model associated with a random bin picking system, according to one or more examples of the present disclosure. FIG. 2 depicts a CAD mesh model 200 of an industrial part. The CAD mesh model 200 is stored in the memory 106. As discussed above, the CAD mesh model includes critical feature vertices 202, 204, and 206 as indicated on the CAD mesh model 200 of the industrial part.

[0044] For example, each critical feature vertex 202, 204, and 206 is selected from the CAD mesh model of the industrial part. The critical feature vertices may be selected manually or automatically detecting surface normal and features. Each critical feature vertex is represented using a 6DoF pose. The 6DoF pose of the critical feature vertices includes 3D translational coordinates (x, y, z). The 3D translational coordinates of the critical feature vertices are relative to the origin of the CAD mesh model. Each critical feature vertex is also assigned a 3D rotation is to form a 6DoF pose forthat critical feature vertex also relative to the origin of the CAD mesh model of the industrial part.

[0045] The predicted 6DoF pose of the industrial part consists of 3-dimensional (3D) translation (tx, ty, tz) and 3D rotation (rx, ry, rz). The 3D rotation may be expressed in many forms like Euler angles, quaternion (w,x,y,z), and rotation matrix (3x3 rotation matrix). The 6DoF pose of the industrial part reflects the translation and rotation of the industrial part in a certain coordinate frame, such as camera coordinate frame or robot base frame. In some cases, the vision-based pose estimation model predicts the 6D pose of the industrial part in the camera coordinate frame.

[0046] FIG. 3 illustrates exemplary diagrams related to a transformation matrix associated with a random bin picking system, according to one or more examples of the present disclosure. Once the images of the industrial part are captured by the imaging devices 114, the pose estimation model 110 uses the image for predicting a pose of the industrial part. Once the image of a bin with a random distribution of parts is captured, industrial parts present in the random bin are identified by the perception system 108. The pose prediction model 110 predicts a 6DoF pose of an identified industrial part. The predicted 6DoF pose of the identified industrial part is in a camera coordinate frame of a camera of the imaging devices 114. FIG. 3 shows the coordinate systems 302 and 304 associated with two different cameras of the imagine devices 114. For example, coordinate system 302 may be associated with a right camera and coordinate system 304 may be associated with a left camera. A 4x4 homogenous transformation matrix may be used to transform the coordinate system of the left camera coordinate frame 304 and the right camera coordinate frame 302 to the robot baseframe 312. The robot base frame 312 is a coordinate frame associated with the robotic arm 102. Transferring the different coordinate systems of the cameras 302 and 304 to the robot base frame 312 allows to provide accurate instructions to the robotic arm 102 to pick up the industrial part that is randomly distributed in the bin. In some cases, the 6DoF pose of the identified industrial part 310 from the camera coordinate frame 302 and 304 is transferred to the robot base frame 312 using the following formula: phase > phase y p camera r zi \ part ‘ camera ‘ part I1 / where Ppart is the 6DoF pose of the industrial part in the robot base frame 312,is the 6DoF pose of a camera of the imaging devices 114 with respect to the robot base frame 312, and Ppa™erais the 6DoF pose of the identified industrial part in the coordinate frame 302 or 304 of the right camera or the left camera.

[0047] In some other embodiments, 6DoF pose of critical feature vertices of the industrial part may also be determined with respect to the center of the industrial part as shown in the CAD model of the industrial part. The 6DoF pose of the critical feature vertices may be converted from the model coordinate system of the CAD model of the industrial part to the robot base frame 312 using the following formula: phase > phase y pcamera y ppart r critical feature vertexrcamerarpart Critical feature vertexwhere P^iticai feature vertice 'sthe 6D0F pose of a critical feature vertex with respect to the center of the industrial part as shown in the CAD mesh model. In some embodiments, the 6DoF pose of a critical feature vertex consists of its translational coordinates in the CAD model and a custom defined orientation depending on a picking angle of a robotic arm 102. In some embodiments, a 6DoF pose of the industrial part may be represented as the following matrix:where an, an, ais, a2i, a22, a23, a3i, a32, and a33 represent a 3x3 rotation matrix and ai4, a24, and a34 represent a translation matrix.

[0048] FIG. 4 illustrates exemplary diagrams related to a transformation matrix associated with a random bin picking system, according to one or more examples of the present disclosure. Diagram 400 of FIG. 4 shows a 6DoF pose of the critical feature vertices 402 and 404 with respect to the robot base frame 312 after the 6DoF pose was convertedfrom the model coordinate frame to the robot base frame 312 using the transformation matrix described above. Once the 6DoF pose of the critical feature vertices are converted from the model coordinate frame of the CAD mesh model to the robot base frame, the controller 104 may instruct the robotic arm 102 to approach each of the critical feature vertices of the identified industrial part. Once the tool tip of the robotic arm approaches a critical feature vertex, an operator and / or end user may determine a distance between the position of the tool tip and the coordinates of the respective critical vertex as derived from the CAD model of the identified industrial part. The coordinates of the respective critical vertex may be converted from a 6DoF pose in the CAD model that is based on a model coordinate frame to a robot base frame using the transformation matrices as described. In case the distance measured between the tool tip of the robotic arm 102 and the respective coordinates of the critical feature vertices in the robot base frame is less than a position accuracy threshold distance, the pose estimation model 110 is deemed to be accurate. Alternatively, the pose estimation model is deemed to be inaccurate.

[0049] In cases where the verification of the pose estimation model 110 is not performed by using the robotic arm 102 to approach the critical vertices of the industrial part, the verification of the pose estimation model 110 may be performed by back projecting the critical feature vertices on images of the identified industrial parts as captured by the imaging devices 114. When performing the back-projection, the coordinates of the 6DoF pose of the critical feature vertices that are in the model coordinate frame, with respect to the center of the industrial part, are converted to 2D pixel coordinates so that they can be back-projected onto images of the identified industrial part as captured by imaging devices 114. Elements 306 and 308 of FIG. 3 show an image space associated with the coordinate frames 302 and 304 respectively. The image space 306 is generated by transforming the 3D coordinate frame 302 associated with the right camera to a 2-dimensional (2D) image space. Similarly, the image space 308 is generated by transforming the 3D coordinate frame 304 associated with the right camera to a 2-dimensional (2D) image space. The transformation of the 6DoF pose to the 2D image space is performed using the following formula:.1. where w represents a scaling factor, x and y represent the 2D pixel coordinates of the industrial part in an image space 306 or 308 that are generated after transforming the X Y Zcoordinates of the industrial part that are provided in the model coordinate frame with respect to the center of the industrial part, and P represents a camera matrix that is generated by multiplying camera intrinsics K with a 6D pose of the industrial part within a camera coordinate system 302 or 304. In some cases, P is generated using the following formula:P = K[Rt] (5) where K is an intrinsics matrix associated with a camera of the imaging devices 114.Exemplary matrix K may be represented as:Fxrepresents a focal length in pixels in a horizontal direction, fyrepresents focal length in pixels in a vertical direction, cxrepresents a principal point and / or optical center in the horizontal direction, cyrepresents a principal point and / or optical center in the vertical direction and s represents a skew coefficient. The skew coefficient s is zero for image axes that are perpendicular. The [R t] matrix represents the Pparterawhich is the 6DoF pose of the identified industrial part in the coordinate frame 302 or 304 of the right camera or the left camera, as shown by equation (3).

[0050] Therefore, as shown by the equation below, the coordinates of the critical feature vertices as shown in the CAD model of an industrial part may be transformed into 2D pixel coordinates for back-projection on an image of the industrial part captured by the imaging devices 114 by:World Coordinates [X Y Z]-> Camera coordinates [Xc Yc Zc]-*Pixel coordinates [x y] (7)

[0051] The pixel coordinates that are generated using the above equations can help convert the 6DoF pose of the critical feature vertices identified in the CAD model of the industrial part and back-project the vertices on to images of the industrial part that are captured by the imaging devices 114 as they are randomly distributed in the bin. When the 6D pose of the industrial part is accurately predicted, the back-projected critical feature vertices using the predicted 6D pose should align well with the corresponding critical features shown on the images captured. The pixel distance between the back-projected checkpoints superimposed on the images of the identified industrial part and reference critical feature vertices of the identified industrial part is measured. The better alignment between these two sets of critical feature vertices, the more accurate the predicted pose is. In case the distance measured between back-projected critical feature vertices and the critical feature vertices isless than a position accuracy threshold distance, the pose estimation model 110 is deemed to be accurate. Alternatively, the pose estimation model is deemed to be inaccurate. The computation of the pixel distance and the comparison of the sets of critical feature vertices may be performed manually or by using controller 104.

[0052] FIG. 5 illustrates a robotic arm performing verification of a pose estimation model associated with a random bin picking system, according to one or more examples of the present disclosure. Layout 500 of FIG. 5 depicts a robotic arm 102 approaching a critical feature vertex on an identified industrial part randomly disposed in a bin 506. The approach of the tool tip of robotic arm 102 stops at point 502 above the identified industrial part. The approach of the tool tip of the robotic arm 102 may be performed based on a 6DoF coordinate of a critical feature vertex received from a CAD mesh model of the industrial part. Using a predicted 6DoF pose of the identified industrial part, 6DoF coordinates of critical feature vertices may be predicted, which may be provided to the robot arm 102. The robot arm 102 may be configured to approach the predicted critical feature vertices of the identified industrial part based on the predicted 6DoF pose of the critical feature vertices. The position of the tool tip of the robot arm 102 as it approaches the identified industrial part may be compared with coordinates of the respective critical feature vertex that are derived from the CAD mesh model 550 of the industrial part.

[0053] For example, point 504 on the identified industrial part may correspond to a critical feature vertex 552 as identified in the mesh model 550 of the industrial part. The 6DoF pose of the critical feature vertex is first calculated in the model coordinate frame and saved along with the CAD mesh model. The coordinates of the critical feature vertex 552 are converted from the model coordinate frame to the robot base frame as described with respect to FIGS. 3-5 above. The conversion of the coordinates may be performed based on images of the identified industrial part that are captured by imaging devices 114. In some cases, the coordinates of the 6DoF pose of the critical feature vertex 552 in the robot base frame may be compared with the position of the tool tip of the robotic arm 102. In case the distance between the coordinates of the 6DoF pose of the critical feature vertex converted to the robot base frame and the position A of the tool tip is less than a positioning accuracy distance threshold, the pose estimation model 110 is deemed to be accurate. Alternatively, if the distance between the coordinates of the 6DoF pose of the critical feature vertex converted to the robot base frame and the position A of the tool tip is greater than a positioning accuracy distance threshold, the pose estimation model 110 is deemed to be inaccurate. In some embodiments, the 6DoF pose A may consist of estimated 3D translation with default straight down approaching orientation or may consist of estimated 6DoF pose of the respective critical feature vertex.

[0054] FIG. 6 illustrates verifying the pose estimation model associated with the random bin picking system by back-projecting critical feature vertices, according to one or more examples of the present disclosure. In some cases, the verification of a pose estimation model 110 may be performed without a robot arm 102, by back-projecting critical feature vertices on images of the identified industrial part as captured by imaging devices 114.

[0055] Image 600 depicts an image of an identified industrial part, as captured by imaging devices 114. The image 600 may be provided to a pose estimation model 110 to predict a 6DoF pose of the identified industrial part. The predicted 6DoF pose of the identified industrial part in the image 600 may be used to predict 6DoF poses for the critical feature vertices as identified in the CAD model of the identified industrial part. As shown in image 600 of FIG. 6, points 602, 604, 606, and 608, correspond to critical feature vertices of the identified industrial part.

[0056] CAD model 650 of FIG. 6 depicts a CAD model of the industrial part. As discussed previously, the CAD model 650 of the industrial part includes critical feature vertices 652, 654, 656, and 658. A 6DoF pose may be associated with each critical vertex 652, 654, 656, and 658. The 6DoF poses of the critical feature vertices may be converted to 2D pixel coordinates that are then back-projected on the image 600 of the identified industrial part.

[0057] The critical feature vertices 602, 604, 606, and 608 are compared with the 2D pixel coordinates of back-projection of the critical feature vertices 652, 654, 656, and 658 from the CAD model 650. In some embodiments, using only translation coordinates of the selected checkpoints, the predicted 6D pose of the industrial part in the camera coordinate frame, and the calibrated camera intrinsics, the critical feature vertices in the CAD model can be back- projected onto the images captured by the camera. When the 6D pose of the industrial part is accurately predicted, the back-projected critical feature vertices using the predicted 6D pose should align well with the corresponding critical features showing on the images captured.

[0058] In case the distance (measured in pixels) between the critical feature vertices in the image of the industrial part and the 2D pixel coordinates of the back-projection of the critical feature vertices is less than a positioning accuracy distance threshold, the pose estimation model 110 is deemed to be accurate. Alternatively, if the distance between the pixels of the critical feature vertices in the image of the industrial part and the 2D pixel coordinates of the back-projection of the critical feature vertices is greater than a positioning accuracy distance threshold, the pose estimation model 110 is deemed to be inaccurate.

[0059] FIG. 7 illustrates predicting critical feature vertices of an industrial part, according to one or more examples of the present disclosure. In some embodiments, the critical feature vertices of the industrial part may be selected by the computing accelerator 112 of system100. The critical feature vertices are based on their ease of visibility, distinguishability, and perceptibility. The computing accelerator 112 may be configured to analyze the CAD model 700 of the industrial part to determine critical feature vertices such as 702, 704, and / or 706.

[0060] FIG. 8 is a block diagram of an exemplary system or device 800 within the system 100 such as the controller 104 or computing accelerator 112. The system 800 includes a processor 104, such as a central processing unit (CPU), and / or logic, that executes computer executable instructions for performing the functions, processes, and / or methods described herein. In some examples, the computer executable instructions are locally stored and accessed from a non-transitory computer readable medium, such as storage 810, which may be a hard drive or flash drive. Read Only Memory (ROM) 806 includes computer executable instructions for initializing the processor 804, while the random-access memory (RAM) 808 is the main memory for loading and processing instructions executed by the processor 804. The network interface 812 may connect to a wired network or cellular network and to a local area network or wide area network. The system 800 may also include a bus 802 that connects the processor 804, ROM 806, RAM 808, storage 810, and / or the network interface 812. The components within the system 800 may use the bus 802 to communicate with each other. The components within the system 800 are merely exemplary and might not be inclusive of every component within the controller 104. Additionally, and / or alternatively, the system 800 may further include components that might not be included within every entity of system 800. For instance, in some examples, the controller 104 might not include a network interface 812.

[0061] FIG. 9 illustrates a process performed by a controller as part of a random bin picking system, according to one or more examples of the present disclosure. However, it will be recognized that any of the following blocks may be performed in any suitable order and that the process 900 may be performed in any environment and by any suitable computing device and / or controller.

[0062] At 902, a set of critical feature vertices are selected in a CAD model of an industrial part. In some embodiments, the critical feature vertices, may be assigned to the CAD mesh model of industrial part that is stored in memory 106.

[0063] At 904, a pose estimation model is applied to images of the industrial part. For example, imaging devices 114 of system 100 shown in FIG. 1 may include multiple cameras arranged at different angles to capture images of a bin in which the industrial parts are randomly distributed.

[0064] At 906, the alignment of the back-projected critical feature vertices is observed. In some embodiments, as part of the additional verification, the controller 104 may project the6D0F pose of critical feature vertices of the industrial part, that are derived from the images of the industrial part on the image of the industrial part captured by the imaging devices 114. Once the 6D0F pose of checkpoints is identified and back-projected on the image, a user and / or operator may perform a visual check of the 6D0F pose of checkpoints in comparison with the respective critical feature vertices identified from the CAD mesh model of the industrial part.

[0065] At 908, an accuracy of the performance of the pose estimation model is determined. In some embodiments, an operator and / or engineer associated may examine the movement of the tool -tip of the robotic arm 102 to see whether the robotic arm approaches the critical feature vertices of the industrial part as defined in the CAD mesh model associated with the part.

[0066] FIG. 10 illustrates a process performed by a controller as part of a random bin picking system, according to one or more examples of the present disclosure. However, it will be recognized that any of the following blocks may be performed in any suitable order and that the process 1000 may be performed in any environment and by any suitable computing device and / or controller.

[0067] At 1002, a set of accuracy checkpoints are selected. In some embodiments, the accuracy checkpoints may be assigned to the CAD mesh model of industrial parts that are stored in the memory 106.

[0068] At 1004, a trained pose estimation model is applied to pictures taken by the imaging devices 114. Imaging devices 114 of system 100 shown in FIG. 1 may include multiple cameras arranged at different angles to capture images of a bin in which the industrial parts are randomly distributed.

[0069]

[0070] At 1006, a touch service is performed using calibrated robot TCP. In some embodiments, the controller 104 sends the 6DoF pose of checkpoints to the robotic arm 102. A pre-defined program in the robotic arm 102 may be used to move a tool-tip of the robotic arm 102 to physically touch each of the identified 6DoF pose of checkpoints on the industrial part in the bin.

[0071] At 1008, an accuracy of the performance of the pose estimation model is evaluated. In some embodiments, an operator and / or associated engineer may examine the movement of the tool-tip of the robotic arm 102 to see whether the robotic arm touches the critical feature vertices of the industrial part as defined in the CAD mesh model associated with the part.

[0072] FIG. 11 illustrates a process performed by a controller as part of a random bin picking system, according to one or more examples of the present disclosure. However, it will be recognized that any of the following blocks may be performed in any suitable order and thatthe process 1000 may be performed in any environment and by any suitable computing device and / or controller. Process 1000 is different from process 900 because process 900 includes an additional verification step before the touch service is performed.

[0073] At 1102, a set of accuracy checkpoints are selected. In some embodiments, the accuracy checkpoints may be assigned to the CAD mesh model of industrial parts that are stored in the memory 106.

[0074] At 1104, a trained pose estimation model is applied to pictures taken by the imaging devices 114. Imaging devices 114 of system 100 shown in FIG. 1 may include multiple cameras arranged at different angles to capture images of a bin in which the industrial parts are randomly distributed.

[0075] At 1106, the alignment of the back-projected accuracy of checkpoints is observed. In some embodiments, as part of the additional verification, the controller 104 may project the 6DoF pose of checkpoints of the industrial part, that are derived from the images of the industrial part on the image of the industrial part captured by the imaging devices 114. Once the 6DoF pose of checkpoints is identified and back-projected on the image, a user and / or operator may perform a visual check of the 6DoF pose of checkpoints in comparison with the respective critical feature vertices identified from the CAD mesh model of the industrial part.

[0076] At 1108, a touch service is performed using calibrated robot TCP. In some embodiments, the controller 104 sends the 6DoF pose of checkpoints to the robotic arm 102. A pre-defined program in the robotic arm 102 may be used to move a tool-tip of the robotic arm 102 to physically touch each of the identified 6DoF pose of checkpoints on the industrial part in the bin.

[0077] At 1110, an accuracy of the performance of the pose estimation model is evaluated. In some embodiments, an operator and / or associated engineer may examine the movement of the tool center point of the robotic arm 102 to see whether the robotic arm touches the critical feature vertices of the industrial part as defined in the CAD mesh model associated with the part.

[0078] While subject matter of the present disclosure has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive. Any statement made herein characterizing the invention is also to be considered illustrative or exemplary and not restrictive as the invention is defined by the claims. It will be understood that changes and modifications may be made, by those of ordinary skill in the art, within the scope of thefollowing claims, which may include any combination of features from different embodiments described above.

[0079] The terms used in the claims should be construed to have the broadest reasonable interpretation consistent with the foregoing description. For example, the use of the article “a” or “the” in introducing an element should not be interpreted as being exclusive of a plurality of elements. Likewise, the recitation of “or” should be interpreted as being inclusive, such that the recitation of “A or B” is not exclusive of “A and B,” unless it is clear from the context or the foregoing description that only one of A and B is intended. Further, the recitation of “at least one of A, B and C” should be interpreted as one or more of a group of elements consisting of A, B and C, and should not be interpreted as requiring at least one of each of the listed elements A, B and C, regardless of whether A, B and C are related as categories or otherwise. Moreover, the recitation of “A, B and / or C” or “at least one of A, B or C” should be interpreted as including any singular entity from the listed elements, e.g., A, any subset from the listed elements, e.g., A and B, or the entire list of elements A, B and C.

Claims

CLAIMSWhat is claimed is:

1. A method for validation of a vision-based pose estimation model, the method comprising: selecting a set of critical features associated with an industrial part; capturing, using a vision system, one or more images of the industrial part present placed in a bin; predicting, using a pose prediction model, a six-degree of freedom (6D0F) pose of the industrial part within the bin, wherein predicting the 6D0F pose of the industrial part within the bin comprises predicting a 6D0F pose for each critical feature of the set of critical features associated with a computer-aided design (CAD) model of the industrial part; converting an assigned 6D0F pose of each critical feature of the set of critical features from the CAD model of the industrial part to predicted 2D coordinates in an image space based on calibrated camera intrinsics and the predicted 6D0F pose of the industrial part; projecting the predicted 2D coordinates on the one or more images of the industrial part; and evaluating an accuracy of the pose prediction model based on comparing a distance between each critical feature of the set of critical features identified in the one or more images of the industrial part and the corresponding predicted 2D coordinates in the one or more images of the industrial part with a position accuracy distance threshold.

2. The method of claim 1, wherein converting the assigned 6DoF pose of each critical feature of the set of critical features to predicted two-dimensional (2D) coordinates in the image space is based on a calibrated camera intrinsics and the predicted 6DoF pose of the part.

3. The method of claim 1, wherein the one or more images of the industrial part along with the predicted 2D coordinates are displayed in a graphical user interface (GUI).

4. The method of claim 1, further comprising: determining that the pose prediction model is accurate based on determining that the distance between each critical feature of the set of critical features and the corresponding predicted 2D coordinates in the one or more images of the industrial part is less than the position accuracy distance threshold; andconverting the predicted 6D0F pose of each critical feature of the set of critical features of the industrial part from a camera coordinate frame to a robot base frame based on the predicted 6D0F pose of the industrial part and a homogeneous transformation matrix; and providing the converted predicted 6D0F pose of each critical feature of the set of critical features to a robotic arm, wherein the robotic arm is configured to move a tool-tip mechanically coupled to the robotic arm to approach at least one critical feature of the set of critical features of the industrial part.

5. The method of claim 1, further comprising: discarding the predicted 6DoF pose; and determining that the pose prediction model is inaccurate based on determining that the distance between the set of critical features and the predicted 2D coordinates in the one or more images of the industrial part is greater than the position accuracy distance threshold.

6. The method of claim 1, wherein the position accuracy distance threshold is empirically determined based on the industrial part and a gripper design.

7. A method for validation of a vision-based pose estimation model, the method comprising: selecting a set of critical features associated with an industrial part; capturing, using a vision system, one or more images of the industrial part present placed in a bin; predicting, using a pose prediction model, a 6DoF pose of the industrial part within the bin, wherein predicting the 6DoF pose of the industrial part within the bin comprises predicting a 6DoF pose for each critical feature of a set of critical features associated with a computer- aided design (CAD) model of the industrial part; converting the predicted 6DoF pose of each critical feature of the set of critical features of the industrial part from a camera coordinate frame to a robot base frame based on the predicted 6DoF pose of the industrial part and a homogeneous transformation matrix; providing the converted predicted 6DoF pose of each critical feature to a robotic arm, wherein the robotic arm is configured to move a tool-tip mechanically coupled to the robotic arm to approach the industrial part on at least one critical feature of the set of critical features; and evaluating an accuracy of the pose estimation model based on a distance measured between a position of the tool-tip of the robotic arm with respect to a predicted 6DoF pose ofthe at least one critical feature and a corresponding reference position of the at least one critical feature on the industrial part.

8. The method of claim 7, further comprising: determining that the pose prediction model is accurate based on determining that the distance between the tool-tip of the robotic arm and the corresponding reference position of the at least one critical feature on the industrial part is less than a position accuracy distance threshold.

9. The method of claim 7, further comprising: discarding the predicted 6DoF pose; and determining that the pose prediction model is inaccurate based on determining that the distance between the tool-tip of the robotic arm and the corresponding reference position of the at least one critical feature on the industrial part is greater than a position accuracy distance threshold.

10. The method of claim 7, wherein the position accuracy distance threshold is empirically determined based on the industrial part and a gripper design.

11. A method for validation of a vision-based pose estimation model, the method comprising: selecting a set of critical features associated with an industrial part; capturing, using a vision system, one or more images of the industrial part present placed in a bin; predicting, using a pose prediction model, a six-degree-of-freedom (6DoF) pose of the industrial part within the bin, wherein predicting the 6DoF pose of the industrial part within the bin comprises predicting a 6DoF pose for each critical feature of the set of critical features associated with a computer-aided design (CAD) model of the industrial part; converting an assigned 6DoF pose of each critical feature of the set of critical features from the CAD model of the industrial part to predicted two-dimensional (2D) coordinates in an image space based on the predicted 6DoF pose of the industrial part; projecting the predicted 2D coordinates on the one or more images of the industrial part; determining that a distance between each critical feature of the set of critical features identified in the one or more images of the industrial part and the corresponding predicted 2Dcoordinates in the one or more images of the industrial part is less than a position accuracy distance threshold; converting the predicted 6D0F pose of each critical feature of the set of critical features of the industrial part from a camera coordinate frame to a robot base frame based on the predicted 6D0F pose of the industrial part and a homogenous transformation matrix; providing the converted predicted 6D0F pose of each critical feature to a robotic arm, wherein the robotic arm is configured to move a tool-tip mechanically coupled to the robotic arm to approach at least one critical feature of the set of critical features of the industrial part; and evaluating an accuracy of the pose estimation model based on a distance measured between a position of the tool-tip of the robotic arm with respect to a predicted 6D0F pose of the at least one critical feature and a corresponding predicted position of the at least one critical feature on the industrial part.

12. The method of claim 11, further comprising: determining that the pose prediction model is accurate based on determining that the distance between the tool-tip of the robotic arm and the corresponding reference position of the at least one critical feature on the industrial part is less than a position accuracy distance threshold.

13. The method of claim 11, wherein the one or more images of the industrial part along with the predicted 2D coordinates are displayed in a graphical user interface (GUI).

Images