Pose detection method and device for irregularly shaped parts

Abstract

The application relates to the technical field of machine vision, in particular to a pose detection method and device for irregularly shaped parts, wherein the method comprises the following steps: acquiring an actual image and a CAD model of an irregularly shaped target part; extracting a real outer contour point set of the target part in the actual image under an image coordinate system, and predicting a first relative pose of the target part and a camera; extracting a double-surface discrete contour point set of the target part in the CAD model, projecting the double-surface discrete contour point set to the image coordinate system to obtain a virtual re-projection outer contour point set, calculating a first precision evaluation index according to the two point sets, optimizing the first relative pose to obtain a second relative pose, filtering out undesired contour points in the real outer contour point set according to the second relative pose, calculating a second precision evaluation index, and optimizing the second relative pose to obtain a pose detection result of the target part and the camera. Thus, the problems of poor applicability, precision and robustness of existing visual pose detection algorithms for sheet parts are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine vision, and in particular to a pose detection method and device for irregularly shaped parts. BACKGROUND

[0002] Using machine vision, high-precision 6-D pose measurement of industrial parts can guide the high-precision grasping and assembly of robots, which has significant application significance in robot automatic assembly tasks. This patent mainly considers monocular vision 6-D pose measurement of irregular sheet parts in 3C precision assembly tasks. Compared with general industrial parts, the main difficulties are as follows: (1) The texture of sheet parts is weak, resulting in only the upper and lower surface contours as the most significant features in a single image. Due to the thin thickness, the contour features of the upper and lower surfaces are coupled together in the image, and the method based on geometric feature fitting and pose solving has poor precision; (2) At present, there are few general pose detection algorithms for parts of any irregular shape, and there is a clear contradiction between complex shape and detection accuracy.

[0003] In view of the above difficulties, many scholars have carried out a lot of research, which specifically includes the following aspects: (1) Extracting regular geometric features of parts presented in images, such as 4 or more key points, a set of circles or ellipses, etc., and fitting their equations to calculate the pose by using the principle of perspective projection imaging. However, this method is only suitable for parts of regular shape, and the coupling of features of sheet parts leads to poor precision; (2) Using deep learning method, based on a large amount of labeled real data to train the parameters of neural network, to realize pose estimation. This method is suitable for objects of any shape, but requires a large amount of labeled data in reality (large workload), and the precision is usually poor, only millimeter-level pose estimation can be achieved; (3) A small part of research combines deep learning and traditional vision algorithms, but the application scene is relatively simple, or the pose detection precision for sheet parts without texture is insufficient. SUMMARY

[0004] The present application provides a pose detection method and device for irregularly shaped parts to solve the problems of poor applicability, precision and robustness of existing visual pose detection algorithms for sheet parts.

[0005] The first aspect of the application provides a pose detection method for irregularly shaped parts, comprising the following steps: acquiring an actual image and a CAD model of an irregularly shaped target part; extracting a real outer contour point set of the target part in the image coordinate system in the actual image, and predicting a first relative pose of the target part and the camera according to the real outer contour point set; extracting a double-surface discrete contour point set of the target part in the CAD model, projecting the double-surface discrete contour point set to the image coordinate system to obtain a virtual re-projection outer contour point set, calculating a first precision evaluation index according to the real outer contour point set and the virtual re-projection outer contour point set, and optimizing the first relative pose to obtain a second relative pose according to the first precision evaluation index; filtering out undesired contour points in the real outer contour point set according to the second relative pose, calculating a second precision evaluation index according to the real outer contour point set after filtering out the undesired contour points and the virtual re-projection outer contour point set, and optimizing the second relative pose to obtain a pose detection result of the target part and the camera according to the second precision evaluation index.

[0006] Optionally, the first relative pose of the target part and the camera is predicted according to the real outer contour point set, comprising: inputting the real outer contour point set into a trained deep neural network, and the deep neural network outputs the first relative pose of the target part and the camera.

[0007] Optionally, before the real outer contour point set is input into the trained deep neural network, it further comprises: rendering and collecting simulation data in a simulator; selecting training data from the simulation data according to the difference between the simulation domain and the real domain; iteratively training the deep neural network using the training data until the iterative training stopping condition is met, and stopping the iterative training of the deep neural network.

[0008] Optionally, the double-surface discrete contour point set of the target part in the CAD model is extracted, comprising: placing the axis direction of the CAD model in the target axis direction; calculating the intersection lines of the CAD model with the upper surface and the lower surface respectively; sampling the first feature point set and the second feature point set on the two groups of intersection lines, and performing counterclockwise or clockwise sorting on the first feature point set and the second feature point set; generating the double-surface discrete contour point set according to the sorted first feature point set and second feature point set.

[0009] Optionally, the double-surface discrete contour point set is projected to the image coordinate system to obtain the virtual re-projection outer contour point set, comprising: calculating the first coordinate representation of the first feature point set and the second feature point set in the camera coordinate system; for the two groups of first coordinate representations, calculating the virtual re-projection points in the image coordinate system, and forming two groups of two-dimensional image point sets according to the virtual re-projection points in the image coordinate system; and calculating the virtual re-projection outer contour point set according to the two groups of two-dimensional image point sets.

[0010] Optionally, the calculation formulas of the first precision evaluation index and the second precision evaluation index are:

[0011]

[0012] wherein P ctr is the real contour point set of the target part in the actual image in the image coordinate system; P o is the virtual re-projection contour point set; denotes the number of contour points in P o ; ‖p o -p ctr ‖ denotes the Euclidean distance between two points.

[0013] Optionally, the first relative pose is optimized according to the first precision evaluation index to obtain a second relative pose, including: taking the first precision evaluation index as a nonlinear optimization objective function; constructing an optimization problem according to the optimization objective function, taking the first relative pose as an initial solution of the optimization problem, and solving the optimization problem to obtain the second relative pose.

[0014] Optionally, the second relative pose is optimized according to the second precision evaluation index to obtain a pose detection result of the target part and the camera, including: taking the second precision evaluation index as a nonlinear optimization objective function; constructing an optimization problem according to the optimization objective function, taking the second relative pose as an initial solution of the optimization problem, and solving the optimization problem to obtain the pose detection result of the target part and the camera.

[0015] Optionally, the undesired contour points in the real contour point set are filtered out according to the second relative pose, including: calculating two groups of second coordinate representations in the camera coordinate system according to the second relative pose; performing linear filling between adjacent points of the two groups of second coordinate representations, so that the distance between every two adjacent points is less than a preset pixel, to form a two-dimensional contour point set; obtaining an intersection point set of the two groups of second coordinate representations, calculating the minimum distance from each point in the intersection point set to all points in the two-dimensional contour point set, and generating a distance threshold according to the minimum distance; and filtering out the undesired contour points from the two-dimensional contour point set according to the distance threshold.

[0016] A second aspect of this application provides a pose detection device for irregularly shaped parts, comprising: an acquisition module for acquiring an actual image and a CAD model of the irregularly shaped target part; an extraction module for extracting a set of true outer contour points of the target part in the actual image under an image coordinate system, and predicting a first relative pose between the target part and a camera based on the set of true outer contour points; a first optimization module for extracting a set of discrete contour points on two surfaces of the target part in the CAD model, projecting the set of discrete contour points on two surfaces onto an image coordinate system to obtain a set of virtual reprojected outer contour points, calculating a first accuracy evaluation index based on the set of true outer contour points and the set of virtual reprojected outer contour points, and optimizing the first relative pose based on the first accuracy evaluation index to obtain a second relative pose; and a second optimization module for filtering out unwanted contour points from the set of true outer contour points based on the second relative pose, calculating a second accuracy evaluation index based on the set of true outer contour points after filtering out unwanted contour points and the set of virtual reprojected outer contour points, and optimizing the second relative pose based on the second accuracy evaluation index to obtain a pose detection result between the target part and the camera.

[0017] Therefore, this application has the following beneficial effects:

[0018] This application embodiment acquires an actual image and a CAD model of an irregularly shaped target part; extracts the true outer contour point set of the target part in the image coordinate system from the actual image, and predicts the first relative pose between the target part and the camera; extracts the dual-surface discrete contour point set of the target part in the CAD model, projects it onto the image coordinate system to obtain a virtual reprojected outer contour point set, calculates a first accuracy evaluation index based on the above two point sets, optimizes the first relative pose to obtain a second relative pose, filters out unwanted contour points in the true outer contour point set, calculates a second accuracy evaluation index, optimizes the second relative pose to obtain the pose detection result between the target part and the camera, achieving high-precision pose detection for thin sheet parts of arbitrary cross-sectional shape, and can be quickly deployed and applied, improving the accuracy of shaft hole alignment in robot automatic assembly. Therefore, it solves the problems of poor applicability, accuracy, and robustness of existing visual pose detection algorithms for thin sheet parts.

[0019] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0020] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0021] Figure 1 This is a flowchart of a pose detection method for irregularly shaped parts according to an embodiment of this application;

[0022] Figure 2An automatic simulation data acquisition flowchart according to an embodiment of the present application is provided;

[0023] Figure 3 A deep neural network architecture diagram for rough pose estimation according to an embodiment of the present application is provided;

[0024] Figure 4 A specific flowchart of a deployed pose detection scenario and stage 1 according to an embodiment of the present application is provided;

[0025] Figure 5 A whole flowchart of an online pose optimization stage according to an embodiment of the present application is provided;

[0026] Figure 6 A calculation flowchart of precision evaluation index MRCM and its visualization according to an embodiment of the present application are provided;

[0027] Figure 7 An example diagram of a pose detection device for irregularly shaped parts according to an embodiment of the present application is provided. DETAILED DESCRIPTION

[0028] Embodiments of the present application are described in detail below with reference to examples shown in the accompanying drawings, in which the same or similar numerals indicate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0029] A pose detection method and device for irregularly shaped parts according to an embodiment of the present application are described below with reference to the accompanying drawings. In view of the problems of poor precision and heavy workload of the prior art robot automatic assembly mentioned in the background art, the present application provides a pose detection method for irregularly shaped parts, in which the actual image and CAD model of the irregularly shaped target part are obtained; the real contour point set of the target part in the image coordinate system in the actual image is extracted, and the first relative pose of the target part and the camera is predicted; the double-surface discrete contour point set of the target part in the CAD model is extracted, and the virtual re-projection contour point set is obtained by projecting it to the image coordinate system; the first precision evaluation index is calculated according to the above two point sets, and the first relative pose is optimized to obtain the second relative pose, by which the undesired contour points in the real contour point set are filtered out, the second precision evaluation index is calculated, and the second relative pose is optimized to obtain the pose detection result of the target part and the camera, thereby realizing high-precision pose detection of any cross-sectional shape sheet part, and the method can be quickly deployed and applied to improve the precision of robot automatic assembly shaft hole alignment. Thus, the problems of poor applicability, precision and robustness of existing visual pose detection algorithms for sheet parts are solved.

[0030] Specifically,Figure 1 A flowchart of a pose detection method for irregularly shaped parts provided by an embodiment of the present application.

[0031] As shown in the figure, the pose detection method for irregularly shaped parts includes the following steps: Figure 1

[0032] In step S101, an actual image and a CAD model of the irregularly shaped target part are obtained.

[0033] It can be understood that the actual image is a visual representation of the target part in the real environment obtained by a monocular camera, which is generally an RGB image, i.e., a color image composed of red, green, and blue color channels; the CAD model is a computer-aided design model, which is a three-dimensional digital representation of the target part created using computer software, and can accurately reflect the geometry, size, and shape of the part.

[0034] In step S102, a real contour point set of the target part in the image coordinate system in the actual image is extracted, and a first relative pose of the target part and the camera is predicted according to the real contour point set.

[0035] Among them, the real contour point set is a set of visible outermost boundary points extracted from the actual image of the target part, which defines the shape of the part; the real contour point set of the target part in the image coordinate system in the actual image can be extracted using adaptive threshold segmentation, deep learning, etc.; the first relative pose refers to the preliminary estimated position and attitude of the target part relative to the camera, which is described by a 6-D pose, which includes six independent parameters to describe the position and attitude of the target part in three-dimensional space, including three translation components and three rotation components.

[0036] It can be understood that the real contour point set of the target part can be extracted from the actual image, for example, using adaptive threshold segmentation, deep learning, etc. to accurately obtain the real contour point set of the target part, and a rough but fast first relative pose is predicted according to the point set, which serves as the basis for subsequent precise positioning optimization.

[0037] In the embodiment of the present application, the first relative pose of the target part and the camera is predicted according to the real contour point set, which includes: inputting the real contour point set into a trained deep neural network, and the deep neural network outputs the first relative pose of the target part and the camera.

[0038] Among them, the deep neural network is a multi-layer artificial neural network structure that can learn and represent complex nonlinear mapping relationships, and the deep neural network of the embodiment of the present application has been trained using simulation data, and the training method will be described in detail below, which will not be repeated here. ​

[0039] It can be understood that the real outer contour point set obtained by the embodiment of the application is input into a deep neural network that has been trained, and the deep neural network outputs an estimation result of the first relative pose, i.e., the initial position and attitude of the target part relative to the camera.

[0040] In the embodiment of the application, before the real outer contour point set is input into the trained deep neural network, the method further includes: rendering and collecting simulation data in a simulator; selecting training data from the simulation data according to the difference between the simulation domain and the real domain; and iteratively training the deep neural network using the training data until a stop condition of the iterative training is met, and stopping the iterative training of the deep neural network.

[0041] The simulator is a software tool for simulating a real environment, can generate virtual images and scenes for algorithm training and testing, and can be a Blender simulation rendering engine or the like. The simulation data is data generated by the simulator. The stop condition of the iterative training is a set of predefined rules or indicators, and when these conditions are met, the training process will be terminated. The stop condition of the iterative training can be that the training error no longer significantly decreases or that the maximum number of iterations is reached, and is specifically set according to actual conditions, which is not limited here.

[0042] It can be understood that the method for training the deep neural network in the embodiment of the application is as follows: a simulator, such as a Blender simulation rendering engine, is used to render and collect a large amount of simulation data. Considering that there is a certain domain difference between the simulation domain and the real domain, features with small differences in the two domains are selected as training data. The selected simulation data is used to iteratively train the deep neural network until a preset stop condition of the iterative training is met, such as training error convergence or reaching a specified maximum number of iterations, which represents that the deep neural network has been fully trained.

[0043] It should be noted that the embodiment of the application designs the input features and network architecture of the deep neural network. After the deep neural network is trained based on the data generated by pure simulation rendering, it can be directly deployed in actual images without any fine-tuning, which greatly increases the deployment efficiency.

[0044] In step S103, a double-surface discrete contour point set of the target part in the CAD model is extracted, the double-surface discrete contour point set is projected to an image coordinate system to obtain a virtual re-projection outer contour point set, a first precision evaluation index is calculated according to the real outer contour point set and the virtual re-projection outer contour point set, and a second relative pose is obtained by optimizing the first relative pose according to the first precision evaluation index.

[0045] The double-surface discrete contour point set refers to a set of discrete points on the upper surface and lower surface contours extracted from a CAD model of the target part; the method for calculating the first precision evaluation index according to the real outer contour point set and the virtual re-projection outer contour point set, and the method for obtaining the second relative pose by optimizing the first relative pose according to the first precision evaluation index will be described in detail below, and thus will not be described herein.

[0046] It can be understood that the embodiments of the present application can extract a double-surface discrete contour point set from a CAD model of a target part, project the double-surface discrete contour point set to an image coordinate system by using the perspective projection principle to form a virtual re-projection outer contour point set, calculate a first precision evaluation index by using the real outer contour point set and the virtual re-projection outer contour point set, and then optimize the initially estimated first relative pose by using the first precision evaluation index to obtain a more accurate second relative pose.

[0047] In the embodiments of the present application, obtaining the second relative pose by optimizing the first relative pose according to the first precision evaluation index includes: taking the first precision evaluation index as a nonlinear optimization objective function; constructing an optimization problem according to the optimization objective function, taking the first relative pose as an initial solution of the optimization problem, and solving the optimization problem to obtain the second relative pose.

[0048] The nonlinear optimization objective function optimizes the pose estimation by minimizing the MRCM, the optimization is iteratively performed within a certain range near the initial solution, the sequential least squares programming method is used for solving, the numerical method is used for estimating the gradient, and finally the second relative pose is calculated.

[0049] It can be understood that the embodiments of the present application take the first precision evaluation index as a nonlinear optimization objective function, take the first relative pose as an initial solution of the optimization problem, optimize the pose estimation by minimizing the MRCM, iteratively perform the optimization within a certain range near the initial solution, use the sequential least squares programming method for solving, use the numerical method for estimating the gradient, and thus obtain the second relative pose.

[0050] In the embodiments of the present application, extracting the double-surface discrete contour point set of the target part in the CAD model includes: placing the axis direction of the CAD model in the target axis direction; calculating the intersection lines of the CAD model and the upper surface and the lower surface respectively; sampling the first feature point set and the second feature point set on the two sets of intersection lines, and performing counterclockwise or clockwise sorting on the first feature point set and the second feature point set; and generating the double-surface discrete contour point set according to the sorted first feature point set and second feature point set.

[0051] It can be understood that the embodiments of the present application adjust the axis direction of the CAD model to the predetermined target axis, for example, if the target axis is the Z axis of the coordinate system, the lower surface of the target part is located on the Z=0 plane, and the upper surface is located on the Z=h plane, the intersection lines of the CAD model and the two planes are calculated respectively to obtain the profiles of the upper surface and the lower surface, uniform sampling is performed on the two intersection lines to obtain the first feature point set and the second feature point set, each point set is sorted counterclockwise or clockwise, and finally, the complete double-surface discrete profile point set is constructed according to the sorted first feature point set and the second feature point set.

[0052] In the embodiments of the present application, the double-surface discrete profile point set is projected into the image coordinate system to obtain a virtual re-projection outer profile point set, including: calculating the first coordinate representation of the first feature point set and the second feature point set in the camera coordinate system; for the two groups of first coordinate representations, calculating the virtual re-projection points in the image coordinate system, forming two groups of two-dimensional image point sets according to the virtual re-projection points in the image coordinate system; and calculating the virtual re-projection outer profile point set according to the two groups of two-dimensional image point sets.

[0053] The camera coordinate system is a three-dimensional coordinate system established with the camera as the origin, which can describe the position of the object relative to the camera; the virtual re-projection point is a corresponding point on the image plane obtained by converting the point in the three-dimensional space to the image plane through the perspective projection principle; and the virtual re-projection outer profile point set is calculated by using the computer graphics method based on the two groups of two-dimensional image point sets.

[0054] It can be understood that the embodiments of the present application convert the first feature point set and the second feature point set extracted from the CAD model into the camera coordinate system to obtain the first coordinate representation, calculate the virtual re-projection points of the two groups of first coordinate representations in the image coordinate system, thereby obtaining the two groups of two-dimensional image point sets, and calculate the virtual re-projection outer profile point set by using the computer graphics method based on the two groups of two-dimensional image point sets.

[0055] In step S104, the undesired profile points in the real outer profile point set are filtered out according to the second relative pose, the second accuracy evaluation index is calculated according to the real outer profile point set after filtering out the undesired profile points and the virtual re-projection outer profile point set, and the second relative pose is optimized according to the second accuracy evaluation index to obtain the pose detection result of the target part and the camera.

[0056] The undesired profile points are the points in the real outer profile point set that do not belong to the upper surface or lower surface profile of the target part, but are caused by the side edge of the part or other interference factors.

[0057] It can be understood that the embodiments of the present application use the second relative pose to analyze the real outer contour point set, identify and remove those undesirable contour points that do not reflect the upper surface or lower surface features of the part, then compare the filtered real outer contour point set with the virtual re-projection outer contour point set, quantify their matching degree by calculating the second precision evaluation index between the two, and optimize and improve the second relative pose according to the second precision evaluation index, so as to obtain the high-precision pose detection result of the target part relative to the camera.

[0058] In the embodiments of the present application, the calculation formula of the first precision evaluation index and the second precision evaluation index is:

[0059]

[0060] wherein, P ctr is the real outer contour point set of the target part in the image coordinate system in the actual image; P o is the virtual re-projection outer contour point set; represents the number of contour points in P o ; ‖p o -p ctr ‖ represents the Euclidean distance between two points.

[0061] In the embodiments of the present application, the pose detection result of the target part and the camera obtained by optimizing the second relative pose according to the second precision evaluation index includes: taking the second precision evaluation index as a nonlinear optimization objective function; constructing an optimization problem according to the optimization objective function, taking the second relative pose as the initial solution of the optimization problem, and solving the optimization problem to obtain the pose detection result of the target part and the camera.

[0062] It can be understood that the embodiments of the present application take the second precision evaluation index as a nonlinear optimization objective function, and take the second relative pose as the initial solution of the optimization problem, and optimize the pose estimation by minimizing the MRCM. The optimization is iteratively performed within a certain range near the initial solution, and the sequential least squares programming method is used for solving. The numerical method is used to estimate the gradient, so as to obtain the pose detection result of the target part and the camera.

[0063] In the embodiments of the present application, filtering out the undesirable contour points in the real outer contour point set according to the second relative pose includes: calculating two groups of second coordinate representations in the camera coordinate system according to the second relative pose; performing linear filling between adjacent points of the two groups of second coordinate representations, so that the distance between every two adjacent points is less than a preset pixel, to form a two-dimensional contour point set; obtaining an intersection set of the two groups of second coordinate representations, calculating the minimum distance from each point in the intersection set to all points in the two-dimensional contour point set, and generating a distance threshold according to the minimum distance; filtering out the undesirable contour points from the two-dimensional contour point set according to the distance threshold.

[0064] wherein, the preset pixel is set according to actual conditions, which is not specifically limited here.

[0065] It can be understood that, according to the second relative pose, the application embodiment calculates two sets of second coordinate representations of the upper surface and lower surface contour point sets of the target part in the camera coordinate system, linearly fills the adjacent points in the two sets of coordinate representations, ensures that the distance between every two adjacent points is less than a preset pixel value, thereby forming a more dense two-dimensional contour point set, and finding a set of intersection points between the two sets of contour point sets, calculating the minimum distance of each point in the set of intersection points to all points in the entire two-dimensional contour point set, and based on the minimum distance, setting a distance threshold value, which determines the strictness of filtering out undesired contour points. Finally, according to the distance threshold value, remove those points from the two-dimensional contour point set that are too close to the intersection points, thereby realizing the filtering out of undesired contour points from the two-dimensional contour point set.

[0066] According to the pose detection method for irregularly shaped parts proposed in the application embodiment, the actual image and the CAD model of the irregularly shaped target part are obtained; the real outer contour point set of the target part in the image coordinate system is extracted from the actual image, and the first relative pose of the target part and the camera is predicted; the double-surface discrete contour point set of the target part in the CAD model is extracted, and the virtual re-projection outer contour point set is obtained by projecting it to the image coordinate system; the first accuracy evaluation index is calculated according to the above two point sets, and the first relative pose is optimized to obtain the second relative pose, according to which the undesired contour points in the real outer contour point set are filtered out, the second accuracy evaluation index is calculated, and the second relative pose is optimized to obtain the pose detection result of the target part and the camera, realizing high-precision pose detection of any cross-sectional shape sheet part, and can be quickly deployed and applied, and the precision of robot automatic assembly shaft hole alignment is improved.

[0067] The pose detection method for irregularly shaped parts is further described below through a specific embodiment.

[0068] First, the offline deployment and preparation stage of the algorithm is specifically described, which is referred to as the preparation stage, which will prepare the required parameters for the subsequent two-stage online pose detection algorithm, which includes the following steps:

[0069] Step 1, render and collect a large amount of simulation data in the simulator, and the specific process is as shown in Figure 2 .

[0070] The CAD model of the wafer part is exported in STL format, and the STL format model is imported into the Blender simulation rendering engine. A monocular camera is generated at the origin (the camera optical axis direction points to the positive direction of the Z axis of the world coordinate system), and the internal parameters of the monocular camera and the consistent used in the real scene are set. A white point light source is generated at the origin, and its radiation range and illumination intensity are set.

[0071] Batch data collection is automatically performed by writing a script program. For 10000 cycles, in each cycle, the program automatically changes the three-dimensional position (represented by x, y, z) and three-dimensional pose (represented by XYZ Euler angle rx, ry, rz) of the wafer STL model within a certain range, and calls the Blender built-in renderer to render a part image, which is stored in the form of data pairs (1 RGB image ~ 1 set of 6-D pose x, y, z, rx, ry, rz). Generally, considering that the grabbing error will not exceed ±5mm, ±5°, the pose of the wafer STL model in the simulation can be set to ±10mm, ±10°, thereby improving the robustness of the data.

[0072] Step 2, quickly train the neural network based on simulation data

[0073] In order to perform simulation-to-reality direct migration of the neural network, features with small differences between the simulation domain and the real domain need to be selected as the input of the deep neural network, and the overall outer contour point set of the wafer part is selected as the input of the deep neural network. In order to obtain the outer contour point set of the wafer part in the image, 10000 groups of images collected in step 1 are binarized using an adaptive threshold segmentation algorithm, the largest connected contour in each image is obtained using a connected domain extraction algorithm, the two-dimensional pixel coordinates of each point in the contour are extracted, and uniform downsampling is performed to obtain 10000 groups of point set data with the same dimension (n x 2).

[0074] Considering the unordered nature of the two-dimensional contour point set, the designed neural network structure feature extractor needs to adapt to the unordered nature of the input, and the neural network structure is as shown in Figure 3 The deep neural network performs point-by-point advanced feature extraction based on 3-layer 1-dimensional convolution with shared weights, and finally extracts the global features (1 x 1024) of the point set through maximum pooling. By using a 3-layer perceptron to decode the global features, the 6-dimensional pose (x, y, z, rx, ry, rz) of the wafer part is output. Batch normalization is used at each layer of the network, and the Relu activation function is used.

[0075] With the foregoing 10000 groups of data (input: n x 2 ~ output: 1 x 6), the neural network can be trained end-to-end, the batch data amount for training is 64, the initial learning rate is 0.01, and a linear learning rate scheduler is used for scheduling, and the network can quickly converge in a short time.

[0076] Step 3, sampling double-surface discrete contour points from the CAD of the sheet part

[0077] In order to smoothly perform the subsequent online phase 2 (pose optimization), two sets of feature point sets are extracted from the CAD model in advance, denoted as and and Each of the m three-dimensional contour points in the coordinate contains m three-dimensional contour points, the former is the uniform sampling of the upper surface contour points in the CAD model, and the latter is the uniform sampling of the lower surface contour points in the CAD model. The specific extraction method is as follows:

[0078] (1) The axis direction of the sheet part is placed in the Z-axis direction, and the lower surface of the sheet part is located on the Z=0 plane, and the upper surface is located on the Z=h plane (h is the thickness of the sheet).

[0079] (2) The intersection lines of the STL model and the Z=0, Z=h planes are calculated using the OCC library;

[0080] (3) Calculate the total length L of the intersection line, calculate the point spacing L / m of the uniform sampling in the circumference direction, and perform uniform sampling on the two intersection lines to obtain two sets of point sets and

[0081] (4) The point sets in and are sorted counterclockwise or clockwise: for sheet parts with convex cross-sectional shape, the sorting can be directly based on the polar angle; for sheet parts with concave cross-sectional shape, a nearest neighbor point connection strategy considering the point angle direction is used to realize the sorting.

[0082] Next, the online execution phase 1 of the algorithm is described, which will be used for the estimation of the pose detection coarse solution. The pre-trained neural network in the offline phase is directly migrated to the real scene for use to perform the estimation of the pose detection coarse solution. The specific process of the deployed pose detection scene and phase 1 is shown in Figure 4 .

[0083] Step 1, deploy the actual pose detection scene

[0084] In the process of precise assembly of 3C products, the system architecture for pose measurement using a monocular camera is as follows Figure 3As shown. The monocular camera is fixed in the environment, and its relative pose to the robotic arm base remains unchanged. Hand-eye calibration can be performed using a checkerboard calibration plate. A suction cup mounted at the end of the robotic arm picks up a sheet part to be inspected and moves it directly above the hole to be assembled. The monocular camera can then observe the sheet part at an angle. Subsequently, an RGB image of the sheet part is captured, which serves as the raw input for pose detection. The key objective of the algorithm proposed in this patent is to measure the pose of the sheet part relative to the camera. Then, based on the extrinsic parameter matrix obtained from hand-eye calibration, the pose of the sheet part relative to the robotic arm is calculated to guide the movement of the robotic arm and perform shaft-hole alignment.

[0085] Step 2: Transfer application of neural networks to perform coarse-scale estimation of pose detection.

[0086] In step 2 of the preparation phase, a deep neural network trained using simulation data is input to n two-dimensional coordinates of the outer contour points of the sheet-like part extracted from the image. Although the lighting, color, texture, and other information of the part presented in the image differ greatly between simulation and reality, the difference in the coordinates of a specific outer contour point is only at the pixel level. Therefore, the weights of the deep neural network trained in simulation in step 2 of the preparation phase can be directly applied to real-world data without any fine-tuning.

[0087] First, image processing is performed on the actual RGB image of the thin-sheet part acquired in step 1 of phase 2 to extract the outer contour point set P of the thin-sheet part from the actual image. ctr (1) In structured lighting environments, the processing of the outer contour point set can be based on traditional image processing algorithms. Adaptive threshold segmentation is used to obtain a binary image; the binary region is filtered using perimeter threshold, area threshold, and shape similarity to obtain the thin sheet part as the foreground image; the outer contour point set of the thin sheet part is obtained using a connected component contour extraction algorithm. This method has higher detection accuracy, but poor robustness. (2) In unstructured complex lighting environments, the processing of the outer contour point set requires the use of deep learning-based segmentation algorithms, such as using a pre-trained SegmentAnything (SAM) large model to perform region segmentation of the thin sheet part, and then using a connected component contour extraction algorithm to obtain the outer contour point set of the thin sheet part. This method is suitable for complex environments and has high robustness, but the accuracy of the contour decreases.

[0088] P ctr By inputting the deep neural network trained offline, pose prediction can be performed directly. This coarse solution for pose detection is denoted as pose. coarse .

[0089] The online execution phase 2 of the algorithm is described below, which will be used for iterative optimization of pose detection accuracy, including the design and calculation of accuracy evaluation index (MRCM), the design and solution of the primary optimization problem, and the design and implementation of the secondary optimization strategy. The overall flow of online execution phase 2 is shown in Figure 5 .

[0090] Step 1, Design and calculation of accuracy evaluation index (MRCM)

[0091] Although the 6-D pose error of the aforementioned initial solution cannot be quantitatively evaluated, it can be analyzed by indirect means to guide further optimization of the pose.

[0092] Assuming that the camera's internal parameters are known (focal length f, principal point coordinates u c ,v c ), and the current estimate of the sheet 6-D pose is (x, y, z, rx, ry, rz), then and the virtual re-projection point sets P u and P b in the pixel coordinate system can be calculated according to the perspective projection principle and coordinate transformation principle. When the estimate of the sheet 6-D pose is exactly equal to the true value of the sheet 6-D pose, P ctr should be a subset of P u ∪ P b ; and when the estimate of the sheet 6-D pose is not equal to the true value of the sheet 6-D pose, P ctr should not be aligned with the outer contour point set of P u ∪ P b (mismatch exists). Therefore, the similarity of P ctr and P u ∪ P b will indirectly reflect the current estimation error of the sheet 6-D pose, so the key of the accuracy evaluation index (MRCM) is to quantitatively evaluate the similarity of P ctr and P u ∪ P b , and the specific calculation method is as follows, and the flowchart is shown in Figure 6 .

[0093] (1) Calculate and in the camera coordinate system according to formula 1 and Subsequently, for each point (x C ,y C ,z C ), calculate its virtual re-projection point (u, v) in the pixel coordinate system according to formula 2, forming two sets of two-dimensional image point sets P uand P b ;

[0094]

[0095] (2) Design a computer graphics method and calculate P according to formula 3. u ∪P b The outer contour point set P o ;

[0096]

[0097] in and They are respectively for P u and P b The resulting binarized image is generated by performing a polygon filling algorithm. Points within the filled region have a grayscale value of 1, while points outside the filled region have a grayscale value of 0.

[0098] (3) Calculate P o and P ctr The one-sided chamfer distance is used to obtain the MRCM, as shown in Equation 4. In order for the one-sided chamfer distance to accurately characterize the degree of profile mismatch, the one-sided chamfer distance is calculated from the point set with lower profile point density to the point set with higher profile point density.

[0099]

[0100] in P represents o The number of contour points, ||p o -p ctr || represents the Euclidean distance between two points. MRCM comprehensively evaluates profile P. o and P ctr The average set distance when P o and P ctr When they overlap, MRCM is 0.

[0101] Step 2: Design and solution of the initial optimization problem

[0102] pose obtained in phase 1 coarse To obtain an initial solution to the optimization problem, we construct the optimization problem as shown in Equation 5, using MRCM as the nonlinear objective function:

[0103]

[0104] Optimize within a certain range near the initial solution iThe inner iteration is performed (the range can be taken as 5mm / 5°) and solved by using the sequence least square programming (SLSQP) method. Since the gradient of the objective function cannot be solved analytically, a numerical method is used to estimate the gradient. Finally, the initial optimization solution pose of the pose detection is calculated fine .

[0105] 3. Design and implementation of secondary optimization strategy

[0106] However, in the calculation process of the foregoing MRCM, there is a step of principle simplification: only the profile points of the upper surface and the lower surface of the sheet part are extracted from the CAD model, and the imaging points of the side profile points of the sheet part are actually also present in P ctr Therefore, the principle error of the MRCM will mislead the initial optimization and reduce the accuracy of the optimization. In the case where the sheet cross-sectional size remains unchanged, as the thickness of the sheet part increases, the proportion of the imaging points of the side profile points of the sheet part in P ctr increases gradually, resulting in a continuous decline in the accuracy of pose fine . In order to improve this situation, based on the result of the initial optimization, the imaging points of the side profile that are not expected are filtered out in P ctr , and secondary optimization is performed. The filtering method of the profile points that are not expected is as follows:

[0107] (1) Based on pose fine , P u,1 and P b,1 are calculated by using formula 1. Linear filling is performed between adjacent points of P u,1 and P b,1 , so that the distance between every two adjacent points is less than one pixel, thereby forming a dense two-dimensional profile point set. The intersection set P u,1 of P b,1 and P itc is solved.

[0108] P itc =P u,1 ∩P b,1 (6)

[0109] (2) The minimum distance d itc of each point in P ctr to all points in P i is calculated, and then a set of distance thresholds r i is generated, which will be used to guide the filtering of the profile points. k>1 is a constant for determining the strength of the filtering, and is usually taken as 2 or 3.

[0110]

[0111] (3) From P ctrthe distance p itc,i of the distance p i of the distance p ctr,2 .

[0112]

[0113] After filtering out the undesired contour points, the calculation of MRCM and the iterative optimization of pose accuracy are performed using a method similar to that of stage 2 step 1 and step 2, but the initial solution of the optimization at this time should use pose fine , and finally obtain the final solution of pose detection pose fine,2 , which is the final output of the method.

[0114] Next, referring to the accompanying drawings, a pose detection device for irregularly shaped parts according to an embodiment of the present application is described.

[0115] Figure 7 is a block schematic diagram of the pose detection device for irregularly shaped parts according to an embodiment of the present application.

[0116] As shown in Figure 7 , the pose detection device for irregularly shaped parts 10 includes an acquisition module 201, an extraction module 202, a first optimization module 203, and a second optimization module 204.

[0117] The acquisition module 201 is configured to acquire an actual image and a CAD model of an irregularly shaped target part. The extraction module 202 is configured to extract a real outer contour point set of the target part in the actual image in an image coordinate system, and predict a first relative pose of the target part and a camera according to the real outer contour point set. The first optimization module 203 is configured to extract a double-surface discrete contour point set of the target part in the CAD model, project the double-surface discrete contour point set to the image coordinate system to obtain a virtual re-projection outer contour point set, calculate a first accuracy evaluation index according to the real outer contour point set and the virtual re-projection outer contour point set, and optimize the first relative pose according to the first accuracy evaluation index to obtain a second relative pose. The second optimization module 204 is configured to filter out undesired contour points in the real outer contour point set according to the second relative pose, calculate a second accuracy evaluation index according to the real outer contour point set after filtering out the undesired contour points and the virtual re-projection outer contour point set, and optimize the second relative pose according to the second accuracy evaluation index to obtain a pose detection result of the target part and the camera.

[0118] In the embodiment of the present application, the extraction module 202 is further configured to predict a first relative pose of the target part and the camera according to the real outer contour point set, and input the real outer contour point set into a trained deep neural network. The deep neural network outputs the first relative pose of the target part and the camera.

[0119] In the embodiments of the present application, the training module is further configured to: render and collect simulation data in a simulator before inputting the real outer contour point set into the trained deep neural network; select training data from the simulation data according to the difference between the simulation domain and the real domain; and iteratively train the deep neural network using the training data until the iterative training stopping condition is met, and stop the iterative training of the deep neural network.

[0120] In the embodiments of the present application, the first optimization module 203 is further configured to: optimize the first relative pose to obtain a second relative pose according to the first precision evaluation index, and take the first precision evaluation index as a nonlinear optimization objective function; construct an optimization problem according to the optimization objective function, take the first relative pose as an initial solution of the optimization problem, and solve the optimization problem to obtain the second relative pose.

[0121] In the embodiments of the present application, the first optimization module 203 is further configured to: extract a double-surface discrete contour point set of a target part in a CAD model, and place an axis direction of the CAD model in a target axis direction; calculate intersection lines of the CAD model and upper and lower surfaces respectively; sample the two sets of intersection lines to obtain a first feature point set and a second feature point set, and perform counterclockwise or clockwise sorting on the first feature point set and the second feature point set; and generate the double-surface discrete contour point set according to the sorted first feature point set and the second feature point set.

[0122] In the embodiments of the present application, the first optimization module 203 is further configured to: project the double-surface discrete contour point set to obtain a virtual re-projection outer contour point set in an image coordinate system, and calculate first coordinate representations of the first feature point set and the second feature point set in a camera coordinate system; for the two sets of first coordinate representations, calculate virtual re-projection points in the image coordinate system, form two sets of two-dimensional image point sets according to the virtual re-projection points in the image coordinate system; and calculate the virtual re-projection outer contour point set according to the two sets of two-dimensional image point sets.

[0123] In the embodiments of the present application, the calculation formulas of the first precision evaluation index and the second precision evaluation index are as follows:

[0124]

[0125] wherein P ctr is a real outer contour point set of a target part in an actual image in an image coordinate system; P o is a virtual re-projection outer contour point set; represents the number of contour points in P o ; ‖p o -p ctr ‖ represents the Euclidean distance between two points.

[0126] In the embodiment of the present application, the second optimization module 204 is further configured to: optimize the second relative pose according to the second precision evaluation index to obtain a pose detection result of the target part and the camera, taking the second precision evaluation index as a nonlinear optimization objective function; construct an optimization problem according to the optimization objective function, taking the second relative pose as an initial solution of the optimization problem, and solving the optimization problem to obtain the pose detection result of the target part and the camera.

[0127] In the embodiment of the present application, the second optimization module 204 is further configured to: filter out undesired contour points in the real external contour point set according to the second relative pose, and calculate two sets of second coordinate representations in the camera coordinate system according to the second relative pose; perform linear filling between adjacent points of the two sets of second coordinate representations, so that the distance between every two adjacent points is less than a preset pixel, to form a two-dimensional contour point set; obtain an intersection point set of the two sets of second coordinate representations, calculate the minimum distance from each point in the intersection point set to all points in the two-dimensional contour point set, and generate a distance threshold according to the minimum distance; filter out the undesired contour points from the two-dimensional contour point set according to the distance threshold.

[0128] It should be noted that the foregoing explanation and description of the embodiment of the pose detection method for the irregularly shaped part also applies to the embodiment of the pose detection device for the irregularly shaped part, which will not be described here again.

[0129] The pose detection device for the irregularly shaped part according to the embodiment of the present application obtains the actual image and the CAD model of the irregularly shaped target part; extracts the real external contour point set of the target part in the actual image in the image coordinate system, predicts the first relative pose of the target part and the camera; extracts the double-surface discrete contour point set of the target part in the CAD model, projects it to the image coordinate system to obtain a virtual re-projection external contour point set, calculates the first precision evaluation index according to the two point sets, optimizes the first relative pose to obtain the second relative pose, filters out the undesired contour points in the real external contour point set according to the second relative pose, calculates the second precision evaluation index, optimizes the second relative pose to obtain the pose detection result of the target part and the camera, realizes high-precision pose detection of the arbitrary cross-section shape sheet part, and can be quickly deployed and applied, and improves the precision of the robot automatic assembly shaft hole alignment.

[0130] In the description of the application, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the application. In the description of the application, the illustrative description of the above terms is not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or N embodiments or examples. In addition, different embodiments or examples described in the description of the application and the features of different embodiments or examples can be combined and combined by those skilled in the art without contradiction.

[0131] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the application, the meaning of "N" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0132] Any process or method descriptions in flow charts or otherwise described herein can be understood as representing code modules, segments, or portions of code that include one or more executable instructions for implementing specific logic functions (or steps) in the process, and that the various embodiments of the application can include additional or fewer steps or processes in alternative implementations, as will be appreciated by those skilled in the art. The various embodiments of the application can be implemented in hardware, software, firmware, or a combination thereof, as desired.

[0133] It should be understood that parts of the application can be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. As in another embodiment implemented in hardware, any one or a combination of the following technologies known in the art can be used: discrete logic circuit with logic gate circuit for implementing logic functions on data signals, application specific integrated circuit with suitable combination logic gate circuit, programmable gate array, field programmable gate array, etc.

[0134] Those skilled in the art of the art can understand that the method of implementing the above-mentioned embodiments carries out all or part of the steps, which can be instructed by the program to complete the relevant hardware, and the above-mentioned program can be stored in a computer readable storage medium, which includes one or a combination of the steps of the method embodiment when executed.

[0135] Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary, and are not to be interpreted as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A method for pose detection of irregularly shaped parts, characterized in that, Includes the following steps: Acquire actual images and CAD models of irregularly shaped target parts; Extract the true outer contour point set of the target part in the image coordinate system from the actual image, and predict the first relative pose of the target part and the camera based on the true outer contour point set; Extract the set of discrete contour points of the dual surfaces of the target part in the CAD model, project the set of discrete contour points of the dual surfaces onto the image coordinate system to obtain the virtual reprojected outer contour point set, calculate the first accuracy evaluation index based on the real outer contour point set and the virtual reprojected outer contour point set, and optimize the first relative pose based on the first accuracy evaluation index to obtain the second relative pose. Undesirable contour points are filtered out from the set of real outer contour points based on the second relative pose. A second accuracy evaluation index is calculated based on the set of real outer contour points after filtering out the undesirable contour points and the set of virtual reprojected outer contour points. The second relative pose is optimized based on the second accuracy evaluation index to obtain the pose detection result of the target part and the camera. The step of filtering out undesirable contour points from the set of real outer contour points based on the second relative pose includes calculating two sets of second coordinate representations in the camera coordinate system based on the second relative pose; linearly filling the adjacent points of the two sets of second coordinate representations so that the distance between any two adjacent points is less than a preset pixel to form a two-dimensional contour point set; obtaining the intersection set of the two sets of second coordinate representations; calculating the minimum distance from each point in the intersection set to all points in the two-dimensional contour point set; generating a distance threshold based on the minimum distance; and filtering out undesirable contour points from the two-dimensional contour point set based on the distance threshold.

2. The pose detection method for irregularly shaped parts according to claim 1, characterized in that, The step of predicting the first relative pose between the target part and the camera based on the true outer contour point set includes: The set of real outer contour points is input into a trained deep neural network, and the deep neural network outputs the first relative pose of the target part and the camera.

3. The pose detection method for irregularly shaped parts according to claim 2, characterized in that, Before inputting the real outer contour point set into the trained deep neural network, the following steps are also included: Render and acquire simulation data in the simulator; Training data is selected from the simulation data based on the differences between the simulation domain and the real domain; The deep neural network is iteratively trained using the training data until the iterative training stopping condition is met, at which point the iterative training of the deep neural network is stopped.

4. The pose detection method for irregularly shaped parts according to claim 1, characterized in that, The extraction of the discrete contour point set of the dual surfaces of the target part in the CAD model includes: Position the axis of the CAD model in the direction of the target axis; Calculate the intersection lines between the CAD model and the upper and lower surfaces, respectively; The first feature point set and the second feature point set are obtained by sampling on the two sets of intersection lines, and the first feature point set and the second feature point set are sorted in counterclockwise or clockwise order. A dual-surface discrete contour point set is generated based on the sorted first feature point set and the second feature point set.

5. The pose detection method for irregularly shaped parts according to claim 4, characterized in that, The step of projecting the set of discrete contour points on the dual surfaces onto the image coordinate system to obtain a virtual reprojected outer contour point set includes: Calculate the first coordinate representation of the first feature point set and the second feature point set in the camera coordinate system; For two sets of first coordinate representations, calculate the virtual reprojection points in the image coordinate system, and form two sets of two-dimensional image point sets based on the virtual reprojection points in the image coordinate system. The virtual reprojection outer contour point set is calculated based on the two sets of two-dimensional image point sets.

6. The pose detection method for irregularly shaped parts according to claim 1, characterized in that, The calculation formulas for the first accuracy evaluation index and the second accuracy evaluation index are as follows: in, The set of actual outer contour points of the target part in the actual image under the image coordinate system; For the virtual reprojection of the outer contour point set; express The number of contour points; It represents the Euclidean distance between two points.

7. The pose detection method for irregularly shaped parts according to claim 1, characterized in that, The step of optimizing the first relative pose according to the first accuracy evaluation index to obtain the second relative pose includes: The first accuracy evaluation index is used as the nonlinear optimization objective function; An optimization problem is constructed based on the objective function, and the second relative pose is obtained by solving the optimization problem using the first relative pose as the initial solution.

8. The pose detection method for irregularly shaped parts according to claim 1, characterized in that, The step of optimizing the second relative pose based on the second accuracy evaluation index to obtain the pose detection result of the target part and the camera includes: The second accuracy evaluation index is used as the nonlinear optimization objective function; An optimization problem is constructed based on the objective function. The second relative pose is used as the initial solution of the optimization problem. The optimization problem is solved to obtain the pose detection results of the target part and the camera.

9. A pose detection device for irregularly shaped parts, characterized in that, include: The acquisition module is used to acquire actual images and CAD models of irregularly shaped target parts. The extraction module is used to extract the true outer contour point set of the target part in the actual image in the image coordinate system, and predict the first relative pose of the target part and the camera based on the true outer contour point set; The first optimization module is used to extract the dual-surface discrete contour point set of the target part in the CAD model, project the dual-surface discrete contour point set onto the image coordinate system to obtain the virtual reprojected outer contour point set, calculate the first accuracy evaluation index based on the real outer contour point set and the virtual reprojected outer contour point set, and optimize the first relative pose based on the first accuracy evaluation index to obtain the second relative pose. The second optimization module is used to filter out unwanted contour points from the set of real outer contour points based on the second relative pose, calculate a second accuracy evaluation index based on the set of real outer contour points after filtering out unwanted contour points and the set of virtual reprojected outer contour points, and optimize the second relative pose based on the second accuracy evaluation index to obtain the pose detection result of the target part and the camera. The step of filtering out unwanted contour points from the set of real outer contour points based on the second relative pose includes calculating two sets of second coordinate representations in the camera coordinate system based on the second relative pose; linearly filling adjacent points of the two sets of second coordinate representations such that the distance between any two adjacent points is less than a preset pixel to form a two-dimensional contour point set; obtaining the intersection set of the two sets of second coordinate representations; calculating the minimum distance from each point in the intersection set to all points in the two-dimensional contour point set; generating a distance threshold based on the minimum distance; and filtering out unwanted contour points from the two-dimensional contour point set based on the distance threshold.

Images