Feature identification method suitable for stacked workpieces
By fusing 2D images and 3D point cloud information, the problem of mis-assembly of features in stacked workpieces was solved, achieving high-precision and stable feature recognition and improving production efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-16
- Publication Date
- 2026-03-10
AI Technical Summary
In multi-model co-production, the feature recognition of stacked workpieces carries the risk of misassembly, leading to production line downtime and low assembly efficiency. Existing technologies struggle to achieve high-precision and stable feature recognition under complex working conditions.
By fusing 2D image information with 3D point cloud information, using a 3D vision sensor to acquire images and point cloud data of the workpiece, performing point cloud registration and homography transformation, and combining grayscale template matching, feature recognition of stacked workpieces can be achieved.
It improves the recognition accuracy and stability of stacked workpieces, reduces the risk of misassembly, and increases production efficiency.
Smart Images
Figure CN121640446A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of machine vision detection, and in particular to a feature recognition method suitable for stacked workpieces. BACKGROUND
[0002] Under the trend of increasing popularity of multi-model collinear production, the foolproof technology for misloading of body parts has important application value. Under normal circumstances, the part mistake proofing of different vehicle models is mainly realized by arranging feature points at different positions. The high-precision recognition technology of the structured light three-dimensional vision system is usually used to collect 2D images or 3D point cloud data of the surface of the body part to realize the classification and determination of the vehicle model to which the current part belongs. However, in the actual production environment, due to the complex working conditions, the measurement and recognition face multiple challenges. Especially when the sensor is fixedly installed and the parts are stacked, with the decrease of the part level, the same physical feature is easily affected by perspective in the 2D imaging plane, showing geometric distortion of "big near and small far"; in the 3D point cloud space, the feature may be filled or lost due to occlusion or sparse point cloud, thereby causing misloading risk and triggering alarm, which will require manual intervention to change the type, resulting in prolonged downtime of the production line and affecting the assembly efficiency. SUMMARY
[0003] To solve the above technical problems, the present application provides a feature recognition method suitable for stacked workpieces, which fuses 2D image information and 3D point cloud information, effectively improving the accuracy and stability of online recognition of stacked workpieces.
[0004] Therefore, the technical scheme of the present application is as follows:
[0005] A feature recognition method suitable for stacked workpieces, which uses a 3D vision sensor to obtain the image and surface point cloud data of the stacked workpiece to be measured, and the 3D vision sensor is fixedly installed;
[0006] The method comprises the following steps:
[0007] 1) Defining the uppermost workpiece in the stacked workpiece to be measured as a template workpiece, and the corresponding image and point cloud data are denoted as template image and template point cloud respectively;
[0008] Preselecting the feature peripheral region in the template point cloud as a feature region, extracting the point cloud of the region, and denoted as feature point cloud P; at the same time, the position of the template image corresponding to the region is denoted as template feature region Q;
[0009] From top to bottom, the other workpieces to be measured under the template workpiece are sequentially numbered as the ith workpiece, i = 1, 2…N-1, and N is the total number of the stacked workpieces to be measured;
[0010] Denote the image and point cloud data of the ith workpiece as image C iand point cloud data A i ;
[0011] 2) Point cloud registration is performed on point cloud data A i and the template point cloud to obtain a spatial position transformation matrix of the i-th workpiece to be measured and the template workpiece, denoted as a registration matrix T i ;
[0012] 3) The feature point cloud P is transformed to the spatial scale of point cloud data A i using the registration matrix T i , denoted as point cloud P i ; in combination with the camera intrinsic matrix of the 3D vision sensor, the point cloud P i is back projected to the corresponding image coordinate system to obtain a corresponding two-dimensional point set, denoted as X i ; according to the feature points in the template feature region Q and the point set X i , a homography transformation matrix H is calculated between them.
[0013] 4) The outer contour of the region where X i is located is circled in the image C i to define an ROI region; all points in the ROI region are converted to the plane where the template feature region Q is located using the homography transformation matrix H, denoted as a point set S i ; all points in the point set S i are subjected to bilinear interpolation, and the interpolation results are filled with the gray values after bilinear interpolation within the above ROI range to obtain a feature image at the same scale as the image C i , denoted as a bridge image.
[0014] 5) The bridge image is subjected to gray template matching with the ROI region circled in C i , and whether the i-th workpiece has features in the template feature region is determined according to the matching score.
[0015] Further, the method for converting all points in the ROI region to the plane where the template feature region Q is located in step 4) is as follows: all points in the ROI region are multiplied by the homography transformation matrix H to obtain coordinates representing the sub-pixel positions of the points, and the coordinates corresponding to all points are denoted as a point set S i .
[0016] Further, the point cloud registration method used in step 2) is any one of the following: ① a coarse registration algorithm based on point pair features (PPF); ② a method based on depth map matching to provide an initial value and iterative closest point (ICP) to provide an accurate transformation relationship; ③ a transformation relationship estimation method based on feature matching combined with a RANSAC robust strategy; and ④ a method for estimating the transformation relationship between point clouds using deep learning.
[0017] Further, in step 5), when performing the gray template matching, taking the bridge image as the feature template image, taking the ROI region circled in C i the threshold value, when the matching score is higher than the threshold value, it is considered that the i th workpiece has the feature in the template feature region, otherwise, it is considered that the i th workpiece does not have the feature in the template feature region.
[0018] Further, in step 1), the shape of the pre-selected feature region is a rectangle.
[0019] Further, in step 3), the feature point cloud P is transformed into the point cloud data A i in the same space scale by using the registration matrix T i Further, in step 3), the feature point cloud P is transformed into the point cloud data A i in the same space scale by using the registration matrix T i Further, in step 3), the feature point cloud P is transformed into the point cloud data A
[0020] Further, the pre-selected features are holes, studs, protrusions or grooves.
[0021] For the actual field use scene: the sensor fixed installation, the part is stacked in the material frame, the method provided by the application can guarantee the accuracy of feature judgment under the conditions of inconsistent imaging scales, missing point clouds and filling influence, compared with the traditional feature recognition method which only relies on single information (2D information or 3D information), the application shows stronger robustness and recognition stability under complex working conditions. BRIEF DESCRIPTION OF DRAWINGS
[0022] Figure 1a is the measured scene pattern under the laboratory condition and the working distance of 2300 mm, and is recorded as a template;
[0023] Figure 1b is the local screenshot of the feature region in the template image obtained under the laboratory condition and the working distance of 2300 mm;
[0024] Figure 1c is the local screenshot of the feature region in the template point cloud obtained under the laboratory condition and the working distance of 2300 mm;
[0025] Figure 2a is the measured scene pattern under the laboratory condition and the working distance of 3300 mm, and is recorded as a template;
[0026] Figure 2b is the local screenshot of the feature region in the template image obtained under the laboratory condition and the working distance of 3300 mm;
[0027] Figure 2c is the local screenshot of the feature region in the template point cloud obtained under the laboratory condition and the working distance of 3300 mm;
[0028] Figure 3 A schematic diagram of the bridge image obtained at a working distance of 3300 mm using the method of the present application, step 4);
[0029] Figure 4 A schematic diagram of the feature region obtained at a working distance of 3300 mm.
[0030] Figure 5 A physical diagram of a real measured layer-stacked workpiece is given. DETAILED DESCRIPTION
[0031] The technical solutions of the present application are described in detail below in combination with the drawings and specific embodiments.
[0032] A feature recognition method suitable for layer-stacked workpieces, which uses a 3D vision sensor to obtain the image and surface point cloud data of the layer-stacked workpiece to be measured, and the 3D vision sensor is installed in a fixed position; comprising the following steps:
[0033] 1) Defining the uppermost workpiece in the layer-stacked workpiece to be measured as a template workpiece, and its corresponding image and point cloud data are denoted as template image and template point cloud, respectively;
[0034] Pre-selecting a feature peripheral region in the template point cloud as a feature region, extracting the point cloud of the region, denoted as feature point cloud P; at the same time, the position of the template image corresponding to the region is the template feature region Q; in specific implementation, the pre-selected features are holes, studs, protrusions or grooves; and the shape of the pre-selected feature region is a rectangle.
[0035] From top to bottom, the other workpieces to be measured below the template workpiece are sequentially numbered as the ith workpiece, i = 1, 2…N-1, N being the total number of layer-stacked workpieces to be measured;
[0036] Denote the image and point cloud data of the ith workpiece as image C i and point cloud data A i ;
[0037] 2) Using the point cloud data A i and the template point cloud to perform point cloud registration, obtaining the spatial position transformation matrix of the ith workpiece to be measured and the template workpiece, denoted as registration matrix T i ; in specific implementation, any one of the following point cloud registration methods can be used: ① a coarse registration algorithm based on point pair features (PPF); ② a method based on depth map matching to provide an initial value and iterative closest point (ICP) to provide an accurate transformation relationship; ③ a transformation relationship estimation method based on feature matching combined with RANSAC robust strategy; and ④ a method using deep learning to estimate the transformation relationship between point clouds.
[0038] 3) The feature point cloud P is registered using the registration matrix T i Transformed to point cloud data A i The spatial scale of the point cloud is denoted as P. i In practice, each point in the feature point cloud P is matched with the registration matrix T. i Multiply, and get the result with A i A set of point clouds at the same spatial scale. This is achieved using the registration matrix T. i Multiplying by each point in the feature point cloud P yields the result of A. i A set of point clouds at the same spatial scale is denoted as M. i M can be considered as i With A i These are two sets of points that are basically aligned. This process is a priori step for subsequent elimination of scale inconsistencies in 2D feature images.
[0039] Combining the intrinsic parameter matrix of the 3D vision sensor camera, the point cloud P i Backprojecting onto its corresponding image coordinate system yields the corresponding two-dimensional point set, denoted as X. i In practical implementation, based on the camera imaging model, the sensor camera calibration intrinsic parameter matrix can be used to back-project the point cloud. From a mathematical calculation perspective, this yields a set of two-dimensional sub-pixel points. The image plane where these two-dimensional point sets are located can be considered to be at the same 2D imaging scale as the image at the current guiding position, with minimal scaling effects on the features. This provides prior 2D coordinate information for subsequent bilinear interpolation using homography to obtain the grayscale values at the corresponding positions.
[0040] Based on the feature points and point set X in the template feature region Q i Calculate the homography transformation matrix H between them; in specific implementation, the template feature region Q and the point set X i Given two sets of points located on different image planes, solving for the transformation matrix between the image planes of these two sets of points is the solution for the homography transformation relationship. The method used here is the Hough transform based on RANSAC.
[0041] 4) With X i The outer contour of the region in image C i Define the Region of Interest (ROI) in the middle; transform all points within the ROI into the plane containing the template feature region Q using the homography transformation matrix H, and denote this as the point set S. i In practice, all points within the ROI region can be multiplied by the homography transformation matrix H to obtain the coordinates representing the sub-pixel positions of each point. The coordinates corresponding to all points are denoted as the point set S. i ; Set S i Bilinear interpolation is performed on all points within the ROI. The interpolation result is then filled with the corresponding gray values after bilinear interpolation within the ROI range, resulting in an image C.i The feature image in the same scale is denoted as a bridge image;
[0042] 5) The bridge image is matched with the ROI region in the C i circle to obtain a gray template matching result, and whether the i-th workpiece has a feature in the template feature region is determined according to a matching score. Specifically, the bridge image is taken as a feature template image, and the ROI region in the C i circle is taken as a search region for the gray template matching. When the matching score is higher than a threshold, it is considered that the i-th workpiece has a feature in the template feature region, otherwise, it is considered that the i-th workpiece does not have a feature in the template feature region.
[0043] In combination with FIGS. 1-4, the Figure 1a is a measured scene pattern under laboratory conditions at a working distance of 2300 mm, denoted as a template; Figure 1b is a local screenshot of a feature region in a template image obtained under laboratory conditions at a working distance of 2300 mm; Figure 1c is a local screenshot of a feature region in a template point cloud obtained under laboratory conditions at a working distance of 2300 mm; Figure 2a is a measured scene pattern under laboratory conditions at a working distance of 3300 mm; Figure 2b is a local screenshot of a feature region in a template image obtained under laboratory conditions at a working distance of 3300 mm; Figure 2c is a local screenshot of a feature region in a template point cloud obtained under laboratory conditions at a working distance of 3300 mm; Figure 3 is a schematic diagram of a bridge image obtained at a working distance of 3300 mm by using the method of step 4) of the present application; Figure 4 is a schematic diagram of a feature region obtained at a working distance of 3300 mm. Figure 3 and Figure 4 Through the gray template matching, a matching score of 84.58 is obtained, which is higher than a set matching score threshold (80), and it is considered that the hole feature exists at this position.
[0044] The following table shows a comparison between the present application, a traditional 2D detection method and a traditional 3D detection method for a stacked workpiece (as shown in FIG. 16) with a measured distance of 2350-3300 mm, and a feature of a feature hole (a hole in the red frame). Figure 5 It can be seen that the present application has a very obvious advantage for the detection of the feature hole of the stacked workpiece.
[0045]
[0046] The foregoing description of specific exemplary embodiments of the application is presented for the purpose of illustration and description. It is not intended to be a limitation on the present application nor is it intended to be exhaustive or to be a description of the only embodiments of the present application. Obviously many modifications and variations can be derived from the teachings presented above without departing from the scope of the present application. The exemplary embodiments were chosen for illustration and description which are the best in accordance with the present application, but other embodiments can be equally suitable and can be used in accordance with the scope of the present application. The scope of the present application is not intended to be limited to the selected exemplary embodiments but is intended to be limited only by the claims set forth below and equivalents thereof.
Claims
1. A feature recognition method suitable for stacking workpieces, using a 3D vision sensor to obtain the image and surface point cloud data of the stacked workpieces to be measured, the 3D vision sensor being fixed in position; comprising the following steps: 1) defining the uppermost workpiece in the stacked workpieces to be measured as a template workpiece, and its corresponding image and point cloud data as a template image and a template point cloud, respectively; 2) preselecting a feature peripheral region in the template point cloud as a feature region, extracting the point cloud of the region, and recording it as a feature point cloud P; at the same time, the position of the template image corresponding to the region is a template feature region Q; 3) from top to bottom, sequentially numbering the other workpieces to be measured below the template workpiece as the ith workpiece, i = 1, 2…N-1, N being the total number of stacked workpieces to be measured; 4) the point cloud registration method used in step 2) is any one of: ① a coarse registration algorithm based on point pair features; ② a method based on depth map matching to provide an initial value and iterative closest points to provide an accurate transformation relationship; ③ a transformation relationship estimation method based on feature matching combined with a RANSAC robust strategy; and ④ using deep learning to estimate the transformation relationship between point clouds. characterized in that In step 1), the shape of the preselected feature region is a rectangle. In step 1), the preselected feature is a hole, a stud, a protrusion, or a groove. image, point cloud data of the i-th workpiece is denoted as image C i and point cloud data A i ; 2) using point cloud data A i and the template point cloud, obtaining a spatial position transformation matrix of the i-th workpiece to be measured and the template workpiece, denoted as a registration matrix T i ; 3) Transform the feature point cloud P to the point cloud data A using the registration matrix T i i i ; combine the 3D visual sensor camera intrinsic matrix to project the point cloud P i i i , calculate the homography transformation matrix H between them; 4) X i The outer contour of the region where the image C i is located circumscribes the ROI region; all points in the ROI region are converted to the plane where the template feature region Q is located using the homography transformation matrix H, denoted as the point set S i ; all points in the point set S i are subjected to bilinear interpolation, and the interpolation result is filled with the gray value after bilinear interpolation within the above ROI range to obtain a feature image at the same scale as the image C i , denoted as a bridge image; 5) Bridge image is matched with C i The ROI region is determined and gray scale template matching is performed. The presence of features in the template feature region of the i-th workpiece is determined based on the matching score.
2. The feature recognition method for stacking workpieces in a stack as recited in claim 1, wherein: Step 4) The method for converting all points in the ROI region to the plane where the template feature region Q is located by using the homography transformation matrix H is: multiplying all points in the ROI region by the homography transformation matrix H respectively to obtain coordinates representing sub-pixel positions of the points, and recording the coordinates corresponding to all points as a point set S i .
3. The feature recognition method for use in stacking workpieces in a stack, as recited in claim 1, wherein: 4. The feature recognition method for stacking workpieces in a stack as recited in claim 1, wherein: In step 5), when performing the gray scale template matching, the bridge image is taken as the feature template image, and the ROI region circled in C i The ROI region circled in C is the search region for the gray scale template matching. A threshold is set. When the matching score is higher than the threshold, it is considered that the i-th workpiece has features in the template feature region, otherwise, it is considered that the i-th workpiece does not have features in the template feature region.
5. The feature recognition method for stacking workpieces in a stack as recited in claim 1, wherein: 6. The feature recognition method for stacking workpieces in a stack as recited in claim 1, wherein: Step 3) Transform the feature point cloud P to the point cloud data A by using the registration matrix T i i The method for transforming the feature point cloud P to the point cloud data A in the same space scale is: multiplying each point in the feature point cloud P by the registration matrix T i i to obtain a set of point clouds in the same space scale. 7. The feature recognition method for stacking workpieces in a stack as recited in claim 1, wherein: