A high-reflective part positioning and detecting method based on 3D vision

Abstract

The application discloses a high-reflective part positioning detection method based on 3D vision, which comprises the following steps: firstly, a binocular stereo industrial camera is used to shoot a detection area, and point cloud image information containing a high-reflective part obtained is preprocessed to separate a background; then, the separated high-reflective part point cloud image information is corrected to obtain position information of the high-reflective part in a camera coordinate system; and finally, the position information of the high-reflective part in the camera coordinate system is subjected to coordinate transformation to obtain positioning information of the high-reflective part. The application can avoid the influence of a high-reflective area and realize accurate positioning of the high-reflective part.

Description

Technical Field

[0001] This invention belongs to the field of machine vision measurement, and relates to the positioning and detection technology of highly reflective parts, and in particular to a method for positioning highly reflective parts based on 3D vision. Background Technology

[0002] Currently, in the field of industrial production, 2D vision technology is still the mainstream method for positioning parts. It has the advantages of low cost, simple algorithm and convenient installation. However, the measurement accuracy of 2D vision is easily affected by factors such as the lighting conditions in the working environment and the color and texture conditions of the target to be detected.

[0003] Highly reflective parts are widely used in major national equipment projects in aviation, aerospace, shipbuilding, and energy. During the positioning of these parts, their surfaces typically exhibit high light reflection and high contrast. If a 2D vision camera is used for imaging, the area containing the part in the resulting image will often be overexposed, while other areas will be underexposed. This makes positioning highly reflective parts using only two-dimensional images extremely difficult and results in relatively low accuracy.

[0004] With the development of 3D vision measurement technology and the widespread use of stereo vision cameras, acquiring 3D data of objects has become much more convenient, and 3D vision technology is increasingly being applied in the research and development of automated production equipment. 3D vision inspection technology can identify the spatial position and surface information of objects and is less affected by environmental conditions. In recent years, the development and widespread use of 3D sensors such as LiDAR, 3D scanners, and depth cameras have made acquiring 3D data increasingly easier, which has promoted the rapid development of 3D vision technology. Similarly, using 3D vision methods for high-precision positioning of highly reflective parts has significant research value.

[0005] However, there is currently no research on the precise positioning of highly reflective parts based on 3D vision measurement technology, which is one of the key problems that technicians urgently need to solve. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a 3D vision-based method for locating and detecting highly reflective parts, thereby improving the intelligence and detection accuracy of existing highly reflective part positioning methods.

[0007] To achieve the above objectives, the present invention employs the following technical solutions.

[0008] The present invention provides a method for locating and detecting highly reflective parts based on 3D vision, which includes the following steps:

[0009] Step 1. Acquire point cloud image of highly reflective part

[0010] The inspection area is photographed using a binocular stereo industrial camera. The obtained point cloud image information containing highly reflective parts is preprocessed, and the background is separated to obtain the point cloud image of the highly reflective parts.

[0011] Step 2. Correct the point cloud image of the highly reflective parts.

[0012] Gradually reduce the exposure coefficient of the binocular stereo camera and use the binocular stereo industrial camera to take pictures of the detection area until the gray value of more than 90% of the pixels in the point cloud image area of ​​the high reflective part in the two-dimensional image acquired by the left or right camera is within a given threshold range; further perform threshold segmentation on the two-dimensional image to correct the point cloud image of the high reflective part.

[0013] Step 3. Obtain the position information of the highly reflective part in the camera coordinate system.

[0014] Based on the corrected point cloud image of the highly reflective part, the centroid position information of the highly reflective part in the camera coordinate system is obtained;

[0015] Step 4. Obtain the positioning information of the highly reflective parts on the inspection platform.

[0016] The location information of the highly reflective part is obtained by performing coordinate transformation on the position information of the highly reflective part in the camera coordinate system.

[0017] In step 1 above, the highly reflective part is placed on the inspection platform, and the binocular stereo industrial camera is placed directly above the inspection platform, with the camera window facing the position of the highly reflective part.

[0018] Existing binocular stereo industrial cameras can acquire two-dimensional images of the object under test through their left and right cameras, and can also further acquire three-dimensional point cloud images of the object under test. The intrinsic parameters of the left and right cameras of the binocular stereo industrial camera are calibrated separately using Zhang's calibration method.

[0019] The camera intrinsic parameter matrix is:

[0020]

[0021] Where f / dx and f / dy represent the scale factors on the X and Y axes in the camera coordinate system, respectively, and u0 and v0 represent the x and y coordinates of the intersection of the optical axis and the imaging plane in the pixel plane coordinate system, respectively.

[0022] After obtaining the intrinsic parameter matrix, let the coordinates of a point on the highly reflective part in the camera coordinate system be (x...). i ,y i ,z i If ), then its pixel coordinates are described by a parameter matrix as follows:

[0023]

[0024] In the pixel plane coordinate system, the pixel coordinates on the plane are represented as [u,v] T Then the radial distortion is corrected by formula (3).

[0025]

[0026] For tangential distortion, it is corrected by formula (4):

[0027]

[0028] In the formula, r 2 =u 2 +v 2 k1, k2, k3, p1, p1 represent distortion correction parameters.

[0029] Using Zhang's calibration method, the calibration board was photographed multiple times to obtain multiple sets of formulas (3) and (4). The least squares method was used to solve for five parameters [k1,k2,k3,p1,p2] to correct the distortion of the entire image. That is, formula (5) was used to correct the position of each pixel.

[0030]

[0031] The result in formula (2) is corrected by formula (5) for correcting camera distortion, and the corrected coordinates become [u corrected ,v corrected ] T It transforms from the pixel plane to the normalized plane.

[0032] Before using the cameras, the positions of the left and right cameras can be adjusted using the positioning holes on the detection platform. Specifically, the binocular stereo camera is used to photograph the detection platform, and the coordinates of the three positioning holes on the platform are measured as follows: upper left hole Q1 (x1, y1, z1), lower left hole Q2 (x2, y2, z2), and upper right hole Q3 (x3, y3, z3). By adjusting the camera positions, the upper left and lower left positioning holes of the detection platform are aligned vertically in the camera's field of view, and the upper left and upper right positioning holes are aligned horizontally, i.e., x1 = x2, y1 = y3, z1 = z2 = z3.

[0033] The above operations can eliminate the influence of the binocular stereo industrial camera itself on the measurement process of highly reflective parts.

[0034] Then, a binocular stereo industrial camera is used to photograph the inspection area. Based on the previous operations, the captured images are subjected to coordinate correction and transformation to obtain two-dimensional images and three-dimensional original point cloud image information containing highly reflective parts in the camera coordinate system.

[0035] Then, the original point cloud image information is preprocessed to separate the background, specifically including the following steps:

[0036] Step 1.1 Downsample the point cloud image information containing highly reflective parts, i.e., construct a three-dimensional voxel grid; in the three-dimensional voxel grid, the average coordinates of all points within each voxel are used to characterize the corresponding voxel, and the average coordinate points are obtained.

[0037] Step 1.2 Filter the mean coordinate point cloud image obtained by downsampling;

[0038] Step 1.3 Remove all points in the filtered point cloud image whose z-axis value is greater than the set threshold to obtain the image after background separation, i.e., the point cloud image of the highly reflective parts.

[0039] In step 1.1 above, the point cloud image information containing highly reflective parts is downsampled to construct a three-dimensional voxel grid.

[0040] Here, a conventional voxel grid is used to divide the point cloud image containing reflective parts, and then the voxel mean downsampling method is used to downsample the point cloud image information containing highly reflective parts.

[0041] The coordinates of all points within a voxel are as follows:

[0042]

[0043] Then the mean coordinates can be represented as:

[0044]

[0045] In step 1.2 above, a radius filter is used to filter the mean coordinate point cloud image after downsampling. The principle of radius filtering is as follows: Figure 1 As shown, this means taking any point in space as the center O of a circle. i R can be selected based on the camera's resolution. A sphere is drawn with a given radius R, and the number of point clouds within this sphere is calculated as m. If m is less than a pre-set threshold M, then point O at the center of this sphere is considered to be... i Outlier points are removed. This process is repeated for all points, using them as the center of a sphere, to obtain the filtered point cloud image.

[0046] In step 1.3 above, in the camera coordinate system, the z-axis coordinate of any point on the detection platform (any positioning hole can be selected here) is used as the threshold. Then, all points in the filtered point cloud image with z-axis values ​​greater than the set threshold are removed to obtain the image after background separation, and the point cloud image Q of the high reflectivity part is obtained.

[0047] In step 2 above, the exposure coefficient adjustment includes shutter speed and aperture value. By adjusting the shutter speed and aperture value, the exposure coefficient of the stereo camera is gradually reduced. Then, the stereo industrial camera is used to photograph the detection area, acquiring a 2D image containing the highly reflective part through the left or right camera, and converting the 2D image into a grayscale image. Based on the projection of the point cloud image of the highly reflective part onto the XOY plane, the pixel values ​​of the corresponding Q region of the point cloud image of the highly reflective part in the 2D image are determined, and it is judged whether the grayscale value of more than 90% of the pixels is within a given threshold range. If not, the camera exposure coefficient is further reduced until the grayscale value of more than 90% of the pixels in the Q region of the point cloud image of the highly reflective part in the 2D image acquired by the left or right camera is within the given threshold range. Then, based on the set upper limit of the threshold, the 2D image is further thresholded, and points exceeding the upper limit of the threshold are filtered out and removed from the point cloud image of the highly reflective part, completing the correction of the point cloud image of the highly reflective part, and obtaining the accurate region N of the point cloud image of the highly reflective part.

[0048] In step S3 above, connected component segmentation is performed in the precise region N of the point cloud image of the highly reflective part to obtain the point cloud of each highly reflective part, denoted as L. j B represents the number of points in the point cloud corresponding to the highly reflective part, and the coordinates of the points in the point cloud are (x, y, y). b ,y b ,z b Next, the centroid coordinates K of each part's point cloud in the camera coordinate system are calculated. j :

[0049]

[0050] Connected component segmentation can be performed using conventional methods already disclosed in this field, such as the Two-Pass algorithm and the region growing algorithm.

[0051] In step S4 above, the coordinates Q1, Q2, and Q3 of the three positioning holes on the detection platform in the camera coordinate system, and the centroid coordinates K of the point cloud of the highly reflective part in the camera coordinate system are used as the basis for this step. j Taking the upper left positioning hole Q1 on the inspection platform as the origin, the positioning information - position G - of the high-reflectivity part point cloud on the inspection platform can be represented as:

[0052]

[0053] Where x k ,y k ,z k These are the coordinates of the centroid of the point cloud of the highly reflective component in the camera coordinate system. This indicates the coordinates of the upper left positioning hole in the camera coordinate system.

[0054] Therefore, this invention is based on 3D vision methods, especially using an active stereo camera to directly obtain three-dimensional point cloud image information of the part, unaffected by highly reflective areas of the part, and also less susceptible to ambient lighting. The obtained three-dimensional point cloud image is then processed through downsampling, filtering, correction, and coordinate transformation to achieve precise positioning information for the highly reflective part.

[0055] The method for locating and detecting highly reflective parts based on 3D vision provided by this invention has the following beneficial effects:

[0056] (1) The present invention first uses a binocular stereo industrial camera to acquire point cloud images containing highly reflective parts, then preprocesses the point cloud images to separate the background to obtain point cloud images of highly reflective parts, then corrects the point cloud images of highly reflective parts to obtain the position information of highly reflective parts in the camera coordinate system, and then converts it into the position information of the detection platform; the method can avoid the influence of highly reflective areas and achieve accurate positioning of highly reflective parts.

[0057] (2) The present invention can improve the intelligent positioning level of high reflective parts processing equipment, and no longer rely on manual or low-precision 2D vision to position high reflective parts, thereby improving work efficiency and reducing positioning error. Attached Figure Description

[0058] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a schematic diagram of the radius filtering principle.

[0060] Figure 2 This is a scenario diagram illustrating the application of the 3D vision-based high-reflectivity part positioning and detection method provided in Embodiment 1 of the present invention.

[0061] Figure 3 A top view of the inspection platform where parts are placed.

[0062] Figure 4 This is a schematic diagram of the process for a high-reflectivity part positioning and detection method based on 3D vision provided in Embodiment 1 of the present invention.

[0063] Figure 5 This is a step-by-step diagram of step 1 in the 3D vision-based method for locating and detecting highly reflective parts.

[0064] Figure 6 for Figure 3A 3D point cloud diagram of one of the parts. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0066] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0067] Example 1

[0068] The 3D vision-based method for locating and detecting highly reflective parts provided in this embodiment is applicable to scenario diagrams. Figure 2 It includes a binocular stereo industrial camera 1 (referred to as a 3D camera, which includes a left camera 2 and a right camera 3). Directly below the camera is a detection platform 4, on which highly reflective parts to be located are placed. In this embodiment, the 3D camera 1 is an active binocular camera.

[0069] In practical application, this embodiment first adjusts the camera position so that the camera is positioned within the camera's field of view. Figure 3 The upper left positioning hole 6 (Q1(x1, y1, z1)) and the lower left positioning hole 7 (Q2(x2, y2, z2)) of the testing platform are on the same vertical line, and the upper left positioning hole 6Q1(x1, y1, z1) and the upper right positioning hole 8Q3(x3, y3, z3) are on the same horizontal line.

[0070] Next, a 3D camera 1 is used to capture images, followed by point cloud information processing. The position of the highly reflective part 5 in the camera coordinate system is calculated and then converted into coordinate information of the detection platform relative to the alignment hole. The position information of part 5 is transmitted to the host computer system, which controls the actuator to complete subsequent work on the part. This embodiment specifically includes the following operations:

[0071] a. The highly reflective part 5 is transferred to the testing platform;

[0072] b. The binocular stereo vision camera 1 emits structured light to extract point cloud information on the detection plane;

[0073] c. Based on the acquired point cloud information, the position of the highly reflective part 5 is obtained according to steps 1-3 given above, and the position information is transmitted to the host computer system.

[0074] d. The host computer system controls the actuator to complete the subsequent work on the parts.

[0075] It should be noted that after the parts are positioned on the current inspection plane and the subsequent work on these parts is completed, a new batch of highly reflective parts will be transferred to inspection plane 4, and the above steps ad will be repeated until the positioning of all parts is completed.

[0076] Based on the above analysis, the 3D vision-based method for locating and detecting highly reflective parts provided in this embodiment, such as... Figure 4 and Figure 5 As shown, it includes the following steps:

[0077] Step 1. Acquire point cloud image of highly reflective part

[0078] The inspection area is photographed using a binocular stereo industrial camera. The obtained point cloud image information containing highly reflective parts is preprocessed, and the background is separated to obtain the point cloud image of the highly reflective parts.

[0079] First, the inspection area is photographed using a binocular stereo industrial camera. Based on the previous operations, the captured images are coordinate corrected and transformed to obtain two-dimensional images and three-dimensional original point cloud image information containing highly reflective parts in the camera coordinate system.

[0080] Then, the original point cloud image information is preprocessed to separate the background, specifically including the following steps:

[0081] Step 1.1 Downsample the point cloud image information containing highly reflective parts, i.e., construct a three-dimensional voxel grid; in the three-dimensional voxel grid, the mean coordinate of all points in each voxel is used to represent the corresponding voxel, and the mean coordinate point is obtained.

[0082] A three-dimensional voxel grid is constructed by downsampling the point cloud image information containing highly reflective parts.

[0083] Here, a conventional voxel grid is used to divide the point cloud image containing reflective parts, and then the voxel mean downsampling method is used to downsample the point cloud image information containing highly reflective parts.

[0084] The coordinates of all points within a voxel are as follows:

[0085]

[0086] Then the mean coordinates can be represented as:

[0087]

[0088] Step 1.2 Filter the mean coordinate point cloud image obtained by downsampling.

[0089] In this step, a radius filter is used to filter the mean coordinate point cloud image after downsampling. The principle of radius filtering is as follows: Figure 1 As shown, this means taking any point in space as the center O of a circle. i R can be selected based on the camera's resolution. A sphere is drawn with a given radius R, and the number of point clouds within this sphere is calculated as m. If m is less than a pre-set threshold M, then point O at the center of this sphere is considered to be... i Outlier points are removed. This process is repeated for all points, using them as the center of a sphere, to obtain the filtered point cloud image.

[0090] Step 1.3 Remove all points in the filtered point cloud image whose z-axis value is greater than the set threshold to obtain the image after background separation, and obtain the point cloud map of the highly reflective parts.

[0091] In this step, under the camera coordinate system, the z-axis coordinate (z1, z2 or z3) of any positioning hole on the detection platform is used as the threshold. Then, all points in the filtered point cloud image with z-axis values ​​greater than the set threshold are removed to obtain the image after background separation, thus obtaining the point cloud map of the highly reflective part.

[0092] Step 2. Correct the point cloud image of the highly reflective parts.

[0093] The exposure coefficient of the binocular stereo camera is gradually reduced, and the detection area is photographed using the binocular stereo industrial camera until the gray value of more than 90% of the pixels in the point cloud image area of ​​the high reflective part in the two-dimensional image acquired by the left or right camera is within a given threshold range; further, the point cloud image of the high reflective part is corrected by threshold segmentation of the two-dimensional image.

[0094] The exposure factor adjustment mentioned here includes shutter speed and aperture value. By adjusting the shutter speed and aperture value, the exposure factor of the stereo camera is gradually reduced. Then, the stereo industrial camera is used to photograph the inspection area, acquiring a 2D image containing the highly reflective part through either the left or right camera, and converting the 2D image into a grayscale image. Based on the projection of the point cloud image of the highly reflective part onto the XOY plane, the pixel values ​​of the corresponding Q region of the point cloud image of the highly reflective part in the 2D image are determined, and it is judged whether the grayscale value of more than 90% of the pixels is within a given threshold range. If not, the camera exposure factor is further reduced until the grayscale value of more than 90% of the pixels in the Q region of the point cloud image of the highly reflective part in the 2D image acquired by the left or right camera is within the given threshold range. Then, based on the set upper limit of the threshold, the 2D image is further thresholded, and points exceeding the upper limit are filtered out and removed from the point cloud image of the highly reflective part, completing the correction of the point cloud image of the highly reflective part, and obtaining the accurate region N of the point cloud image of the highly reflective part.

[0095] Step 3. Obtain the position information of the highly reflective part in the camera coordinate system.

[0096] Based on the corrected point cloud image of the highly reflective parts, the centroid position information of the highly reflective parts in the camera coordinate system is obtained. In this step, in the precise region N of the point cloud image of the highly reflective parts, a region growing algorithm (see Automatic Seeded Region Growing Image Segmentation for Medical Image Segmentation: A Brief Review, Neeraj Shrivastava et al. International Journal of Image and Graphics Vol.20, No.3(2020)2050018(21pages)) is used to segment the connected components, and the point cloud of each highly reflective part is denoted as L. j ( Figure 6 (A 3D point cloud diagram of one of the parts is given), where B is the number of points in the point cloud of the corresponding highly reflective part, and the coordinates of the points in the point cloud are (x, y, y). b ,y b ,z b Next, the centroid coordinates K of each part's point cloud in the camera coordinate system are calculated. j :

[0097]

[0098] Step 4. Obtain the positioning information of the highly reflective parts on the inspection platform.

[0099] The location information of the highly reflective part is obtained by performing coordinate transformation on the position information of the highly reflective part in the camera coordinate system.

[0100] In this step, taking the upper left positioning hole Q1 on the inspection platform as the origin, the positioning information - position G - of the high-reflectivity part point cloud on the inspection platform can be represented as:

[0101]

[0102] Where x k ,y k ,z k These are the coordinates of the centroid of the point cloud of the highly reflective component in the camera coordinate system. This indicates the coordinates of the upper left positioning hole in the camera coordinate system.

[0103] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A 3D vision-based high-reflective part positioning detection method, characterized in that, The method comprises the following steps: Step 1. Obtain a high-reflective part point cloud image The detection area is photographed by using a binocular stereo industrial camera, and the obtained high-reflective part point cloud image information is preprocessed to separate the background and obtain the high-reflective part point cloud image. Step 1 specifically comprises the following steps: Step 1.

1. Down-sampling the point cloud image information containing the high-reflective part, that is, constructing a three-dimensional voxel grid; in the three-dimensional voxel grid, the mean coordinates of all points in each voxel are used to represent the corresponding voxel, and the mean coordinate points are obtained; Step 1.

2. Filtering the down-sampled mean coordinate point cloud image; Step 1.

3. Removing all points with a z-axis value greater than a set threshold from the filtered point cloud image to obtain a background-separated image, that is, a high-reflective part point cloud image; Step 2. Correcting the high-reflective part point cloud image Gradually reduce the exposure coefficient of the binocular stereo camera, photograph the detection area by using the binocular stereo industrial camera, and until more than 90% of the pixels in the two-dimensional image obtained by the left camera or the right camera corresponding to the high-reflective part point cloud image region are within a given threshold range; further threshold segmentation is performed on the two-dimensional image to correct the high-reflective part point cloud image; Step 3. Obtain the position information of the high-reflective part in the camera coordinate system Based on the corrected high-reflective part point cloud image, the centroid position information of the high-reflective part in the camera coordinate system is obtained; Step 4. Obtain the positioning information of the high-reflective part on the detection platform The position information of the high-reflective part in the camera coordinate system is transformed to obtain the positioning information of the high-reflective part.

2. The 3D vision-based high-reflective part positioning detection method of claim 1, wherein, In step 1.2, a radius filter is used to filter the down-sampled mean coordinate point cloud image.

3. The 3D vision-based high-reflective part positioning detection method of claim 1, wherein, In the camera coordinate system, the z-axis coordinate of any point on the detection platform is used as a threshold, and then all points with a z-axis value greater than the set threshold are removed from the filtered point cloud image to obtain a background-separated image, that is, a high-reflective part point cloud image.

4. The 3D vision-based high-reflective part positioning detection method of claim 1, wherein, In step 2, according to the set upper threshold, the two-dimensional image is threshold segmented, the points exceeding the upper threshold are screened out, and the high-reflective part point cloud image is corrected by removing the points from the high-reflective part point cloud image.

5. The 3D vision based high-reflective part positioning detection method of claim 1, wherein, Based on the corrected high-reflective part point cloud image, a connected domain segmentation is performed to obtain a point cloud of each high-reflective part, denoted as , B is the number of points in the point cloud of the corresponding high-reflective part, the coordinates of the points in the point cloud are , and then the centroid coordinates of each part point cloud in the camera coordinate system are calculated as : 。 6. The 3D vision based high-reflective part positioning detection method of claim 1, wherein, Taking any point on the detection platform as the coordinate origin, the positioning information of the high-reflective part point cloud centroid in the detection platform is obtained.

Images