Sheet metal part automatic detection method based on multi-view 3D scanning

Abstract

The invention discloses a sheet metal part automatic detection method based on multi-view 3D scanning, and the method comprises the steps: enabling a six-axis mechanical arm to drive a 3D camera to move, cooperating with a rotary table to rotate, and obtaining the point cloud data and image data of the outer surface of a sheet metal part. Through a structured light 3D reconstruction algorithm, a point cloud splicing algorithm based on coding points, a hole center 3D positioning algorithm reconstruction algorithm based on multiple views and an edge 3D reconstruction algorithm based on a part digital model, the index precision of the diameter, the hole circle center distance, the height, the planeness and the like of the high-precision metal plate round hole can reach within 0.1 mm through a structured light 3D reconstruction algorithm, a point cloud splicing algorithm based on coding points, a hole center 3D positioning algorithm reconstruction algorithm based on multiple views and an edge 3D reconstruction algorithm based on a part digital model. The system realized by the invention has good robustness, and solves the problem that the indexes such as height, planeness, different plane apertures and the like cannot be measured by the current pure 2D vision measurement. The problems that pure 3D point cloud processing is not high in splicing precision, and the reconstruction quality of the edge point cloud of the sheet metal part is not high, so that the workpiece measurement precision is not high are solved.

Description

Technical Field

[0001] The present invention relates to the field of industrial automation measurement technology, and in particular to an automated detection method for sheet metal parts based on multi-view 3D scanning. Background Art

[0002] With the development and progress of modern industry, especially in some high-precision processing industries, traditional detection methods are far from meeting the needs of production. Machine vision measurement technology is a non-contact measurement method based on optical imaging, digital image processing, and computer graphics, with a wider measurement range, higher measurement accuracy and efficiency.

[0003] At present, there are contact-type three-coordinate machines in the field of high-precision measurement. Although the three-coordinate machine has high accuracy, its structure is complex and expensive. It has high environmental requirements, time-consuming measurement path planning, and the probe is easily damaged. Non-contact measurement is divided into 2D visual measurement and 3D visual measurement. 2D visual measurement uses a single industrial camera and installs an industrial fixed-focus lens or a center lens. Generally, the workpiece needs to be back-lit. The use of industrial lenses can only be limited to measuring on a calibrated plane. If the workpiece indicators are not on the same plane, they cannot be measured. Generally, multiple cameras are required to measure different parameters of a workpiece. The system is highly complex, and workpiece switching is troublesome and difficult to achieve. In addition, it is impossible to measure parameters such as flatness and out-of-plane distance. With the installation of a center lens, the workpiece size measurement in the direction perpendicular to the camera has high accuracy, but it is also impossible to measure parameters such as flatness and out-of-plane distance.

[0004] 3D visual measurement is to use line laser with motion platform or handheld 3D scanner or structured light 3D camera to reconstruct the workpiece in 3D, and process the acquired point cloud for measurement. Line laser with motion platform has high measurement accuracy, but small field of view, and cannot completely splice the entire outer surface of the workpiece. It is generally used to detect height and flatness. The hole spacing and circular hole parameters depend on the quality of the hole edge point cloud, and the measurement accuracy is also limited. Handheld 3D scanners need to attach reflective targets on and around the workpiece to splice the point cloud. Scanning with a scanner by hand or robot can obtain the point cloud of the inner wall of the circular hole from different angles. The measurement accuracy is high, but the measurement accuracy of small holes or shallow holes also depends on the quality of the hole edge, and the reconstruction accuracy of the edge of the handheld scanner is not high. It is also difficult to achieve high-efficiency automation because each workpiece needs to be attached to switch points. Some workpieces that cannot be touched cannot be attached with reflective points. Manual post-processing of data calculation is also required, and online calculation cannot be performed. The cost of carrying 3D with a robotic arm is also high, and automatic measurement software needs to be written. Pure structured light 3D cameras can be used with robots to reconstruct all point clouds on the outer surface of workpieces, but the reconstruction effect of edges is not good, and the difference can be as small as 1 to 2 pixels. Stitching is usually done through robot calibration, and the stitching accuracy depends on the absolute positioning accuracy of the robot. However, the absolute positioning accuracy of robots is generally not high, usually around 0.5mm, and the cost of robots with high absolute positioning accuracy is very high. Or it depends on the characteristics of the workpiece itself for stitching, but for workpieces with unclear features or symmetrical features, the stitching effect is unpredictable and not very robust. Summary of the invention

[0005] The present invention provides an automated detection method for sheet metal parts based on multi-view 3D scanning, which can achieve high-precision automated measurement of the aperture, hole spacing, height, and flatness of sheet metal processed parts.

[0006] The automatic detection method of sheet metal parts based on multi-view 3D scanning includes the following steps: Step 1: Use the computer software as the main control terminal to communicate with the robot, 3D camera, and turntable; Step 2: manually place the sheet metal part at the center of the turntable and select and start the system according to the type of workpiece to be placed; Step 3: The system controls the robot arm to move with the 3D camera to the planned designated position and stops, cooperates with the rotating turntable to make the workpiece at a suitable angle, and then takes a 2D fill light picture of the workpiece and 24 structured light pictures, and saves the data; Step 4: Use the collected 24 structured light image data to reconstruct a dense point cloud through a structured light 3D reconstruction algorithm; Step 5, stitching the dense point cloud to the turntable coding point coordinate system through the coding point based point cloud stitching algorithm, saving the matrix used for stitching calculation, and displaying the dense point cloud after the reconstruction and stitching in the software interface; Step 6, repeating steps 3 to 5 until the entire outer surface data of the workpiece is photographed and calculated; Step 7: filter the spliced ​​point cloud to remove noise, use the multi-view based hole center 3D positioning algorithm, and calculate the 3D coordinates of the center of each hole in the picture through multiple 2D fill light pictures taken and saved in the previous step, and perform point-to-point registration with the hole positions of the designed digital model to obtain the registered rigid body transformation matrix, and transform the point cloud to the digital model coordinate system; Step 8, according to the captured 2D fill light picture, using the edge 3D reconstruction algorithm based on the part digital model, calculate the actual discrete 3D coordinates of all contours; Step 9, perform least squares plane fitting on all discrete 3D coordinates of each individual circular contour, and correct each discrete 3D coordinate of the circular contour to a plane whose normal vector is equal to (0, 0, 1) according to the fitted plane, and then use the X and Y coordinates of the corrected discrete 3D coordinates of the circular contour to perform least squares fitting on the circle to obtain the aperture value and the coordinates of the hole center; Step 10, restore the corrected hole center coordinates to obtain the hole center coordinates and hole center distance values ​​in the digital model coordinate system; Step 11, transforming the point cloud to the digital model coordinate system through step 7, segmenting and intercepting the point cloud according to the required height area pre-planned under the digital model, performing plane fitting on the intercepted area, and projecting the planned points to the fitted plane to obtain the position coordinates under the digital model, transforming the position coordinates to the turntable plane coordinate system, obtaining the height of this coordinate area on the turntable, and obtaining the height information of the sheet metal part; Step 12, transforming the point cloud into the digital model coordinate system through step 7, segmenting and intercepting the point cloud according to the desired flatness area pre-planned under the digital model, and directly performing flatness calculation on the segmented point cloud to obtain the flatness calculation result; Step 13, put all the required data into a table format, display it in the software, and save it in EXCEL to form a measurement report, and the system operation ends.

[0007] Furthermore, the step 4 uses the collected 24 structured light image data to reconstruct a dense point cloud through a structured light 3D reconstruction algorithm. The structured light 3D reconstruction algorithm is specifically: Design the mechanism light coding pattern according to the resolution of the camera and generate the corresponding coding pattern; Decode the collected structured light patterns to achieve pixel-level structured light image encoding; According to the encoding results and the calibration parameters of the 3D camera, the spatial coordinates corresponding to all encoded pixels are calculated to form a dense 3D point cloud.

[0008] Furthermore, the point cloud stitching algorithm based on the coded points in step 5 is specifically as follows: Extract and identify the code points in the images collected at different positions and obtain the image coordinates and specific codes of the code point center; According to the encoding results, the encoding points in the images with different postures are matched, the basic matrix between two adjacent images is solved, and the relative postures between different images are calculated; Perform global pose optimization, calculate the spatial coordinates of all coded points, and generate a point cloud splicing skeleton; The point clouds obtained at different positions are spliced ​​according to the skeleton to obtain complete point cloud data.

[0009] Furthermore, the step 7 is based on a multi-view hole center 3D positioning algorithm, and the specific processing process is as follows: Extract the hole contour from the images collected at different positions and obtain the image coordinates of the hole center; According to the acquired basic matrix, the epipolar lines are calculated for image matching of the hole centers, so as to achieve matching of the hole centers in all acquired images; The matching results of the hole centers are merged with the matching results of the encoding points, and then a global pose optimization is performed to calculate the 3D coordinates of all hole centers.

[0010] Furthermore, the step 8 is specifically as follows: Extract edge contours from images collected at different positions, discretize the contours at the pixel level, and obtain image discrete points of the contours; According to the hole center coordinates obtained in the hole center positioning algorithm module, the hole center coordinates in the part digital model are aligned to obtain the spatial pose of the digital model relative to all shooting poses; Extract all contours that need to be reconstructed in the digital model, and discretize them to obtain spatial discrete points of the contours; Back-projecting the spatial discrete points back to the corresponding image according to the spatial pose, finding the image contour discrete points closest to the back-projected discrete points, and encoding them; The actual discrete 3D coordinates of all contours are calculated based on the acquired pose parameters.

[0011] The beneficial effects of the present invention are: The present invention solves the problem that 2D vision cannot measure flatness and skewness in the field of non-contact measurement, and the problem that 3D vision cannot measure aperture and hole spacing with high precision and automation. By high-precision splicing of the point cloud by coded points, and then high-precision reconstruction of the hole edge through the idea of ​​2D and 3D fusion, high-precision sheet metal workpiece size information is obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 It is a flow chart design diagram of the system of the present invention; Figure 2 It is the stp model diagram of the standard plate of the present invention; Figure 3 is a digital model diagram of a sheet metal part 1 tested by the present invention; Figure 4 is the detection result of the sheet metal part 2 by the system of the present invention; Figure 5 is the detection result of the sheet metal part 3 by the system of the present invention; Figure 6 It is the detection result of the sheet metal part 4 by the system of the present invention. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0014] The technical solutions provided by various embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0015] The method used in the present invention is based on a camera clamping motion module, a 3D camera module, and a turntable module. The camera clamping motion module includes a 6-axis mechanical arm and a camera fixing tool. The 3D camera module includes a camera, a projector, and a ring light source. The turntable module mainly includes a PLC, a stepping motor, and a disk with two circles of coding points.

[0016] like Figure 1 As shown, the automatic detection method of sheet metal parts based on multi-view 3D scanning includes the following steps: Step 1: Use the computer software as the main control terminal to establish communication connections with the robot, 3D camera, and turntable.

[0017] Step 2: Manually place the sheet metal part at the center of the turntable, select the type of workpiece to be placed, and start the system.

[0018] Step 3: The system controls the robot arm to move with the 3D camera to the planned designated position and stops, cooperates with the rotating turntable to make the workpiece at a suitable angle, and then takes a 2D fill light picture of the workpiece and 24 structured light pictures, and saves the data.

[0019] Step 4: Use the collected data to reconstruct the point cloud through the structured light 3D reconstruction algorithm. The specific process of the structured light 3D reconstruction algorithm is as follows: ① Design the structured light coding pattern according to the camera resolution and generate the corresponding coding pattern for structured light projection; ②Decode the collected structured light patterns to achieve pixel-level structured light image encoding; ③ According to the encoding results and the calibration parameters of the 3D camera, the spatial coordinates corresponding to all encoded pixels are calculated to form a dense 3D point cloud.

[0020] Step 5: Use the point cloud stitching algorithm based on coded points to stitch the point cloud to the turntable target point coordinate system with high precision, save the matrix used for stitching calculation, and display the reconstructed point cloud in the software interface. The specific process of the point cloud stitching algorithm based on coded points is as follows: ① Extract and identify the coding points in the images collected at different positions, and obtain the image coordinates and specific codes of the coding point center; ② According to the encoding results, the encoding points in the images of different postures are matched to solve the basic matrix between two adjacent images, and further calculate the relative postures between different images; ③ Perform global pose optimization, calculate the spatial coordinates of all coded points, and generate a point cloud splicing skeleton; ④ The point clouds obtained at different positions are spliced ​​according to the skeleton to obtain complete point cloud data.

[0021] Step 6, repeat steps 3 to 5 until the entire outer surface data of the workpiece is captured and calculated.

[0022] Step 7, filter the stitched point cloud to remove noise. Through the stitching matrix obtained and multiple 2D fill light pictures taken and saved in the previous step, the center coordinates of each hole in the picture are calculated using the multi-view based hole center 3D positioning algorithm. Point-to-point registration is performed with the hole positions of the designed digital model to obtain the registered rigid body transformation matrix, and the point cloud can be transformed into the digital model coordinate system. This algorithm module is used for high-precision 3D positioning of the center of the hole on the part surface. The specific process is as follows: ① Extract the hole contour from the images collected at different positions and obtain the image coordinates of the hole center; ②According to the basic matrix obtained in the stitching algorithm module, the epipolar lines can be calculated for image matching of the hole center, so as to achieve matching of the hole centers in all acquired images; ③ Combine the matching results of the hole centers with the matching results of the encoding points, perform global pose optimization again, and calculate the 3D coordinates of all hole centers.

[0023] Step 8: Based on the captured 2D fill light picture, the actual discrete 3D coordinates of all contours are calculated using the edge 3D reconstruction algorithm based on the part digital model. This algorithm module is used for 3D reconstruction of part edge contours (including hole contours). The specific process is as follows: ① Extract edge contours from images collected at different positions, discretize the contours at the pixel level, and obtain image discrete points of the contours; ②According to the hole center coordinates obtained in the hole center positioning algorithm module, align them with the hole center coordinates in the part digital model to obtain the spatial pose of the digital model relative to all shooting poses; ③ Extract all contours that need to be reconstructed from the digital model, and discretize them to obtain the spatial discrete points of the contours; ④ According to the position of ②, the spatial discrete points in ③ are back-projected into the corresponding image, and the image contour discrete points closest to the back-projected discrete points are found and encoded; ⑤ Calculate the actual discrete 3D coordinates of all contours based on the pose parameters obtained in algorithm module 3.

[0024] Step 9, perform least squares plane fitting on all discrete 3D coordinates of each individual circular contour, and correct each individual circular contour discrete 3D coordinate to a plane whose normal vector is equal to (0, 0, 1) according to the fitted plane, and then perform least squares fitting on the circle using only the X and Y coordinates of the corrected circular contour discrete 3D coordinates to obtain high-precision aperture values ​​and hole center coordinates.

[0025] Step 10, restore the corrected hole center coordinates to obtain the hole center coordinates in the digital-analog coordinate system. After knowing the center coordinates of each hole in the digital-analog coordinate system, obtain a high-precision hole center distance value.

[0026] Step 11, transform the point cloud to the digital model coordinate system through step 7, segment and intercept the point cloud according to the required height area pre-planned in the digital model, perform plane fitting on the intercepted area, and project the planned points to the fitted plane to obtain the position coordinates under the high-precision digital model. Transform this coordinate to the turntable plane coordinate system, obtain the height of this coordinate area on the turntable, and obtain the height information of the sheet metal part.

[0027] Step 12, transforming the point cloud to the digital model coordinate system through step 7, segmenting and intercepting the point cloud according to the desired flatness area pre-planned in the digital model, and directly performing flatness calculation on the segmented point cloud to obtain the flatness calculation result.

[0028] Step 13, put all the required data into a table format, display it in the software, and save it in EXCEL to form a measurement report, and the system operation ends.

[0029] The specific embodiments of the present invention are as follows: First, place a special indium steel tetrahedron for measurement. The actual data of the indium steel tetrahedron has been measured by professional equipment. The standard distance accuracy between the small balls of the indium steel tetrahedron is less than 0.001mm. The robot with a 3D camera reconstructs and splices the standard balls on the indium steel tetrahedron, fits the small balls in the point cloud and measures the distance. The following data is obtained, proving that the accuracy of point cloud reconstruction and splicing is less than 0.1mm. The point cloud reconstruction and splicing data are shown in Table 1: Table 1 like Figure 2 As shown in the figure, a standard plate is prepared, and the coding points are affixed to the standard plate. The aperture, hole spacing, height and flatness of the inspection plate are measured by a three-coordinate machine with an accuracy of 0.001mm.

[0030] The standard plate is placed on the turntable, sampled through this system, and solved using the system algorithm. The results show that the data required by this system is less than 0.1. The reference value represents the actual value measured by the three-coordinate machine, the tolerance represents the set tolerance, and the actual value represents the value calculated by the system software algorithm for the workpiece. If the reference value minus the actual value is within the tolerance range, the data in this row will be displayed as qualified, otherwise it will be unqualified.

[0031] Four different sheet metal parts are prepared, and through the system steps of the present invention, the system is started to automatically sample and calculate the sheet metal parts respectively, and the results are displayed. Figure 3 Digital model for sheet metal part 1, Figure 4 The calculation results of sheet metal 2 software are as follows: Figure 5 The calculation results of sheet metal 3 software are as follows: Figure 6The calculation result of sheet metal 4 software. In the calculation result diagram, the reference value is the design value of the sheet metal drawing, the upper tolerance and the lower tolerance indicate the deviation that the processing designed in the drawing cannot exceed, and the actual value indicates the value calculated by the system. If the reference value minus the actual value does not exceed the design deviation, it will be displayed as qualified, otherwise it will be displayed as unqualified. All the above workpieces have obtained qualified results. Moreover, the present invention can not only measure the hole diameter, hole spacing, flatness, and height information, but also measure the edge-to-edge distance, edge-to-face distance, face-to-face distance, and eccentric hole spacing in space.

[0032] In summary, the present invention solves the problem that 2D vision cannot measure flatness and skewness in the field of non-contact measurement, and the problem that 3D vision cannot measure aperture and hole spacing with high precision and automation. By high-precision splicing of the point cloud by coded points, and then high-precision reconstruction of the hole edge through the idea of ​​2D and 3D fusion, high-precision sheet metal workpiece size information is obtained.

[0033] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. The automatic detection method of sheet metal parts based on multi-view 3D scanning is characterized by: The following steps are involved: Step 1: Use the computer software as the main control terminal to communicate with the robot, 3D camera, and turntable; Step 2: manually place the sheet metal part at the center of the turntable and select and start the system according to the type of workpiece to be placed; Step 3: The system controls the robot arm to move with the 3D camera to the planned designated position and stops, cooperates with the rotating turntable to make the workpiece at a suitable angle, and then takes a 2D fill light picture of the workpiece and 24 structured light pictures, and saves the data; Step 4: Use the collected 24 structured light image data to reconstruct a dense point cloud through a structured light 3D reconstruction algorithm; Step 5, stitching the dense point cloud to the turntable coding point coordinate system through the coding point based point cloud stitching algorithm, saving the matrix used for stitching calculation, and displaying the dense point cloud after the reconstruction and stitching in the software interface; Step 6, repeating steps 3 to 5 until the entire outer surface data of the workpiece is photographed and calculated; Step 7: filter the spliced ​​point cloud to remove noise, use the multi-view based hole center 3D positioning algorithm, and calculate the 3D coordinates of the center of each hole in the picture through multiple 2D fill light pictures taken and saved in the previous step, and perform point-to-point registration with the hole positions of the designed digital model to obtain the registered rigid body transformation matrix, and transform the point cloud to the digital model coordinate system; Step 8, according to the captured 2D fill light picture, using the edge 3D reconstruction algorithm based on the part digital model, calculate the actual discrete 3D coordinates of all contours; Step 9, perform least squares plane fitting on all discrete 3D coordinates of each individual circular contour, and correct each discrete 3D coordinate of the circular contour to a plane whose normal vector is equal to (0, 0, 1) according to the fitted plane, and then use the X and Y coordinates of the corrected discrete 3D coordinates of the circular contour to perform least squares fitting on the circle to obtain the aperture value and the coordinates of the hole center; Step 10, restore the corrected hole center coordinates to obtain the hole center coordinates and hole center distance values ​​in the digital model coordinate system; Step 11, transforming the point cloud to the digital model coordinate system through step 7, segmenting and intercepting the point cloud according to the required height area pre-planned under the digital model, performing plane fitting on the intercepted area, and projecting the planned points to the fitted plane to obtain the position coordinates under the digital model, transforming the position coordinates to the turntable plane coordinate system, obtaining the height of this coordinate area on the turntable, and obtaining the height information of the sheet metal part; Step 12, transforming the point cloud into the digital model coordinate system through step 7, segmenting and intercepting the point cloud according to the desired flatness area pre-planned under the digital model, and directly performing flatness calculation on the segmented point cloud to obtain the flatness calculation result; Step 13, put all the required data into a table format, display it in the software, and save it in EXCEL to form a measurement report, and the system operation ends.

2. The automatic detection method for sheet metal parts based on multi-view 3D scanning according to claim 1 is characterized in that: The step 4 uses the collected 24 structured light image data to reconstruct a dense point cloud through a structured light 3D reconstruction algorithm. The structured light 3D reconstruction algorithm is specifically as follows: Design the mechanism light coding pattern according to the resolution of the camera and generate the corresponding coding pattern; Decode the collected structured light patterns to achieve pixel-level structured light image encoding; According to the encoding results and the calibration parameters of the 3D camera, the spatial coordinates corresponding to all encoded pixels are calculated to form a dense 3D point cloud.

3. The automatic detection method for sheet metal parts based on multi-view 3D scanning according to claim 1 is characterized in that: The point cloud stitching algorithm based on the coded points in step 5 is specifically as follows: Extract and identify the code points in the images collected at different positions and obtain the image coordinates and specific codes of the code point center; According to the encoding results, the encoding points in the images with different postures are matched, the basic matrix between two adjacent images is solved, and the relative postures between different images are calculated; Perform global pose optimization, calculate the spatial coordinates of all coded points, and generate a point cloud splicing skeleton; The point clouds obtained at different positions are spliced ​​according to the skeleton to obtain complete point cloud data.

4. The automatic detection method for sheet metal parts based on multi-view 3D scanning according to claim 1 is characterized in that: The step 7 is based on the multi-view hole center 3D positioning algorithm, and the specific processing process is as follows: Extract the hole contour from the images collected at different positions and obtain the image coordinates of the hole center; According to the acquired basic matrix, the epipolar lines are calculated for image matching of the hole centers, so as to achieve matching of the hole centers in all acquired images; The matching results of the hole centers are merged with the matching results of the encoding points, and then a global pose optimization is performed to calculate the 3D coordinates of all hole centers.

5. The automatic detection method for sheet metal parts based on multi-view 3D scanning according to claim 1 is characterized in that: The step 8 is specifically as follows: Extract edge contours from images collected at different positions, discretize the contours at the pixel level, and obtain image discrete points of the contours; According to the hole center coordinates obtained in the hole center positioning algorithm module, the hole center coordinates in the part digital model are aligned to obtain the spatial pose of the digital model relative to all shooting poses; Extract all contours that need to be reconstructed in the digital model, and discretize them to obtain spatial discrete points of the contours; Back-projecting the spatial discrete points back to the corresponding image according to the spatial pose, finding the image contour discrete points closest to the back-projected discrete points, and encoding them; The actual discrete 3D coordinates of all contours are calculated based on the acquired pose parameters.

Images