Multi-vision automobile flange plate thread detection device and method based on mechanical arm

Abstract

The invention discloses a multi-view vision automobile flange plate thread detection device and method based on a mechanical arm. The detection device is composed of a calibration object, an illumination module, an imaging module and a control module. The calibration object is a checkerboard calibration plate and is used for solving a transformation matrix between the cameras and a transformation matrix between the cameras and the mechanical arm; the illumination module is composed of an LED infrared light source and a light source controller and is used for providing illumination during detection; the imaging module is composed of an industrial camera and a mechanical arm, and images of required flange plate threads are collected through the industrial camera; the control module is used for controlling the imaging module, detecting and positioning of the threads are achieved through the flange plate image collected at the initial position and image processing, the mechanical arm is driven through the positioning result to complete internal thread parameter measurement, and the detection result is transmitted. According to the invention, automatic detection of automobile flange plate threaded holes on an assembly line is realized, the device is simple, the leak detection and false detection rate is low, and the production quality and efficiency of flange plates are ensured.

Description

Technical Field

[0001] The invention relates to the technical field of automobile flange thread detection, in particular to a multi-eye vision automobile flange thread detection device and method based on a mechanical arm. Background Art

[0002] The automobile wheel hub flange is the core component of the engine, and its accuracy has a great impact on the final quality of the car. In actual production, in order to verify whether the threaded holes on the flange are within the reasonable accuracy requirements, most of them rely on manual detection methods. This method has low detection efficiency, and most of the current flange detection methods are contact measurement, which is measured by instruments such as crankshaft comprehensive testers, functional gauges, height gauges, and three-coordinate measuring machines. It is expensive, complicated to operate, and easy to scratch the workpiece, which is difficult to meet the requirements of fast and large-scale online detection in industrial sites. The existing method of detecting flanges can no longer keep up with the pace of modern industrial production, so a more effective and less likely to damage the flange detection method is needed.

[0003] In recent years, visual inspection technology has made great progress in many fields and has been applied to many tasks, and has achieved ideal results. The most widely used is industrial inspection. At present, many companies use robot hand-eye systems to replace manual inspection technology, which not only reduces labor costs, but also improves the accuracy and efficiency of inspection. Combining the robot's hand-eye system with image recognition and processing algorithms can meet the needs of real-time inspection in industrial production. Summary of the invention

[0004] The main purpose of the present invention is to provide a multi-vision automobile flange thread detection device and method based on a robotic arm, which drives an industrial camera to capture images of flange threads through the robotic arm, and completes internal thread defect detection and pitch measurement through image processing methods, thereby overcoming the defects of manual detection, improving detection accuracy and detection speed, and being suitable for modern assembly line production.

[0005] The technical solution to achieve the purpose of the present invention is as follows: On the one hand, the present invention provides a multi-eye vision automobile flange thread detection device based on a robotic arm, including a chessboard calibration plate, a lighting module, an imaging module and a control module;

[0006] The corners of the chessboard calibration plate are provided with asymmetric circular feature points for calibrating camera parameters and calculating the transformation matrix between the cameras;

[0007] The lighting module is used to provide lighting conditions for the detection device;

[0008] The imaging module is used to collect the image of the flange to be detected;

[0009] The control module provides an operable human-machine interaction interface for signal control and image processing to achieve flange internal thread defect detection and pitch measurement.

[0010] On the other hand, the present invention also provides a multi-eye vision automobile flange thread detection method based on a robotic arm, comprising the following steps:

[0011] (1) Calibration process: Extract corner points of the calibration plate images collected by the four cameras, take the corner with two circular feature points as the third corner in the clockwise direction, extract 8x8 feature points in a certain order, and solve the internal parameters of the four cameras; construct an 8x8 standard dot matrix as the standard reference space based on the calibration plate, and obtain the imaging perspective transformation matrix from the three cameras to the standard space; move the robotic arm so that the main camera can capture the entire calibration plate, record multiple sets of robotic arm postures and relative calibration plate images, and obtain the transformation matrix from the camera coordinate system to the robotic arm end coordinate system through Zhang Zhengyou calibration method;

[0012] (2) Flange thread recognition: The end of the robot arm is placed at the starting position, and the flange image is captured by the main camera. Based on the captured image, the threaded hole position is identified and located using the target detection algorithm.

[0013] (3) According to the position of the threaded hole obtained in (2), the camera coordinates of the threaded hole are converted into the basic coordinates of the robot arm, and the serial port signal is output to the robot arm;

[0014] (4) driving the robotic arm to move to the next target position in sequence;

[0015] (5) Arriving at the target position, the internal thread images taken by the three auxiliary cameras are transformed into perspective;

[0016] (6) inspecting the processed image obtained in (5) to determine whether the internal thread has defects and calculate the thread pitch;

[0017] (7) Repeat (4)-(6) until every threaded hole has been inspected.

[0018] Compared with the prior art, the present invention has the following significant advantages: the present invention uses visual processing analysis to complete the detection without touching the internal threads, effectively avoiding damage to the internal threads during the detection process. The use of multiple cameras for detection effectively curbs the negative impact of reflections on detection. The image processing algorithm has good robustness and meets the real-time requirements of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a diagram of a multi-eye vision automobile flange thread detection device based on a robotic arm.

[0020] Figure 2This is a schematic diagram of a camera and a camera stand.

[0021] Figure 3 This is the working flow chart of the multi-eye vision automobile flange thread detection device based on a robotic arm.

[0022] Among them, 1 is a computer, 2 is a six-axis robot arm, 3 is a camera and a camera bracket, 4 is a car flange, and 5 is an infrared LED light source. DETAILED DESCRIPTION

[0023] Combination Figure 1 A multi-eye vision automobile flange thread detection device based on a robotic arm is mainly composed of a chessboard calibration plate, a lighting module, an imaging module and a control module.

[0024] The calibration object is a checkerboard calibration plate, which is used to calibrate the internal and external parameters of the camera, calculate the transformation matrix between the cameras, and the transformation matrix between the main camera and the robotic arm;

[0025] The lighting module consists of an infrared LED light source and a light source controller, and is used for lighting during detection;

[0026] The imaging module consists of four industrial cameras and a robotic arm. The four cameras are fixed to the flange at the end of the robotic arm through a bracket. The four cameras collect images from different angles and transmit them to the control module for image processing.

[0027] The control module provides an operable human-computer interaction interface to control the operation of the entire detection device.

[0028] Furthermore, five asymmetric circular feature points are distributed on the four corners of the chessboard calibration plate, which are used to determine the direction during calibration to ensure that the feature points extracted at different angles can be arranged in the same order; each grid in the chessboard is a black and white square, each grid is 10mm, and the number of grid points is 8x8.

[0029] Furthermore, the infrared LED light source has a wavelength of 850nm. In view of the circular nature of the flange, it is composed of 16 small LED infrared light sources arranged in a circle, which can ensure the uniformity of light on the brake disc.

[0030] Furthermore, the imaging module consists of 4 black-and-white industrial cameras, a camera bracket and a robotic arm. The camera is a black-and-white camera with a resolution of 1292x964. The main camera is equipped with a lens with a focal length of 8mm, and the remaining three cameras are equipped with lenses with a focal length of 25mm. The camera bracket is used to connect the robotic arm and the 4 industrial cameras. The main camera is in the middle and is used to capture the image of the flange body. The auxiliary cameras are distributed around and are used to capture the image of the internal threads of the flange.

[0031] The camera is fixed to the flange at the end of the robotic arm through a camera bracket. The relative position between the cameras is fixed. The main camera is centered and an 8mm focal length lens is used to take photos of the entire flange to complete the positioning of the threaded holes. The three auxiliary cameras are distributed around at an interval of 120 degrees and an angle of 30 degrees with the vertical direction. A 25mm focal length lens is used to take pictures of the internal threads of the threaded holes and transmit them to the computer for analysis and calculation.

[0032] The robotic arm is RobotAnno's SJ602-A robotic arm. The end flange has a variety of threaded interfaces, can be connected to various camera brackets, can freely set the initial position, the repeat positioning accuracy can reach 0.1mm, the positioning accuracy is high, supports C++ secondary development, and the control of the robotic arm can be completed by connecting to the computer via USB.

[0033] like Figure 1 As shown, the control module is integrated in the computer 1, 2 is a six-axis robot arm, 3 is a camera and a camera bracket, 4 is a car flange to be tested, and 5 is an infrared LED light source. Figure 2 This is a schematic diagram of a camera and a camera stand.

[0034] Furthermore, the control module is used to complete signal control and image processing, and the interactive interface can realize the access of detection results, signal control and modification of image processing algorithm parameters.

[0035] Signal control includes camera signal control, light source signal control and serial communication via USB to control the movement of the robotic arm.

[0036] Image processing mainly includes two parts: threaded hole recognition and positioning, and internal thread detection.

[0037] The process of identifying and locating the threaded hole is to extract the edge of the flange plate image taken by the main camera within the set ROI range through canny edge detection, and then use contour screening and positioning to complete the identification and positioning of the threaded hole.

[0038] The internal thread detection process is to use the internal thread images taken by three auxiliary cameras, use the imaging perspective transformation matrix H obtained by previous calibration to map the images to the standard space to correct the distortion, and complete the defect detection and pitch measurement through image processing.

[0039] Based on the above detection device, the present invention also provides a multi-eye vision automobile flange thread detection method based on a robotic arm, such as Figure 3 As shown, the following steps are included:

[0040] (1) Camera calibration process: The present invention first needs to calibrate each camera separately. The calibration accuracy greatly affects the accuracy of subsequent measurements.

[0041] Four cameras are used to collect images of the calibration plate for corner extraction. The corner with two circular feature points is taken as the third corner clockwise. 8x8 feature points are extracted in a fixed order. The internal parameters and distortion coefficients of the camera model are calculated using Zhang Zhengyou calibration method, thereby obtaining the transformation relationship between the pixel coordinate system and the camera coordinate system.

[0042] In order to correct the distortion caused by the non-parallelism between the camera imaging plane and the internal thread plane, the three auxiliary cameras extract corner points of the calibration plate image respectively, extract 8x8 feature points in a certain order, and construct an 8x8 standard dot matrix as the standard reference space according to the calibration plate. The imaging perspective transformation matrix H from the three cameras to the standard space is obtained.

[0043] The robot arm is used to drive the main camera to collect images of the calibration plate, the coordinates of the end of the robot arm are recorded, and 8x8 feature points are extracted in a fixed order. The hand-eye calibration of the robot arm and the camera is completed through the Zhang Zhengyou calibration method, and the coordinate transformation relationship between the camera pixel coordinate system and the spatial robot arm coordinate system is obtained.

[0044] (2) The robotic arm is placed in the initial position, and the flange image is captured by the main camera and transmitted to the computer, and the position of the threaded hole is identified and located by the target recognition algorithm;

[0045] (3) The obtained threaded hole position is converted into the position in the robot coordinate system by using the result of hand-eye calibration. The position information of the six axes of the robot is solved by inverse kinematics analysis of the robot and transmitted back to the robot, so that the robot obtains the position information;

[0046] (4) Move to the next threaded hole target position;

[0047] (5) The internal thread images of the threaded hole are captured by three auxiliary cameras, and the obtained images are corrected by the perspective transformation matrix H to obtain the final internal thread images without distortion.

[0048] (6) Perform internal thread defect detection and parameter detection on the internal thread image obtained in step (5).

[0049] (7) Repeat steps (4) to (6) to complete the thread inspection of the flange.

[0050] The detailed steps of step (2) are as follows:

[0051] (21) Set the center coordinates and radius of the ROI area.

[0052] (22) Use Gaussian filter T to filter the pixels of the original image, where

[0053]

[0054] The filter calculates the weighted average of the pixels around the pixel to obtain the final filtering result.

[0055] (23) Use the Sobel operator to calculate the horizontal gradient G of the image x and the vertical gradient G y , and calculate the magnitude G and direction θ of the gradient, where

[0056]

[0057] (24) Perform non-maximum suppression on the image and iterate through the pixels one by one. If the point is a local maximum in the positive or negative gradient direction, then keep the point; otherwise, set the point to 0.

[0058] (25) Perform morphological processing on the image to fill the gaps within the contours and connect adjacent contours;

[0059] (26) Contour screening and positioning are performed. In the set ROI area, the boundary points of the same connected domain are placed in the same set in the form of pixel coordinates. The minimum horizontal pixel difference min(x), the minimum vertical pixel difference min(y), the maximum horizontal pixel difference max(x), and the maximum vertical pixel difference max(y) in the set are selected to construct the circumscribed rectangle of the connected domain, and the length, width, and center coordinates of the circumscribed rectangle are obtained. The threaded holes are screened out according to the difference between the perimeter of the contour and the length and width of the circumscribed rectangle.

[0060] The detailed steps of step (6) are as follows:

[0061] (61) Bilateral filtering is performed on the thread image to retain the image contour and edge feature information.

[0062] (62) Binarize the filtered image, find the contour, and determine whether there are defects.

[0063] (63) Detect the coordinates of the intersection of the thread flank and the mid-diameter line in the thread edge image and calculate the thread pitch P

[0064]

[0065] Where Q(i) is the coordinate of the i-th intersection point of the median diameter line and the left tooth inclined edge, Q(i+1) is the coordinate of the i+1-th intersection point of the median diameter line and the left tooth inclined edge, Q ′ (i) is the coordinate of the i-th intersection of the median diameter line and the right tooth inclined edge, Q ′ (i+1) is the coordinate of the i+1th intersection point between the median diameter line and the inclined edge of the right tooth, k is the distance ratio between the pixel coordinate system and the world coordinate system, and n is the number of intersection points between the median diameter line and the inclined edge of the left tooth.

Claims

1. A multi-eye vision automobile flange thread detection device based on a robotic arm, characterized in that: It includes a checkerboard calibration plate, an illumination module, an imaging module and a control module; The corners of the chessboard calibration plate are provided with asymmetric circular feature points for calibrating camera parameters and calculating the transformation matrix between the cameras; The lighting module is used to provide lighting conditions for the detection device; The imaging module is used to collect the image of the flange to be detected; The control module provides an operable human-machine interaction interface for signal control and image processing to achieve flange internal thread defect detection and pitch measurement.

2. The multi-eye vision automobile flange thread detection device based on a robotic arm according to claim 1 is characterized in that: The chessboard calibration plate has five asymmetric circular feature points distributed on the four corners, which are used to determine the direction during calibration to ensure that the feature points extracted at different angles can be arranged in the same order; each grid in the chessboard is a square black and white chessboard with 8x8 grid points.

3. The multi-eye vision automobile flange thread detection device based on a robotic arm according to claim 2 is characterized in that: The lighting module is composed of an infrared LED light source and a light source controller, and is used for lighting during detection; the infrared LED light source has a wavelength of 850nm and is composed of 16 small LED infrared light sources arranged in a circle.

4. The multi-eye vision automobile flange thread detection device based on a robotic arm according to claim 2 is characterized in that: The imaging module consists of four black-and-white industrial cameras, a camera bracket and a robotic arm. The main camera uses an 8mm focal length lens, and the remaining three cameras use 25mm focal length lenses. The camera bracket is used to connect the robotic arm and the four industrial cameras. The main camera is in the middle and the auxiliary cameras are distributed around.

5. The multi-eye vision automobile flange thread detection device based on a robotic arm according to claim 4 is characterized in that: We chose RobotAnno's SJ602-A robotic arm, which has six-axis freedom, a repeatability of 0.1mm, and supports C++ secondary development.

6. The multi-eye vision automobile flange thread detection device based on a robotic arm according to claim 4 is characterized in that: The control module is used to complete signal control and image processing. Signal control includes control of camera signals, control of light source signals and control of robot arm movement. The UI interface is used for accessing detection results, signal control and modification of image processing algorithm parameters.

7. The multi-eye vision automobile flange thread detection device based on a robotic arm according to claim 6 is characterized in that: The control module performs flange thread detection based on the collected calibration plate image and flange image, including: Identification and positioning of threaded holes: The flange image taken by the main camera is used to extract the edge within the set ROI range through canny edge detection, and then the threaded holes are identified and positioned using contour screening and positioning; Internal thread detection: The internal thread images taken by three auxiliary cameras are used to obtain the imaging perspective transformation matrix through previous calibration, and the images are mapped to the standard space to correct the distortion. The defect detection and pitch measurement are completed through image processing.

8. A multi-eye vision automobile flange thread detection method based on a robotic arm, characterized in that: The method is implemented based on the multi-eye vision automobile flange thread detection device based on a robotic arm according to any one of claims 1 to 7, and comprises the following steps: (1) Calibration process: Extract corner points of the calibration plate images collected by the four cameras, take the corner with two circular feature points as the third corner in the clockwise direction, extract 8x8 feature points in a certain order, and solve the internal parameters of the four cameras; construct an 8x8 standard dot matrix as the standard reference space based on the calibration plate, and obtain the imaging perspective transformation matrix from the three cameras to the standard space; move the robotic arm so that the main camera can capture the entire calibration plate, record multiple sets of robotic arm postures and relative calibration plate images, and obtain the transformation matrix from the camera coordinate system to the robotic arm end coordinate system through Zhang Zhengyou calibration method; (2) Flange thread recognition: The end of the robot arm is placed at the starting position, and the flange image is captured by the main camera. Based on the captured image, the threaded hole position is identified and located using the target detection algorithm. (3) According to the position of the threaded hole obtained in (2), the camera coordinates of the threaded hole are converted into the basic coordinates of the robot arm, and the serial port signal is output to the robot arm; (4) driving the robotic arm to move to the next target position in sequence; (5) Arriving at the target position, the internal thread images taken by the three auxiliary cameras are transformed into perspective; (6) inspecting the processed image obtained in (5) to determine whether the internal thread has defects and calculate the thread pitch; (7) Repeat (4)-(6) until every threaded hole has been inspected.

9. The method according to claim 8, characterized in that Step (2) specifically includes: (21) Set the center coordinates and radius of the ROI area; (22) Use Gaussian filter T to filter the pixels of the original image, where The weighted average value of the pixels around the pixel point is calculated through the filter to obtain the final filtering result; (23) Use the Sobel operator to calculate the horizontal gradient G of the image x and the vertical gradient G y , and calculate the magnitude G and direction θ of the gradient, where (24) Perform non-maximum suppression on the image, traverse the pixel points one by one, and if the point is the local maximum in the positive or negative gradient direction, retain the point, otherwise set the point to 0; (25) Perform morphological processing on the image to fill the gaps within the contours and connect adjacent contours; (26) Contour screening and positioning are performed. In the set ROI area, the boundary points of the same connected domain are placed in the same set in the form of pixel coordinates. The minimum horizontal pixel difference, the minimum vertical pixel difference, the maximum horizontal pixel difference, and the maximum vertical pixel difference in the set are selected to construct the circumscribed rectangle of the connected domain, and the length, width, and center coordinates of the circumscribed rectangle are obtained. The threaded holes are screened out according to the difference between the perimeter of the contour and the length and width of the circumscribed rectangle.

10. The method according to claim 8, characterized in that Step (6) specifically includes: (61) Perform bilateral filtering on the thread image to retain the image contour and edge feature information; (62) Binarize the filtered image, find the contour, and determine whether there are defects; (63) Detect the coordinates of the intersection of the thread flank and the mid-diameter line in the thread edge image and calculate the thread pitch P: Where Q(i) is the coordinate of the i-th intersection point of the median diameter line and the left tooth inclined edge, Q(i+1) is the coordinate of the i+1-th intersection point of the median diameter line and the left tooth inclined edge, Q ′ (i) is the coordinate of the i-th intersection of the median diameter line and the right tooth inclined edge, Q ′ (i+1) is the coordinate of the i+1th intersection point between the median diameter line and the inclined edge of the right tooth, k is the distance ratio between the pixel coordinate system and the world coordinate system, and n is the number of intersection points between the median diameter line and the inclined edge of the left tooth.

Images