Multi-view vision automobile flange thread detection device and method based on mechanical arm

Abstract

The application discloses a kind of multi-vision automobile flange plate thread detection device and method based on mechanical arm, and detection device is by calibration object, illumination module, imaging module and control module composition;The calibration object is checkerboard calibration board, for solving the transformation matrix between camera and the transformation matrix between camera and mechanical arm;Illumination module is LED infrared light source and light source controller composition, for providing illumination when detecting;Imaging module is by industrial camera and mechanical arm, and the image of required flange plate thread is collected by industrial camera;Control module is used to control imaging module, and the flange plate image collected by starting position is detected and positioned by image processing to realize thread, and the result of positioning drives mechanical arm to complete internal thread parameter measurement, and transmission detection result.The application realizes the automation detection of automobile flange plate thread hole on assembly line, and the device is simple, and the miss detection and misjudgment rate are low, guarantee the production quality and efficiency of flange plate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automobile flange thread detection, and in particular to a multi-view vision automobile flange thread detection device and method based on a mechanical arm. BACKGROUND

[0002] The automobile hub flange is a core component of the engine, and its precision has a great influence on the final automobile quality. In actual production, in order to verify whether the thread holes on the flange are within a reasonable precision requirement range, a manual detection method is mostly relied on. This method has low detection efficiency, and most current flange detection methods are contact measurement, which is measured by a crankshaft comprehensive detector, a functional gauge, a height gauge, a three-coordinate measuring machine and other instruments. The method is expensive, complex to operate, easy to scratch the workpiece, and difficult to meet the requirements of online detection, rapidness and large batch in industrial field. The existing flange detection method cannot keep up with the pace of modern industrial production, and therefore a more effective detection method which is not easy to damage the flange is needed.

[0003] In recent years, visual detection technology has developed rapidly in many fields and has been applied to many tasks, and ideal results have been achieved. The most widely used is the industrial detection direction. At present, many enterprises use the hand-eye system of the robot to replace the manual detection technology, which not only reduces the labor cost, but also improves the detection accuracy and efficiency. The combination of the hand-eye system of the robot and the image recognition and processing algorithm can meet the real-time detection requirements in industrial production. SUMMARY

[0004] The main purpose of the present application is to provide a multi-view vision automobile flange thread detection device and method based on a mechanical arm. The image of the flange thread is photographed by the mechanical arm driving the industrial camera, the internal thread defect detection and the pitch measurement are completed by the image processing method, the defects of manual detection are overcome, the detection precision and speed are improved, and the device is suitable for modern assembly line production.

[0005] The technical solution for achieving the purpose of the present application is as follows: on the one hand, the present application provides a multi-view vision automobile flange thread detection device based on a mechanical arm, which comprises a checkerboard calibration plate, an illumination module, an imaging module and a control module.

[0006] The checkerboard calibration plate is provided with asymmetric circular ring feature points at the corners, which are used for calibrating the camera parameters and calculating the transformation matrix between the cameras.

[0007] The illumination module is used to provide illumination conditions for the detection device.

[0008] The imaging module is used to collect the images of the flanges to be detected.

[0009] The control module provides an operable human-machine interface for signal control and image processing, and realizes flange inner thread defect detection and pitch measurement.

[0010] In another aspect, the application also provides a multi-view vision automobile flange thread detection method based on a mechanical arm, comprising the following steps:

[0011] (1) Calibration process: corner points of images of a calibration board collected by four cameras are extracted, an edge corner with two circular ring features is taken as a third edge corner in a clockwise direction, 8x8 feature points are extracted in a certain order, and internal parameters of the four cameras are solved; an 8x8 standard point array is constructed as a standard reference space according to the calibration board, and an imaging perspective transformation matrix of three camera auxiliary cameras to the standard space is solved; the main camera is moved to shoot the entire calibration board, and multiple sets of mechanical arm poses and the corresponding calibration board pictures are recorded, and a transformation matrix of the camera coordinate system to the mechanical arm end coordinate system is solved by Zhang Zhengyou calibration method;

[0012] (2) Flange thread identification: the end of the mechanical arm is placed at the starting position, the flange picture is shot by the main camera, the thread hole position is identified and located by a target detection algorithm according to the shot picture;

[0013] (3) According to the position of the thread hole obtained in (2), the camera coordinates of the obtained thread hole are converted into mechanical arm base coordinates, and a serial port signal is output to the mechanical arm;

[0014] (4) sequentially driving the mechanical arm to move to the next target position;

[0015] (5) reaching the target position, performing perspective transformation on the inner thread pictures shot by the three auxiliary cameras;

[0016] (6) detecting the processed images obtained in (5) to determine whether the inner thread has defects and calculate the pitch;

[0017] (7) repeating (4)-(6) until each thread hole is detected.

[0018] Compared with the prior art, the application has the following advantages: the application uses visual processing analysis to complete detection without contacting the inner thread, effectively avoiding damage to the inner thread during detection. Multi-camera detection effectively suppresses the negative impact of reflection on detection. The image processing algorithm has good robustness and meets the real-time requirements of the system. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a multi-view vision automobile flange thread detection device based on a mechanical arm.

[0020] Figure 2is a schematic diagram of a camera and a camera holder.

[0021] Figure 3 is a mechanical arm-based multi-view vision automobile flange thread detection device workflow diagram.

[0022] wherein, 1 is a computer, 2 is a six-axis mechanical arm, 3 is a camera and a camera holder, 4 is an automobile flange, 5 is an infrared LED light source. DETAILED DESCRIPTION

[0023] In combination Figure 1 A mechanical arm-based multi-view vision automobile flange thread detection device mainly consists of a checkerboard calibration board, an illumination module, an imaging module, and a control module.

[0024] The calibration object is a checkerboard calibration board, which is used to calibrate the internal and external parameters of the camera, calculate the transformation matrix between the cameras, and calculate the transformation matrix between the main camera and the mechanical arm.

[0025] The illumination module consists of an infrared LED light source and a light source controller, which is used for illumination during detection.

[0026] The imaging module consists of four industrial cameras and a mechanical arm. The four cameras are fixed on the flange at the end of the mechanical arm through a holder. The four cameras collect images from different angles and transmit them to the control module for image processing.

[0027] The control module provides an operable human-computer interaction interface to control the operation of the entire detection device.

[0028] Further, the checkerboard calibration board has five asymmetric circular feature points distributed at its four corners, which are used to determine the direction during calibration to ensure that the feature points extracted at different angles can be arranged in the same order. Each cell in the checkerboard is a black and white square, and the size of each cell is 10 mm, with 8x8 grid points.

[0029] Further, the infrared LED light source has a wavelength of 850 nm and is composed of 16 small LED infrared light sources arranged in a circle to ensure the uniformity of the light on the brake disc.

[0030] Further, the imaging module consists of four black and white industrial cameras, a camera holder, and a mechanical arm. The camera is a black and white camera with a resolution of 1292x964. The main camera is equipped with an 8mm focal length lens, and the remaining three cameras are equipped with a 25mm focal length lens. The camera holder is used to connect the mechanical arm and the four industrial cameras. The main camera is in the center to capture the main image of the flange, and the auxiliary cameras are distributed around to capture the internal thread image of the flange.

[0031] The camera is fixed on the flange at the end of the mechanical arm through the camera support, the relative pose between the camera and the camera is fixed, the main camera is in the center, the 8mm focal length lens is selected to shoot the whole flange plate to complete the positioning of the threaded hole, three auxiliary cameras are distributed at an interval of 120 degrees around the flange plate, and the included angle with the vertical direction is 30 degrees, and the 25mm focal length lens is selected to shoot the thread picture in the threaded hole, and the thread picture is transmitted to the computer for analysis and calculation.

[0032] The mechanical arm is an SJ602-A mechanical arm of RobotAnno, the end flange has rich threaded interfaces, can connect various camera supports, can freely set the initial position, the repeated positioning accuracy can reach 0.1mm, the positioning accuracy is high, supports c++ secondary development, and the control of the mechanical arm can be completed through the connection of the computer through the USB.

[0033] As shown in Figure 1 The control module is integrated in the computer 1, 2 is a six-axis mechanical arm, 3 is a camera and a camera support, 4 is a flange plate to be measured, and 5 is an infrared LED light source. Figure 2 It is a schematic view of the camera and the camera support.

[0034] Further, the control module is used for completing signal control and image processing, and the interactive interface can realize the access of detection results, the control of signals and the modification of image processing algorithm parameters.

[0035] The signal control includes the control of the camera signal, the control of the light source signal and the control of the movement of the mechanical arm through the serial communication of the USB.

[0036] The image processing mainly includes thread hole recognition and positioning, and internal thread detection.

[0037] The thread hole recognition and positioning are as follows: through the flange plate picture shot by the main camera, in the set ROI range, the edge is extracted through canny edge detection, and then the thread hole recognition and positioning are completed through contour screening and positioning.

[0038] The internal thread detection is as follows: through the internal thread pictures shot by the three auxiliary cameras, the imaging perspective transformation matrix H obtained through previous calibration is used to map the pictures to the standard space to correct the distortion, and the defect detection and the pitch measurement are completed through image processing.

[0039] Based on the above detection device, the application also provides a multi-view vision automobile flange thread detection method based on a mechanical arm, as shown in Figure 3 The method comprises the following steps:

[0040] (1) Camera calibration process: the application first needs to calibrate each camera respectively, and the calibration accuracy greatly affects the accuracy of subsequent measurement.

[0041] Four cameras were used to acquire images of the calibration board for corner point extraction. The corner with two circular feature points was taken as the third corner clockwise. 8x8 feature points were extracted in a fixed order. The internal parameters and distortion coefficients of the camera model were calculated by Zhang Zhengyou calibration method, thereby obtaining the transformation relationship between the pixel coordinate system and the camera coordinate system.

[0042] To correct the distortion caused by the non-parallelism between the camera imaging plane and the internal thread plane, corner points were extracted from the calibration board image by the three auxiliary cameras respectively. 8x8 feature points were extracted in a certain order. An 8x8 standard point matrix was constructed based on the calibration board as a standard reference space, and the imaging perspective transformation matrix H from the three cameras to the standard space was obtained.

[0043] The robotic arm drives the main camera to acquire images of the calibration plate, records the coordinates of the robotic arm's end effector, and extracts 8x8 feature points in a fixed order. The hand-eye calibration of the robotic arm and the camera is completed using the Zhang Zhengyou calibration method, and the coordinate transformation relationship between the camera pixel coordinate system and the spatial robotic arm coordinate system is obtained.

[0044] (2) Place the robotic arm in the initial position, capture an image of the flange with the main camera and transmit it to the computer, and use the target recognition algorithm to identify and locate the position of the threaded hole;

[0045] (3) The position of the threaded hole obtained by hand-eye calibration is converted into the position in the coordinate system of the robotic arm. The position information of the six axes of the robotic arm is solved by the inverse kinematics analysis of the robotic arm and transmitted back to the robotic arm. The robotic arm obtains the position information.

[0046] (4) Move to the next target position of the threaded hole;

[0047] (5) Take pictures of the internal thread of the threaded hole with three auxiliary cameras, and perform image correction on the obtained images through perspective transformation matrix H to obtain the final internal thread image without distortion.

[0048] (6) Perform internal thread defect detection and parameter detection on the internal thread image obtained in step (5).

[0049] (7) Repeat steps (4)-(6) to complete the thread inspection of the flange.

[0050] The detailed steps of step (2) are as follows:

[0051] (21) Set the center coordinates and radius of the ROI region.

[0052] (22) Use a Gaussian filter T to filter the pixels of the original image, where

[0053]

[0054] The final filtering result is obtained by calculating a weighted average of the pixels surrounding a given pixel using a filter.

[0055] (23) Calculate the horizontal gradient G of the image using the Sobel operator. x and vertical gradient G y And calculate the magnitude G and direction θ of the gradient, where

[0056]

[0057] (24) Perform non-maximum suppression on the image by going through each pixel one by one. If the pixel is a local maximum in the positive or negative gradient direction, keep the pixel; otherwise, set the pixel to 0.

[0058] (25) Perform morphological processing on the image to fill the gaps in the contours and connect adjacent contours.

[0059] (26) Perform contour filtering and positioning. In the set ROI area, the boundary points of the same connected region will be placed in the same set in the form of pixel coordinates. Select the minimum horizontal pixel difference min(x), the minimum vertical pixel difference min(y), the maximum horizontal pixel difference max(x), and the maximum vertical pixel difference max(y) in the set. Construct the bounding rectangle of the connected region and obtain the length, width and center coordinates of the bounding rectangle. Filter out the threaded holes according to the perimeter of the contour and the difference between the length and width of the bounding rectangle.

[0060] The detailed steps of step (6) are as follows:

[0061] (61) Perform bilateral filtering on the thread image to preserve the image contour and edge feature information.

[0062] (62) Binarize the filtered image, find the contour, and determine if there are any defects.

[0063] (63) Detect the coordinates of the intersection point between the thread flank and the pitch diameter line in the thread edge image, and calculate the pitch P.

[0064]

[0065] Where Q(i) is the coordinate of the i-th intersection point of the median diameter and the left tooth inclined side, and Q(i+1) is the coordinate of the (i+1)-th intersection point of the median diameter and the left tooth inclined side. ′ (i) represents the coordinates of the i-th intersection point between the median diameter and the right tooth inclined edge, Q ′ (i+1) is the coordinate of the (i+1)th intersection point of the median diameter line and the right tooth inclined edge, k is the distance ratio between the pixel coordinate system and the world coordinate system, and n is the number of intersection points of the median diameter line and the left tooth inclined edge.

Claims

1. A multi-view vision-based method for inspecting automotive flange threads using a robotic arm, characterized in that, The method includes the following steps: (1) Calibration process: Corner points are extracted from the calibration board images acquired by the four cameras. The corner with two circular feature points is taken as the third corner in the clockwise direction. 8x8 feature points are extracted in a certain order, and the intrinsic parameters of the four cameras are solved. An 8x8 standard point matrix is ​​constructed based on the calibration board as a standard reference space. The imaging perspective transformation matrix from the three auxiliary cameras to the standard space is obtained. The robotic arm is moved so that the main camera can capture the entire calibration board. Multiple sets of robotic arm poses and images relative to the calibration board are recorded. The transformation matrix from the camera coordinate system to the robotic arm end coordinate system is obtained by Zhang Zhengyou calibration method. (2) Flange thread recognition: The end effector of the robotic arm is positioned at the starting position, and an image of the flange is captured by the main camera. Based on the captured image, the location of the threaded hole is identified and located using a target detection algorithm; specifically including: (21) Set the center coordinates and radius of the ROI region; (22) Use a Gaussian filter T to filter the pixels of the original image, where The final filtering result is obtained by calculating a weighted average of the pixels surrounding a given pixel using a filter. (23) Calculate the horizontal gradient G of the image using the Sobel operator. x and vertical gradient G y And calculate the magnitude G and direction θ of the gradient, where (24) Perform non-maximum suppression on the image by going through each pixel one by one. If the pixel is a local maximum in the positive or negative gradient direction, keep the pixel; otherwise, set the pixel to 0. (25) Perform morphological processing on the image to fill the gaps in the contours and connect adjacent contours. (26) Perform contour filtering and positioning. In the set ROI area, the boundary points of the same connected region will be placed in the same set in the form of pixel coordinates. Select the minimum horizontal pixel difference, the minimum vertical pixel difference, the maximum horizontal pixel difference, and the maximum vertical pixel difference within the set to construct the bounding rectangle of the connected region. Obtain the length, width, and center coordinates of the bounding rectangle. Filter out threaded holes according to the perimeter of the contour and the difference between the length and width of the bounding rectangle. (3) Based on the position of the threaded hole obtained in (2), convert the camera coordinates of the threaded hole into the basic coordinates of the robotic arm and output a serial port signal to the robotic arm. (4) Move the robotic arm to the next target position in sequence; (5) Upon reaching the target location, perform perspective transformation on the internal thread images captured by the three auxiliary cameras; (6) Detect the processed image obtained in (5) to determine if there are defects in the internal thread and calculate the thread pitch; specifically including: (61) Perform bilateral filtering on the thread image to preserve the image contour and edge feature information; (62) Binarize the filtered image, find the contour, and determine if there are any defects; (63) Detect the coordinates of the intersection point between the thread flank and the pitch diameter line in the thread edge image, and calculate the pitch P: Where Q(i) is the coordinate of the i-th intersection point of the median diameter and the left tooth inclined side, and Q(i+1) is the coordinate of the (i+1)-th intersection point of the median diameter and the left tooth inclined side. ' (i) represents the coordinates of the i-th intersection point between the median diameter and the right tooth inclined edge, Q ' (i+1) is the coordinate of the (i+1)th intersection point of the median diameter line and the right tooth inclined edge, k is the distance ratio between the pixel coordinate system and the world coordinate system, and n is the number of intersection points of the median diameter line and the left tooth inclined edge. (7) Repeat (4)-(6) until every threaded hole has been inspected.

2. A multi-view vision-based automotive flange thread inspection device based on a robotic arm, characterized in that, Based on the method described in claim 1, the device includes a checkerboard calibration plate, an illumination module, an imaging module, and a control module; The checkerboard calibration board has asymmetrical circular feature points at its corners, which are used to calibrate camera parameters and calculate the transformation matrix between each camera. The lighting module is used to provide illumination for the detection device; The imaging module is used to acquire images of the flange to be inspected; The control module provides an operable human-machine interface for signal control and image processing, enabling the detection of defects in the internal threads of the flange and the measurement of the thread pitch.

3. The multi-view vision automotive flange thread inspection device based on a robotic arm according to claim 2, characterized in that, The checkerboard calibration board has five asymmetrical circular feature points distributed at its four corners, which are used to determine the direction during calibration and ensure that the feature points extracted at different angles can be arranged in the same order; each grid in the checkerboard is a square black and white checkerboard with 8x8 grid points.

4. The multi-view vision automotive flange thread inspection device based on a robotic arm according to claim 3, characterized in that, The lighting module consists of an infrared LED light source and a light source controller, and is used for illumination during detection. The infrared LED light source has a wavelength of 850nm and is composed of 16 small infrared LEDs arranged in a circle.

5. The multi-view vision automotive flange thread inspection device based on a robotic arm according to claim 3, characterized in that, The imaging module consists of four monochrome industrial cameras, a camera bracket, and a robotic arm. The main camera uses an 8mm focal length lens, and the remaining three cameras use 25mm focal length lenses. The camera bracket is used to connect the robotic arm to the four industrial cameras, with the main camera in the center and the auxiliary cameras distributed around it.

6. The multi-view vision automotive flange thread inspection device based on a robotic arm according to claim 5, characterized in that, Choose RobotAnno's SJ602-A robotic arm, which has six degrees of freedom, a repeatability of 0.1mm, and supports C++ secondary development.

7. The multi-view vision automotive flange thread inspection device based on a robotic arm according to claim 5, characterized in that, The control module is used to perform signal control and image processing. Signal control includes the control of camera signals, the control of light source signals, and the control of robotic arm movement. The UI interface is used to store and retrieve detection results, control signals, and modify image processing algorithm parameters.

8. The multi-view vision automotive flange thread inspection device based on a robotic arm according to claim 7, characterized in that, The control module performs flange thread detection based on the acquired calibration plate image and flange image, including: Threaded hole identification and positioning: Using the flange image captured by the main camera, within the set ROI range, edges are extracted using Canny edge detection, and then contour filtering and positioning are used to complete the identification and positioning of threaded holes. Internal thread inspection: Images of the internal thread are captured by three auxiliary cameras. The imaging perspective transformation matrix is ​​obtained through prior calibration. The images are mapped to a standard space to correct distortion. Defect detection and pitch measurement are completed through image processing.

Images