Body-in-white autonomous welding method based on machine vision

Abstract

The invention discloses a body-in-white autonomous welding method based on machine vision, relates to the field of automobile manufacturing and body-in-white welding, and solves the problems that the welding quality is influenced due to the fact that an existing welding robot depends on welding conditions, the flexibility is poor, the working efficiency of welding is reduced, the production cycle of a product is prolonged, and finally the production cost is increased. According to the method, the camera is calibrated, and the internal reference and the distortion coefficient of the camera are obtained; after obtaining a vehicle body image, extracting two-dimensional coordinates of centroids of feature points; according to the mass center two-dimensional coordinates and the feature point mass center three-dimensional coordinates of the vehicle body model, the coordinate conversion relation from a vehicle body coordinate system to a camera coordinate system is solved; hand-eye calibration is conducted, and the coordinate conversion relation from a camera coordinate system to a mechanical arm coordinate system is calibrated; the position coordinates of the welding spots are converted into a mechanical arm coordinate system from a vehicle body coordinate system, and a mechanical arm conducts welding. According to the method, the positions of all the welding spots do not need to be recognized, the calculated amount is reduced, and the requirement for hardware is lowered accordingly.

Description

Technical Field

[0001] The invention relates to the fields of automobile manufacturing and body-in-white welding, and in particular to an autonomous body-in-white welding method, which is applicable to an autonomous body-in-white welding method based on machine vision on an automobile production line. Background Art

[0002] The automobile manufacturing industry is one of the indispensable pillar industries in the modern economy. It not only promotes technological innovation and industrial development, but also provides employment for millions of people around the world. With the increasing demand for transportation in society, the automobile manufacturing industry plays an important role in promoting economic growth and improving people's quality of life. As an important part of the car, the body in white is connected by resistance spot welding of hundreds of thin sheet stamping parts. Each body in white requires about 3,000 to 6,000 resistance welding points. Therefore, the body in white usually requires a large number of welding operations to ensure the connection strength and stability between the various components. In the traditional production mode, the welding method is manual welding, but the welding site environment is harsh and the work intensity is relatively high. In order to improve the welding quality, welding robots have emerged and are widely used in various welding fields, making important contributions to the realization of welding mechanization and automation.

[0003] However, the usage of welding robots is extremely simple, and they mainly rely on the teaching and reproducing function. That is, workers need to repeatedly perform teaching operations for different workpieces, and then the robot will repeat the welding process. At the same time, the mechanical positioning method used for positioning weldments is expensive and has poor flexibility. On the one hand, this working mode reduces the efficiency of welding and prolongs the production cycle of products, thus increasing the production cost of the factory and is not conducive to improving the competitiveness of enterprises; on the other hand, this method is more dependent on welding conditions, and assembly errors and thermal deformations of the same batch will affect the welding quality. The problem is that the flexibility is poor and it cannot respond to external changes accordingly. Summary of the invention

[0004] In order to solve the problems that the existing welding robots rely on welding conditions, which affects the welding quality, have poor flexibility, reduce welding efficiency, extend the product production cycle and ultimately increase production costs, the present invention provides a white body autonomous welding method based on machine vision to address the problem of poor flexibility in autonomous welding using mechanical positioning.

[0005] A white body autonomous welding method based on machine vision is implemented by the following steps:

[0006] Step 1: Calibrate the camera to obtain the camera's intrinsic parameters and distortion coefficients;

[0007] Step 2: Use the camera calibrated in step 1 to take a picture of the vehicle body, process the acquired vehicle body image, and extract the two-dimensional coordinates of the centroid of the feature points;

[0008] Step 3: Calculate the coordinate transformation relationship from the vehicle body coordinate system to the camera coordinate system based on the two-dimensional coordinates of the centroid of the feature point obtained in step 2 and the three-dimensional coordinates of the centroid of the feature point obtained based on the vehicle body model;

[0009] Step 4: Perform hand-eye calibration to calibrate the coordinate transformation relationship from the camera coordinate system to the robotic arm coordinate system;

[0010] Step 5: Convert the position coordinates of the welding point from the vehicle body coordinate system to the robot arm coordinate system, and then use the robot arm to perform welding.

[0011] Beneficial effects of the present invention:

[0012] The autonomous body-in-white welding method described in the present invention solves the coordinate system conversion problem, that is, through the coordinate conversion between the body coordinate system, the camera coordinate system and the robotic arm coordinate system, so that the weld point position coordinates in the camera coordinate system can be converted to the robotic arm coordinate system, and finally robotic arm welding is realized.

[0013] The white-body autonomous welding method of the present invention uses machine vision in the white-body autonomous welding, and no longer needs to use a precision positioning fixture, which reduces production costs. At the same time, due to the addition of vision, the method of the present invention can respond to external changes accordingly, improving flexibility. At the same time, this method does not need to identify the positions of all welding points, but only needs to select a few feature points for identification, and then combined with the body model, the conversion relationship from the body coordinate system to the camera coordinate system can be solved, which reduces the amount of calculation and is more convenient and quick. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A diagram showing the conversion relationship of various coordinate systems in a body-in-white autonomous welding method based on machine vision according to the present invention;

[0015] Figure 2 This is a flow chart of a body-in-white autonomous welding method based on machine vision according to the present invention;

[0016] Figure 3 This is a schematic diagram of the eye outside the hand. DETAILED DESCRIPTION

[0017] Combination Figures 1 to 3 This embodiment describes a white body autonomous welding method based on machine vision. In this method, the hardware used includes an industrial camera, an industrial computer, a robotic arm and a white body. Figure 1As shown, the overall idea of ​​the method described in this embodiment is the coordinate system conversion problem, that is, the coordinate conversion between the vehicle body coordinate system, the camera coordinate system and the robot arm coordinate system. Figure 2 As shown, this method is based on Figure 1 The coordinate transformation is determined. First, the camera is calibrated, and then the camera is used to shoot the car body, image processing is performed, and then the coordinate transformation relationship from the car body to the camera is solved. The algorithm used is the PnP algorithm, and then the weld point position coordinates in the car body coordinate system can be transformed to the camera coordinate system. After that, the hand-eye calibration is performed to solve the coordinate transformation relationship from the camera to the robotic arm. In this way, the weld point position coordinates in the camera coordinate system can be transformed to the robotic arm coordinate system, and finally the robotic arm welding is realized.

[0018] The specific process of the welding method described in this embodiment is as follows:

[0019] Step 1. First, you need to calibrate the camera to obtain its camera intrinsic parameters and distortion coefficients to prepare for subsequent work.

[0020] In this embodiment, a chessboard is used as a calibration plate, and a camera is used to take pictures of the chessboard. The chessboard keeps changing its posture, and about 10 to 20 pictures are taken. The taken images are processed, and the Zhang Zhengyou calibration method is used to obtain the camera's internal parameters and distortion coefficients.

[0021] Step 2. Pre-select several feature points on the vehicle body. Since the 3D model of the vehicle body is known, the 3D coordinates of the centroid of these feature points are also known. Use a calibrated camera to take a picture of the vehicle body, use the internal parameters and distortion coefficients obtained in the previous calibration to dedistort the image, then convert it into a grayscale image, and then perform OSTU segmentation to transform it into a binary image. Based on the binary image, extract the centroid pixel coordinates of the feature points. Specifically:

[0022] Take a photo of the vehicle body, obtain the vehicle body image and process it, first remove the distortion, then grayscale and binarize it, and extract the two-dimensional coordinates of the centroid of the feature points;

[0023] In this embodiment, a calibrated camera is used to take a picture of the vehicle body, and then the obtained vehicle body image is first dedistorted, then grayed into a grayscale image, and then OSTU segmentation is used to obtain a binary image, and finally the connected domain is obtained. After the connected domain is obtained, the two-dimensional coordinates of the center of mass of each connected domain are extracted.

[0024] Step 3. Since the centroid 3D coordinates of the feature points are known, image processing obtains the 2D coordinates of the pixels. The PnP algorithm is used to solve the camera pose, that is, the coordinate transformation relationship from the vehicle coordinate system to the camera coordinate system, including the rotation matrix R and the translation vector T, that is, the camera pose (R, T). Specifically:

[0025] According to the obtained two-dimensional coordinates of the center of mass and the three-dimensional coordinates of the center of mass of the feature points provided by the vehicle body model, the PnP algorithm is used to calculate the coordinate transformation relationship from the vehicle body coordinate system to the camera coordinate system;

[0026] Step 4. Convert the welding point position coordinates in the vehicle coordinate system to the camera coordinate system, and then perform hand-eye calibration to solve the coordinate transformation relationship from the camera to the robotic arm;

[0027] In this embodiment, hand-eye calibration is performed to calibrate the hand-eye relationship, that is, the relationship between the camera and the robotic arm, such as Figure 3 As shown, the eye-outside-hand mode is used here. The camera 1 is fixed on the bracket 2, and the calibration plate 3 is fixed at the end of the robotic arm 4. The relative positions remain unchanged. By continuously changing the posture of the robotic arm, the camera takes multiple groups of calibration plate images and transforms them into solving the problem of solving the AX=XB equation group. The transformation relationship between the camera coordinate system and the robotic arm base coordinate system can be calibrated by solving the equation.

[0028] Among them, A is the coordinate transformation from the end of the robot to the base, which can be given by the robot, and represents the product of the coordinate transformation from the base to the end of the robot during two photos. Each time the posture of the robot is changed, a set of data will be transmitted, representing the coordinate offsets in the three directions of XYZ and the rotation angles around the three axes of XYZ. The three offsets correspond to the translation vector, and the three angles need to be processed to obtain the required rotation matrix form. Because the rotation matrix used is the product of the rotation matrices around the three axes, the three rotation angles are respectively substituted into their respective rotation matrix expressions, and then the three matrices are multiplied in sequence to obtain the desired rotation matrix, and then the rotation matrix and the translation vector are spliced ​​together to obtain the transformation matrix M mentioned in the previous derivation.

[0029] B is the coordinate transformation from the camera to the calibration plate, which represents the product of the transformation relationship between the camera and the chessboard when taking two photos. Because the chessboard is connected to the end of the robotic arm, the chessboard moves every time the robotic arm changes its posture, and the chessboard image taken by the camera is different. Therefore, it is necessary to find the transformation relationship between the chessboard and the camera after the robotic arm changes its posture each time. After taking the image, the image is processed to extract the corner points. The three-dimensional world coordinates of the corner points are matched one by one with the two-dimensional pixel coordinates of the corner points extracted by image recognition, and the posture transformation of the chessboard and the camera, that is, the rotation matrix and the translation vector, is solved. Finally, the two are spliced ​​to obtain the transformation matrix M.

[0030] Then X is the transformation matrix to be solved, that is, the coordinate transformation relationship between the camera coordinate system and the robot base coordinate system. Because the coordinate transformation from the robot base to the end has been obtained and can be converted into A, and the coordinate transformation from the camera coordinate system to the chessboard coordinate system has been obtained and can be converted into B, it is possible to solve the AX=XB equation group to obtain X, that is, the coordinate transformation matrix from the camera to the robot base. Specifically:

[0031] It can be represented by a 4*4 homogeneous matrix M, which is obtained by combining a 3*3 rotation matrix R and a 3*1 translation vector T.

[0032] In this implementation, the principle formula of the eye outside the hand is as follows, which converts the coordinate transformation relationship from the camera to the base of the robotic arm into the result of multiplying multiple transformation relationships:

[0033]

[0034] in, The transformation matrix representing the end of the robot to the base of the robot can be obtained by the robot itself. Represents the transformation matrix from the calibration plate to the end of the robotic arm. Since the positions of the calibration plate and the end are fixed, the matrix is ​​a constant. Represents the transformation matrix from the camera to the calibration plate, which can be obtained by the PnP algorithm.

[0035] In this embodiment, since a group of images need to be captured, the above formula can be written as a series of formulas:

[0036]

[0037] Where i = 1, 2, 3...n, and n is the number of shots.

[0038] Transform it and we get:

[0039]

[0040] because is a constant, so we can get the equation:

[0041]

[0042] Transform again and we get:

[0043]

[0044] Similarly, we can get n-2 more formulas, which can be transformed into the problem of solving AX=XB, as follows: Seen as Seen as It is the X that is being sought.

[0045] Among them, A is the coordinate transformation from the end of the robot to the base, which can be given by the robot, and B is the coordinate transformation from the camera to the calibration plate, which can also be calculated. Then we can also calculate X, which is the coordinate transformation matrix from the camera coordinate system to the robot base coordinate system.

[0046] Step 5. Finally, according to the coordinate system transformation relationship solved in step 4, the welding point position coordinates on the vehicle body coordinate system can be first transferred to the camera coordinate system, and then transferred to the robot coordinate system. After the robot obtains the coordinates, it welds the welding point.

[0047] In this implementation, based on the coordinate transformation relationship from the vehicle coordinate system to the camera coordinate system, the position coordinates of the welding point can be transferred to the camera coordinate system, and then the hand-eye calibration obtains the coordinate transformation relationship from the camera coordinate system to the robot coordinate system, and the position coordinates of the welding point can be transferred to the robot coordinate system, and the robot starts working. The specific conversion is as follows:

[0048] First, the welding point position coordinates are transferred from the vehicle body coordinate system to the camera coordinate system:

[0049]

[0050] Where (X,Y,Z) T It represents the coordinates of the welding point in the vehicle body coordinate system, (X1, Y1, Z1) T It represents the coordinates of the welding point in the camera coordinate system, (R, T) represents the camera pose (R and T are rotation matrices and translation vectors respectively, and here they are concatenated to form a new matrix, which represents the camera pose, that is, the coordinate transformation relationship from the vehicle body coordinate system to the camera coordinate system). It is a rigid body transformation, that is, the coordinate transformation from the vehicle body to the camera solved by the PnP algorithm in step C.

[0051] Then the welding point position coordinates are transferred from the camera coordinate system to the robot arm coordinate system:

[0052]

[0053] Where (X2, Y2, Z2) T It represents the coordinates of the welding point in the robot coordinate system, and (R1, T1) represents the coordinate transformation from the camera to the robot, which is obtained by the hand-eye calibration in step 4.

[0054] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0055] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A white body autonomous welding method based on machine vision, characterized by: The method is implemented by the following steps: Step 1: Calibrate the camera to obtain the camera's intrinsic parameters and distortion coefficients; Step 2: Use the camera calibrated in step 1 to take a picture of the vehicle body, process the acquired vehicle body image, and extract the two-dimensional coordinates of the centroid of the feature points; Step 3: Calculate the coordinate transformation relationship from the vehicle body coordinate system to the camera coordinate system based on the two-dimensional coordinates of the centroid of the feature point obtained in step 2 and the three-dimensional coordinates of the centroid of the feature point obtained based on the vehicle body model; Step 4: Perform hand-eye calibration to calibrate the coordinate transformation relationship from the camera coordinate system to the robotic arm coordinate system; Step 5: Convert the position coordinates of the welding point from the vehicle body coordinate system to the robot arm coordinate system, and then use the robot arm to perform welding.

2. The method for autonomous body-in-white welding based on machine vision according to claim 1, characterized in that: In step 1, a chessboard is used as a calibration plate, and a camera is used to take pictures of the chessboard. The chessboard continuously changes its posture, and multiple chessboard images are obtained. The captured images are processed to obtain the camera's internal parameters and distortion coefficients.

3. The method for autonomous body-in-white welding based on machine vision according to claim 1, characterized in that: In step 2, the vehicle body image is dedistorted and then converted into a grayscale image. OSTU segmentation is used to obtain a binary image. Finally, the connected domain is obtained and the centroid coordinates of each connected domain are extracted.

4. The method for autonomous body-in-white welding based on machine vision according to claim 1, characterized in that: In step three, the camera pose is calculated using the PnP algorithm based on the three-dimensional coordinates of the centroid of the feature points obtained from the vehicle body model, that is, the coordinate transformation relationship from the vehicle body coordinate system to the camera coordinate system.

5. The method for autonomous body-in-white welding based on machine vision according to claim 1, characterized in that: In step 4, the eye-outside-hand calibration method is adopted. The camera is fixed on the bracket and the calibration plate is fixed on the end of the robotic arm. The relative positions remain unchanged. Multiple sets of images are taken by continuously changing the posture of the robotic arm to obtain the coordinate transformation relationship from the camera to the base of the robotic arm.

6. The method for autonomous body-in-white welding based on machine vision according to claim 5 is characterized in that: The principle formula of the eye outside the hand is as follows. The coordinate transformation relationship from the camera to the base of the robotic arm is transformed into the result of multiplying multiple transformation relationships, which is expressed as follows: In the formula, is the transformation matrix from the end of the robot to the base of the robot, is the transformation matrix from the calibration plate to the end of the robotic arm, is the transformation matrix from the camera to the calibration plate. For multiple groups of images, the above formula can be rewritten as: Where i = 1, 2, 3...n, n is the number of shots; transform it into: because is a constant, we can get the equation: Transform again and get: Similarly, we can get n-2 formulas, which can be transformed into the problem of solving AX=XB: Will As A, As B, is the X to be determined; that is, the coordinate transformation matrix from the camera coordinate system to the robot base coordinate system.

7. The method for autonomous body-in-white welding based on machine vision according to claim 1, characterized in that: In step 5, according to the coordinate transformation relationship from the camera coordinate system to the robot coordinate system obtained in step 4, the weld point position coordinates on the vehicle body coordinate system are first transformed to the camera coordinate system, and then transformed to the robot coordinate system. After the robot obtains the coordinates, it welds the weld point.

8. The method for autonomous body-in-white welding based on machine vision according to claim 7, characterized in that: The welding point position coordinates are converted from the vehicle body coordinate system to the camera coordinate system: Among them, (X,Y,Z) T is the welding point coordinates in the vehicle body coordinate system, (X1, Y1, Z1) T is the welding point coordinate in the camera coordinate system, (R, T) is the camera pose, that is, the coordinate transformation from the vehicle body to the camera. Convert the welding point position coordinates from the camera coordinate system to the robot arm coordinate system: Among them, (X2, Y2, Z2) T is the welding point coordinate in the robot coordinate system, and (R1, T1) is the coordinate transformation from camera to robot.

Images