Special vehicle spraying gun pose detection method based on deep learning

Abstract

The invention belongs to the technical field of spraying processes, and discloses a special vehicle spraying gun pose detection method based on deep learning, comprising the following steps: step 1, setting an area array CCD camera coordinate system as a global coordinate; step 2, calibrating a relation equation parameter between a two-dimensional coordinate (u, v) and a three-dimensional coordinate (x, y, z) of the laser point in the two-dimensional image under a global coordinate system; and step 3, loading the FFDNet rapid denoising pre-training model by using PyTorch to carry out rapid denoising processing on the two-dimensional image. According to the method, the accuracy of spraying gun pose calculation is improved on the whole, a foundation is established for real-time dynamic adjustment and correction of the spraying gun pose, and therefore the spraying quality is improved.

Description

Technical Field

[0001] The present invention belongs to the field of spraying process technology, and in particular relates to a method for detecting the posture of a special vehicle spray gun based on deep learning. Background Art

[0002] Existing spray gun posture detection systems primarily consist of a point laser emitter, an area array CCD camera, and an image processing unit. The point laser emitter projects a laser point onto the surface of the object being measured, while the area array CCD camera captures the image of the projected laser point. The image processing unit, connected to the camera, pre-processes the laser point image and calculates relevant coordinate parameters, plane parameters, distance parameters, and angle parameters. This system is used to monitor the spray gun's posture parameters in real time during the spraying process, and uses feedback to adjust and correct the gun's position and posture relative to the surface being sprayed.

[0003] In a streamlined inspection process, the center location of the laser spot on the 2D image captured by the camera is crucial for pose calculation and directly determines the accuracy of real-time pose adjustments. Existing systems use a Gaussian filter to smooth the 2D image to remove noise. The Canny algorithm then extracts the edges of the laser spot, generating a binary image of the spot's edges. Finally, hardware within the programmable logic device (PLD) determines the coordinates of the laser line's center.

[0004] First, for spray painting of special vehicles, due to the complex surface structures and conditions of special vehicles, traditional Gaussian filtering technology is difficult to adapt to the denoising of various complex surface images. At the same time, the effect of Gaussian filtering on nonlinear and non-uniform noise is also insufficient. Secondly, the Canny edge detection algorithm is sensitive to noise. In the case of high noise or poor filtering effect, the detection results will be greatly affected. At the same time, the Canny algorithm is not accurate enough when processing complex edge structures, such as curved or bifurcated edges. Summary of the Invention

[0005] The technical problem to be solved by the present invention is that the traditional Gaussian filtering technology is difficult to adapt to the denoising processing of various complex surface images of special vehicles. At the same time, the effect of Gaussian filtering for nonlinear and non-uniform noise is also difficult to meet the needs, and the Canny algorithm is not accurate enough when processing complex edge structures.

[0006] In order to solve the above technical problems, the specific technical solutions of the present invention are as follows:

[0007] A method for detecting the posture of a special vehicle spray gun based on deep learning, characterized by comprising the following steps:

[0008] Step 1: Set the coordinate system of the area array CCD camera to the global coordinate system;

[0009] Step 2: calibrate the relationship equation parameters between the two-dimensional coordinates (u, v) and three-dimensional coordinates (x, y, z) of the laser point in the two-dimensional image in the global coordinate system;

[0010] Step 3: Use PyTorch to load the FFDNet fast denoising pre-trained model to perform fast denoising on the two-dimensional image;

[0011] Step 4: Use PyTorch to load the ENet efficient convolutional neural network pre-training model to perform edge segmentation and extraction on the two-dimensional image to obtain a binarized spot edge image;

[0012] Step 5: Use Hough transform method to determine the coordinates (u, v) of the spot center;

[0013] Step 6: Calculate the three-dimensional coordinates (x, y, z) of the laser point in the camera coordinate system through the calibrated relationship equation;

[0014] Step 7: Construct a plane equation of the surface of the object to be measured, i.e., a reference plane equation, through the three-dimensional coordinates of at least three laser points;

[0015] Step 8. Calculate the distance from the origin of the camera coordinate system to the reference plane;

[0016] Step 9: Calculate the angle between the Z axis of the camera coordinate system and the normal vector of the reference plane to determine the angle of the spray gun;

[0017] Step 10: Based on the relative coordinate relationship between the spray gun and the area array CCD camera, the distance and angle of the camera relative to the surface of the object to be measured are converted into the distance and angle of the spray gun relative to the surface of the object to be measured, thereby completing the posture detection.

[0018] Furthermore, in step 2, the conversion relationship between the two-dimensional coordinates (u, v) of the laser point in the two-dimensional image and the three-dimensional coordinates (x, y, z) in the global coordinate system is calibrated as follows:

[0019] According to the pinhole imaging principle, the corresponding relationship between (u, v) and (x, y, z) is:

[0020]

[0021] Among them, f x , f y , u0, v0 are the intrinsic parameters of the area array CCD camera, which can be obtained by calibration:

[0022] Assume that the angle between the line connecting the two laser points projected by the point laser emitter and the imaging plane of the area array CCD camera is α, the intersection point is (x0, y0, z0), and the angle between the projection of the line connecting the two laser points on the imaging plane of the area array CCD camera and the global coordinate x-axis is β. Using the trigonometric function relationship, we can get:

[0023]

[0024] Substituting formula (3) into formula (1) yields:

[0025]

[0026] Let a = f x x0, but

[0027] Substituting formula (4) into formula (2) yields:

[0028]

[0029] Let m = f y y0, but

[0030] In equations (6) and (7), z is the distance from the laser point to the optical center plane. Since the optical center plane is difficult to determine, the z value cannot be accurately measured. Assuming that the distance between a reference plane A and the optical center plane is c, the distance between the light point and the reference plane A can be accurately measured and is set as z1, then:

[0031]

[0032] During the calibration process, the target is moved at a fixed distance within the measurement range. During the movement, the position of the laser point in the image (u i , v i ), and then use a high-precision ruler to measure the distance z1 from the laser point to the reference plane;

[0033] All measured u i , v i Substituting the values ​​of and z1 into equations (7) and (8), the parameters a, b, c, m, and n can be estimated, and the relationship between (u, v) and (x, y, z) is:

[0034]

[0035] make but

[0036]

[0037] make but

[0038] Therefore, the relationship equation between (u, v) and (x, y, z) is as follows:

[0039]

[0040] Further, step seven is specifically as follows:

[0041] Assume that the coordinates of the laser point in the camera coordinate system are (x, y, z), and construct two vectors v from the three laser point coordinates 1 and v 2 , let the plane equation be:

[0042] Ax+By+Cz+D=0;

[0043] Using the two vectors v above 1 =[x 1 ,y 1 ,z 1 ] T and v 2 =[x 2 ,y 2 ,z 2 ] T Constructing the plane equation, we get:

[0044] A=y 1 z 2 -y 2 z 1

[0045] B=x 2 z 1 -x 1 z 2

[0046] C=x 1 y 2 -y 1 x 2

[0047] D=-Ax 1 -By 1 -Cz 1 .

[0048] Furthermore, in step eight, after the spray gun position changes, the distance from the origin of the camera coordinate system (x0, y0, z0) to the reference plane is:

[0049] Furthermore, step nine is as follows: 1 and v 2 Perform cross multiplication to obtain the normal vector n of the reference plane2 ;

[0050] Therefore, assuming that the Z-axis unit vector is n 1 , reference plane normal vector n 2 , the angle between the two normal vectors is:

[0051]

[0052] The present invention has the following advantages: In the two-dimensional image processing stage, the deep learning algorithm models FFDNet (Fast and Flexible Denoising Network) and ENet (Efficient Neural Network) based on artificial neural networks are introduced. Compared with traditional image denoising and edge detection algorithms, the pre-trained model based on deep learning can greatly improve the denoising ability, improve the edge detection effect, and adapt to the feature extraction of the surface image of the complex structure of special vehicles through pre-training learning of a large amount of data. At the same time, through directional and targeted model fine-tuning technology, the model can perform well in specific types of surface image processing work. Secondly, by adopting Hough transform for edge recognition, the light spot contour line can be fitted more accurately, thereby obtaining more accurate center point coordinates, laying the foundation for subsequent coordinate transformation and calculation. By combining artificial neural network deep learning technology, the present invention improves the accuracy of spray gun posture calculation as a whole, lays the foundation for real-time dynamic adjustment and correction of spray gun posture, thereby improving the spraying quality. DETAILED DESCRIPTION

[0053] In order to better understand the purpose, structure and function of the present invention, the present invention is further described in detail below in conjunction with embodiments.

[0054] The purpose of this embodiment is to improve the flexibility and effectiveness of denoising and edge detection of two-dimensional light spot images through the FFDNet (Fast and Flexible Denoising Network) and ENet (Efficient Neural Network) deep learning models, so that the entire detection process can more flexibly adapt to the complex surface structure conditions of special vehicles, thereby obtaining high-precision, high-quality posture state parameters in real time and improving spraying quality.

[0055] The detection method described in this embodiment is divided into 10 steps, and the detailed steps are as follows:

[0056] 1. Set the coordinate system of the area array CCD camera to global coordinates;

[0057] 2. The conversion relationship between the two-dimensional coordinates (u, v) of the laser point in the two-dimensional image and the three-dimensional coordinates (x, y, z) in the global coordinate system is calibrated as follows:

[0058] According to the pinhole imaging principle, the corresponding relationship between (u, v) and (x, y, z) is:

[0059]

[0060]

[0061] Among them, f x , f y , u0, v0 are the intrinsic parameters of the area array CCD camera, which can be obtained by calibration:

[0062] Assume that the angle between the line connecting the two laser points projected by the point laser emitter and the imaging plane of the area array CCD camera is α, the intersection point is (x0, y0, z0), and the angle between the projection of the line connecting the two laser points on the imaging plane of the area array CCD camera and the global coordinate x-axis is β. Using the trigonometric function relationship, we can get:

[0063]

[0064] Substituting formula (3) into formula (1) yields:

[0065]

[0066] Let a = f x x0, but

[0067] Substituting formula (4) into formula (2) yields:

[0068]

[0069] Let m = f y y0, but

[0070] In equations (6) and (7), z is the distance from the laser point to the optical center plane. Since the optical center plane is difficult to determine, the z value cannot be accurately measured. Assuming that the distance between a reference plane A and the optical center plane is c, the distance between the light point and the reference plane A can be accurately measured and is set as z1, then:

[0071]

[0072] During the calibration process, the target is moved at a fixed distance within the measurement range. During the movement, the position of the laser point in the image (ui, vi) is calculated using an image processing algorithm, and then the distance z1 from the laser point to the reference plane is measured using a high-precision ruler.

[0073] All measured u i , v i Substituting the values ​​of and z1 into equations (7) and (8), the parameters a, b, c, m, and n can be estimated, and the relationship between (u, v) and (x, y, z) is:

[0074]

[0075] make but

[0076]

[0077] make but

[0078] Therefore, the relationship equation between (u, v) and (x, y, z) is as follows:

[0079]

[0080] 3. Use PyTorch to load the FFDNet (Fast and Flexible Denoising Network) fast denoising pre-trained model to perform fast denoising on two-dimensional images;

[0081] 4. Use PyTorch to load the ENet (Efficient Neural Network) efficient convolutional neural network pre-training model to perform edge segmentation and extraction on the two-dimensional image to obtain the binarized spot edge image;

[0082] 5. Use Hough transform method to determine the coordinates of the spot center position (u, v);

[0083] 6. Calculate the three-dimensional coordinates (x, y, z) of the laser point in the camera coordinate system through the calibrated relationship equation;

[0084] 7. Construct the plane equation of the surface of the object to be measured, i.e. the reference plane equation, through the three-dimensional coordinates of the laser points (at least three);

[0085] Assume that the coordinates of the laser point in the camera coordinate system are (x, y, z), and construct two vectors v from the three laser point coordinates 1 and v 2 , let the plane equation be:

[0086] Ax+By+Cz+D=0

[0087] Using the two vectors v above 1 =[x 1 ,y 1 ,z 1 ] T and v 2 =[x 2 ,y 2 ,z 2 ] T Constructing the plane equation, we get:

[0088] A=y 1 z 2 -y 2 z 1

[0089] B=x 2 z 1 -x 1 z 2

[0090] C=x 1 y 2 -y 1 x 2

[0091] D=-Ax 1 -By 1 -Cz 1

[0092] 8. Calculate the distance from the origin of the camera coordinate system to the reference plane;

[0093] After the gun position changes, the distance from the origin of the camera coordinate system (x0, y0, z0) to the reference plane is:

[0094]

[0095] 9. Calculate the angle between the Z axis of the camera coordinate system and the normal vector of the reference plane to determine the angle of the spray gun.

[0096] V 1 and v 2 Perform cross multiplication to obtain the normal vector n of the reference plane 2 .

[0097] Therefore, assuming that the Z-axis unit vector is n 1 , reference plane normal vector n 2 , the angle between the two normal vectors is:

[0098]

[0099] 10. According to the relative coordinate relationship between the spray gun and the area array CCD camera, the distance and angle of the camera relative to the surface of the object to be measured are converted into the distance and angle of the spray gun relative to the surface of the object to be measured, thereby completing the posture detection

[0100] The key points of this embodiment are:

[0101] 1. The two-dimensional laser point images captured by the area array CCD camera are first denoised using the FFDNet (Fast and Flexible Denoising Network) fast denoising pre-training model;

[0102] 2. The denoised two-dimensional laser point image is subjected to edge segmentation extraction using the ENet (Efficient Neural Network) efficient convolutional neural network pre-training model to obtain a binarized spot edge image.

[0103] 3. Use Hough transform to detect the circular contour of the laser spot and determine the two-dimensional coordinates of the center position.

[0104] Although the embodiments of the present invention are described in conjunction with the accompanying drawings, it is apparent to those skilled in the art that several variations and improvements may be made without departing from the principles of the present invention, and these should also be considered to fall within the scope of protection of the present invention.

Claims

1. A method for detecting the posture of a special vehicle spray gun based on deep learning, characterized in that: The steps include: Step 1: Set the coordinate system of the area array CCD camera to the global coordinate system; Step 2: calibrate the relationship equation parameters between the two-dimensional coordinates (u, v) and three-dimensional coordinates (x, y, z) of the laser point in the two-dimensional image in the global coordinate system; Step 3: Use PyTorch to load the FFDNet fast denoising pre-trained model to perform fast denoising on the two-dimensional image; Step 4: Use PyTorch to load the ENet efficient convolutional neural network pre-training model to perform edge segmentation and extraction on the two-dimensional image to obtain a binarized spot edge image; Step 5: Use Hough transform method to determine the coordinates (u, v) of the spot center; Step 6: Calculate the three-dimensional coordinates (x, y, z) of the laser point in the camera coordinate system through the calibrated relationship equation; Step 7: Construct a plane equation of the surface of the object to be measured, i.e., a reference plane equation, through the three-dimensional coordinates of at least three laser points; Step 8. Calculate the distance from the origin of the camera coordinate system to the reference plane; Step 9: Calculate the angle between the Z axis of the camera coordinate system and the normal vector of the reference plane to determine the angle of the spray gun; Step 10: Based on the relative coordinate relationship between the spray gun and the area array CCD camera, the distance and angle of the camera relative to the surface of the object to be measured are converted into the distance and angle of the spray gun relative to the surface of the object to be measured, thereby completing the posture detection.

2. The method for detecting the posture of a special vehicle spray gun based on deep learning according to claim 1, characterized in that: In step 2, the conversion relationship between the two-dimensional coordinates (u, v) of the laser point in the two-dimensional image and the three-dimensional coordinates (x, y, z) in the global coordinate system is calibrated as follows: According to the pinhole imaging principle, the corresponding relationship between (u, v) and (x, y, z) is: Among them, f x , f y , u0, v0 are the intrinsic parameters of the area array CCD camera, which can be obtained by calibration: Assume that the angle between the line connecting the two laser points projected by the point laser emitter and the imaging plane of the area array CCD camera is α, the intersection point is (x0, y0, z0), and the angle between the projection of the line connecting the two laser points on the imaging plane of the area array CCD camera and the global coordinate x-axis is β. Using the trigonometric function relationship, we can get: Substituting formula (3) into formula (1) yields: Let a = f x x0, but Substituting formula (4) into formula (2) yields: Let m = f y y0, but In equations (6) and (7), z is the distance from the laser point to the optical center plane. Since the optical center plane is difficult to determine, the z value cannot be accurately measured. Assuming that the distance between a reference plane A and the optical center plane is c, the distance between the light point and the reference plane A can be accurately measured and is set as z1, then: During the calibration process, the target is moved at a fixed distance within the measurement range. During the movement, the position of the laser point in the image (u i , v i ), and then use a high-precision ruler to measure the distance z1 from the laser point to the reference plane; All measured u i , v i Substituting the values ​​of and z1 into equations (7) and (8), the parameters a, b, c, m, and n can be estimated, and the relationship between (u, v) and (x, y, z) is: make but make but Therefore, the relationship equation between (u, v) and (x, y, z) is as follows:

3. The method for detecting the posture of a special vehicle spray gun based on deep learning according to claim 2, characterized in that: Step seven is as follows: Assume that the coordinates of the laser point in the camera coordinate system are (x, y, z), and construct two vectors v from the three laser point coordinates 1 and v 2 , let the plane equation be: Ax+By+Cz+D=0; Using the two vectors v above 1 =[x 1 ,y 1 ,z 1 ] T and v 2 =[x 2 ,y 2 ,z 2 ] T Constructing the plane equation, we get: A=y 1 z 2 -y 2 z 1 B=x 2 z 1 -x 1 z 2 C=x 1 and 2 -and 1 x 2 D=-Ax 1 -By 1 -Cz 1 。 4. The method for detecting the posture of a special vehicle spray gun based on deep learning according to claim 3, characterized in that: In step 8, after the spray gun position changes, the distance from the origin of the camera coordinate system (x0, y0, z0) to the reference plane is:

5. The method for detecting the posture of a special vehicle spray gun based on deep learning according to claim 4, characterized in that: Step nine is as follows: 1 and v 2 Perform cross multiplication to obtain the normal vector n of the reference plane 2 ; Therefore, assuming that the Z-axis unit vector is n 1 , reference plane normal vector n 2 , the angle between the two normal vectors is:

Images