Visual inspection method and system for automobile gluing product

Abstract

The invention discloses a visual inspection method and system for automobile gluing products, and belongs to the technical field of automatic visual inspection. Comprising the following steps: S1, carrying out initial parameter setting on three visual detection cameras, adopting the three visual detection cameras to be arranged at equal height, and taking pictures along the length direction of a glued product to obtain an initial image; s2, uploading the initial images acquired by the plurality of visual inspection cameras to a server, fusing the images and acquiring a final image; s3, setting a threshold value, and performing edge detection on the fused image; in the step S2, the plurality of initial image fusion steps are included. According to the visual detection method and system for the automobile gluing product, the three cameras are adopted to sequentially shoot partially-overlapped images, then an algorithm is utilized to fuse image overlapping areas, the overlapped parts of the images are aligned, the images are fused to ensure color consistency and natural transition, the image processing speed is high, and the visual detection accuracy is high. And compared with a monovision detection system, parallel computing can be adopted, the system needs lower computing power, and the detection efficiency is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of automated vision inspection, and more specifically, to a vision inspection method and system for automotive gluing products. Background Art

[0002] During the production of automotive gluing products, it is necessary to detect the gluing positions or graphic defects. In the context where manual means do not meet the current production capacity, the current method of using vision inspection for automotive gluing products has emerged. However, since the size of automotive gluing products used inside the vehicle is very large, usually with a length of 2.5m - 3.0m and a width exceeding 1m, it is difficult for a single vision inspection system image to accommodate the entire product of this size. It is necessary to drive the camera to move and take pictures through a moving mechanism. However, this inspection method has two problems. First, the cost is high, and at least a moving mechanism with a camera and multiple sensors are required on the device. Second, the size of the generated initial images is large, the image generation is slow, and serial computing is used for preprocessing, which requires a large amount of computing, wasting computing power and delaying the production inspection time at the same time.

[0003] Based on the above problems, we propose a vision inspection method and system for automotive interior gluing products, which adopts a three-camera vision inspection method to fuse the images generated by each camera. On the premise of solving the above problems, it can ensure the elimination of geometric distortion and perspective differences between cameras, align the overlapping parts of the images and fuse the images to ensure color consistency and natural transition, generate processed images quickly, and can use parallel computing, greatly reducing the computing power and ensuring the inspection efficiency. Summary of the Invention

[0004] The present invention provides a vision inspection method and system for automotive gluing products to overcome the problems of high cost and low efficiency of a single vision inspection system when detecting defects of large-size gluing products as mentioned in the above background art.

[0005] To solve the above technical problems, the technical solution of the present invention is as follows:

[0006] A vision inspection method for automotive gluing products includes the following steps:

[0007] S1: Perform initial parameter settings on three vision inspection cameras, and use the three vision inspection cameras to be set at the same height and take pictures along the length direction of the gluing product to obtain initial images;

[0008] S2: Upload the initial images obtained by multiple vision inspection cameras to the server to fuse the images and obtain the final image;

[0009] S3: Set a threshold and perform edge detection on the fused image; where

[0010] The fusion step of multiple initial images in step S2 includes

[0011] S21: Obtain the internal parameter matrix and distortion coefficients of the camera, perform calibration using the checkerboard calibration method, correct the image, and eliminate lens distortion;

[0012] S22: Extract the feature points existing in different initial images and perform feature matching to determine the correspondence of multiple initial images;

[0013] S23: Deduce the spatial mapping relationship of different initial images, calculate the homography matrix of the left and right vision detection cameras relative to the middle vision detection camera respectively using the homography matrix, perform perspective transformation on the image, and project the left and right initial images onto the plane coordinate system of a middle vision detection camera for image alignment;

[0014] S24: Weight and average the pixel values of the two images in the overlapping area, adjust the overall tone and contrast of the image according to the brightness and color differences in the overlapping area, ensure the color consistency after stitching, and perform stitching to generate a fused image;

[0015] S25: Crop the black edges of the fused image, adjust the ratio and angle of the stitched image, and generate the final image.

[0016] Preferably, the image distortion correction in step S21 includes radial distortion correction and tangential distortion correction, specifically:

[0017] Radial distortion:

[0018] Δx r = x(k1r 2 + k2r 4 + k3r 6 ), Δy r = y(k1r 2 + k2r 4 + k3r 6 );

[0019] Where, (x, y) are the pixel coordinates in the initial image, (Δx r , Δy r ) are the corrected pixel coordinates, r is the distance between the pixel coordinates and the image center, and k1, k2, k3 are the radial distortion coefficients;

[0020] Tangential distortion:

[0021] Δx t = 2p1xy + p2(r 2 + 2x 2 ), Δy t = p1(r 2 + 2y 2 ) + 2p2xy;

[0022] Among them, p1 and p2 are tangential distortion coefficients.

[0023] Preferably, in the step S22, the feature point extraction method is to extract using the SIFT or ORB algorithm.

[0024] Preferably, the feature matching steps in the step S22 are as follows:

[0025] S221: Ratio test: Compare the best match with the second-best match for each point, and remove unreliable matches;

[0026] S222: Use the RANSAC algorithm to obtain effective sample data, and remove outliers among the matching points to improve the robustness of the matching.

[0027] Preferably, the homography matrix form adopted by the image transformation method in the step S23 is:

[0028]

[0029] Through normalization processing, the coordinates of the mapped feature points are:

[0030]

[0031] Two adjacent initial images require at least four feature matching points, and the linear solution equation composed of the identity matrix is as follows:

[0032] x′ i (h 31 x i +h 32 y i +h 33 ) = h 11 x i +h 12 y i +h 13 ,

[0033] y′ i (h 31 x i +h 32 y i +h 33 ) = h 21 x i +h 22 y i +h 23 .

[0034] Preferably, in the step S24, two adjacent pictures are set as I1 and I2, the pixel values of I1 and I2 in the overlapping area are P1 and P2 respectively, and the weighted average formula of the fused pixel value P is:

[0035] P(x, y) = w1(x, y)·P1(x, y) + w2(x, y)·P2(x, y),

[0036] Correct the brightness and color differences in the overlapping area of the pictures, adjust the overall tone and contrast of the image. Let the average brightness of pictures I1 and I2 be μ1 and μ2, and the way to adjust the brightness of picture I2 is:

[0037] I′2(x, y) = I2(x, y) + (μ1 - μ2).

[0038] Preferably, in step S24 of the above method, multi-band fusion technology is used for image stitching.

[0039] The present invention also provides a vision inspection system for automotive gluing products, which includes three vision inspection cameras arranged in parallel, a control unit, a light source, and an image processing module.

[0040] Preferably, the image processing module includes at least one memory for storing the algorithm system of the above vision inspection system; and at least one processor for executing the algorithms in the memory.

[0041] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0042] The present invention provides a vision inspection method and system for automotive gluing products. When performing vision inspection on large-sized automotive gluing products, three cameras are used to sequentially capture partially overlapping images, and then an algorithm is used to fuse the overlapping areas of the images, and finally a complete image is generated. This can ensure the elimination of geometric distortion and perspective differences between cameras, align the overlapping parts of the images and fuse the images to ensure color consistency and natural transition. The image processing speed is fast. Compared with a single vision inspection system, there is no need to set up a camera movement mechanism and various sensors, which reduces costs. At the same time, compared with directly calculating a whole large-sized picture, the generated processed image has a fast speed, parallel computing can be used, and the computing power required by the system is lower, ensuring the detection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0044] Figure 1 It is a schematic diagram of steps S1 - S3 of the present invention;

[0045] Figure 2 It is a schematic diagram of steps S21 - S25 of the present invention;

[0046] Figure 3 Schematic diagram of the fused image of the present invention;

[0047] Figure 4 Schematic diagram of the image after feature extraction obtained by the left camera of the present invention;

[0048] Figure 5 Schematic diagram of the image after feature extraction obtained by the middle camera of the present invention;

[0049] Figure 6 Schematic diagram of the image after feature extraction obtained by the right camera of the present invention;

[0050] Figure 7 Schematic diagram of the light source of the present invention. Detailed implementation manners

[0051] In order to better understand the purpose, structure and function of the present invention, the technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific preferred embodiments.

[0052] Embodiment 1:

[0053] As Figure 1 and Figure 2 shown, the present invention provides a visual inspection method for automotive gluing products, including the following steps:

[0054] S1: Perform initial parameter settings on three visual inspection cameras, and take initial images by setting the three visual inspection cameras at the same height and photographing along the length direction of the gluing product. The visual inspection cameras are CCD or CMOS.

[0055] S2: Upload the initial images obtained by multiple visual inspection cameras to the server to fuse the images and obtain the final image.

[0056] S3: Set a threshold value, perform edge detection on the fused image, and the edge detection adopts the canny edge detection algorithm. The detected defect types include the gluing shape or the gluing trajectory.

[0057] More specifically, the multiple initial image fusion steps in step S2 include,

[0058] S21: Obtain the internal parameter matrix and distortion coefficient of the camera. The internal parameter matrix is a parameter describing the focal length and the offset of the optical axis. Specific parameters need to be viewed according to the camera model. The checkerboard calibration method is used for calibration to correct the image and eliminate lens distortion;

[0059] S22: Extract the feature points existing in different initial images and perform feature matching to determine the correspondence of multiple initial images;

[0060] S23: Derive the spatial mapping relationships of different initial images. Calculate the homography matrices of the left and right visual detection cameras relative to the middle visual detection camera respectively using homography matrices, perform perspective transformation on the images, and project the left and right initial images onto the plane coordinate system of a middle visual detection camera for image alignment;

[0061] S24: Weight and average the pixel values of the two images in the overlapping area. Adjust the overall hue and contrast of the images according to the brightness and color differences in the overlapping area, ensure the color consistency after stitching, and perform stitching to generate a fused image;

[0062] S25: Crop the black edges of the fused image, adjust the ratio and angle of the stitched image, and generate the final image.

[0063] Furthermore, the image distortion correction in step S21 includes radial distortion correction and tangential distortion correction, specifically:

[0064] Radial distortion:

[0065] Δx r = x(k1r 2 + k2r 4 + k3r 6 ), Δy r = y(k1r 2 + k2r 4 + k3r 6 );

[0066] Among them, (x, y) are the pixel coordinates in the initial image, (Δx r , Δy r ) are the corrected pixel coordinates, r is the distance between the pixel coordinates and the image center, and k1, k2, and k3 are the radial distortion coefficients;

[0067] Tangential distortion:

[0068] Δx t = 2p1xy + p2(r 2 + 2x 2 ), Δy t = p1(r 2 + 2y 2 ) + 2p2xy;

[0069] Among them, p1 and p2 are the tangential distortion coefficients.

[0070] Furthermore, in step S22, the feature point extraction method uses the SIFT or ORB algorithm for extraction. Among them, the SIFT algorithm is a convolution operation of the original image I(x, y) and a two-dimensional Gaussian function G(x, y, σ) with a variable scale. Specifically,

[0071] L(x, y, σ) = G(x, y, σ) * I(x, y),

[0072] Thus, the scale-variable Gaussian function is,[[]]

[0073] Finally, the feature points are found.[[]]

[0074] Furthermore, the feature matching step in the step S22 is as follows:[[]]

[0075] S221: Ratio test: Compare the ratio of the best match to the second-best match for each point, and remove unreliable matches;[[]]

[0076] S222: The RANSAC algorithm obtains effective sample data, removes outliers from the matching points, and improves the robustness of the matching.[[]]

[0077] Furthermore, the homography matrix form adopted by the image transformation method in the step S23 is:[[]]

[0078]

[0079] Through normalization processing, the coordinates of the mapped feature points are:[[]]

[0080]

[0081] Two adjacent initial images require at least four feature matching points, and the unit matrix forms a linear solution equation as follows:[[]]

[0082] x′ i (h 31 x i +h 32 y i +h 33 ) = h 11 x i +h 12 y i +h 13 ,

[0083] y′ i (h 31 x i +h 32 y i +h 33 ) = h 21xi +h 22 y i +h 23 .

[0084] Furthermore, in the step S24, two adjacent pictures are set as I1 and I2, the pixel values of I1 and I2 in the overlapping area are P1 and P2 respectively, and the weighted average formula for the fused pixel value P is:[[]]

[0085] P(x, y) = w1(x, y)·P1(x, y) + w2(x, y)·P2(x, y),

[0086] Correct the brightness and color differences in the overlapping area of the pictures, adjust the overall tone and contrast of the images. Let the average brightness of pictures I1 and I2 be μ1 and μ2, and the method for adjusting the brightness of picture I2 is as follows:

[0087] I′2(x, y) = I2(x, y) + (μ1 - μ2).

[0088] Furthermore, in step S24, multi-band fusion technology is used for image stitching, and the multi-band fusion technology is the pyramid algorithm.

[0089] Next, specific embodiments will be combined to show the fusion of images in the three-camera system. The glue-applying product in the embodiment is an automotive glue-applying product with a length of 2.8 m and a width of 1 m, as Figures 3-6 shown Figure 4 is the image obtained after preprocessing by the left camera. Two feature points are extracted on the right side of the image. Figure 6 is the image obtained after preprocessing by the right camera. Figure 5 is the image obtained after preprocessing by the middle camera. Four feature points are extracted in the up and down directions on both sides of the image. Among them, the two feature points on the left correspond to Figure 4 the feature points obtained by the left camera in; the two feature points on the right correspond to the feature points obtained by the right camera. After fusing the images, the complete glue-applying product image as shown in Figure 3 is finally obtained. As can be seen from Figure 3 the feature points on both sides of the image of the middle camera are completely fused with the feature points of the left and right cameras, without geometric distortion and perspective difference, and the overlapping parts of the aligned images and the fused image colors are kept consistent and transition naturally.

[0090] On the basis of the above steps, to improve efficiency, the GPU module of OpenCV can be used to accelerate computationally intensive operations such as feature point extraction, feature matching, and image transformation, and the image resolution can be reduced according to needs to reduce the computational burden.

[0091] Embodiment 2:

[0092] On the basis of the above embodiment, a vision detection system for an automotive glue-applying product of the present invention is used to execute any of the above glue-applying product vision detection methods, and includes three parallel vision detection cameras, a control unit, a light source, and an image processing module.

[0093] As Figure 7As shown, due to the size problem of the glue-coated product, color differences will occur when multiple cameras take pictures using traditional light sources. Therefore, in this application, a light source with a length of 3.0 m and a width of 1.2 m is set up, a black light-absorbing background is adopted, and multiple light strips are arranged along the length direction of the product to ensure stable light, ensure that the color differences of the photos obtained by different cameras are small, and reduce the difficulty of subsequent post-processing.

[0094] Furthermore, the image processing module includes at least one memory for storing the algorithm system of the above visual detection system; and at least one processor for executing the algorithms in the memory.

[0095] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, rather than limitations on the implementation manners of the present invention. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to enumerate all the implementation manners here. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A visual inspection method for automotive glue-coated products, characterized in that: The steps include: S1: Initial parameter settings are performed for three visual inspection cameras. The three visual inspection cameras are set at equal heights and take photos along the length direction of the glue-coated product to obtain an initial image. S2: Upload the initial images obtained by multiple visual inspection cameras to the server to fuse the images and obtain the final image; S3: Set the threshold and perform edge detection on the fused image; The multiple initial image fusion steps in step S2 include: S21: Obtain the camera's internal parameter matrix and distortion coefficient, calibrate using a chessboard calibration method, correct the image, and eliminate lens distortion; S22: extracting feature points existing in different initial images and performing feature matching to determine the corresponding relationship between the multiple initial images; S23: deriving the spatial mapping relationship between different initial images, using homography matrices to respectively calculate the homography matrices of the left and right visual detection cameras relative to the middle visual detection camera, performing perspective transformation on the images, and projecting the left and right initial images into a plane coordinate system of the middle visual detection camera for image alignment; S24: performing weighted averaging of pixel values ​​of the two images in the overlapping area, adjusting the overall hue and contrast of the images according to the brightness and color differences of the overlapping area to ensure color consistency after stitching, and performing stitching to generate a fused image; S25: Crop the black edges of the fused image, adjust the scale and angle of the stitched image, and generate a final image.

2. The visual inspection method for automotive glue coating products according to claim 1, characterized in that: The image distortion correction in step S21 includes radial distortion correction and tangential distortion correction, specifically: Radial distortion: Δx r =x(k1r 2 +k2r 4 +k3r 6 ),Δy r =y(k1r 2 +k2r 4 +k3r 6 ); Where (x, y) is the pixel coordinate in the original image, (Δx r , Δy r ) is the corrected pixel coordinate, r is the distance between the pixel coordinate and the image center, k1, k2, k3 are radial distortion coefficients; Tangential distortion: Δx t =2p1xy+p2(r 2 +2x 2 ),Δy t =p1(r 2 +2y 2 )+2p2xy; Among them, p1 and p2 are the tangential distortion coefficients.

3. The visual inspection method for automotive glue coating products according to claim 1, characterized in that: In the step S22, the feature point extraction method is SIFT or ORB algorithm.

4. The method for visual inspection of automotive glue-coated products according to claim 3, characterized in that: The feature matching steps in step S22 are as follows: S221: Ratio test: compare the best match and the second best match of each point to remove unreliable matches; S222: The RANSAC algorithm obtains valid sample data, removes outliers from matching points, and improves the robustness of the matching.

5. The visual inspection method for automotive glue coating products according to claim 1, characterized in that: The homography matrix form adopted by the image transformation method in step S23 is: After normalization, the coordinates of the mapped feature points are: Two adjacent initial images require at least four feature matching points, and the linear solution equation composed of the unit matrix is ​​as follows: x′ i (h 31 x i +h 32 y i +h 33 )=h 11 x i +h 12 y i +h 13 , y′ i (h 31 x i +h 32 y i +h 33 )=h 21 x i +h 22 y i +h 23 .。 6. The method for visual inspection of automotive glue-coated products according to claim 1, characterized in that: In step S24, the two adjacent pictures are set as I1 and I2, the pixel values ​​of I1 and I2 in the overlapping area are P1 and P2 respectively, and the weighted average formula of the fused pixel value P is: P(x, y)=w1(x, y)·P1(x, y)+w2(x, y)·P2(x, y), Correct the brightness and color differences in the overlapping areas of the images, adjust the overall tone and contrast of the image, and assume that the average brightness of images I1 and I2 is μ1 and μ2. The brightness of image I2 is adjusted as follows: I′2(x,y)=I2(x,y)+(μ1-μ2).

7. The visual inspection method for automotive glue coating products according to claim 6, characterized in that: The image stitching in step S24 adopts multi-band fusion technology.

8. A visual inspection system for automotive glue-coated products, used to perform any of the visual inspection methods for glue-coated products according to claims 1 to 7, characterized in that: It includes three visual inspection cameras arranged in parallel, a control unit, a light source and an image processing module.

9. The automotive glue coating product visual inspection system according to claim 8, characterized in that: The image processing module includes at least one memory for storing the algorithm system of any one of the visual inspection systems of claims 1-7; and at least one processor for executing the algorithm of the memory.

Images