Glue line detection method based on machine vision

Abstract

The present invention provides a glue line detection method based on machine vision, comprising: obtaining multiple training images; establishing a glue line recognition model based on the multiple training images, and performing semantic segmentation training; obtaining multiple standard sample images; extracting the glue line contours of the multiple standard sample images according to the glue line recognition model to obtain a standard glue line path; obtaining a sample image to be tested; extracting the glue line contour of the sample image to be tested according to the glue line recognition model to obtain a glue line path to be tested; calculating the offset value between the glue line contour of the standard sample image and the glue line contour of the sample image to be tested, so as to determine whether the glue line path to be tested fits the standard glue line path. The present invention solves the problems of high difficulty and large error in glue line detection in the prior art.

Description

Technical Field

[0001] The present invention relates to the technical field of module assembly process, and in particular to a glue line detection method based on machine vision. Background Art

[0002] During the assembly process of the camera module, multiple glue drawing is involved, such as fixing the color filter on the bracket, assembling the photosensitive component and the lens, etc. Therefore, the quality of the glue line is an important indicator affecting the assembly quality of the camera module. Conventional glue line detection is through manual detection such as broken glue, different glue widths, excessive glue in some areas, etc. The quality of glue line detection depends on the inspection level of the quality inspector. The consistency of multiple batches of glue detection is poor and the inspection efficiency is low. Existing camera module products have many extreme specifications, large differences in appearance, and many risks of local morphological characteristics. The difficulty of glue line detection for more complex glue line shapes (such as many changes in glue width and different glue line contours) is further increased.

[0003] That is to say, the existing technology of glue line detection has the problems of high detection difficulty and large error. Summary of the invention

[0004] The main purpose of the present invention is to provide a glue line detection method based on machine vision to solve the problems of high difficulty and large error in glue line detection in the prior art.

[0005] In order to achieve the above-mentioned objectives, the present invention provides a glue line detection method based on machine vision, including: obtaining multiple training images; establishing a glue line recognition model based on the multiple training images, and performing semantic segmentation training; obtaining multiple standard sample images; extracting the glue line contours of the multiple standard sample images according to the glue line recognition model to obtain a standard glue line path; obtaining a sample image to be tested; extracting the glue line contour of the sample image to be tested according to the glue line recognition model to obtain the glue line path to be tested; calculating the offset value between the glue line contour of the standard sample image and the glue line contour of the sample image to be tested, so as to determine whether the glue line path to be tested fits the standard glue line path.

[0006] Furthermore, in the process of establishing a glue line recognition model based on multiple training images and performing semantic segmentation training, it includes: obtaining a template image; extracting the first feature of the template image and establishing a glue line recognition model; obtaining the glue line areas of all training images as a data set; and deep learning semantic segmentation of the data set to train the glue line recognition model.

[0007] Furthermore, in the process of extracting the first feature of the template image and establishing the glue line recognition model, it includes: obtaining the first area of ​​the template image; obtaining the first feature of the template image in the first area; establishing a basic coordinate system according to the first feature of the template image; obtaining the first features of multiple training images; matching the first features of the multiple training images with the basic coordinate system to obtain a unified coordinate system.

[0008] Furthermore, in the process of obtaining the glue line regions of all training images as the data set, it includes: selecting the straight segments of the glue lines of the training images as the glue line regions of the training images.

[0009] Furthermore, in the process of extracting the glue line contours of multiple standard sample images according to the glue line recognition model to obtain the standard glue line path, it includes: acquiring the area to be identified of the standard sample image; performing glue line segmentation on the area to be identified according to the glue line recognition model to obtain the glue line contour of the standard sample image; and taking the union of the glue line contours of all standard sample images to obtain the standard glue line path.

[0010] Furthermore, in the process of performing glue line segmentation on the area to be identified according to the glue line recognition model to obtain the glue line contour of the standard sample image, it includes: outputting a binary image of the standard sample image according to the glue line recognition model; dividing the coordinate points with the same pixel values ​​in the binary image into the same area to perform glue line segmentation on the area to be identified.

[0011] Furthermore, in the process of calculating the offset value of the glue line contour of the standard sample image and the glue line contour of the sample image to be tested to determine whether the glue line path to be tested fits the standard glue line path, it includes: comparing the coordinate coincidence of the standard glue line path and the glue line path to be tested; if the number of coordinate misalignments between the standard glue line path and the glue line path to be tested is lower than a preset standard, it is determined that the glue line path to be tested fits the standard glue line path, and the sample image to be tested is a preliminary qualified image.

[0012] Furthermore, the glue line detection method based on machine vision also includes: obtaining an inner contour to be measured and an outer contour to be measured of the glue line contour of the preliminary qualified image; obtaining a standard inner contour and a standard outer contour of the glue line contour of the standard sample image; calculating an inner offset value between the inner contour to be measured and the standard inner contour, and an outer offset value between the outer contour to be measured and the standard outer contour; if both the inner offset value and the outer offset value are within a preset range, the glue line width of the preliminary qualified image is judged to be qualified.

[0013] Furthermore, in the process of calculating the inner offset value of the inner contour to be measured and the standard inner contour, and the outer offset value of the outer contour to be measured and the standard outer contour, it includes: obtaining a first direction and a second direction perpendicular to each other in the plane of the preliminary qualified image, at least a part of the glue line of the preliminary qualified image extends along the first direction, and at least another part of the glue line of the preliminary qualified image extends along the second direction; obtaining a detection direction, the detection direction is along the first direction or the second direction; obtaining a calculation direction perpendicular to the detection direction; taking all coordinate points in the calculation direction, and generating a detection line along the detection direction; calculating the difference between the intersection point of the detection line and the glue line contour of the preliminary qualified image, and the intersection point of the detection line and the glue line contour of the standard sample image; and taking the average of the differences of all coordinate points as the offset value.

[0014] Furthermore, after calculating the inner offset value between the inner contour to be measured and the standard inner contour, and the outer offset value between the outer contour to be measured and the standard outer contour, it also includes: obtaining the horizontal and vertical scale factors of the visual system; according to the horizontal and vertical scale factors of the visual system, converting the inner offset value, the outer offset value and the glue line width of the preliminary qualified image.

[0015] Furthermore, in the process of obtaining the horizontal and vertical scale factors of the visual system, the following steps are included: obtaining the size L of the actual standard sample; obtaining the number of pixels A of the photosensitive chip of the shooting camera in a single direction; obtaining the field of view fitting coefficient F; and the horizontal and vertical scale factors N of the visual system satisfy:

[0016] Formula (1).

[0017] Furthermore, before establishing a glue line recognition model based on multiple training images and performing semantic segmentation training, it also includes: obtaining a test machine; calibrating the accuracy of the test machine; calibrating the brightness of the image recognized by the test machine; and calibrating the software operating environment of the test machine.

[0018] Applying the technical solution of the present invention, the glue line detection method based on machine vision includes: step S10: obtaining multiple training images; step S20: establishing a glue line recognition model based on the multiple training images, and performing semantic segmentation training; step S30: obtaining multiple standard sample images; step S40: extracting the glue line contours of the multiple standard sample images according to the glue line recognition model to obtain the standard glue line path; step S50: obtaining the sample image to be tested; step S60: extracting the glue line contour of the sample image to be tested according to the glue line recognition model to obtain the glue line path to be tested; step S70: calculating the offset value between the glue line contour of the standard sample image and the glue line contour of the sample image to be tested, so as to determine whether the glue line path to be tested fits the standard glue line path.

[0019] Through step S10 and step S20, a glue line recognition model is established for semantic segmentation training, which can greatly improve the recognition accuracy and the accuracy of the recognition result. The semantic segmentation training helps step S30 and step S40 to more accurately segment the glue line area and the non-glue line area, extract a more accurate glue line contour of the standard sample image, and reduce the recognition error. Through step S50 and step S60, the glue line contour of the sample image to be tested is extracted using the glue line recognition model, and step S70 is used to compare the offset value of the glue line contour of the sample image to be tested with the glue line contour of the sample image to be tested, and the difference in the glue line contours of the two is more accurately compared at various locations, so as to determine whether the glue line contour of the sample image to be tested is qualified.

[0020] Specifically, semantic segmentation training can assign a category label to each pixel in the image, so that the glue line recognition model can accurately identify and distinguish the glue line area and non-glue line area in the training image, and the recognition of complex glue line structures can more accurately capture its edges and shapes. Through deep learning training, the glue line recognition model can learn and understand complex image features, including lighting changes, background noise, texture differences, etc., which is conducive to the glue line recognition model in practical applications. It can detect and identify glue lines more stably, even in changing production and inspection environments. It has high recognition accuracy and effectively reduces recognition errors caused by image blur, reflection, shadow, etc. The detection process of the glue line recognition model obtained through continuous training can be fully automated without human intervention, which greatly improves the detection speed and efficiency, and can quickly process a large number of samples, which is of great significance for improving the detection capabilities of production lines and reducing labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings constituting a part of the present application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0022] Figure 1 A schematic diagram showing an angle of a camera module of an optional embodiment of the present invention when applying glue;

[0023] Figure 2 Shows Figure 1 A schematic diagram of another angle of the camera module in FIG.

[0024] Figure 3 A schematic diagram of a training image according to an optional embodiment of the present invention is shown;

[0025] Figure 4 Shows Figure 3 The first characteristic diagram in FIG.

[0026] Figure 5 Shows Figure 3Schematic diagram of the first area in;

[0027] Figure 6 Shows Figure 3 Schematic diagram of the first glue line area in FIG.

[0028] Figure 7 Shows Figure 3 Schematic diagram of the second glue line area;

[0029] Figure 8 A schematic diagram of a standard sample image of an optional embodiment of the present invention is shown;

[0030] Fig. 9 Shows Figure 8 Schematic diagram of the standard glue line area in;

[0031] Fig.10 Shows Figure 8 Schematic diagram of the standard glue line area and boundary outline;

[0032] Fig.11 A schematic diagram of calculating the glue line width of a sample image to be tested according to an optional embodiment of the present invention is shown;

[0033] Fig.12 Shows Fig.11 Schematic diagram of the detection area selected in the middle box;

[0034] Fig.13 Shows Fig.11 Schematic diagram of glue line profile offset calculation in;

[0035] Fig.14 Shows Fig.11 Schematic diagram of safety distance in ;

[0036] Fig.15 A flow chart showing a glue line detection scheme according to an optional embodiment of the present invention;

[0037] Fig.16 A flow chart of a glue line detection method based on machine vision according to any optional embodiment of the present invention is shown.

[0038] The above drawings include the following reference numerals:

[0039] 10. glue; 20. lens holder; 30. circuit board; 40. photosensitive chip; 50. color filter; 60. first feature; 70. first area; C. area to be identified; D. standard glue line area; E. glue line area to be tested; F. detection area; R1. first boundary contour; R11. first pixel point; R2. second boundary contour; R21. second pixel point; L1. first vertical line; L2. second vertical line; D_R1. first glue line contour of standard sample image; E_R1. first glue line contour of sample image to be tested; 80. first glue line area; 90. second glue line area; w. width direction; h. height direction. DETAILED DESCRIPTION

[0040] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0041] It should be noted that, unless otherwise specified, all technical and scientific terms used in this application have the same meanings as commonly understood by ordinary technicians in the technical field to which this application belongs.

[0042] In the present invention, unless otherwise specified, the directional words used, such as "up, down, top, bottom", usually refer to the directions shown in the drawings, or to the components themselves in the vertical, perpendicular or gravity directions; similarly, for ease of understanding and description, "inside and outside" refer to the inside and outside relative to the outline of each component itself, but the above-mentioned directional words are not used to limit the present invention.

[0043] In order to solve the problems of high difficulty and large error in glue line detection in the prior art, the present invention provides a glue line detection method based on machine vision.

[0044] like Figures 1 to 16 As shown, the glue line detection method based on machine vision includes: step S10: obtaining multiple training images; step S20: establishing a glue line recognition model based on the multiple training images, and performing semantic segmentation training; step S30: obtaining multiple standard sample images; step S40: extracting the glue line contours of the multiple standard sample images according to the glue line recognition model to obtain the standard glue line path; step S50: obtaining the sample image to be tested; step S60: extracting the glue line contour of the sample image to be tested according to the glue line recognition model to obtain the glue line path to be tested; step S70: calculating the offset value between the glue line contour of the standard sample image and the glue line contour of the sample image to be tested, so as to determine whether the glue line path to be tested fits the standard glue line path.

[0045] Through step S10 and step S20, a glue line recognition model is established for semantic segmentation training, which can greatly improve the recognition accuracy and the accuracy of the recognition result. The semantic segmentation training helps step S30 and step S40 to more accurately segment the glue line area and the non-glue line area, extract a more accurate glue line contour of the standard sample image, and reduce the recognition error. Through step S50 and step S60, the glue line contour of the sample image to be tested is extracted using the glue line recognition model, and step S70 is used to compare the offset value of the glue line contour of the sample image to be tested with the glue line contour of the sample image to be tested, and the difference in the glue line contours of the two is more accurately compared at various locations, so as to determine whether the glue line contour of the sample image to be tested is qualified.

[0046] Specifically, semantic segmentation training can assign a category label to each pixel in the image, so that the glue line recognition model can accurately identify and distinguish the glue line area and non-glue line area in the training image, and the recognition of complex glue line structures can more accurately capture its edges and shapes. Through deep learning training, the glue line recognition model can learn and understand complex image features, including lighting changes, background noise, texture differences, etc., which is conducive to the glue line recognition model in practical applications. It can detect and identify glue lines more stably, even in changing production and inspection environments. It has high recognition accuracy and effectively reduces recognition errors caused by image blur, reflection, shadow, etc. The detection process of the glue line recognition model obtained through continuous training can be fully automated without human intervention, which greatly improves the detection speed and efficiency, and can quickly process a large number of samples, which is of great significance for improving the detection capabilities of production lines and reducing labor costs.

[0047] Specifically, in the process of establishing a glue line recognition model based on multiple training images and performing semantic segmentation training, it includes: obtaining a template image; extracting the first feature of the template image to establish a glue line recognition model; obtaining the glue line area of ​​all training images as a data set; deep learning semantic segmentation of the data set to train the glue line recognition model. The first feature is a feature with obvious and stable contrast that exists on all images. Using it as the basis for building a glue line recognition model is conducive to quickly locating the glue line area on the image, and can ensure that the glue line recognition model can accurately capture the key information of the glue line in the subsequent recognition process, thereby improving the accuracy of glue line recognition. For modules with different shapes and specifications, the glue line recognition model based on the first feature can better adapt to various glue line forms and maintain high detection performance even in the case of product diversification.

[0048] Optionally, the data set includes a training set and a validation set, the training set is used as an object for deep learning semantic segmentation to form a glue line recognition model, and the validation set is used as an object to verify the recognition accuracy of the glue line recognition model.

[0049] Specifically, in the process of extracting the first feature of the template image and establishing the glue line recognition model, the process includes: obtaining the first area of ​​the template image; obtaining the first feature of the template image in the first area; establishing a basic coordinate system according to the first feature of the template image; obtaining the first features of multiple training images; matching the first features of multiple training images with the basic coordinate system to obtain a unified coordinate system. The establishment of a unified coordinate system helps to improve the standardization and normalization of glue line detection, ensures that the glue line contours between different images can be accurately compared and analyzed, and is suitable for production lines that require batch detection and quality control. By selecting the first feature in the first area, the established basic coordinate system provides a positioning reference, which helps to quickly locate and identify the first feature of the training image and improve the speed of detection and recognition. By matching the first features of multiple training images with the basic coordinate system to obtain a unified coordinate system, it is ensured that the glue line features of all training images are described in the same coordinate frame, which helps the glue line recognition model understand and learn the cross-image consistency of the glue line during the training process and improve the generalization ability of the model. By performing coordinate transformation and matching of feature points in a unified coordinate system, the offset value of the glue line contour can be calculated more accurately, the detection error can be effectively controlled, and the detection accuracy can be improved.

[0050] Specifically, in the process of obtaining the glue line area of ​​all training images as a data set, the process includes: selecting the straight segment of the glue line of the training image as the glue line area of ​​the training image. The straight segment of the glue line is used as the glue line recognition area, and the glue line corner area is removed, thereby improving the recognition accuracy and reducing the recognition amount.

[0051] Specifically, in the process of extracting the glue line contours of multiple standard sample images according to the glue line recognition model to obtain the standard glue line path, the process includes: obtaining the to-be-recognized area of ​​the standard sample image; performing glue line segmentation on the to-be-recognized area according to the glue line recognition model to obtain the glue line contour of the standard sample image; and taking the union of the glue line contours of all standard sample images to obtain the standard glue line path. By collecting and taking the union of the glue line contours of all standard sample images, the recognition error is reduced and the accuracy of the standard glue line path is improved.

[0052] Specifically, in the process of performing glue line segmentation on the area to be identified according to the glue line recognition model to obtain the glue line contour of the standard sample image, it includes: outputting a binary image of the standard sample image according to the glue line recognition model; dividing the coordinate points with the same pixel value in the binary image into the same area to perform glue line segmentation on the area to be identified. Since the glue line recognition model divides the area to be identified of the standard sample image into a glue line area and a non-glue line area through semantic segmentation, the obtained binary image contains coordinate information, pixel information, and a mark of whether it is a glue line area, etc., which can effectively improve the extraction accuracy of the glue line contour and reduce noise interference.

[0053] Specifically, in the process of calculating the offset value of the glue line contour of the standard sample image and the glue line contour of the sample image to be tested to determine whether the glue line path to be tested fits the standard glue line path, it includes: comparing the coordinate coincidence of the standard glue line path and the glue line path to be tested; if the number of coordinate misalignments between the standard glue line path and the glue line path to be tested is lower than the preset standard, it is determined that the glue line path to be tested fits the standard glue line path, and the sample image to be tested is a preliminary qualified image. By making a preliminary rough judgment to screen the glue line path to be tested, and then performing fine calculations on the preliminary qualified image, the amount of precise calculations is reduced, the calculation time is saved, and the detection and recognition efficiency is greatly improved.

[0054] Specifically, the glue line detection method based on machine vision also includes: obtaining the inner contour to be measured and the outer contour to be measured of the glue line contour of the preliminary qualified image; obtaining the standard inner contour and the standard outer contour of the glue line contour of the standard sample image; calculating the inner offset value between the inner contour to be measured and the standard inner contour, and the outer offset value between the outer contour to be measured and the standard outer contour; if the inner offset value and the outer offset value are both within the preset range, it is determined that the width of the glue line of the preliminary qualified image is qualified. By calculating the offset values ​​for the inner contour and the outer contour respectively, the detection accuracy of the inner and outer contour positions of the glue line and the width of the glue line is guaranteed.

[0055] Specifically, in the process of calculating the inner offset value between the inner contour to be measured and the standard inner contour, and the outer offset value between the outer contour to be measured and the standard outer contour, the process includes: obtaining a first direction and a second direction perpendicular to each other in the plane of the preliminary qualified image, at least a part of the glue line of the preliminary qualified image extends along the first direction, and at least another part of the glue line of the preliminary qualified image extends along the second direction; obtaining a detection direction, the detection direction is along the first direction or the second direction; obtaining a calculation direction perpendicular to the detection direction; taking all coordinate points in the calculation direction, and generating a detection line along the detection direction; calculating the difference between the intersection of the detection line and the glue line contour of the preliminary qualified image, and the intersection of the detection line and the glue line contour of the standard sample image; taking the average of the differences of all coordinate points as the offset value. This calculation method based on the difference of coordinate points can accurately evaluate the deviation between the inner contour to be measured and the standard inner contour, and the deviation between the outer contour to be measured and the standard outer contour, and improve the accuracy and reliability of detection. Using the average of the differences of all coordinate points as the offset value effectively reduces the deviation caused by individual special values. It should be noted that the first direction and the second direction are usually the width direction and the height direction, which are easy to identify and calculate.

[0056] Specifically, after calculating the inner offset value between the inner contour to be measured and the standard inner contour, and the outer offset value between the outer contour to be measured and the standard outer contour, it also includes: obtaining the horizontal and vertical scale factors of the visual system; converting the inner offset value, the outer offset value, and the glue line width of the preliminary qualified image according to the horizontal and vertical scale factors of the visual system. For cameras with different resolutions, the number of pixels occupied by the glue line width is different, but the actual value of the glue line width is certain. By converting the horizontal and vertical scale factors of the visual system, the glue line width and the offset value are output as actual quantitative values ​​as a universal measurement standard, which can ensure the consistency of the detection data when cameras with different resolutions recognize images.

[0057] Specifically, the process of obtaining the horizontal and vertical scale factors of the visual system includes: obtaining the size L of the actual standard sample; obtaining the number of pixels A of the photosensitive chip of the shooting camera in a single direction; obtaining the field of view fitting coefficient F; the horizontal and vertical scale factors N of the visual system satisfy:

[0058] Formula (1).

[0059] Since there is field curvature in the image when the camera takes the image, the pixel value needs to be compensated and corrected. Through formula (1), a more accurate horizontal and vertical scale factor of the visual system can be obtained, thereby improving the accuracy of the glue line width conversion.

[0060] Specifically, before establishing a glue line recognition model based on multiple training images and conducting semantic segmentation training, it also includes: obtaining a test machine; calibrating the accuracy of the test machine; calibrating the brightness of the image recognized by the test machine; and calibrating the software operating environment of the test machine. Since the brightness detection on the image may not match the actual situation due to local reflections, it is necessary to manually adjust the brightness of the local area. For example, by performing brightness detection on the image, the brightness value of the specified area of ​​the current image can be quantified, and the brightness that makes the outline of the area to be tested on the image clearer is set as the detection brightness. If the image is partially reflected at a certain brightness, the brightness needs to be lowered to make the boundary between the glue line outline and the background more obvious. When calibrating the accuracy of the test machine, take a photo of the same calibration block and unify the width of one grid of the calibration block to a uniform size. In addition, due to various errors caused by the lack of software environment on different machines, the software operating environment of the machine needs to be supplemented.

[0061] The following is a further description of the glue line detection method based on machine vision of the present application in combination with the glue line detection of a camera module. In this embodiment, it involves the glue line detection on the photosensitive component of the camera module. Figures 1 to 3As shown, the photosensitive component includes a circuit board 30, a photosensitive chip 40, a lens holder 20 and a color filter 50. The lens holder 20 is provided with a light window, and the color filter 50 is installed in the light window. The lens holder 20 is glued around the light window and bonded to the lens module (if the camera module is a fixed focus module, the lens module is a lens; if the camera module is a zoom module, the lens module is a motor) through glue 10.

[0062] This embodiment detects the glue line on the mirror base 20, which includes the following steps in sequence:

[0063] 1. Machine calibration, mainly includes accuracy calibration of the test machine, brightness calibration of the recognition image, and calibration of the software operating environment.

[0064] 2. The preparation work before testing is mainly divided into three parts: establishing the glue line recognition model, formulating the standard glue line path and calibrating the machine consistency.

[0065] 2.1 Establishment of glue line identification model:

[0066] 2.1.1 Collect at least 30 training images with glue lines, denoted as P1, P2, ... Pn, where n ≥ 30;

[0067] 2.1.1 Determine the production positioning template (basic coordinate system):

[0068] 2.1.1.1 Select any one of the training images, assuming that the selected image is P1, and select the first feature 60 on the image P1. The first feature 60 is used to construct a template. The first feature 60 is a feature with obvious and stable contrast value existing in each training image. In this embodiment, the first feature 60 is the outline of the color filter 50;

[0069] 2.1.1.2 As Figure 4 and Figure 5 As shown, a first area 70 is framed on P1, and the first area 70 includes a first feature 60;

[0070] 2.1.1.3 Identify the first feature 60 in the first area 70 and establish a positioning template (basic coordinate system), that is, construct a basic coordinate system based on the first feature 60;

[0071] 2.1.2 Template matching:

[0072] Specifically, the first features 60 in P2, ...Pn are identified, and the first features 60 in P2, ...Pn are matched with the positioning template (basic coordinate system) of P1, that is, the first features 60 in the remaining training images are used to construct the same coordinate system as P1, so as to correct the pose (relative to the camera field of view) of the object or image captured by the camera;

[0073] In the template matching process, by comparing the similarity between the first feature 60 of the positioning template P1 and the position of each first feature 60 in the image to be searched P2, ... Pn, the position with the highest similarity (or the smallest difference) is found, thereby determining the position of the positioning template in the image to be searched, and then affine transforming the position of the first feature 60 in the current image P2, ... Pn to the position of the first feature 60 of the positioning template P1, unifying the coordinate system, and thus eliminating the position difference between different products. The calculation of similarity can be achieved by a variety of methods, including but not limited to calculating the difference in pixel values, calculating correlation coefficients, etc.

[0074] 2.1.3 Constructing the Dataset:

[0075] like Figure 6 and Figure 7 As shown, the first glue line area 80 and the second glue line area 90 in all training images P1, P2, ... Pn are manually accurately labeled (specifically, the glue line area is framed and the data is not fuzzy processed) as a data set for training the glue line recognition model;

[0076] In order to improve the recognition accuracy and reduce the recognition amount, the corner area of ​​the glue line is removed, and the straight section of the glue line is used as the glue line area, that is, the first glue line area 80 is optimized to the second glue line area 90.

[0077] 2.1.4 Training glue line recognition model:

[0078] Use deep learning semantic segmentation dataset to train glue line recognition model;

[0079] Among them, the data set includes a training set and a validation set. The training set is used as the object of deep learning semantic segmentation to form a glue line recognition model, and the validation set is used as the object to verify the recognition accuracy of the glue line recognition model.

[0080] 2.2 Standard glue line production:

[0081] 2.2.1 Select standard samples and collect standard sample images M1, M2, ..., Mn, where n is a positive integer, usually 3-5;

[0082] 2.2.2 Use drawing tools to correct the glue line image contour of the standard sample (such as correcting unclear jagged edges to clearer straight lines, adjusting the range of the glue line contour, etc.) so that the standard part can be closer to the median of the control range;

[0083] 2.2.3 Using the positioning model established by P1, match the templates of the standard sample images M1, M2, ..., Mn to the positioning model;

[0084] 2.2.4 As Figure 8As shown, select the areas C1, C2, ..., Cn to be identified on the standard sample image;

[0085] Specifically, the radial dimension of the area C to be identified is 1.2-1.5 times the outer contour of the glue line, which can achieve both identification accuracy and identification efficiency;

[0086] 2.2.5 Use the glue line recognition model to identify the glue line area of ​​the standard sample image:

[0087] Run the glue line recognition model obtained in step 2.1.4 to perform semantic segmentation of the glue line in the area to be recognized C. The semantic segmentation stage has marked the area to be recognized C of the standard sample image as the glue line area and the non-glue line area, and the output is a binary image. The binary image contains coordinate information, pixel information, and whether it is a glue line area, etc., that is, the pixel value (0 or 1) corresponding to each coordinate point (x, y) on the standard sample image (x=0,1,2, ..., w-1, y=0,1,2, ..., h-1; where w and h are the width and height of the standard sample image respectively), with 0 representing the background and 1 representing the glue line;

[0088] Next, the pixel threshold with a value of 1 is segmented to form the glue line area, that is, the standard glue line area D1, D2, ..., Dn on the current standard sample image to be tested.

[0089] 2.2.6 Forming a standard glue line path:

[0090] Specifically, the standard glue line areas D1, D2, ..., Dn of multiple standard sample images M1, M2, ..., Mn are fused (the union of the standard glue line areas D in multiple standard sample images) to generate a standard glue line path (standard part).

[0091] 3. Image inspection of the first sample to be tested.

[0092] 3.1 Collect the image of the sample to be tested;

[0093] 3.2 Using the positioning template established by P1, match the template of the sample image to be tested to the positioning template;

[0094] 3.3 Select the area to be tested of the sample image to be tested, and use the glue line recognition model to identify the glue line path to be tested of the sample image to be tested;

[0095] 3.4 Compare the test glue line path on the current test sample image N with the standard glue line path, and compare whether the coordinates (x1, y1) of the test glue line path on the test sample image coincide with the coordinates (x0, y0) of the standard glue line path on the standard sample image (only need to make a judgment of whether or not). If the number of inconsistent coordinates is 0 or the proportion of inconsistent coordinates is lower than the set threshold, it is judged that the glue line contour is fitted (contour consistency judgment passed) and enter step 3.5; if the proportion of inconsistent coordinates reaches the threshold, it is judged that the glue line contour is not fitted (contour consistency judgment failed);

[0096] 3.5 Glue line width and position offset detection:

[0097] Glue line width detection: In the framed area to be detected, by extracting the first boundary contour R1 and the second boundary contour R2 of the glue line path to be tested of the current sample image to be tested, take all points on the first boundary contour R1, and calculate the distance distlist1 from the point to the second boundary contour R2 on the other side point by point. Similarly, take all points on the second boundary contour R2, and calculate the distance distlist2 from the point to the first boundary contour R1 point by point. Combine the two sets of data to calculate the mean (distlist1+distlist2) / 2, which is used as the value of the glue line width distlist.

[0098] by Fig.11 For example, take the first pixel point R11 on the first boundary contour R1, distlist1 is the first perpendicular line L1 from the first pixel point R11 on the first boundary contour R1 to the second boundary contour R2, distlist2 is the second perpendicular line L2 from the second pixel point R21 corresponding to the first pixel point R11 on the second boundary contour R2 to the first boundary contour R1, then the glue line width from the first pixel point R11 to the second pixel point R21 is (distlist1+ distlist2) / 2.

[0099] Position offset calculation: Set the direction of the detected glue line. In the selected detection area F, extract the boundary contour of the standard glue line area D (the two side contours are respectively represented by the first glue line contour D_R1 of the standard sample image and the second glue line contour D_R2 of the standard sample image) and the boundary contour of the glue line area E to be tested on the current sample image to be tested (the two side contours are respectively represented by the first glue line contour E_R1 of the sample image to be tested and the second glue line contour E_R2 of the sample image to be tested). Take the corresponding contours of D and E to calculate the offset. The specific method is as follows:

[0100] For example, if D_R1 and E_R1 are taken, in the width w direction or height h direction (depending on the set detection glue line direction) of the selected detection area F, here it is assumed that the detection direction is the height h direction, then all points p (p=0,1,2,…,w-1) on the width w are taken to generate a straight line along the height h direction, and the intersection points pointlistD11 and pointlistE11 with the contours D_R1 and E_R1 are calculated respectively, and the difference (positive or negative) of the point set pointlist11=pointlistE11-pointlist D11 (point by point subtraction) is calculated respectively, and the average of the differences is taken as the offset value of E_R1 relative to D_R1;

[0101] by Fig.13 For example, the offset value of the glue line profile E_R1 of the sample image to be tested relative to the glue line profile D_R1 of the standard sample image is pointlist1=(pointlist11+ pointlist12+…+ pointlist1n) / n;

[0102] The offset value of the glue line profile E_R2 of the sample image to be tested relative to the glue line profile D_R2 of the standard sample image is pointlist2=(pointlist21+ pointlist22+…+ pointlist2n) / n;

[0103] like Fig.14 As shown, the external safety distance G1=H1+ pointlist1, where H1 is the standard external safety distance;

[0104] Internal safety distance G2 = H2 + pointlist2, where H2 is the standard internal safety distance; (H1 and H2 are design parameters)

[0105] 3.6 Convert pixel value (pix) to actual value (µm):

[0106] Import the horizontal and vertical scale factors of the visual system, convert the measured glue width (pix) and offset (pix), and output the actual quantized value (µm). For cameras with different resolutions, the number of pixels occupied by the glue width is different, but the actual value of the glue width is certain, ensuring the consistency of the camera detection data;

[0107] Since there is field curvature when the camera captures the image (because the lens is round and the photosensitive chip is square), the pixel value needs to be compensated to obtain a more accurate horizontal and vertical scale factor of the visual system.

[0108] The horizontal and vertical scale factor N (corrected) of the visual system = the size of the actual calibration object L / the number of pixels A of the photosensitive chip of the shooting camera in a single direction (horizontal X or vertical Y) × the field of view fitting coefficient F, where the field of view fitting coefficient F is converted by the curve fitting method.

[0109] 3.7 Determine whether the glue line meets the glue width control: Determine whether the external safety distance G1 and the internal safety distance G2 are within the preset range. If they are within the preset range, the glue width control is determined to be qualified and proceed to step 3.8; if they are outside the preset range, the glue width control is determined to be unqualified and proceed to step 3.9.

[0110] 3.8 The three-dimensional testing equipment determines whether the glue line meets the following requirements:

[0111] If the three-dimensional judgment is in compliance, the glue line dispensing parameters are indicated to be adopted. If the three-dimensional judgment is not in compliance, go to step 3.9.

[0112] Among them, the three-dimensional inspection equipment is a standard equipment for dimensional inspection, which can detect the length, width and height of the glue line. The inspection time is long, and only the first sample that has passed the software test and glue width control is inspected.

[0113] 3.9 Determine whether there is an error in the software detection of the glue line:

[0114] If there is an error in the glue line detection by the software, check and correct the link that caused the detection error, and re-perform the glue line detection (re-execute the steps from the link where the error occurred).

[0115] Generally, the error in the glue line detected by the software is most likely due to an error in the manually labeled data set, which requires manual re-labeling and updating of the data set. If there is no error in the glue line detected by the software, it will indicate adjusting the glue line dispensing parameters.

[0116] It should be noted that step 3.9 is adopted in the debugging and verification stage before the first sample and software testing.

[0117] 4. Image detection of other samples to be tested.

[0118] Select 15 samples (or other values) as the interval, collect the image of the sample to be tested once every 15 samples, repeat steps 3.1-3.7, and in step 3.7, determine whether the external safety distance G1 and the internal safety distance G2 are within the preset range. If they are within the preset range, the glue width control is determined to be qualified; if they are outside the preset range, the glue width control is determined to be unqualified.

[0119] Fig.14 This is a flow chart of the glue line detection solution of this embodiment, wherein the blue box is executed by software, and the green box is executed outside the software (using other machines or manual).

[0120] When testing the sample to be tested, the glue line detection method of the present application first roughly determines whether the glue line contour is fitted, and then performs fine calculations. According to the fitting result of the glue line contour, it is determined whether to perform glue width and position offset detection. It should be noted that "determining whether the glue line contour is fitted" specifically refers to comparing the glue line area on the sample image to be tested and the standard sample image, and comparing whether the coordinates of the glue line area on the sample image to be tested and the coordinates on the standard sample image coincide with each other. If there is no overlap or the ratio is lower than the threshold, the glue line contour is judged to be fitted; "glue width and position offset detection" refers to calculating the difference between the pixel values ​​at each coordinate on the glue line area on the sample image to be tested and the standard sample image. Compared with the prior art that directly performs glue width and position offset detection, the present application adds a step of judging whether the glue line contour is fitted, and preliminarily screens the number of qualified samples to be tested, thereby reducing the amount of precise calculations, saving calculation time, and improving the efficiency of glue line detection.

[0121] In addition, compared with the rough recognition and one-size-fits-all threshold segmentation image recognition method in the prior art, the present application adopts a semantic recognition model with higher recognition accuracy for recognition, which can make the establishment of the recognition model and the recognition results of the test samples more accurate, improve the glue line detection accuracy, and the artificial neural network model is trained on precise data sets, and can develop a boundary function for accurately identifying glue line areas and non-glue line areas.

[0122] Furthermore, after recognizing the image, the present application converts the pixel value (pix) detected by the recognition system into the actual glue width value (µm), and uses the glue width value as a universal measurement standard, so that cameras with different resolutions can ensure the consistency of detection data when recognizing the image.

[0123] Obviously, the above-described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0124] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0125] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0126] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A glue line detection method based on machine vision, characterized in that: The glue line detection method based on machine vision is used for glue line detection in a camera module, and includes: Obtain multiple training images; Establishing a glue line recognition model based on the plurality of training images, and performing semantic segmentation training; Acquire multiple standard sample images; Extracting the glue line contours of the plurality of standard sample images according to the glue line recognition model to obtain a standard glue line path; Acquire an image of a sample to be tested; Extracting the glue line contour of the sample image to be tested according to the glue line recognition model to obtain the glue line path to be tested; Calculating the offset value between the glue line profile of the standard sample image and the glue line profile of the sample image to be tested, so as to determine whether the glue line path to be tested fits the standard glue line path; Wherein, in the process of calculating the offset value between the glue line contour of the standard sample image and the glue line contour of the sample image to be tested to determine whether the glue line path to be tested fits the standard glue line path, it includes: Comparing the coordinate coincidence between the standard glue line path and the glue line path to be tested; If the number of non-overlapping coordinates between the standard glue line path and the glue line path to be tested is lower than a preset standard, it is determined that the glue line path to be tested fits the standard glue line path, and the sample image to be tested is a preliminary qualified image; The glue line detection method based on machine vision also includes: Acquire the inner contour to be measured and the outer contour to be measured of the glue line contour of the preliminary qualified image; Acquire a standard inner contour and a standard outer contour of the glue line contour of the standard sample image; Calculating an inner offset value between the inner contour to be measured and the standard inner contour, and an outer offset value between the outer contour to be measured and the standard outer contour; If the inner offset value and the outer offset value are both within a preset range, it is determined that the glue line width of the preliminary qualified image is qualified.

2. The glue line detection method based on machine vision according to claim 1 is characterized in that: The process of establishing a glue line recognition model according to the plurality of training images and performing semantic segmentation training includes: Get the template image; Extracting the first feature of the template image and establishing the glue line recognition model; Acquire the glue line areas of all the training images as a data set; Deep learning semantic segmentation is performed on the dataset to train the glue line recognition model.

3. The glue line detection method based on machine vision according to claim 2 is characterized in that: The process of extracting the first feature of the template image and establishing the glue line recognition model includes: Acquire a first region of the template image; Acquire a first feature of the template image in the first area; Establishing a basic coordinate system according to the first feature of the template image; Acquire a first feature of a plurality of the training images; The first features of the plurality of training images are matched with the basic coordinate system to obtain a unified coordinate system.

4. The glue line detection method based on machine vision according to claim 2 is characterized in that: The process of obtaining the glue line regions of all the training images as the data set includes: selecting the straight segments of the glue lines of the training images as the glue line regions of the training images.

5. The glue line detection method based on machine vision according to claim 1, characterized in that: The process of extracting the glue line contours of the plurality of standard sample images according to the glue line recognition model to obtain the standard glue line path includes: Acquiring the area to be identified of the standard sample image; Performing glue line segmentation on the area to be identified according to the glue line identification model to obtain a glue line contour of the standard sample image; The glue line contours of all the standard sample images are combined to obtain the standard glue line path.

6. The glue line detection method based on machine vision according to claim 5 is characterized in that: The process of performing glue line segmentation on the area to be identified according to the glue line identification model to obtain the glue line contour of the standard sample image includes: Outputting a binary image of the standard sample image according to the glue line recognition model; Coordinate points with the same pixel value in the binary image are divided into the same area, so as to perform glue line segmentation on the area to be identified.

7. The glue line detection method based on machine vision according to claim 1, characterized in that: The process of calculating the inner offset value between the inner contour to be measured and the standard inner contour, and the outer offset value between the outer contour to be measured and the standard outer contour includes: Acquire a first direction and a second direction perpendicular to each other in the plane of the preliminary qualified image, wherein at least a portion of the glue line of the preliminary qualified image extends along the first direction, and at least another portion of the glue line of the preliminary qualified image extends along the second direction; Acquire a detection direction, where the detection direction is along the first direction or the second direction; Obtaining a calculation direction perpendicular to the detection direction; Take all coordinate points in the calculation direction and generate a detection line along the detection direction; Calculating the difference between the intersection point of the detection straight line and the glue line contour of the preliminary qualified image and the intersection point of the detection straight line and the glue line contour of the standard sample image; The average of the differences of all the coordinate points is taken as the offset value.

8. The glue line detection method based on machine vision according to claim 1, characterized in that: After the process of calculating the inner offset value between the inner contour to be measured and the standard inner contour, and the outer offset value between the outer contour to be measured and the standard outer contour, the method further includes: Get the horizontal and vertical scale factors of the visual system; The inner offset value, the outer offset value and the glue line width of the preliminary qualified image are converted according to the horizontal and vertical scale factors of the visual system.

9. The glue line detection method based on machine vision according to claim 8, characterized in that: The process of obtaining the horizontal and vertical scale factors of the visual system includes: Get the size L of the actual standard sample; Obtain the number of pixels A of the photosensitive chip of the shooting camera in a single direction; Get the field of view fitting coefficient F; The visual system horizontal and vertical scale factor N satisfies: Formula(1).

10. The glue line detection method based on machine vision according to any one of claims 1 to 9, characterized in that: Before the process of establishing a glue line recognition model according to the plurality of training images and performing semantic segmentation training, the process further includes: Get a test machine; Performing precision calibration on the test machine; Performing brightness calibration on the brightness of the image recognized by the test machine; The software operating environment of the test machine is calibrated.

Images