Two-step circuit board plug-in defect abnormity detection method and system

Abstract

The invention relates to the technical field of industrial vision, in particular to a two-step circuit board plug-in defect anomaly detection method and system for detecting various defects and anomalies of a plug-in of a circuit board on an assembly line, and the method comprises the following steps: manufacturing a template: placing a circuit board with a correct plug-in under the condition that a conveyor belt is static, and according to a collected picture, carrying out the detection of the defects and anomalies of the plug-in of the circuit board; box selection is carried out on the circuit board and the plug-ins needing to be detected, pictures and box coordinates are obtained, and threshold values are set for the circuit board and each plug-in; pictures of the circuit board and the plug-in are templates, frame coordinates of the circuit board and the plug-in are respective detection frames, and the pictures, the frame coordinates and threshold values are stored and used during detection, and detection is executed: two-step detection of the circuit board plug-in is executed, after a real-time image is obtained, board card in-place detection is performed according to the generated circuit board template, and the board card in-place detection is performed according to the generated circuit board template; and performing plug-in abnormality detection on the plug-in in the detection frame according to the generated plug-in template, and outputting a detection result of each plug-in.

Description

Technical field:

[0001] The present invention relates to the field of industrial vision technology, and in particular to a two-step circuit board plug-in defect anomaly detection method and system for detecting multiple defect anomalies of a circuit board plug-in on an assembly line, which can significantly improve detection accuracy and detection efficiency. Background technology:

[0002] Printed circuit boards, or PCBs for short, have a huge production volume and are widely used in the field of electronic products due to their low cost and high stability. At present, most domestic electronics factories use assembly line manual insertion for circuit board plug-ins. However, due to operational errors, manual insertion of circuit boards often results in multiple pieces, missing pieces, reversed pieces, and wrong pieces. Therefore, it is necessary to perform abnormal detection of plug-in defects on the board before entering the soldering furnace to avoid scrapping the board after soldering.

[0003] There are currently two mainstream solutions for automated circuit board plug-in defect anomaly detection systems: 1. Use air pumps, infrared sensors and cameras to rely on traditional digital image processing methods for detection. 2. Use only cameras and rely on deep learning neural network target detection and tracking algorithms for detection. The former requires the configuration of air pumps outside the camera and host, and the line needs to be stopped during detection. Although the latter can first determine the board position through target detection and tracking algorithms, and then determine plug-in defects through classification algorithms, the accuracy rate is guaranteed to a certain extent, but the algorithm used is relatively complex, and the computing power requirements are high. It requires a high-configuration graphics card and the computing speed is slow. Both have certain shortcomings in terms of cost and efficiency.

[0004] In addition to these two solutions, there are also solutions that only use cameras and host computers and utilize traditional image processing algorithms for detection, but their detection process and the technology used are relatively simple. Some of their detections are only performed on the entire circuit board, and there are many false alarms due to individual differences in the boards. In addition, it is impossible to perform comprehensive detection on the designated plug-in area, and it is also difficult to detect subtle defects such as reversed parts. Some directly match and detect the plug-in, discarding the relative position information between the circuit board and the plug-in. When there are the same type of plug-ins in similar positions or similar areas of the circuit board, confusion is easily caused. Both situations have high false alarms and missed alarms, which is not conducive to use.

[0005] The common feature of the above three solutions is that they cannot achieve both detection accuracy and efficiency. Summary of the invention:

[0006] In view of the shortcomings and deficiencies in the prior art, the present invention proposes a two-step circuit board plug-in defect anomaly detection method and system that first performs board in-place detection and then performs plug-in anomaly detection, which integrates digital image processing and deep learning neural networks and can significantly improve detection accuracy and detection efficiency.

[0007] The present invention is achieved by the following measures:

[0008] A two-step circuit board plug-in defect anomaly detection method, characterized in that a two-step detection of the circuit board and the plug-in is performed in the detection stage, comprising the following steps:

[0009] Step 1: Make a template: When the conveyor belt is stationary, place the circuit board with the correct plug-in. According to the collected image, select the circuit board and the plug-in to be detected, obtain the image and frame coordinates, and set thresholds for the circuit board and each plug-in. The image of the circuit board and the plug-in is the template, and the frame coordinates of the circuit board and the plug-in are their respective detection frames. The image, frame coordinates, and threshold are all stored and used for detection.

[0010] Step 2: Execute detection: Perform a two-step detection of the circuit board plug-in. After obtaining the real-time image, first perform a board in place detection according to the generated circuit board template, and then perform a plug-in abnormality detection on the plug-in in the detection box according to the generated plug-in template, and output each plug-in detection result; if a circuit board plug-in defect occurs, output the abnormal defect information and process it in a targeted manner according to the needs. The targeted processing includes alarming or controlling the conveyor belt to pause. The abnormal defects of the circuit board plug-in include: missing parts, multiple parts, wrong parts, and reversed parts. Otherwise, obtain the latest real-time image again to continue to determine whether the board is in place and detect the next board.

[0011] In step 1 of the present invention, firstly, it is necessary to place the circuit board with the correct plug-in when the conveyor belt is stationary, collect the RGB picture of the circuit board with the correct plug-in, use the image acquisition module to collect the real-time picture with the help of the camera, select the circuit board on the picture, record the RGB picture of the selection box as the circuit board template image Ct, record the horizontal and vertical coordinates Cx1, Cy1, Cx2, Cy2 of the upper left and lower right of the selection box and use the box as the circuit board detection box, and set the detection threshold Cv for the circuit board;

[0012] After that, keep the circuit board still, continue to select n plug-ins to be tested in the figure in turn, record the RGB image of each plug-in as the template image P1t, P2t, ..., Pnt, record the horizontal and vertical coordinates of the upper left and lower right rectangular boxes of each plug-in (P1x1, P1y1, P1x2, P1y2), (P2x1, P2y1, P2x2, P2y2), ..., (Pnx1, Pny1, Pnx2, Pny2) and use them as plug-in detection boxes, set detection thresholds P1v, P2v, ... Pnv for each plug-in;

[0013] Finally, the above pictures, frame coordinates, and thresholds are stored persistently for use during detection.

[0014] The board in place detection in step 2 of the present invention specifically includes:

[0015] Step 2-1: According to the real-time image of the camera and the position of the circuit board detection frame Cx1, Cy1, Cx2, Cy2, crop and obtain the real-time image Cs within the detection frame;

[0016] Step 2-2: The real-time image Cs in the detection frame is preprocessed in the same manner as the circuit board template image Ct recorded in step 1, wherein the preprocessing method includes one or more of the following: (1) scaling the image size to a specified value using nearest neighbor interpolation or bilinear interpolation;

[0017] (2) Perform blur processing, and perform Gaussian filtering, mean filtering, or median filtering on the image;

[0018] (3) Perform brightness normalization and perform a linear transformation on the image pixels: y = k*x, where k = a / b, a is the target brightness value to be normalized, such as 127, and b is the overall brightness value of the image, that is, the average of all pixel values;

[0019] After preprocessing, the circuit board template image Ctp and the real-time image Csp in the detection frame are obtained;

[0020] Step 2-3: The preprocessed images Ctp and Csp are subjected to feature extraction in the same way. The feature extraction method is one of the following, or the intersection of the results of the following methods, or the union of the results of the following methods:

[0021] (1) Directly perform binarization processing, including adaptive threshold or OTSU threshold,

[0022] (2) Perform Harris corner point detection, then perform binarization on the corner point features to obtain the corner point features.

[0023] (3) Perform Canny edge detection to obtain edge features;

[0024] After feature extraction, the circuit board feature Ctf and the real-time feature Csf in the detection frame are obtained;

[0025] Step 2-4: Use the same pre-trained image classification neural network to extract features from the pre-processed images Ctp and Csp. The neural network model can be selected from one of the following:

[0026] (1) VGG16 or VGG16 with batch normalization, retaining the rest of the model except the last classifier, the classifier contains 3 fully connected layers, and the first two fully connected layers are followed by activation function layers and dropout layers,

[0027] (2) VGG19 or VGG19 with batch normalization, retaining the remaining part of the model except the last classifier, the classifier contains 3 fully connected layers, and the first two fully connected layers are followed by activation function layers and dropout layers;

[0028] (3) ResNet18 or ResNet34 or ResNet50 or ResNet101 or ResNet152 model, retain the remaining part of the model except the last fully connected layer;

[0029] After feature extraction, the circuit board feature Ctd and the real-time feature Csd in the detection frame are obtained;

[0030] Step 2-5: Calculate the similarity Cuf of the features Ctf and Csf extracted by the two image processing methods, and the evaluation index is cosine similarity or intersection ratio; calculate the similarity Cud of the features Ctd and Csd extracted by the two neural network methods, and the evaluation index is cosine similarity or Euclidean distance similarity; take the minimum or maximum value or weighted average of the similarities Cuf and Cud of the features extracted by the two methods according to the actual detection effect to obtain the final similarity Cu. In particular, if one of the weights is 0, it means that this method is not used to extract features.

[0031] Then one of the corresponding steps 2-3 and 2-4 can be omitted;

[0032] Step 2-6: Compare the similarity Cu with the circuit board threshold Cv set in step 1. If Cu≥Cv, it is considered that the board is in the detection frame and the plug-in abnormality detection is started. Otherwise, execute step 2-1 from the beginning.

[0033] The plug-in anomaly detection in steps 2-6 of the present invention specifically includes:

[0034] Step 2-6 (a): After obtaining the real-time image, taking the i-th plug-in as an example, the range of the detection frame Pix1, Piy1, Pix2, Piy2 is expanded to obtain the expanded detection frame, and the actual image Pis of the first plug-in is obtained by cropping according to the real-time image of the camera and the coordinates of the expanded detection frame. The specific method of expanding the detection frame is to reduce the horizontal and vertical coordinates of the upper left point of the rectangular frame, increase the horizontal and vertical coordinates of the lower right point of the rectangular frame, and increase the area of ​​the rectangular frame. For example, the detection frame is changed to Pix1-d, Piy1-d, Pix2+d, Piy2+d, where d>0;

[0035] Execute this method for n plug-ins, expand the detection frame, and crop according to the real-time image of the camera and the coordinates of the expanded detection frame to obtain the actual images P1s, P2s, ..., Pns of each plug-in;

[0036] Step 2-6 (b): Taking the i-th plug-in as an example, the real-time plug-in image Pis and the plug-in template image Pit recorded in step 1 are subjected to the same preprocessing to obtain the preprocessed plug-in actual image Pisp and the preprocessed template image Pitp;

[0037] Preprocessing methods include one or more of the following:

[0038] (1) Perform gamma transformation. For images with overall brightness (including the mean of all pixel values) that are too low or too high, perform gamma transformation with gamma values ​​<1 or >1 respectively to improve contrast.

[0039] (2) Perform blur processing, apply Gaussian filtering, mean filtering, or median filtering to the image to reduce individual differences;

[0040] (3) Performing corrosion processing, corroding the image once or multiple times to highlight the image feature details;

[0041] (4) Remove highlights, filter the overly bright areas (i.e. areas with too high pixel values), and then use Poisson transform to make them transition smoothly with the surrounding areas, thereby reducing the interference of lighting conditions on detection;

[0042] This method is executed for n plug-ins to obtain the actual images P1sp, P2sp, ..., Pnsp and template images P1tp, P2tp, ..., Pntp of the n plug-ins after processing;

[0043] Step 2-6(c): Taking the i-th plug-in as an example, the plug-in template image Pitp processed in step 2-5(b) is rotated k times to obtain k plug-in template images (Pit1, Pit2, ..., Pitk) after rotation, where the values ​​of the rotation degrees t1, t2, ..., tk are evenly distributed according to the skew angle range allowed by the plug-in. When the plug-in is allowed to skew ±10 degrees, k is 4, and t1 to t4 are -10, -5, 5, 10 respectively;

[0044] Execute this method for n plug-ins to obtain n groups of plug-in template graphs (P1t1, P1t2, ..., P1tk), (P2t1, P2t2, ..., P2tk), ..., (Pnt1, Pnt2, ..., Pntk) after rotation.

[0045] Step 2-6(d): Taking the i-th plug-in as an example, the plug-in template image Pitp preprocessed in step 2-6(b) is rotated 180 degrees to obtain the plug-in template image Pitr after 180 degrees rotation; this method is performed on n plug-ins to obtain the plug-in template images P1tr, P2tr, ..., Pntr after 180 degrees rotation;

[0046] Step 2-6(e): Taking the i-th plug-in as an example, calculate the similarity Piu0 between the actual plug-in image Pisp preprocessed in step 2-6(b) and the template image Pitp processed in step 2-6(b). The similarity calculation method is to perform template matching on the two, and the indicator used is one of the normalized cross-correlation or normalized correlation coefficient;

[0047] This method is executed for n plug-ins to obtain the similarities P1u0, P2u0, ... Pnu0 between n plug-ins and the template;

[0048] Step 2-6(f): Taking the i-th plug-in as an example, calculate the similarity Piu1, Piu2, ..., Piuk of the plug-in actual image Pisp preprocessed in step 2-6(b) and the k template images Pit1, Pit2, ..., Pitk rotated in step 2-6(c). The similarity calculation method and index are the same as step 2-6(e);

[0049] Execute this method for n plug-ins to obtain n plug-ins, and the similarity value arrays of each plug-in and k rotation templates are (P1u1, P1u2, ..., P1uk), (P2u1, P2u2, ..., P2uk), ..., (Pnu1, Pnu2, ..., Pnuk);

[0050] Step 2-6(g): Taking the i-th plug-in as an example, calculate the similarity Piur between the actual plug-in image Pisp processed in step 2-6(b) and the template image Pitr processed by 180 degree rotation in step 2-6(d). The similarity calculation method and index are the same as step 2-6(e);

[0051] Execute this method for n plug-ins to obtain the similarity value arrays P1ur, P2ur, ... Pnur between n plug-ins and their reverse templates;

[0052] Step 2-6(h): Taking the i-th plug-in as an example, take the similarity Piu0 obtained in step 2-6(e) and the maximum value Piu among the k similarities Piu1, Piu2, ..., Piuk obtained in step 2-6(f), compare Piu with the plug-in threshold Piv set in step 1, if Piu ≥ Piv, proceed to the subsequent step 2-6(i), otherwise record that the i-th plug-in has a defect anomaly; this step is used to detect the anomalies of missing parts, insufficient parts or wrong parts;

[0053] Process all n plug-ins in this way to determine whether each plug-in has any missing, insufficient or wrong parts.

[0054] Step 2-6(i): Taking the i-th plug-in as an example, compare the similarity Piu0 obtained in step 2-6(e) with the similarity Piur obtained in step 2-6(g). If Piu0+e≥Piur, the first plug-in has no abnormality. Otherwise, it is recorded as a defect abnormality in the first plug-in, and there is a counter-part. e is the allowable value, which can be adjusted according to the actual situation and is generally set to 0. All plug-ins that have no abnormality in step 2-6(h) are processed in this way to determine whether there is a counter-part abnormality. At this point, the detection results of whether each plug-in is abnormal are obtained.

[0055] If all plug-ins are normal, the next round will be executed from step 2-1 to detect subsequent circuit boards. If there is an abnormality in a plug-in, the recorded plug-in abnormal defect result will be output, and the alarm will sound and light alarm or control the conveyor belt to stop as needed.

[0056] The present invention also proposes a two-step circuit board plug-in defect anomaly detection system, which is characterized by being provided with a camera, a host, a display, a conveyor belt, and an alarm. The detection system configured on the host for executing the two-step circuit board plug-in defect anomaly detection method as described above includes:

[0057] Image acquisition module: obtains data through the camera.

[0058] PCB and plug-in template generation module: Generates and stores PCB and plug-in template images, detection frames and thresholds based on box selection for use in the detection process.

[0059] Board in place detection module: to determine whether the circuit board is in place during detection; plug-in abnormality detection module: to determine whether the plug-in has defects or abnormalities after the circuit board is in place;

[0060] Control processing module: If the plug-in is detected to be abnormal, this module will control the sound and light alarm to sound an alarm, or control the conveyor belt to stop according to actual needs.

[0061] Compared with the prior art, the present invention has high detection efficiency and low cost: the system does not need to stop the line for board detection, and does not use deep learning. The detection speed is faster, it can support faster line speed, and can effectively improve the efficiency of the production line. No deep learning is used, so there is no need for an independent graphics card. Non-stop detection saves cylinders, sensors and other devices; good applicability and ease of use: it can cope with various situations such as lighting changes and reflections, is suitable for various plug-ins such as terminals and capacitors, and can detect various abnormal defects including missing parts, wrong parts, multiple parts, and reversed parts. The use of digital image processing algorithms does not require model training, and the template making process is simple; high accuracy: the system algorithm performs various processes on the plug-in, highlights the characteristics of the plug-in, reduces individual differences, and only detects the plug-in within the selection range, ultimately achieving low false alarms and low missed alarms under the premise of detecting various defects. Description of the drawings:

[0062] Attached Figure 1 It is the overall flow chart of the present invention

[0063] Attached Figure 2 It is a flow chart of the detection performed by the present invention.

[0064] Attached Figure 3 It is a working flow chart of the board in place detection module in the present invention.

[0065] Attached Figure 4 It is a workflow diagram of the plug-in anomaly detection module in the present invention. Specific implementation method:

[0066] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0067] As attached Figure 1 And attached Figure 2 As shown, the present invention proposes a two-step circuit board plug-in defect anomaly detection method, which mainly includes the following steps:

[0068] Step 1: Make a template: When the conveyor is stationary, place the circuit board with the correct plug-in, select the circuit board and the plug-in to be detected according to the collected image, obtain the image and frame coordinates, and set thresholds for the circuit board and each plug-in; the image of the circuit board and the plug-in is the template, and the frame coordinates of the circuit board and the plug-in are their respective detection frames. The image, frame coordinates, and thresholds are all stored and used when testing;

[0069] Step 2: Execute the detection: perform a two-step detection of the circuit board plug-in. After acquiring the real-time image, firstly perform the board in place detection according to the generated circuit board template, then perform the plug-in abnormality detection on the plug-in in the detection frame according to the generated plug-in template, and output the detection results of each plug-in; if there is a circuit board plug-in defect, output the abnormal defect information and handle it according to the demand (such as alarm or control the conveyor belt to pause, etc.), the circuit board plug-in abnormal defect includes: missing parts, multiple parts, wrong parts, and reversed parts; otherwise, obtain the latest real-time image again to continue to determine whether the board is in place and detect the next board;

[0070] Example:

[0071] This example proposes a low-cost circuit board plug-in defect anomaly detection system, in which the system equipment includes a camera, a host, a display, a conveyor belt, an alarm (including an alarm light), and a corresponding detection method is configured on the host. The detection system based on the detection method consists of the following modules:

[0072] Image acquisition module: obtains data through the camera.

[0073] PCB and plug-in template generation module: Generates and stores PCB and plug-in template images, detection frames and thresholds based on box selection for use in the detection process.

[0074] Board in place detection module: to determine whether the circuit board is in place during the test. Plug-in abnormality detection module: to determine whether the plug-in has defects or abnormalities after the circuit board is in place.

[0075] Control processing module: If the plug-in is detected to be abnormal, this module will control the sound and light alarm to sound an alarm, or control the conveyor belt to stop according to actual needs.

[0076] In this example, in order to realize the defect detection of circuit board plug-ins, the equipment deployment is first performed, that is, step 1: deploying cameras, host computers, displays, conveyor belts, alarm lights (devices) and other equipment, setting up cameras above the conveyor belts, and obtaining real-time data captured by the cameras, connecting the conveyor belts to the PLC, allowing the conveyor belts to run and accept corresponding control instructions, and connecting alarm lights (devices) as needed, and then performing the following operations:

[0077] Step 2: Make a template:

[0078] Step 2.1: When the conveyor belt is stationary, place the circuit board with the correct plug-in within the camera acquisition range;

[0079] Step 2.2: Use the image acquisition module to collect real-time images with the help of a camera, select the circuit board on the image, record the RGB image of the selection box as the circuit board template image Ct, record the horizontal and vertical coordinates Cx1, Cy1, Cx2, Cy2 of the upper left and lower right of the rectangular selection box and use the box as the circuit board detection box, and set the detection threshold Cv for the circuit board;

[0080] Step 2.3: Keep the circuit board still, and continue to select n plug-ins to be tested in the figure in turn, record the RGB image of each plug-in as the template image P1t, P2t, ..., Pnt, record the horizontal and vertical coordinates of the upper left and lower right of each plug-in rectangular selection box (P1x1, P1y1, P1x2, P1y2), (P2x1, P2y1, P2x2, P2y2), ..., (Pnx1, Pny1, Pnx2, Pny2) and use it as the plug-in detection box, and set the detection threshold P1v, P2v, ... Pnv for each plug-in. Step 2.4: Persistently store the pictures, box coordinates, and threshold data in the above steps 2.2 and 2.3 for use during detection;

[0081] Step 3: Carry out the detection process, and the camera collects real-time pictures;

[0082] Step 4: Perform board in-place detection in the detection process:

[0083] Step 4.1: Get a real-time circuit board image; according to the camera real-time image and the circuit board detection frame position Cx1, Cy1, Cx2, Cy2, crop to obtain the real-time image Cs within the detection frame;

[0084] Step 4.2: Preprocessing: The real-time image Cs in the detection frame and the circuit board template image Ct recorded in step 2.2 are both preprocessed in the same manner, and the preprocessing methods include one or more of the following:

[0085] ① Scaling, using nearest neighbor interpolation or bilinear interpolation to change the image size to the specified value;

[0086] ② Perform blur processing, and perform Gaussian filtering, mean filtering, or median filtering on the image;

[0087] ③ Normalize the brightness and perform linear transformation on the image pixels, y = k*x, where

[0088] k = a / b, where a is the target brightness value to be normalized, such as 127, and b is the overall brightness value of the image, that is, the mean of all pixel values;

[0089] After preprocessing, the circuit board template image Ctp and the real-time image Csp in the detection frame are obtained;

[0090] Step 4.3: Feature extraction: The preprocessed images Ctp and Csp are subjected to feature extraction in the same way. The feature extraction method is one of the following, or the intersection of the results of the following methods, or the union of the results of the following methods:

[0091] ① Directly perform binarization processing, including adaptive threshold or OTSU threshold,

[0092] ② Perform Harris corner point detection, then perform binarization on the corner point features to obtain the corner point features.

[0093] ③Perform Canny edge detection to obtain edge features;

[0094] After feature extraction, the circuit board feature Ctf and the real-time feature Csf in the detection frame are obtained.

[0095] Step 4.4: Calculate similarity; calculate the similarity Cu of the two features Ctf and Csf, and the evaluation index is one of the intersection-over-union ratio, mean square error or cosine similarity;

[0096] Step 4.5: Compare thresholds; compare the similarity Cu with the circuit board threshold Cv set in step 1. If Cu≥Cv, it is considered that the board is in the detection frame and step 5 plug-in abnormality detection is started. Otherwise, step 3 is executed to start the next round of board in place detection;

[0097] Step 5: Perform plug-in anomaly detection in the detection process. The following description takes the first plug-in as an example. The process performs the following processing on each plug-in:

[0098] Step 5.1: Get the plug-in image; expand the range of the detection frame P1x1, P1y1, P1x2, P1y2 to obtain the expanded detection frame. According to the real-time image of the camera and the coordinates of the expanded detection frame, crop and obtain the actual image P1s of the first plug-in. The specific way to expand the detection frame is to reduce the horizontal and vertical coordinates of the upper left point of the rectangular frame, increase the horizontal and vertical coordinates of the lower right point of the rectangular frame, and increase the area of ​​the rectangular frame. For example, the detection frame is changed to P1x1-d, P1y1-d, P1x2+d, P1y2+d, where d>0;

[0099] Step 5.2: Preprocessing: Perform the same preprocessing on the real-time plug-in image P1s and the plug-in template image P1t recorded in step 1 to obtain the preprocessed plug-in actual image P1sp and the preprocessed template image P1tp, wherein the preprocessing method includes one or more of the following:

[0100] ① Perform gamma transformation, for the overall brightness including the mean of all pixel values, too low and too high

[0101] ② Perform gamma transformation on the pictures with gamma values ​​<1 and >1 to improve the contrast; ② Perform blur processing, perform Gaussian filtering, mean filtering or median filtering on the pictures to weaken individual differences;

[0102] ③ Carry out corrosion processing, corrode the image once or multiple times to highlight the image feature details;

[0103] ④ Remove highlights, filter the overly bright areas (i.e. areas with too high pixel values), and then use Poisson transform to make them transition smoothly with the surrounding areas, thus reducing the interference of lighting conditions on detection;

[0104] Step 5.3: Rotation enhancement;

[0105] Step 5.3.1: Rotate the plug-in template image P1tp preprocessed in step 5.2 k times to obtain k rotated plug-in template images (P1t1, P1t2, ..., P1tk), where the values ​​of the rotation degrees t1, t2, ..., tk are evenly distributed according to the skew angle range allowed by the plug-in. For example, if the plug-in is allowed to skew ±10 degrees, k can be 4, and t1 to t4 are -10, -5, 5, 10 respectively;

[0106] Step 5.3.2: Rotate the plug-in template image P1tp preprocessed in step 5.2 by 180 degrees to obtain a plug-in template image P1tr rotated by 180 degrees;

[0107] Step 5.4: Calculate similarity;

[0108] Step 5.4.1: Calculate the similarity P1u0 between the actual plug-in image P1sp preprocessed in step 5.2 and the template image P1tp preprocessed in step 5.2. The similarity calculation method is to perform template matching between the two, and the indicator used is one of the normalized cross-correlation or normalized correlation coefficient;

[0109] Step 5.4.2: Calculate the similarity P1u1, P1u2, ..., P1uk of the actual plug-in image P1sp preprocessed in step 5.2 and the k template images P1t1, P1t2, ..., P1tk rotated in step 5.3.1. The similarity calculation method and index are the same as step 5.4.1;

[0110] Step 5.4.3: Calculate the similarity P1ur between the actual plug-in image P1sp preprocessed in step 5.2 and the template image P1tr rotated 180 degrees in step 5.3.2. The similarity calculation method and index are the same as step 5.4.1;

[0111] Step 5.5: Compare thresholds;

[0112] Step 5.5.1: Take the similarity P1u0 obtained in step 5.4.1 and the maximum value P1u among the k similarities P1u1, P1u2, ..., P1uk obtained in step 5.4.2, and compare P1u with the plug-in threshold P1v set in step 2.3. If P1u ≥ P1v, proceed to the subsequent step 5.5.2, otherwise, record that the plug-in has a defect abnormality; this step is used to detect the abnormality of missing parts, fewer parts or wrong parts; Step 5.5.2: Compare the similarity P1u0 obtained in step 5.4.1 with the similarity P1ur obtained in step 5.4.3. If P1u0+e ≥ P1ur, the plug-in has no abnormality, otherwise, record that the plug-in has a defect abnormality. This step is used to detect the abnormality of missing parts, fewer parts or wrong parts; where e is the allowable value, which can be adjusted according to the actual situation and is generally set to 0;

[0113] If all the plug-ins in this step are normal, the next round of testing will start from step 3 to test the subsequent circuit boards; if any plug-in is abnormal, go to step 6;

[0114] Step 6: Output the plug-in defect results recorded in step 5.5, and give an audible and visual alarm or control the conveyor belt to pause as required.

[0115] Compared with the prior art, the present invention has high detection efficiency and low cost: the system does not need to stop the line for board detection, and does not use deep learning. The detection speed is faster, it can support faster line speed, and can effectively improve the efficiency of the production line. No deep learning is used, so there is no need for an independent graphics card. Non-stop detection saves cylinders, sensors and other devices; good applicability and ease of use: it can cope with various situations such as lighting changes and reflections, is suitable for various plug-ins such as terminals and capacitors, and can detect various abnormal defects including missing parts, wrong parts, multiple parts, and reversed parts. The use of digital image processing algorithms does not require model training, and the template making process is simple; high accuracy: the system algorithm performs various processes on the plug-in, highlights the characteristics of the plug-in, reduces individual differences, and only detects the plug-in within the selection range, ultimately achieving low false alarms and low missed alarms under the premise of detecting various defects.

Claims

1. A two-step circuit board plug-in defect anomaly detection method, characterized in that: The inspection phase performs a two-step inspection of the circuit board and the plug-in, including the following steps: Step 1: Make a template: When the conveyor belt is stationary, place the circuit board with the correct plug-in. According to the collected image, select the circuit board and the plug-in to be detected, obtain the image and frame coordinates, and set thresholds for the circuit board and each plug-in. The image of the circuit board and the plug-in is the template, and the frame coordinates of the circuit board and the plug-in are their respective detection frames. The image, frame coordinates, and threshold are all stored and used for detection. Step 2: Execute detection: Perform a two-step detection of the circuit board plug-in. After obtaining the real-time image, first perform a board in place detection according to the generated circuit board template, and then perform a plug-in abnormality detection on the plug-in in the detection box according to the generated plug-in template, and output each plug-in detection result; if a circuit board plug-in defect occurs, output the abnormal defect information and process it in a targeted manner according to the needs. The targeted processing includes alarming or controlling the conveyor belt to pause. The abnormal defects of the circuit board plug-in include: missing parts, multiple parts, wrong parts, and reversed parts. Otherwise, obtain the latest real-time image again to continue to determine whether the board is in place and detect the next board.

2. The two-step circuit board plug-in defect anomaly detection method according to claim 1, characterized in that: In the step 1, first, it is necessary to place the circuit board with the correct plug-in when the conveyor belt is stationary, collect the RGB picture of the circuit board with the correct plug-in, use the image acquisition module to collect the real-time picture with the help of the camera, select the circuit board on the picture, record the RGB picture of the selection box as the circuit board template image Ct, record the horizontal and vertical coordinates Cx1, Cy1, Cx2, Cy2 of the upper left and lower right of the selection box and use the box as the circuit board detection box, and set the detection threshold Cv for the circuit board; After that, keep the circuit board still, continue to select n plug-ins to be detected in the figure in turn, record the RGB image of each plug-in as the template image P1t, P2t, ..., Pnt, record the horizontal and vertical coordinates of the upper left and lower right of each plug-in rectangular box (P1x1, P1y1, P1x2, P1y2), (P2x1, P2y1, P2x2, P2y2), ..., (Pnx1, Pny1, Pnx2, Pny2) and use it as the plug-in detection box, set the detection threshold P1v, P2v, ... Pnv for each plug-in; finally, the above pictures, box coordinates, and thresholds are all persistently stored for use in detection.

3. The two-step circuit board plug-in defect anomaly detection method according to claim 1, characterized in that: The board in place detection in step 2 specifically includes: Step 2-1: According to the real-time image of the camera and the position of the circuit board detection frame Cx1, Cy1, Cx2, Cy2, crop and obtain the real-time image Cs within the detection frame; Step 2-2: The real-time image Cs in the detection frame is preprocessed in the same manner as the circuit board template image Ct recorded in step 1, and the circuit board template image Ctp and the real-time image Csp in the detection frame are obtained after preprocessing, wherein the preprocessing method includes one or more of the following: Method 1: Scaling, using nearest neighbor interpolation or bilinear interpolation to change the image size to the specified value; Method 2: Perform blur processing, and perform Gaussian filtering, mean filtering, or median filtering on the image; Method 3: Normalize the brightness and perform a linear transformation on the image pixels: y = k*x, where k = a / b, a is the target brightness value to be normalized, such as 127, and b is the overall brightness value of the image, that is, the average value of all pixel values; After preprocessing, the circuit board template image Ctp and the real-time image Csp in the detection frame are obtained; Step 2-3: The preprocessed images Ctp and Csp are subjected to feature extraction in the same way. The feature extraction method is one of the following, or the intersection of the results of the following methods, or the union of the results of the following methods: The first method is to directly perform binarization processing, including adaptive threshold or OTSU threshold. The second method is to perform Harris corner point detection and then perform binarization processing on the corner point features to obtain the corner point features. The third method: perform Canny edge detection to obtain edge features; After feature extraction, the circuit board feature Ctf and the real-time feature Csf in the detection frame are obtained; Step 2-4: Extract features of the preprocessed images Ctp and Csp through the same pre-trained image classification neural network. The neural network model is selected as one of the following: Model 1: VGG16 or VGG16 with batch normalization. Keep the rest of the model except the last classifier. The classifier contains 3 fully connected layers, and the first two fully connected layers are followed by activation function layers and dropout layers. Model 2: VGG19 or VGG19 with batch normalization, retaining the rest of the model except the last classifier. The classifier contains 3 fully connected layers, and the first two fully connected layers are followed by activation function layers and dropout layers. Model 3: ResNet18 or ResNet34 or ResNet50 or ResNet101 or ResNet152 model, retaining the rest of the model except the last fully connected layer; After feature extraction, the circuit board feature Ctd and the real-time feature Csd in the detection frame are obtained; Step 2-5: Calculate the similarity Cuf of the features Ctf and Csf extracted by the two image processing methods, and the evaluation index is cosine similarity or intersection-over-union ratio; calculate the similarity Cud of the features Ctd and Csd extracted by the two neural network methods, and the evaluation index is cosine similarity or Euclidean distance similarity; take the minimum or maximum value or weighted average of the similarities Cuf and Cud of the features extracted by the two methods according to the actual detection effect to obtain the final similarity Cu. In particular, if one of the weights is 0, it means that the feature is not extracted by this method, and one of the corresponding steps 2-3 and 2-4 can be omitted; Step 2-6: Compare the similarity Cu with the circuit board threshold Cv set in step 1. If Cu≥Cv, it is considered that the board is in the detection frame and the plug-in abnormality detection is started. Otherwise, execute step 2-1 from the beginning.

4. The two-step circuit board plug-in defect anomaly detection method according to claim 3 is characterized in that: The plug-in anomaly detection in steps 2-6 specifically includes: Step 2-6 (a): After obtaining the real-time image, taking the i-th plug-in as an example, the range of the detection frame Pix1, Piy1, Pix2, Piy2 is expanded to obtain the expanded detection frame, and the actual image Pis of the first plug-in is obtained by cropping according to the real-time image of the camera and the coordinates of the expanded detection frame. The specific method of expanding the detection frame is to reduce the horizontal and vertical coordinates of the upper left point of the rectangular frame, increase the horizontal and vertical coordinates of the lower right point of the rectangular frame, and increase the area of ​​the rectangular frame. For example, the detection frame is changed to Pix1-d, Piy1-d, Pix2+d, Piy2+d, where d>0; Execute this method for n plug-ins, expand the detection frame, and crop according to the real-time image of the camera and the coordinates of the expanded detection frame to obtain the actual images P1s, P2s, ..., Pns of each plug-in; Step 2-6 (b): Taking the i-th plug-in as an example, the real-time plug-in image Pis and the plug-in template image Pit recorded in step 1 are subjected to the same preprocessing to obtain the preprocessed plug-in actual image Pisp and the preprocessed template image Pitp; Preprocessing methods include one or more of the following: (1) Perform gamma transformation. For images with overall brightness (including the mean of all pixel values) that are too low or too high, perform gamma transformation with gamma values ​​<1 or >1 respectively to improve contrast. (2) Perform blur processing, apply Gaussian filtering, mean filtering, or median filtering to the image to reduce individual differences; (3) Performing corrosion processing, corroding the image once or multiple times to highlight the image feature details; (4) Remove highlights, filter the overly bright areas (i.e. areas with too high pixel values), and then use Poisson transform to make them transition smoothly with the surrounding areas, thereby reducing the interference of lighting conditions on detection; This method is executed for n plug-ins to obtain the actual images P1sp, P2sp, ..., Pnsp and template images P1tp, P2tp, ..., Pntp of the n plug-ins after processing; Step 2-6(c): Taking the i-th plug-in as an example, the plug-in template image Pitp processed in step 2-6(b) is rotated k times to obtain k rotated plug-in template images (Pit1, Pit2, ..., Pitk), where the values ​​of the rotation degrees t1, t2, ..., tk are evenly distributed according to the skew angle range allowed by the plug-in. When the plug-in is allowed to skew ±10 degrees, k is 4, and t1 to t4 are -10, -5, 5, 10 respectively; Execute this method for n plug-ins to obtain n groups of plug-in template images (P1t1, P1t2, ..., P1tk), (P2t1, P2t2, ..., P2tk), ..., (Pnt1, Pnt2, ..., Pntk) after rotation, with each group of k plug-in template images; Step 2-6 (d): Taking the i-th plug-in as an example, perform a 180-degree rotation transformation on the plug-in template image Pitp preprocessed in step 2-6 (b) to obtain a 180-degree rotated plug-in template image Pitr; Execute this method for n plug-ins to obtain each plug-in template image P1tr, P2tr, ..., Pntr after 180-degree rotation; Step 2-6(e): Taking the i-th plug-in as an example, calculate the similarity Piu0 between the actual plug-in image Pisp preprocessed in step 2-6(b) and the template image Pitp processed in step 2-6(b). The similarity calculation method is to perform template matching on the two, and the indicator used is one of the normalized cross-correlation or normalized correlation coefficient; This method is executed for n plug-ins to obtain the similarities P1u0, P2u0, ... Pnu0 between n plug-ins and the template; Step 2-6(f): Taking the i-th plug-in as an example, calculate the similarity Piu1, Piu2, ..., Piuk of the plug-in actual image Pisp preprocessed in step 2-6(b) and the k template images Pit1, Pit2, ..., Pitk rotated in step 2-6(c). The similarity calculation method and index are the same as step 2-6(e); Execute this method for n plug-ins to obtain n plug-ins, and the similarity value arrays of each plug-in and k rotation templates are (P1u1, P1u2, ..., P1uk), (P2u1, P2u2, ..., P2uk), ..., (Pnu1, Pnu2, ..., Pnuk); Step 2-6(g): Taking the i-th plug-in as an example, calculate the similarity Piur between the actual plug-in image Pisp processed in step 2-6(b) and the template image Pitr processed by 180 degree rotation in step 2-6(d). The similarity calculation method and index are the same as step 2-6(e); Execute this method for n plug-ins to obtain the similarity value arrays P1ur, P2ur, ... Pnur between n plug-ins and their reverse templates; Step 2-6(h): Taking the i-th plug-in as an example, take the similarity Piu0 obtained in step 2-6(e) and the maximum value Piu among the k similarities Piu1, Piu2, ..., Piuk obtained in step 2-6(f), compare Piu with the plug-in threshold Piv set in step 1, if Piu ≥ Piv, proceed to the subsequent step 2-6(i), otherwise record the defect anomaly of the first plug-in; this step is used to detect the anomalies of missing parts, insufficient parts or wrong parts; Process all n plug-ins in this way to determine whether each plug-in has any missing, insufficient or wrong parts. Step 2-6(i): Taking the i-th plug-in as an example, compare the similarity Piu0 obtained in step 2-6(e) with the similarity Piur obtained in step 2-6(g). If Piu0+e≥Piur, the first plug-in has no abnormality. Otherwise, it is recorded as a defect abnormality in the first plug-in, and there is a counter-part. e is the allowable value, which can be adjusted according to the actual situation and is generally set to 0. All plug-ins that have no abnormality in step 2-6(h) are processed in this way to determine whether there is a counter-part abnormality. At this point, the detection results of whether each plug-in is abnormal are obtained. If all plug-ins are normal, the next round will be executed from step 2-1 to detect subsequent circuit boards. If there is an abnormality in a plug-in, the recorded plug-in abnormal defect result will be output, and the alarm will sound and light alarm or control the conveyor belt to stop as needed.

5. A two-step circuit board plug-in defect anomaly detection system, characterized in that: A detection system is provided with a camera, a host, a display, a conveyor belt, and an alarm. The detection system is configured on the host to execute the two-step circuit board plug-in defect anomaly detection method as claimed in any one of claims 1 to 4, including: Image acquisition module: obtains data through the camera. PCB and plug-in template generation module: Generates and stores PCB and plug-in template images, detection frames and thresholds based on box selection for use in the detection process. Board in place detection module: determines whether the circuit board is in place during the detection. Plug-in abnormality detection module: used to determine whether the plug-in has defects or abnormalities after the circuit board is in place; Control processing module: If the plug-in is detected to be abnormal, this module will control the sound and light alarm to sound an alarm, or control the conveyor belt to stop according to actual needs.

Images