A mask solder joint defect detection method based on AI and machine vision algorithms

Abstract

The present invention discloses a method for detecting mask solder joint defects based on AI and machine vision algorithms, belonging to the field of mask defect detection. In the deep learning algorithm, the image acquired by the original camera is enhanced through an image enhancement model, and the solder joint ROI area is effectively extracted using an object detection model, and then sent to semantic segmentation and classification models for detecting defects such as solder joint dirt, half solder joints, and knots. In the traditional machine vision algorithm, the self-designed SJDD1 and SJDD2 algorithms are used to process the solder joint ROI image to detect whether there are defects such as burrs, earband overrun, and piercing. Finally, the results of the five algorithms are integrated to obtain a conclusion on whether there are defects in the mask solder joints. Therefore, the algorithm has extremely high robustness and accuracy. The present invention combines the deep learning algorithm with the self-designed traditional machine vision algorithm for effective detection of mask solder joint defects, effectively saving computing power and time consumption.

Description

Technical Field

[0001] The present invention relates to the technical field of mask defect detection, and particularly relates to a method for detecting mask solder joint defects based on AI and machine vision algorithms. Background Art

[0002] In recent years, people's demand for masks has been continuous and increasing day by day. At the same time, in order to effectively respond to virus transmission, the quality requirements for masks are also an important demand indicator, which has promoted mask manufacturers to conduct strict defect detection on masks and eliminate defective masks.

[0003] Solder joints exist in masks of any specification. The solder joint is the area formed by pressing the ear straps onto the formed pieces, which is related to whether the mask can be worn normally and the overall stability during wearing. Therefore, the detection of solder joint defects is an extremely important part of mask defect detection. At present, on the production lines of mask manufacturers, defect detection is generally carried out manually. However, the subjective factors have a greater impact during manual detection, and the standards are difficult to unify. Moreover, manual detection for a long time is prone to visual fatigue, which not only affects the detection results but also has low efficiency. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for detecting mask solder joint defects based on AI and machine vision algorithms to solve the problems in the background art.

[0005] To solve the above technical problems, the present invention provides a method for detecting mask solder joint defects based on AI and machine vision algorithms, including:

[0006] Step 1: Input the Ori image containing the mask directly collected by the camera into the pre-trained deep learning image enhancement and improvement model CAN++ for brightness enhancement and noise reduction processing, and output a processed Proc image with the same size as the Ori image;

[0007] Step 2: Use the pre-trained deep learning object detection model YOLOv5 to identify the solder joint ROI in the Proc image, and obtain the center point, length, and width data of the solder joint ROI;

[0008] Step 3: Based on the recognition results, perform defect detection on the number and spacing of solder joints not conforming to the standard;

[0009] Step 4: Based on the center point, length, and width data of the solder joint ROI obtained in Step 2, extract the solder joint ROI image on the Proc image;

[0010] Step 5: Use the pre-trained deep learning semantic segmentation model Mask-RCNN, the deep learning classification model Inceptionv4, and the designed traditional machine vision algorithm to detect defects in the solder joint ROI images respectively;

[0011] Step 6: Comprehensively judge whether there are defects in the mask solder joints based on the semantic segmentation and classification detection results in Step 5.

[0012] In one implementation, the improvement method of the deep learning image enhancement improvement model CAN++ in Step 1 includes:

[0013] Concatenate the first-layer convolution result of the CAN network model with the fourth-layer convolution result by concat as the input of the fifth-layer convolution;

[0014] Concatenate the second-layer convolution result of the CAN network model with the fifth-layer convolution result by concat as the input of the sixth-layer convolution;

[0015] Concatenate the third-layer convolution result of the CAN network model with the sixth-layer convolution result by concat as the input of the seventh-layer convolution.

[0016] In one implementation, the training process of the deep learning image enhancement improvement model CAN++ in Step 1 includes:

[0017] (a) Fix the camera to collect images of different mask samples. For the same mask sample, collect two images respectively under low exposure time and low ambient brightness, and long exposure time and high ambient brightness, and use them as the comparison between the model input and the model output. Build a dataset for model training by collecting a certain number of paired images;

[0018] (b) Divide the constructed dataset into a training set, a validation set, and a test set according to the ratio of 8:1:1;

[0019] (c) Use the training set to train the image enhancement improvement model CAN++, use the validation set to evaluate the performance of the model during training, and based on the validation set results, see if it is necessary to adjust the hyperparameters of the model training. Then use the test set for testing. When the test results meet the requirements in both the PSNR and SSIM metrics, it passes the test, and finally obtain the deep learning enhancement improvement model CAN++ that meets the requirements.

[0020] In one implementation, the training process of the deep learning object detection model YOLOv5 in Step 2 includes:

[0021] (a) Use the annotation tool labelme to annotate the solder joint targets for the mask Proc images processed by the image enhancement improvement model CAN++, thereby constructing a dataset;

[0022] (b) Divide the constructed dataset into a training set, a validation set, and a test set according to a ratio of 8:1:1;

[0023] (c) Use the training set to train the object detection model YOLOv5, use the validation set to evaluate the performance of the model during training, and based on the results of the validation set, see if it is necessary to adjust the hyperparameters of the model training. Then use the test set for testing. When the test results can accurately identify and locate the solder joint area, it passes the test, and finally obtain the deep learning object detection model YOLOv5 that meets the requirements.

[0024] In one implementation, the defect detection of the number and spacing of solder joints not meeting the requirements based on the recognition results in step 3 includes:

[0025] Process the Proc image based on the deep learning object detection model YOLOv5 to obtain the center point coordinates, length, and width values of all solder joints in the Proc image. Statistically analyze this data to obtain the number of solder joints. At the same time, measure the distance between solder joints based on the center point coordinates of each solder joint to detect whether the distance between solder joints meets the requirements.

[0026] In one implementation, step 5 includes:

[0027] Step 5-1: Perform semantic segmentation on the solder joint ROI image based on the deep learning semantic segmentation model Mask-RCNN to detect whether there is a dirty area, and both the pixel area and confidence of this area meet the preset requirements;

[0028] Step 5-2: Classify the solder joint ROI image based on the deep learning classification model Inceptionv4 to obtain the OK, NG, and OT classification results of the solder joint ROI image; where OK means no defect, NG means there is a defect, and OT means it is occluded and cannot be judged whether there is a defect;

[0029] Step 5-3: Detect burrs and earband exceeding defects on the solder joint ROI image based on the designed traditional machine vision algorithm SJDD1;

[0030] Step 5-4: Detect piercing and dirty defects on the solder joint ROI image based on the designed traditional machine vision algorithm SJDD2.

[0031] In one implementation, the training process of the deep learning semantic segmentation model Mask-RCNN in step 5 includes:

[0032] (a) Collect the solder joint ROI images containing the dirty areas, and use the labeling tool labelme to label the dirty areas on the solder joints, so as to construct a dataset;

[0033] (b) Divide the constructed dataset into a training set, a validation set and a test set according to the ratio of 8:1:1;

[0034] (c) Use the training set to train the semantic segmentation model Mask-RCNN, use the validation set to evaluate the performance of the model during the training process, and based on the results of the validation set, see if it is necessary to adjust the hyperparameters of the model training. Then use the test set for testing. When the test results can accurately find the dirty areas on the solder joints, it passes the test, and finally obtain the deep learning semantic segmentation model Mask-RCNN that meets the requirements.

[0035] In one implementation, the training process of the deep learning classification model Inceptionv4 in step 5 includes:

[0036] (a) Collect the extracted solder joint ROI images, and classify them into three categories: OK, NG, and OT, so as to construct a dataset;

[0037] (b) Divide the constructed dataset into a training set, a validation set and a test set according to the ratio of 8:1:1;

[0038] (c) Use the training set to train the classification model Inceptionv4, use the validation set to evaluate the performance of the model during the training process, and based on the results of the validation set, see if it is necessary to adjust the hyperparameters of the model training. Then use the test set for testing. When the test results can accurately classify the three types of solder joints: OK, NG, and OT, it passes the test, and finally obtain the deep learning classification model Inceptionv4 that meets the requirements.

[0039] In one implementation, the processing process of the traditional machine vision algorithm SJDD1 designed in step 5 for defect detection of the solder joint ROI images includes:

[0040] (a) Separate the three RGB channels of the color solder joint ROI image to obtain three single-channel images, namely ROI_R, ROI_G, and ROI_B;

[0041] (b) Subtract the single-channel images ROI_R and ROI_G to obtain the subtracted single-channel image ROI_SUB;

[0042] (c) Perform threshold segmentation and erosion and dilation operations on the image ROI_SUB to obtain the image ROI_OD;

[0043] (d) Screen the remaining target areas in the image ROI_OD by area to obtain the un-welded earband areas;

[0044] (e) Search for the un-welded earband target OBJ_CENTER that is located at the center of the solder joint ROI area and meets the requirements of both area and rectangularity in a loop based on the coordinates and rectangularity information of the earband areas screened in step (d), and this target is unique;

[0045] (f) Taking the unique earband target OBJ_CENTER screened in step (e) as a reference, search among all the un-welded earband areas screened in step (d). If there is a target OBJ_RIGHT whose X-axis direction is on the right side of the unique earband target OBJ_CENTER and whose area meets the requirements, it is considered that there is a defect in this solder joint of the mask.

[0046] In one implementation, the processing process of the traditional machine vision algorithm SJDD2 designed in step 5 for defect detection of the solder joint ROI image includes:

[0047] (a) Separate the RGB three channels of the color solder joint ROI image to obtain three single-channel images, namely ROI_R, ROI_G, and ROI_B;

[0048] (b) Convert the three single-channel images ROI_R, ROI_G, and ROI_B of RGB into HSV channel images, namely ROI_H, ROI_S, and ROI_V;

[0049] (c) Perform threshold segmentation on the brightness channel image ROI_V to obtain the area targets with lower brightness;

[0050] (d) Perform average threshold screening on each of the area targets with lower brightness obtained in step (c). If the average threshold of the area target is lower than a set screening threshold, it is considered that this area target is a penetrated area and is a blackened penetrated situation; if the average threshold of the area target is higher than the set screening threshold, then perform step (e);

[0051] (e) Extract the regional image on the solder joint ROI image according to the coordinates of the area targets remaining after screening in step (d), and then send the extracted regional image into a pre-trained MLP classifier for processing to determine whether there is a pressing-through situation in this area;

[0052] Based on the results of steps (d) and (e), it is concluded whether there are pressing-through and contamination defects in this solder joint.

[0053] A method for detecting mask solder joint defects based on AI and machine vision algorithms provided by the present invention. In the deep learning algorithm, the image collected by the original camera is enhanced through an image enhancement model, and then the target detection model is used to effectively extract the solder joint ROI area, which is then sent to the semantic segmentation and classification models for detecting defects such as solder joint dirt, half solder joints, and knots. In the traditional machine vision algorithm, the self-designed SJDD1 and SJDD2 algorithms are used to process the solder joint ROI image to detect whether there are defects such as burrs, earband overrun, and piercing. Finally, the results of the five algorithms are comprehensively combined to obtain a conclusion on whether there are defects in the mask solder joints. Therefore, the algorithm has extremely high robustness and accuracy. In addition, the deep learning algorithm is combined with the self-designed traditional machine vision algorithm for effective detection of mask solder joint defects. By using different algorithms to detect different types of solder joint defects, the advantages of the deep learning algorithm for detecting tasks with complex and variable imaging content are exerted, and the advantages of the traditional machine vision algorithm for detecting tasks with relatively fixed shapes are also exerted. The two algorithms are cleverly combined, effectively saving computing power and time. It replaces manual detection of mask defects, greatly reducing labor costs and improving production efficiency at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 FIG. is a flowchart of a method for detecting mask solder joint defects based on AI and machine vision algorithms provided by the present invention.

[0055] Figure 2 FIG. is a schematic diagram of an Ori image including a mask.

[0056] Figure 3 FIG. is a schematic diagram of a Proc image obtained after processing the Ori image by an image enhancement algorithm.

[0057] Figure 4 FIG. is a schematic diagram of identifying the solder joint ROI by a target detection algorithm.

[0058] Figure 5 FIG. is a schematic diagram of extracting the solder joint ROI image.

[0059] Figure 6 FIG. is a schematic diagram of detecting dirt on the solder joint ROI by a semantic segmentation algorithm.

[0060] Figure 7 FIG. is a schematic diagram of detecting burrs and earband overrun on the solder joint ROI by the algorithm SJDD1.

[0061] Figure 8 FIG. is a schematic diagram of detecting piercing on the solder joint ROI by the algorithm SJDD2. DETAILED DESCRIPTION OF THE INVENTION

[0062] The following further elaborates in detail a method for detecting mask solder joint defects based on AI and machine vision algorithms proposed by the present invention in combination with the accompanying drawings and specific embodiments. According to the following description, the advantages and features of the present invention will be clearer. It should be noted that the accompanying drawings are all in a very simplified form and use non-precise scales, only for conveniently and clearly assisting in explaining the purpose of the embodiments of the present invention.

[0063] The present invention provides a method for detecting mask solder joint defects based on AI and machine vision algorithms. The flowchart is as Figure 1 shown, and specifically includes the following steps:

[0064] Step 1: Input the Ori image containing the mask directly collected by the camera into the pre-trained deep learning image enhancement improvement model CAN++ for processing. The model outputs a processed Proc image with the same size as the initial Ori image. The Proc image has enhanced brightness and reduced noise. The Ori image is as Figure 2 shown, and the Proc image is as Figure 3 shown;

[0065] The improvement method of the deep learning image enhancement improvement model CAN++ is specifically as follows:

[0066] The CAN network model consists of a total of 8 convolutional layers. The input and output of the network model are RGB three-channel images with exactly the same size. The first convolutional result of the CAN network model is concatenated with the fourth convolutional result by the concat method as the input of the fifth convolution; the second convolutional result of the CAN network model is concatenated with the fifth convolutional result by the concat method as the input of the sixth convolution; the third convolutional result of the CAN network model is concatenated with the sixth convolutional result by the concat method as the input of the seventh convolution.

[0067] The training process of the deep learning image enhancement improvement model CAN++ is specifically as follows:

[0068] (a) Fix the camera to collect images of different mask samples. For the same mask sample, collect two images respectively at a low exposure time of 2000 us, low ambient brightness and a long exposure time of 40000 us, high ambient brightness, and use them as the comparison between the model input and the model output. Build a dataset for model training by collecting 5000 pairs (i.e., 10000 images) of paired images;

[0069] (b) Divide the constructed dataset into a training set, a validation set, and a test set according to a ratio of 8:1:1;

[0070] (c) Use the training set to train the image enhancement improvement model CAN++. Use the validation set to evaluate the performance of the model during training, and based on the validation set results, see if it is necessary to adjust the hyperparameters of the model training. Then use the test set for testing. When the test results are greater than 0.91 in both PSNR (Peak Signal to Noise Ratio) and SSIM (Structural Similarity) metrics, the test is passed, and finally a CAN++ pre-trained model that meets the requirements is obtained.

[0071] Step 2: Use the pre-trained deep learning object detection model YOLOv5 to identify the solder joint ROI of the image Proc. The effect is as Figure 4 shown, and the data of the center point (x, y), length l, and width w of the solder joint ROI are obtained;

[0072] The training process of the deep learning object detection model YOLOv5 is as follows:

[0073] (a) Use the annotation tool labelme to annotate the solder joint targets (soldered_dot) for a total of 10,000 mask images processed by the image enhancement improvement model CAN++, thus constructing a dataset;

[0074] (b) Divide the constructed dataset into a training set, a validation set, and a test set according to a ratio of 8:1:1;

[0075] (c) Use the training set to train the object detection model YOLOv5. Use the validation set to evaluate the performance of the model during training, and based on the validation set results, see if it is necessary to adjust the hyperparameters of the model training. Then use the test set for testing. When the test results can accurately identify and locate the solder joint area, that is, when the IOU (Intersection over Union) value meets the threshold requirements, the test is passed, and finally a YOLOv5 pre-trained model that meets the requirements is obtained.

[0076] Step 3: Perform defect detection for the number and spacing of solder joints that do not match based on the recognition results. First, process the image Proc based on the object detection model YOLOv5, and the center point coordinates of all solder joints in the image Proc can be obtained as {(x0, y0), (x1, y1),...., (x n , y n )} and lengths {l0, l1,...., l n} and widths {w0, w1,...., w n} value. By counting this data, the number of solder joints, i.e., the n value, can be obtained. At the same time, if there are only 2 detected solder joint targets, i.e., n = 1, it means that the number of solder joints is correct. Then, the distance between the solder joints can be measured based on the center point coordinates of the two solder joints. The calculation formula is Check whether Distances meet the requirements.

[0077] Step 4: Based on the center point (x, y) of the solder joint ROI, the length l, and the width w data obtained in Step 2, extract the solder joint ROI image on the image Proc. The extraction effect is as Figure 5 shown. To accurately detect whether the ear straps welded on the solder joints exceed the solder joint area, and the exceeded length and area, when extracting the solder joint ROI image, the length and width values of the extraction area can be appropriately increased. The length can be increased by 30 pixel values, and the width can be increased by 20 pixel values;

[0078] Step 5: Use the pre-trained deep learning semantic segmentation model Mask-RCNN, classification model Inceptionv4, and the designed traditional machine vision algorithm to detect defects in the solder joint ROI image:

[0079] (5-1) Based on the deep learning semantic segmentation model Mask-RCNN, perform semantic segmentation on the solder joint ROI image. When it is detected that there is a dirty area and the pixel area of this area is greater than 20 pixels and the average confidence level is greater than 0.85, this area is considered as the solder joint dirty area. The semantic segmentation effect is shown in Figure 6;

[0080] The training process of the deep learning semantic segmentation model Mask-RCNN is as follows:

[0081] (a) Collect 5000 solder joint ROI images containing dirty areas, and use the annotation tool labelme to annotate the dirty areas on the solder joints, thereby constructing a dataset;

[0082] (b) Divide the constructed dataset into a training set, a validation set, and a test set according to a ratio of 8:1:1;

[0083] (c) Use the training set to train the semantic segmentation model Mask-RCNN, use the validation set to evaluate the performance of the model during the training process, and based on the validation set results, see if it is necessary to adjust the hyperparameters of the model training. Then use the test set for testing. When the test results can accurately find the dirty areas on the solder joints, it passes the test, and finally obtain a Mask-RCNN pre-trained model that meets the requirements.

[0084] (5-2) The classification model Inceptionv4 based on deep learning classifies the solder joint ROI images to obtain the classification results of OK, NG, and OT for the solder joint ROI. Among them, OK represents a solder joint without defects, NG represents solder joints with defects such as half solder joints, crescent solder joints, dot deviation, white solder joints, knotting, and ultrasonic pressing of solder joints, and OT represents that the solder joint is blocked and it is impossible to determine whether there are defects;

[0085] The training process of the deep learning classification model Inceptionv4 is as follows:

[0086] (a) Collect the extracted solder joint ROI images and classify them into three categories: OK, NG, and OT. Among them, OK is a solder joint without defects, NG is a solder joint with defects, and OT is a solder joint that is blocked and it is impossible to determine whether there are defects, so as to construct a data set;

[0087] (b) Divide the constructed data set into a training set, a validation set, and a test set according to the ratio of 8:1:1;

[0088] (c) Use the training set to train the classification model Inceptionv4, use the validation set to evaluate the performance of the model during the training process, and based on the validation set results, see if it is necessary to adjust the hyperparameters of the model training. Then use the test set for testing. When the test results can classify the three types of solder joints OK, NG, and OT with an accuracy rate of 99%, it passes the test, and finally obtain the Inceptionv4 pre-training model that meets the requirements.

[0089] (5-3) Detect defects such as burrs and earband overrun on the solder joint ROI images based on the designed traditional machine vision algorithm SJDD1;

[0090] The processing process of the designed traditional machine vision algorithm SJDD1 for detecting defects such as burrs and earband overrun on the solder joint ROI images is as follows:

[0091] (a) Separate the three RGB channels of the color solder joint ROI image to obtain three single-channel images, namely ROI_R, ROI_G, and ROI_B;

[0092] (b) Subtract the single-channel images ROI_R and ROI_G to obtain the subtracted single-channel image ROI_SUB;

[0093] (c) Perform threshold segmentation and erosion and dilation operations on the image ROI_SUB with a range of (77, 255) to obtain the image ROI_OD;

[0094] (d) Screen the remaining target areas in the image ROI_OD according to the minimum area of 400 pixel values to obtain the earband area that has not been soldered, such asFigure 7 As shown in the figure, two target regions are selected;

[0095] (e) Search for the target selected in step (d) cyclically according to its coordinates and rectangularity in the image, and find the un-welded earband target OBJ_CENTER located at the center of the solder joint ROI region, and its area and rectangularity both meet certain requirements. At the same time, this target is unique. Specifically, as shown in Figure 7 the region surrounded by the left periphery in the figure;

[0096] (f) Take the unique target OBJ_CENTER selected in step (e) as the reference, and search among all the targets selected in step (d). If there is a target OBJ_RIGHT whose X-axis direction is on the right side of OBJ_CENTER and whose area meets certain requirements. Specifically, as shown in Figure 7 the region surrounded by the right periphery in the figure, it is considered that there are burrs and the earband exceeds the defect in this solder joint of the mask.

[0097] (5-4) Detect defects such as penetration and dirt on the solder joint ROI image based on the designed traditional machine vision algorithm SJDD2;

[0098] The processing process of detecting penetration and dirt defects on the solder joint ROI image by the designed traditional machine vision algorithm SJDD2 is as follows:

[0099] (a) Separate the three RGB channels of the color solder joint ROI image to obtain three single-channel images, namely ROI_R, ROI_G, and ROI_B;

[0100] (b) Convert the three single-channel images ROI_R, ROI_G, and ROI_B of RGB into HSV channel images, namely ROI_H, ROI_S, and ROI_V;

[0101] (c) Perform threshold segmentation on the brightness channel image ROI_V with a range of (0, 91) to obtain the region target with lower brightness;

[0102] (d) Perform average threshold screening on each of the region targets obtained in step (c). If the average threshold of the region target is lower than a set screening threshold, it is considered that the region target is a penetrated area and is a blackened penetration situation. Specifically, as shown in Figure 8 the figure. If the average threshold of the region target is higher than the set screening threshold, then execute step (e);

[0103] (e) Extract the regional image on the solder joint ROI image according to the coordinates of the remaining region targets screened in step (d), and then send the extracted regional image to a pre-trained MLP classifier for processing to determine whether there is a penetration situation in this region;

[0104] (f) Combine the results of steps (d) and (e) to determine whether there are any situations such as penetration or contamination at the solder joint.

[0105] (6) Combine the results of the semantic segmentation, classification and other detection algorithms in steps three and five to determine whether there are defects in the mask solder joints, and output the solder joint defects detected by each algorithm.

[0106] The above description is only a description of the preferred embodiments of the present invention and does not limit the scope of the present invention in any way. Any changes or modifications made by those of ordinary skill in the art of the present invention based on the above disclosure shall fall within the scope of protection of the claims.

Claims

1. A method for detecting mask solder joint defects based on AI and machine vision algorithms, characterized in that, Including: Step 1: Input the Ori image containing masks directly collected by the camera into the pre-trained deep learning image enhancement and improvement model CAN++ for brightness enhancement and noise reduction processing, and output a processed Proc image with the same size as the Ori image; Step 2: Use the pre-trained deep learning object detection model YOLOv5 to identify the solder joint ROI in the Proc image, and obtain the center point, length, and width data of the solder joint ROI; Step 3: Perform defect detection on the number and spacing of solder joints based on the recognition results; Step 4: Based on the center point, length, and width data of the solder joint ROI obtained in Step 2, extract the solder joint ROI image on the Proc image; Step 5: Use the pre-trained deep learning semantic segmentation model Mask-RCNN, deep learning classification model Inception v4, and the designed traditional machine vision algorithm to perform defect detection on the solder joint ROI image, including detecting burrs and earband overrun defects on the solder joint ROI image based on the designed traditional machine vision algorithm SJDD1 and detecting penetration defects on the solder joint ROI image based on the designed traditional machine vision algorithm SJDD2; Step 6: Comprehensively judge whether there are defects in the mask solder joints based on the semantic segmentation and classification detection results in Step 5; The deep learning image enhancement and improvement model CAN++ includes: Concatenate the first-layer convolution result of the CAN network model with the fourth-layer convolution result by concat as the input of the fifth-layer convolution; Concatenate the second-layer convolution result of the CAN network model with the fifth-layer convolution result by concat as the input of the sixth-layer convolution; Concatenate the third-layer convolution result of the CAN network model with the sixth-layer convolution result by concat as the input of the seventh-layer convolution; The processing process of the designed traditional machine vision algorithm SJDD1 for defect detection on the solder joint ROI image includes: (a) Separate the three RGB channels of the color solder joint ROI image to obtain three single-channel images, namely ROI_R, ROI_G, and ROI_B; (b) Subtract the single-channel images ROI_R and ROI_G to obtain the subtracted single-channel image ROI_SUB; (c) Perform threshold segmentation and erosion and dilation operations on the image ROI_SUB to obtain the image ROI_OD; (d) Screen the image ROI_OD according to the area to obtain the un-soldered earband area; (e) Circularly search for the earband area screened in step (d) according to its coordinates and rectangularity information in the image to find the un-soldered earband target OBJ_CENTER located at the center of the solder joint ROI area and meeting the requirements of area and rectangularity, and this target is unique; (f)Taking the only earband target OBJ_CENTER selected in step (e) as a benchmark, search among all the un-welded earband areas selected in step (d). If there is a target OBJ_RIGHT whose X-axis direction is on the right side of the only earband target OBJ_CENTER and whose area meets the requirements, it is considered that there are burrs and the earband exceeds the defect in this solder joint of the mask. The processing procedure for defect detection of the solder joint ROI image by the traditional machine vision algorithm SJDD2 based on the design includes: (a)Separate the RGB three channels of the color solder joint ROI image to obtain three single-channel images, namely ROI_R, ROI_G, and ROI_B. (b)Convert the three single-channel images ROI_R, ROI_G, and ROI_B of RGB into HSV channel images, namely ROI_H, ROI_S, and ROI_V. (c)Perform threshold segmentation on the brightness channel image ROI_V to obtain the area target with lower brightness. (d)Perform average threshold screening on each of the area targets with lower brightness obtained in step (c). If the average threshold of the area target is lower than a set screening threshold, it is considered that the area target is a penetrated area and is a blackened penetration situation; if the average threshold of the area target is higher than the set screening threshold, then execute step (e). (e)Extract the area image on the solder joint ROI image according to the coordinates of the area targets remaining after screening in step (d), and then send the extracted area image to the pre-trained MLP classifier for processing to determine whether there is a penetration situation in this area. Based on the results of steps (d) and (e), it is determined whether there is a penetration defect in this solder joint.

2. The method for detecting mask solder joint defects based on AI and machine vision algorithms according to claim 1, characterized in that, The training process of the deep learning image enhancement improvement model CAN++ in step 1 includes: (a)Fix the camera to collect images of different mask samples. For the same mask sample, collect two images respectively under low exposure time and low ambient brightness and under long exposure time and high ambient brightness, which are used as the comparison between the model input and the model output. Build a dataset for model training by collecting a certain number of paired images. (b)Divide the constructed dataset into a training set, a validation set, and a test set according to the ratio of 8:1:

1. (c)Use the training set to train the image enhancement improvement model CAN++. Use the validation set to evaluate the performance of the model during the training process, and based on the results of the validation set, see if it is necessary to adjust the hyperparameters of the model training. Then use the test set for testing. When the test results meet the requirements in both the PSNR and SSIM indicators, it passes the test, and finally obtain the deep learning enhancement improvement model CAN++ that meets the requirements.

3. The method for detecting mask solder joint defects based on AI and machine vision algorithms according to claim 2, characterized in that, The training process of the deep learning object detection model YOLOv5 in step 2 includes: (a)Use the annotation tool labelme to annotate the solder joint targets of the mask Proc image processed by the image enhancement improvement model CAN++ to build a dataset. (b)Divide the constructed dataset into a training set, a validation set, and a test set according to the ratio of 8:1:

1. (c)Train the object detection model YOLOv5 using the training set, use the validation set to evaluate the performance of the model during training, and based on the results of the validation set, see if it is necessary to adjust the hyperparameters of the model training. Then use the test set for testing. When the test results can accurately identify and locate the solder joint area, it passes the test, and finally obtain the deep learning object detection model YOLOv5 that meets the requirements.

4. The method for detecting mask solder joint defects based on AI and machine vision algorithms according to claim 3, wherein, The defect detection of the number and spacing of solder joints not meeting the requirements based on the recognition results in step 3 includes: Process the Proc image based on the deep learning object detection model YOLOv5 to obtain the center point coordinates, length, and width values of all solder joints in the Proc image. Statistically analyze this data to obtain the number of solder joints. At the same time, measure the distance between solder joints based on the center point coordinates of each solder joint to detect whether the distance between solder joints meets the requirements.

5. The method for detecting mask solder joint defects based on AI and machine vision algorithms according to claim 4, wherein, Step 5 includes: Step 5-1: Perform semantic segmentation on the solder joint ROI image based on the deep learning semantic segmentation model Mask-RCNN to detect whether there is a dirty area, and both the pixel area and confidence of this area meet the preset requirements. Step 5-2: Classify the solder joint ROI image based on the deep learning classification model Inception v4 to obtain the OK, NG, OT classification results of the solder joint ROI image; where OK means no defect, NG means there is a defect, and OT means it is occluded and cannot determine whether there is a defect.

6. The method for detecting mask solder joint defects based on AI and machine vision algorithms according to claim 5, wherein, The training process of the deep learning semantic segmentation model Mask-RCNN in step 5 includes: (a)Collect solder joint ROI images containing dirty areas, and use the annotation tool labelme to annotate the dirty areas on the solder joints, thus constructing a dataset. (b)Divide the constructed dataset into a training set, a validation set, and a test set according to the ratio of 8:1:

1. (c)Train the semantic segmentation model Mask-RCNN using the training set, use the validation set to evaluate the performance of the model during training, and based on the results of the validation set, see if it is necessary to adjust the hyperparameters of the model training. Then use the test set for testing. When the test results can accurately find the dirty areas on the solder joints, it passes the test, and finally obtain the deep learning semantic segmentation model Mask-RCNN that meets the requirements.

7. The method for detecting mask solder joint defects based on AI and machine vision algorithms according to claim 6, wherein, The training process of the deep learning classification model Inception v4 in step 5 includes: (a)Collect the extracted solder joint ROI images and classify them into three categories: OK, NG, and OT, thus constructing a dataset. (b)Divide the constructed dataset into a training set, a validation set, and a test set according to the ratio of 8:1:

1. (c)Train the classification model Inception v4 using the training set, use the validation set to evaluate the performance of the model during training, and based on the results of the validation set, see if it is necessary to adjust the hyperparameters of the model training. Then use the test set for testing. When the test results can accurately classify the three types of solder joints: OK, NG, and OT, it passes the test, and finally obtain the deep learning classification model Inception v4 that meets the requirements.

Images