PCBA defect detection data set manufacturing method based on deep learning
Through steps such as screening, manufacturing defects, image fusion and labeling preprocessing, the problem of making PCBA defect detection data sets in the prior art has been solved, the recognition of deep learning models and data set quality are improved, and the production process is simplified.
Patent Information
- Application Number
- CN202410529885.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-29
- Publication Date
- 2024-11-08
AI Technical Summary
The prior art is difficult to effectively produce high-quality PCBA defect detection data sets, resulting in insufficient recognition of deep learning models in defect detection, and the data set production process requires a lot of manpower and time investment.
A method for producing PCBA defect detection data set based on deep learning, including screening PCBA boards, artificial defects, sub-region shooting, image fusion and labeling preprocessing, through constructing weight factors, bilateral pyramid decomposition, Gaussian and Las Vegas pyramids Fusion and manual PS steps to generate high-quality image datasets.
It improves the recognition of PCBA defect detection by deep learning models, simplifies the data set production process, reduces manpower and time investment, and ensures the quality and diversity of the data set.
Smart Images

Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] The invention relates to the field of image processing, and in particular to an improved image fusion method for improving the image quality of a defective PCBA board. Background Art
[0002] Printed Circuit Board (PCB), as a carrier for installing electronic components and connecting circuits, is an important component of various types of electronic products. The development level of its production and manufacturing has played a fundamental role in the technological progress of industries such as computers, consumer electronics, communication equipment, automotive electronics and industrial equipment. As PCBA is accelerating towards miniaturization, thinness and density, manual visual inspection and online electrical inspection are difficult to meet the needs of the electronic circuit manufacturing industry for PCBA quality control due to limited sight and electrical contact.
[0003] In recent years, deep learning algorithms represented by Convolutional Neural Network (CNN) have achieved great success in computer vision fields such as face recognition, medical imaging and autonomous driving, attracting a large number of researchers to try to combine deep learning algorithms with visual defect detection, and have achieved gratifying results and started to promote their application. Deep learning has the advantages of automatic feature extraction, end-to-end and strong portability, and is suitable for PCBA defect detection.
[0004] Before building a PCBA defect detection system based on deep learning, a high-quality dataset is essential. However, due to the wide variety of PCBA defects and the different characteristics of each defect, it is not easy to make a realistic and diverse dataset. It requires a lot of manpower and time to collect, clean and annotate a large number of data samples. At the same time, in the process of making a dataset, it is necessary to have a deep understanding of the source and characteristics of the data, and to ensure the balance, diversity and representativeness of the dataset so that the model can perform well in various situations. Therefore, a good dataset is crucial for PCBA defect detection. Summary of the invention
[0005] Purpose of the invention: To provide a reasonable and effective PCBA dataset preparation solution to improve the recognition of PCBA defect detection by deep learning models.
[0006] Technical solution: The method for preparing a PCBA defect detection data set based on deep learning described in the present invention includes: (1) Conditional screening of PCBA boards provided by the factory; (2) Artificially create defects on the screened boards; (3) Photograph defective PCBA boards in different areas; (4) Take photos, perform algorithm fusion, and generate high-quality images; (5) Perform labeling preprocessing on the improved images; (6) Label the images using LabelImg software.
[0007] Furthermore, in step (1), defective PCBA boards are screened; the scrapped PCBA (without main control chip) samples provided by the factory are screened, and the selected samples must meet the following characteristics: 1. A large number of component types (resistors, capacitors, diodes, inductors, transistors, etc.) and no damaged components; 2. A clean background (no stains, scratches, scale); 3. No exposed copper and no tin on the solder joints; 4. The board size is appropriate, not too large or too small, and must meet the machine shooting area.
[0008] Furthermore, in step (2), artificial defects are made on the screened boards: artificial defects are made on the screened PCBA boards, and the defect standards are based on the samples provided by the factory (machine errors cause various defects on the PCBA boards). During the manufacturing process, we first preheat the heating table to 285 degrees (preheat 265 degrees for defective soldering), then use tweezers to place the board on the heating table until the solder paste on the board melts, and then use tweezers to divide the board into 8 areas and make seven defects in each area. Different defects have different process requirements as follows; 1. Tombstone: Tombstones are divided into single-leg tombstones, side tombstones and oblique tombstones. I take the single-leg tombstone as an example, and the other types are the same. The main difference is that when making a side tombstone, we need to lift the middle of the component and stay for 5s, and for the oblique tombstone, we need to lift the diagonal and stay for 5s; during the production of the single-leg tombstone, we need to fix the board with tweezers, and then use another pair of tweezers to pry one leg of the component and stay for 3s, and finally form a single-leg 10~90 degree tombstone. 2. Missing parts: We can first check whether there are enough missing parts on the board. If not, we have to create missing parts manually. First, we need to use tweezers to remove the components on the board in a 90-degree direction and keep the solder joints. The solder joints are the types of missing parts. But when removing the components, do not drag them at will, which will cause the solder joints to be connected. Such solder joints cannot be considered as missing parts. 3. Tinning: Tinning is different from the other three defects. We need to apply the solder paste evenly first (use as little solder paste as possible, and the tinning generated later will be more connected). Near standard) between two components or between two solder joints, and then put it on a heating table preheated to 265 degrees, until the solder becomes a solder point connected to the components and the solder joints at both ends, and then take the board off the heating table and put it aside to cool. Finally, a defect of bridging tin will be generated between the components and the solder joints coated with solder paste. 4. Offset: We need to use tweezers to move the components to deviate from the original pads. Components with small deviation angles (that is, slight deviation) are mainly produced by us, which are more in line with the actual situation and meet the conditions of machine manufacturing defects.
[0009] Furthermore, in step (4), the image fusion algorithm is used. The image fusion method is to fuse two images of the same area with different exposure and focal lengths. The detailed implementation steps are as follows: Construct weight factors: Calculate the focal length factor, exposure factor and color factor according to the two input images and output them;
[0010] In the formula, B is the focal length factor, α and β are adjustment factors, which are set to 0.4 and 0.6 respectively, k is the number of input images; D is the entropy factor, J and K are the length and width of the 3x3 moving window rectangle, is an arbitrary element of the moving window, , , They represent the average grayscale value, the maximum grayscale value and the minimum grayscale value in the moving window respectively. E is the energy factor, i is a grayscale value from 0 to 255, which represents the probability of the grayscale value occurring in all grayscale values; The formula for calculating the exposure factor is as follows:
[0011] in the formula is the exposure factor, k is the number of input images, is the grayscale value of the input k-th image, Here, 0.2 is the Gaussian function parameter; u is the adaptive exposure adjustment factor. The tuning parameter is set to 0.5. is the grayscale average of the kth image; the calculation formula for the color saturation factor is as follows: in is the color saturation factor, , , They correspond to the three channel values of R, G, and B of the kth image respectively.
[0012] Weight factor fusion: For constructing the input of the weight factor module , , , we need to fuse and normalize; the specific formula is as follows: Where k represents the number of different images, Represents the weight parameter of each image, is the normalized weight map.
[0013] Weighted bilateral pyramid decomposition: for multiple input weight graphs Recorded as , then we need to bring in the weight optimization formula to get the image of the first layer after refinement , and then Cycle through bilateral filtering and downsampling m times to get an m-layer image , the detailed calculation formula is as follows;
[0014]
[0015]
[0016]
[0017] in For the k-th image, there are m layers of weight optimization graph. is a 15x15 sliding window, and are the arbitrary value and mean of the guide graph under the sliding window, See formula 10 for the solution, See formula 11 for the solution, where , , They are the values of the RGB channels of the k-th original image; l and e represent the length and width of the original image; Gaussian and Laplace pyramid decomposition: construct Gaussian pyramid and Laplace pyramid for the two original images, denoted as and as well as and (See attached for solution steps. Figure 5 ), the relationship between n and m is as follows: Where m and n represent the number of layers of Gaussian pyramid and Laplacian pyramid respectively.
[0018] Gaussian and Laplace pyramid fusion: For the weighted bilateral pyramid decomposition module, (Since the number of input original images is two, we take and ) and the Gaussian and Laplace pyramid decomposition modules , , Do the product and you get and The specific formula is as follows:
[0019] Laplace detail enhancement: high-frequency details passed in by the Gaussian and Laplace pyramid fusion modules Gaussian Pyramid , first Grayscale processing is performed , and then the high frequency details and Substitute into the Laplace detail enhancement formula to get , the solution formula is as follows:
[0020] Laplace reconstruction: passed in by the Gaussian and Laplace pyramid fusion module First, Gaussian blur and upsampling are performed to obtain And add the first layer of high-frequency details passed in by the Laplace detail enhancement module , get the image after the first layer fusion , and then As the previous step Image, repeat the above operation n times (see formula 13 for details), and finally get the fused image H(x,y).
[0021] Furthermore, in step (5), artificial manufacturing defects are performed on the screened boards: image labeling preprocessing. The high-quality images after image fusion need to be manually PSed. The specific steps are as follows: First, use the drawing software to open the image and fill the suspected feature area with the background color 1. The pin area of the chip has been removed from the board 2. Too many missing parts features are reserved on the board 3. The area with complex background (exposed copper, grooves, dirt and scratches, etc.) 4. The tin marks and defect features on the board caused by the manufacturing defect process. After manual PS, the image needs to be resized to 1080x1280 resolution and sorted for easy training. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 It is a processing flow chart of the invention provided by the embodiment of the present application; Figure 2 This is a flow chart of the image fusion algorithm in the embodiment of the present application; Figure 3 These are four defect original images in the embodiments of the present application; Figure 4 This is the defective PS image in the embodiment of the present application.
[0023] Figure 5 This is the Gaussian pyramid and Laplacian pyramid solution process in the embodiment of the present application (decomposing two layers as an example). DETAILED DESCRIPTION
[0024] The present invention will be further described below in conjunction with the accompanying drawings.
[0025] (1) Screening of defective PCBA boards. The scrapped PCBA (without main control chip) samples provided by the factory are screened. The selected samples must meet the following characteristics: 1. A large number of component types (resistors, capacitors, diodes, inductors, transistors, etc.) and no damaged components; 2. Clean background (no stains, scratches, scale); 3. No exposed copper and no tin on solder joints; 4. The board size is appropriate, not too large or too small, and must meet the machine shooting area.
[0026] (2) Step 2: Create PCBA defects. For the screened PCBA boards, artificial defects are created. The defect standards are based on the samples provided by the factory (machine errors cause various defects on the PCBA boards). During the manufacturing process, we first preheat the heating table to 285 degrees (preheat 265 degrees for defective soldering), then use tweezers to place the board on the heating table until the solder paste on the board melts, and then use tweezers to divide it into 8 areas and create seven defects in each area. Different defects have different process requirements as follows (see attached for defective samples). Figure 3 ). 1. Tombstone: Tombstones are divided into single-leg tombstones, side tombstones and oblique tombstones. I take the single-leg tombstone as an example, and the other types are the same. The main difference is that when we use the side tombstone, we need to lift the middle of the component and stay for 5s. For the oblique tombstone, we need to lift the diagonal and stay for 5s. In the production process of the single-leg tombstone, we need to fix the board with tweezers, and then use another pair of tweezers to pry one leg of the component and stay for 3s, and finally form a single-leg 10~90 degree tombstone. 2. Missing parts: We can first see if there are enough missing parts on the board. If not, we have to artificially create missing parts. First, we need to use tweezers to remove the components on the board in a 90-degree direction and retain the solder joints. The solder joints are the types of missing parts, but when removing the components, do not drag them at will, which will cause the solder joints to be connected. Such solder joints cannot be used as types of missing parts. 3. Tin bridging: Tin bridging is different from the other three defects. We need to first apply the solder paste evenly (use as little solder paste as possible, the tin bridging generated later will be closer to the standard) between two components or two solder joints, and then place them on a heating table preheated to 265 degrees until the solder becomes a solder point connected to the components and the solder joints. Then take the board off the heating table and set it aside to cool. Finally, a tin bridging defect will be generated between the components and solder joints coated with solder paste. 4. Offset: We need to use tweezers to move the components to deviate from the original pads. We mainly produce components with small deviation angles (that is, slight deviation), which are more in line with the actual situation and meet the conditions of machine manufacturing defects.
[0027] (3) Machine photography: We need to control the PLC programmer to move the machine to achieve board positioning. The specific operations are as follows: First, use the PLC control system to move the machine's XY axis to the camera shooting area. If the image quality on the display screen does not meet the defect production requirements, it is necessary to move the XY axis again until a suitable area is found and the coordinates of the area are located and counted. This operation is repeated 8 times until the entire PCBA board is traversed. After positioning the board, we need to take photos of the finished defective board in 8 areas and store them in the computer. This operation is automatically completed by the machine. We only need to follow a fixed process. After each photo is taken, the defective board is replaced to facilitate the next assembly line operation of the machine.
[0028] (4) Image fusion. The image fusion method is to fuse two images of the same area with different exposure and focal length. The specific process is as follows: Figure 2 As shown, the detailed implementation steps are as follows: Construct weight factors: Calculate the focal length factor, exposure factor and color factor according to the two input images and output them; The calculation formula for the focal length factor is as follows:
[0029] In formula (1), B is the focal length factor, α and β are adjustment factors, which are set to 0.4 and 0.6 respectively, and k is the number of input images. D is the entropy factor, J and K are the length and width of the 3x3 moving window rectangle, and C(x, y) is any element of the moving window. , , They represent the average grayscale value, the maximum grayscale value and the minimum grayscale value in the moving window respectively; E is the energy factor, i is a grayscale value from 0 to 255, indicating the probability of the grayscale value occurring in all grayscale values; The formula for calculating the exposure factor is as follows:
[0030] in the formula is the exposure factor, k is the number of input images, is the grayscale value of the input k-th image, Here, 0.2 is the Gaussian function parameter; u is the adaptive exposure adjustment factor. The tuning parameter is set to 0.5. is the grayscale average of the k-th image; The calculation formula for the color saturation factor is as follows:
[0031] in is the color saturation factor, , , They correspond to the three channel values of R, G, and B of the kth image respectively; Weight factor fusion: For constructing the input of the weight factor module , , , we need to fuse and normalize The specific formula is as follows;
[0032] Where k represents the number of different images, Represents the weight parameter of each image, is the normalized weight map; weighted bilateral pyramid decomposition: for multiple input weight maps Recorded as , then we need to bring in the weight optimization formula 9 to get the image of the first layer after refinement , and then Cycle through bilateral filtering and downsampling m times to get an m-layer image The detailed calculation formula is as follows:
[0033] in For the k-th image, there are m layers of weight optimization graph. is a 15x15 sliding window, and are the arbitrary value and mean of the guide graph under the sliding window, See formula 10 for the solution, See formula 11 for the solution, where , , are the values of the RGB channels of the kth original image. In formula 12, l and e represent the length and width of the original image.
[0034] Gaussian and Laplace pyramid decomposition: construct Gaussian pyramid and Laplace pyramid for the two original images, denoted as and as well as and (See attached for solution steps. Figure 5 ), the relationship between n and m is as follows:
[0035] Where m and n represent the number of layers of Gaussian pyramid and Laplacian pyramid respectively; Gaussian and Laplace pyramid fusion: For the weighted bilateral pyramid decomposition module, (Since the number of input original images is two, we take and ) and the Gaussian and Laplace pyramid decomposition modules , , , Do the product and you get and The specific formula is as follows:
[0036] Laplace detail enhancement: high-frequency details passed in by the Gaussian and Laplace pyramid fusion modules Gaussian Pyramid , first Grayscale processing is performed (See formula 11 for grayscale processing), and then the high-frequency details and Substitute into the Laplace detail enhancement formula to get , the solution formula is as follows:
[0037] Laplace reconstruction: passed in by the Gaussian and Laplace pyramid fusion module First, Gaussian blur and upsampling are performed to obtain And add the first layer of high-frequency details passed in by the Laplace detail enhancement module , get the image after the first layer fusion , and then treat it as the previous step Image, repeat the above operation n times (see formula 13 for details), and finally get the fused image H(x,y).
[0038] (5) Image labeling preprocessing. The high-quality images after image fusion need to be manually processed by PS (see Appendix Figure 4 ), the specific steps are as follows: first use drawing software to open the image and use the background color to fill the suspected feature area 1. The pin area of the chip has been removed from the board 2. Too many missing parts features are reserved on the board 3. Areas with complex backgrounds (exposed copper, gutters, dirt and scratches, etc.) 4. Tin marks and defective features on the board caused by the production defect process; after manual PS, the image needs to be resized to 1080x1280 resolution and sorted for easy training.
[0039] (6) Labeling. After resizing, the images need to be labeled using Labelmg. The specific steps are as follows. First, use the software to open the folder where the images are stored, and set the folder path after labeling and set it to YOLO format. Then we need to zoom in on each defect and use annotation boxes to fit each defect for calibration. When calibrating, it is necessary to include the features of the four labels: 1. The solder points of missing parts 2. The components and solder points of the monument 3. The tin points and components of the tin connection 4. The offset components and solder points.
Claims
1. A method for preparing a PCBA defect detection dataset based on deep learning, characterized in that: include: (1) Screen the PCBA boards provided by the factory as required; (2) After screening, the board is artificially manufactured with defects; (3) Photograph defective PCBA boards in different areas; (4) Take photos, perform algorithm fusion, and generate high-quality images; (5) Perform labeling preprocessing on the improved images; (6) Label the images using LabelImg software.
2. According to the method for preparing a PCBA defect detection dataset based on deep learning in claim 1, it is characterized in that In step (4), the focal length factor, exposure factor and color factor of the two images are calculated; the calculation formula for the focal length factor is as follows: In the formula, B is the focal length factor, α and β are adjustment factors, which are set to 0.4 and 0.6 respectively, k is the number of input images, D is the entropy factor, J and K are the length and width of the 3x3 moving window rectangle, and C(x,y) is any element of the moving window. , , They represent the average grayscale value, the maximum grayscale value and the minimum grayscale value in the moving window respectively. E is the energy factor, i is a grayscale value from 0 to 255, which represents the probability of the grayscale value occurring in all grayscale values; The formula for calculating the exposure factor is as follows: in the formula is the exposure factor, k is the number of input images, is the grayscale value of the input k-th image, Here, 0.2 is the Gaussian function parameter; u is the adaptive exposure adjustment factor, which is set to 0.
5. is the grayscale average of the kth image; the calculation formula for the color saturation factor is as follows: in is the color saturation factor, They correspond to the three channel values of R, G, and B of the kth image respectively. The method for preparing a PCBA defect detection dataset based on deep learning according to claim 1 is characterized in that in step (4), for the weighted bilateral pyramid decomposition process in the image fusion algorithm process: For multiple input weight maps Recorded as , it is necessary to bring in the weight optimization formula to get the image of the first layer after refinement , and then The detailed calculation formula for obtaining an m-layer image by cyclically using bilateral filtering and downsampling m times is as follows; in For the k-th image, there are m layers of weight optimization graph. is a 15x15 sliding window, and are the arbitrary value and mean of the guide graph under the sliding window, respectively. , , are the values of the three RGB channels of the kth original image; l and e represent the length and width of the original image. According to the method for preparing a PCBA defect detection dataset based on deep learning in claim 1, it is characterized in that in step (4), for the Laplace detail enhancement algorithm in the image fusion algorithm process: For the high-frequency details passed in by the Gaussian and Laplace pyramid fusion modules Gaussian Pyramid , first Grayscale processing is performed , and then the high frequency details and Substitute into the Laplace detail enhancement formula to get , the solution formula is as follows: The method for preparing a PCBA defect detection dataset based on deep learning according to claim 1 is characterized in that in step (4), the weight factor fusion in the image fusion algorithm process is: For the input of the weight factor module , , , we need to fuse and normalize ; The specific formula is as follows; Where k represents the number of different images, Represents the weight parameter of each image, is the normalized weight map.
3. The method for preparing a PCBA defect detection dataset based on deep learning according to claim 1, characterized in that In step (2), the screened boards are artificially made to have defects: During the manufacturing process, we first preheat the heating table to 285 degrees (preheat to 265 degrees for defect soldering), then use tweezers to place the board on the heating table until the solder paste on the board melts, and then use tweezers to divide the board into 8 areas and make seven defects in each area. Different defects have different manufacturing methods.
4. According to the method for preparing a PCBA defect detection data set based on deep learning in claim 2, it is characterized in that: Different methods of making defects:
1. Tombstone: Tombstones are divided into single-leg tombstones, side tombstones and oblique tombstones; among them, take the single-leg tombstone as an example, and the other types are similar. The main difference is that when making the side tombstone, we need to lift the middle of the component and stay for 5s, and for the oblique tombstone, we need to lift the diagonal and stay for 5s; in the production process of the single-leg tombstone, we need to use tweezers to fix the board, and then use another pair of tweezers to pry one leg of the component and stay for 3s, and finally form a single-leg 10~90 degree tombstone; 2. Missing parts: We can first see if there are enough missing parts on the board. If not, we have to artificially create missing parts. First, we need to use tweezers to remove the components on the board in a 90-degree direction and retain the solder joints. The solder joints are the types of missing parts, but when removing the components, do not drag them at will. , which will cause the solder joints to have ligatures, and such solder joints cannot be considered as missing parts; 3. Tin ligature: Tin ligature is different from the other three defects. We need to first apply the solder paste evenly (use as little solder paste as possible, the ligature generated later will be closer to the standard) between two components or between two solder joints, and then place them on a heating table preheated to 265 degrees until the solder becomes a solder point connecting the components and the solder joints. Then take the board off the heating table and set it aside for cooling. Finally, a ligature defect will be generated between the components and solder joints coated with solder paste; 4. Offset: We need to use tweezers to move the components to deviate from the original pads. Components with small deviation angles (that is, slight deviation) are our main products, which are more in line with actual conditions and meet the conditions of machine manufacturing defects.
5. The method for preparing a PCBA defect detection dataset based on deep learning according to claim 1, characterized in that In step (5), the improved image is pre-processed by labeling: the high-quality image after image fusion needs to be manually Photoshopped. The specific steps are as follows: First, open the image with a drawing software and fill the suspected feature area with the background color; 1. The pin area of the chip has been removed from the board; 2. Too many missing features are reserved on the board; 3. The area with complex background (exposed copper, grooves, dirt and scratches, etc.); 4. The defective process of manufacturing leads to tin marks and defective features on the board. After manual PS, the image needs to be resized to 1080x1280 resolution and sorted for training convenience.
