A solder defect detection method based on classification model of point cloud and image processing

Abstract

The present invention belongs to the technical field of solder defect detection, specifically, a solder defect detection method based on a classification model of point cloud and image processing, comprising: translating in a direction according to the collected original image of the tray, dividing the translated image into four parts, obtaining a single roughly positioned PCB board image, correcting and positioning the roughly positioned PCB board image, and obtaining a precisely positioned PCB board image; performing three-dimensional reconstruction on a single solder image in the precisely positioned PCB board image to establish a 3D point cloud image of the solder, obtaining the three-dimensional features of the solder through the 3D point cloud image, and calculating the volume of the solder, wherein the three-dimensional features include the bottom surface area and height of the solder; and classifying and identifying solder defects by using the sklearn-XGBoost classification model using the obtained two-dimensional features and three-dimensional features. The method solves the problem of insufficient defect detection accuracy, missed detection, and false detection caused by the tiny size of the solder, the reflective surface condition, and the complex and changeable welding condition.

Description

Technical Field

[0001] The present invention belongs to the technical field of solder defect detection, and specifically relates to a solder defect detection method based on a classification model of point cloud and image processing. Background Art

[0002] PCB boards are an important part of modern electronic devices. They are used to support and connect electronic components such as integrated circuits, capacitors, resistors, etc. The main function of PCB boards is to provide reliable electrical connections and support electrical components, so that electronic devices are stable and reliable when working. They are widely used in various fields of daily life, such as consumer electronics, transportation, medical equipment, industrial equipment, etc. The quality of printed circuit board solder directly affects the quality of electronic products. Defects in solder can cause problems with electronic products or even scrap them, resulting in serious losses.

[0003] As industrial production continues to increase its requirements for high speed and high precision, traditional manual inspection and simple automated solder inspection technology have been unable to match actual production needs. However, due to the tiny size of solder, the reflective surface and the complex and changeable welding conditions, existing technologies are difficult to effectively overcome these problems, resulting in insufficient defect detection accuracy, frequent missed detections and false detections. In order to overcome these challenges, it is necessary to seek more advanced visual inspection technologies to more accurately identify and analyze solder on circuit boards. Summary of the invention

[0004] The purpose of the present invention is to provide a solder defect detection method based on a classification model of point cloud and image processing, so as to solve the problems of insufficient defect detection accuracy, missed detection and false detection caused by the tiny size of solder, surface reflective conditions and complex and changeable welding conditions.

[0005] The present invention is achieved in this way.

[0006] A solder defect detection method based on a classification model of point cloud and image processing, the method comprising:

[0007] S1 is based on the original image of the pallet collected. The image after translation is divided into four parts to obtain a single coarse positioning PCB board image, and the coarse positioning PCB board image is corrected and positioned to obtain a fine positioning PCB board image;

[0008] S2 extracts solder contours from a single solder image in the precisely positioned PCB board image to obtain two-dimensional features of the solder, where the two-dimensional features include perimeter, area, and roundness;

[0009] S3 uses a single solder image to perform three-dimensional reconstruction to establish a 3D point cloud image of the solder, obtains the three-dimensional features of the solder through the 3D point cloud image, and calculates the volume of the solder, wherein the three-dimensional features include the bottom surface area and height of the solder;

[0010] S4 will obtain two-dimensional features and three-dimensional features to classify and identify solder defects through the sklearn-XGBoost classification model.

[0011] Further: S1 specifically includes:

[0012] Extract the ROI area of ​​the original image of the tray, apply a Gaussian blur process with a Gaussian kernel size of 15 x 15 to the ROI area, binarize the processed image, and perform a morphological closing operation on the binarized image using a 10x10 rectangular structure element to obtain a binary image with smooth and clear boundaries;

[0013] Find the connected domain contour of the binary image and retrieve the outermost contour. Use compression method to store the contour and save the hierarchical information of each contour. Find the only contour of interest by morphologically screening the height and width of the contour and store it in the variable contour.

[0014] The upper left point of the original image of the tray is the coordinate origin, and the vertical direction is Axis, horizontal Axis, establish XY two-dimensional coordinate system, make a A straight line is traversed from top to bottom to find the first intersection point F1 with the contour of interest. The same method is used to find the same point on the standard image to obtain the standard intersection point F1_s. The first intersection point F1 obtained is compared with the standard intersection point F1_s. Based on these two points, the original image of the pallet is translated up and down as a whole, so that the first intersection point F1 coincides with the standard intersection point F1_s to obtain the first new image. The first new image is divided into four equal parts to obtain four roughly positioned PCB board images.

[0015] Furthermore: two ROI areas are taken at fixed positions of the roughly positioned PCB board image, and the center positions of the two ROI areas are respectively marked as the first feature point F2 and the second feature point F3, which are used as two feature points on the roughly positioned PCB board image. The coordinates of these two points are compared with the first standard feature point F2_s and the first standard feature point F3_s at the corresponding positions on the standard image, and two-dimensional translation and rotation are performed to obtain the precisely positioned PCB board image.

[0016] Further: performing translation and rotation in two dimensions includes:

[0017] The coarse positioning PCB board image is translated so that the first feature point F2 is translated to the coordinate position corresponding to the first standard feature point F2_s corresponding to the standard image;

[0018] The angle between the first feature point F2 and the second feature point F3 is calculated through the coordinates of the two points, and the angle between the line connecting the two points and the horizontal line. The angle between the line connecting the first standard feature point F2_s and the second standard feature point F3_s on the standard image and the horizontal line is also calculated. The direction and angle of rotation of the roughly positioned PCB board image around the first feature point F2 are determined through the difference between the two angles. After rotation, a second new image is obtained. The position coordinates of the solder joints in the second new image are consistent with those on the standard image. According to the solder joint positions on the standard image, the solder joint positions of the second new image are obtained, and the precisely positioned PCB board image is obtained.

[0019] Further: In S2, the method of extracting the solder contour of a single solder image in the precisely positioned PCB board image and obtaining the two-dimensional features of the solder specifically includes:

[0020] Perform contour extraction on a single solder image;

[0021] The solder perimeter is obtained by calculating the total number of contour pixels. ;

[0022] Calculate the number of pixels inside the contour to get the solder area ;

[0023] Calculate the roundness of solder using solder area and solder perimeter : .

[0024] Further: S3 specifically includes:

[0025] Under the condition of XY two-dimensional coordinate system, a three-dimensional coordinate system is established by making a Z axis perpendicular to the XY plane at the origin;

[0026] set up , The first The coordinates of the points, For the Points Axis coordinates, For the Points Axis coordinates, For the Points Axis coordinates, where , is the total number of points in the 3D point cloud data. The solder is fitted into a conical model, and the vertex coordinates of the conical model are , , , calculate the bottom radius of the conical model through 3D point cloud data and high ;

[0027] Calculate the center point , find the center point of the bottom surface of the cone model in the 3D point cloud data , with this center point As the center of the cone base:

[0028] ,

[0029] Calculate the base radius of the conical model and high Includes: Calculate the distance from all points on the bottom surface of the conical model in the 3D point cloud data to the center point The average distance is taken as the base radius , and The maximum value of the axis coordinate is taken as the height :

[0030] ,

[0031] Indicates the center point of The coordinates of the axes, Indicates the center point of The coordinates of the axes, , They are the bottom points of the conical model in the 3D point cloud data of the solder. , Axis coordinates, where , is the total number of points on the bottom surface of the cone model in the 3D point cloud data:

[0032] ,

[0033] According to the volume formula of the cone, calculate the volume of the solder: .

[0034] Furthermore: before classifying and identifying solder defects, a database of solder images and a sklearn-XGBoost classification model are established, and the two-dimensional features and three-dimensional features obtained from the database are converted into feature vectors as training samples and added to the sklearn-XGBoost classification model for training to obtain a trained sklearn-XGBoost classification model.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The present invention combines point cloud processing with image processing, and aims to obtain PCB solder images using image processing methods of machine vision. The method of the present invention obtains the position information of the solder on the PCB board and a single solder image in real time by accurately matching the correct PCB board, that is, generating a detection area; then the solder contour is extracted by image processing methods to calculate the solder contour perimeter, area and roundness, and a single solder image is three-dimensionally reconstructed to obtain a point cloud image, so as to reduce the amount of calculation and improve the detection efficiency; and the obtained point cloud image is processed by fitting a cone to estimate the volume of the solder; a database of solder images is established through features, and the sklearn-XGBoost classification model is trained using the database to converge, and the solder features are classified using the trained sklearn-XGBoost classification module to obtain the detection results of solder defects, and the solder defect detection work is completed efficiently and accurately, meeting the production requirements of automation, high speed and high precision. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A flowchart of a method provided by an embodiment of the present invention;

[0038] Figure 2 A schematic structural diagram of an original image of a tray provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0040] See also Figure 1 As shown, a solder defect detection method based on a classification model of point cloud and image processing includes:

[0041] S1 is based on the original image of the pallet collected. The image after translation is divided into four parts to obtain a single coarse positioning PCB board image, and the coarse positioning PCB board image is corrected and positioned to obtain a fine positioning PCB board image;

[0042] S2 extracts solder contours from a single solder image in the precisely positioned PCB board image to obtain two-dimensional features of the solder, where the two-dimensional features include perimeter, area, and roundness;

[0043] S3 uses a single solder image to perform three-dimensional reconstruction to establish a 3D point cloud image of the solder, obtains the three-dimensional features of the solder through the 3D point cloud image, and calculates the volume of the solder, wherein the three-dimensional features include the bottom surface area and height of the solder;

[0044] S4 will obtain two-dimensional features and three-dimensional features to classify and identify solder defects through the sklearn-XGBoost classification model.

[0045] In S1, the ROI area (Region of Interest) of the original image of the tray is extracted. The ROI area selected here can be the upper half, see Figure 2 As shown in the figure, it is a schematic diagram of the original image of the tray. The outermost rectangle 1 is the range of the camera shooting. The photographed tray 2 and the PCB board 4 located in the tray 2 are located in the range of the camera shooting. The tray 2 is in a tilted state. The solid circle 3 and the solid oval in the PCB board 4 are solder. The two hollow circles in the PCB board are inherent features on the PCB board. Figure 2 The upper half of the image is taken as the ROI area, that is, the part including the upper edge line of the tray. A Gaussian blur operation with a Gaussian kernel size of 15 x 15 is applied to the ROI area. The processed image is binarized and a 10x10 rectangular structure element is used to perform a morphological closing operation on the binarized image to obtain a binary image with a smooth and clear boundary.

[0046] Find the connected domain contour of the binary image and retrieve the outermost contour. Use a simple compression method to store the contour. At the same time, the hierarchical information of each contour is also saved. By screening these contours in terms of height, width, etc., the only contour of interest is found and stored in the variable contour. The contour with the widest width, that is, the contour of the area corresponding to the upper edge of the tray, is used as the only contour of interest.

[0047] Take the upper left point of the original image of the pallet as the coordinate origin, the vertical direction as the y-axis, and the horizontal direction as the x-axis. Draw a straight line with x=0, traverse the straight line from top to bottom, and find the first intersection with the contour of interest saved in the previous step and record it as F1. Use the same method to find the same point on the standard image to obtain the standard intersection F1_s. The standard image here is a pre-existing image. The position coordinates of the welding point to be detected and the feature point on the standard image, namely the standard intersection F1_s, are all determined. Through the above steps, we can know the precise coordinates of the first intersection F1 and the standard intersection F1_s. According to the difference between the vertical coordinates of the first intersection F1 and the standard intersection F1_s, determine the translation direction and distance of the image to be detected. If Then the image to be detected is translated downward as a whole, otherwise it is translated upward, so that the coordinates of the first intersection F1 are equal to the coordinates of the standard intersection F1-s in value. After the translation up and down, a new image will be obtained. A small part of the new image will be covered on one side of the translation direction, that is, above or below the image, and the extra part on the other side will be replaced by a black screen. The translated new image is divided into four parts to obtain four roughly positioned PCB board images.

[0048] Take two ROI areas of appropriate size at fixed positions on the coarse positioning PCB board image. The position and size of the taken ROI area are the inherent features of each PCB board image and the standard image, for example Figure 2 In the PCB board image shown, the inherent two standard circles can be used as ROI areas, and of course other features can also be selected as ROI areas. When taking the ROI area, it is necessary to consider the left and right offsets. Finally, the center positions of the two ROI areas are marked as the first feature point F2 and the second feature point F3, respectively. At the same time, the coordinates of these two feature points are compared with the same two feature points on the standard image, namely the first standard feature point F2_s and the first standard feature point F3_s, and two-dimensional translation is performed to obtain a precisely positioned PCB board image. In one embodiment:

[0049] Take the ROI area for the coarse positioning PCB board image, and select two circular objects. Use the Hough circle detection method to detect the two circular objects in the coarse positioning PCB board image as the ROI area. You can find a circle that meets a certain radius condition and store the center of the circle in hough (x, y). Apply a Gaussian blur operation to the ROI area, binarize the processed image, erode and then dilate the ROI area, find the connected domain of the processed image, extract the contour, and save it in the variable contours. Traverse each extracted contour, filter the morphological data such as the area, aspect ratio, and circularity of the contour, obtain a unique contour for each ROI area, save the center of the circle inscribed in the contour, and assign the center coordinates to center1 (x, y). If there is no contour that meets the conditions, assign the variable hough (x, y) to center1 (x, y). The processing steps for another ROI area are the same as above, and center2 (xy) is obtained. The center1 and center2 obtained are the first feature point F2 and the second feature point F3. The coordinates of the first feature point F2 are obtained, and the roughly positioned PCB board image can be translated to make the first feature point F2 translated to the coordinate position corresponding to the same feature point of the standard image, namely the first standard feature point F2_s. At the same time, the angle between the line connecting the two points and the horizontal line can be calculated through the coordinates of the first feature point F2 and the second feature point F3. Similarly, the angle between the line connecting the first standard feature point F2_s and the second standard feature point F3_s on the standard image and the horizontal line can be obtained. The direction and angle of rotation of the roughly positioned PCB board image around the first feature point F2 can be determined by the difference between the two angles. After rotation, a second new image will be obtained. The position coordinates of the solder joint to be detected in the second new image will be consistent with the position coordinates of the solder joint to be detected on the standard image, and the position of the solder joint to be detected on the standard image is known. In this way, the position of the solder joint of the roughly positioned PCB board image is known, and the positioning is completed.

[0050] According to the initial positions of the four roughly positioned PCB board images, the positions of the respective standard images are compared, and the ROI area taken by each roughly positioned PCB board image can be the same or different, and four precisely positioned PCB board images can be obtained.

[0051] In S2, the method for extracting the solder contour of a single solder image in the precisely positioned PCB board image and obtaining the two-dimensional features of the solder specifically includes:

[0052] Perform contour extraction on a single solder image;

[0053] The solder perimeter is obtained by calculating the total number of contour pixels. ;

[0054] Calculate the number of pixels inside the contour to get the solder area ;

[0055] Calculate the roundness of solder using solder area and solder perimeter : .

[0056] S3 specifically includes:

[0057] Under the condition of XY two-dimensional coordinate system, a three-dimensional coordinate system is established by making a Z axis perpendicular to the XY plane at the origin;

[0058] set up , The first The coordinates of the points, For the Points Axis coordinates, For the Points Axis coordinates, For the Points Axis coordinates, where , is the total number of points in the 3D point cloud data. The solder is fitted into a conical model, and the vertex coordinates of the conical model are , , , calculate the bottom radius of the conical model through 3D point cloud data and high ;

[0059] Calculate the center point , find the center point of the bottom surface of the cone model in the 3D point cloud data , with this center point As the center of the cone base:

[0060] ,

[0061] Calculate the base radius of the conical model and high Includes: Calculate the distance from all points on the bottom surface of the conical model in the 3D point cloud data to the center point The average distance is taken as the base radius , and The maximum value of the axis coordinate is taken as the height :

[0062] ,

[0063] Indicates the center point of The coordinates of the axes, Indicates the center point of The coordinates of the axes, , They are the bottom points of the conical model in the 3D point cloud data of the solder. , Axis coordinates, where , is the total number of points on the bottom surface of the cone model in the 3D point cloud data:

[0064] ,

[0065] According to the volume formula of the cone, calculate the volume of the solder: .

[0066] In S4, the XGBoost multi-classification problem is implemented based on the sklearn library. XGBoost uses a distributed forward additive model framework with fast detection speed and high accuracy.

[0067] Before classifying and identifying solder defects, the embodiment of the present invention establishes a database of solder images and a sklearn-XGBoost classification model, and converts the features obtained in the database into feature vectors as training samples and adds them to the sklearn-XGBoost classification model for training. The upsampling technology is combined to process unbalanced data, and the hyperparameters are fine-tuned through learning curve analysis and grid search. Finally, the sklearn-XGBoost classification model is converged to complete the training;

[0068] The volume, circumference, roundness, area and height features of the solder contour are imported into the sklearn-XGBoost classification model to obtain the classification results, complete the solder quality inspection and output the results.

[0069] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A solder defect detection method based on a classification model of point cloud and image processing, characterized in that: The method includes: S1 is based on the original image of the pallet collected. The image after translation is divided into four parts to obtain a single coarse positioning PCB board image, and the coarse positioning PCB board image is corrected and positioned to obtain a fine positioning PCB board image; S2 extracts the solder contour of a single solder image in the precisely positioned PCB board image to obtain two-dimensional features of the solder, wherein the two-dimensional features include perimeter, area and roundness; S3 uses a single solder image to perform three-dimensional reconstruction to establish a 3D point cloud image of the solder, obtains the three-dimensional features of the solder through the 3D point cloud image, and calculates the volume of the solder, wherein the three-dimensional features include the bottom surface area and height of the solder; S4 will obtain two-dimensional features and three-dimensional features to classify and identify solder defects through the sklearn-XGBoost classification model; S1 specifically includes: Extract the ROI area of ​​the original image of the tray, apply a Gaussian blur process with a Gaussian kernel size of 15 x 15 to the ROI area, binarize the processed image, and perform a morphological closing operation on the binarized image using a 10x10 rectangular structure element to obtain a binary image with smooth and clear boundaries; Find the connected domain contour of the binary image and retrieve the outermost contour. Use compression method to store the contour and save the hierarchical information of each contour. Find the only contour of interest by morphologically screening the height and width of the contour and store it in the variable contour. The upper left point of the original image of the tray is the coordinate origin, and the vertical direction is Axis, horizontal Axis, establish XY two-dimensional coordinate system, make a A straight line is traversed from top to bottom to find the first intersection point F1 with the contour of interest. The same method is used to find the same point on the standard image to obtain the standard intersection point F1_s. The first intersection point F1 obtained is compared with the standard intersection point F1_s. Based on these two points, the original image of the pallet is translated up and down as a whole, so that the first intersection point F1 coincides with the standard intersection point F1_s to obtain the first new image. The first new image is divided into four equal parts to obtain four roughly positioned PCB board images.

2. The solder defect detection method based on classification model of point cloud and image processing according to claim 1 is characterized in that: Take two ROI areas at fixed positions of the roughly positioned PCB board image, and mark the center positions of the two ROI areas as the first feature point F2 and the second feature point F3 respectively, as the two feature points on the roughly positioned PCB board image. Use these two points to compare the coordinates of the first standard feature point F2_s and the first standard feature point F3_s at the corresponding positions on the standard image, perform two-dimensional translation and rotation, and obtain the precisely positioned PCB board image.

3. The solder defect detection method based on point cloud and image processing classification model according to claim 2 is characterized in that: Two-dimensional translation and rotation include: The coarse positioning PCB board image is translated so that the first feature point F2 is translated to the coordinate position corresponding to the first standard feature point F2_s corresponding to the standard image; The angle between the first feature point F2 and the second feature point F3 is calculated through the coordinates of the two points, and the angle between the line connecting the two points and the horizontal line. The angle between the line connecting the first standard feature point F2_s and the second standard feature point F3_s on the standard image and the horizontal line is also calculated. The direction and angle of rotation of the roughly positioned PCB board image around the first feature point F2 are determined through the difference between the two angles. After rotation, a second new image is obtained. The position coordinates of the solder joints in the second new image are consistent with those on the standard image. According to the solder joint positions on the standard image, the solder joint positions of the second new image are obtained, and the precisely positioned PCB board image is obtained.

4. The solder defect detection method based on classification model of point cloud and image processing according to claim 1 is characterized in that: In S2, the method for extracting the solder contour of a single solder image in the precisely positioned PCB board image and obtaining the two-dimensional features of the solder specifically includes: Perform contour extraction on a single solder image; The solder perimeter is obtained by calculating the total number of contour pixels. ; Calculate the number of pixels inside the contour to get the solder area ; Calculate the roundness of solder using solder area and solder perimeter : .

5. The solder defect detection method based on classification model of point cloud and image processing according to claim 1 is characterized in that: S3 specifically includes: Under the condition of XY two-dimensional coordinate system, a three-dimensional coordinate system is established by making a Z axis perpendicular to the XY plane at the origin; set up , The first The coordinates of the points, For the Points Axis coordinates, For the Points Axis coordinates, For the Points Axis coordinates, where , is the total number of points in the 3D point cloud data. The solder is fitted into a conical model, and the vertex coordinates of the conical model are , , , calculate the bottom radius of the conical model through 3D point cloud data and high ; Calculate the center point , find the center point of the bottom surface of the cone model in the 3D point cloud data , with this center point As the center of the cone base: ， Calculate the base radius of the conical model and high Includes: Calculate the distance from all points on the bottom surface of the conical model in the 3D point cloud data to the center point The average distance is taken as the base radius , and The maximum value of the axis coordinate is taken as the height : ， Indicates the center point of The coordinates of the axes, Indicates the center point of The coordinates of the axes, , They are the bottom points of the conical model in the 3D point cloud data of the solder. , Axis coordinates, where , is the total number of points on the bottom surface of the cone model in the 3D point cloud data: ， According to the volume formula of the cone, calculate the volume of the solder: .

6. The solder defect detection method based on classification model of point cloud and image processing according to claim 1 is characterized in that: Before classifying and identifying solder defects, a database of solder images and a sklearn-XGBoost classification model are established, and the two-dimensional features and three-dimensional features obtained from the database are converted into feature vectors as training samples and added to the sklearn-XGBoost classification model for training to obtain the trained sklearn-XGBoost classification model.

Images