An efficient and fast binocular 3D point cloud solder joint defect detection method

Abstract

The present invention relates to an efficient and fast binocular 3D point cloud solder joint defect detection method, the detection method comprising: establishing a binocular vision system to efficiently and quickly collect point clouds of view 1 and view 2 of solder joints, which are more complete than the 3D point cloud solder joints collected by a monocular vision system; designing a 3D template matching method to locate the binocular 3D point cloud solder joints, the matching method comprising: acquiring point clouds of view 1 and view 2 of a printed circuit board based on a semantic segmentation method, aligning point clouds of view 1 and view 2 based on homogeneous coordinate transformation, and registering the aligned point clouds and a standard template based on a fast point feature histogram method; constructing a 3D point cloud solder joint defect detection technology based on a fine-grained method, the technology predicts key areas of solder joints through global features, and extracts features of key areas for defect classification, thereby realizing efficient and fast solder joint defect classification, which is of great significance to the quality inspection of printed circuit boards in industry.

Description

Technical Field

[0001] The invention belongs to the technical field of circuit board solder spot defect detection, and relates to an efficient and rapid binocular 3D point cloud solder spot defect detection method. Background Art

[0002] With the rapid development of science and technology, printed circuit boards are widely used in all walks of life. In the production and manufacturing process of printed circuit boards, the welding quality inspection of solder joints is a key link. Traditional solder joint defect detection mainly relies on manual inspection, that is, the operator judges whether the soldering of the printed circuit board is qualified by combining pre-specified standards with experience. Manual inspection involves subjective evaluation, and repetitive labor leads to low efficiency and high cost of manual inspection. Therefore, automatic inspection technology is applied to defect detection of solder joints. The existing automatic inspection technology is mainly based on image defect classification. The quality of the image is directly related to the result of defect detection, so the image-based method usually has high requirements for the light source of the detection environment.

[0003] Recently, due to the emergence of low-cost scanners and high-speed computing devices, point clouds have been widely used in many fields. Compared with images, point clouds provide rich geometry, shape and spatial information to characterize 3D objects, and the acquisition of point clouds is insensitive to light sources. Defect detection of printed circuit boards is actually the shape detection of solder joints in 3D structures. The characterization of solder joint shapes by 3D point clouds is better than that by 2D images. And with the breakthrough development of deep learning on point clouds, deep neural networks can process point cloud data efficiently and quickly, which provides a basis for practical production applications.

[0004] The existing automatic detection methods for solder joint defects are mainly based on image data, using multi-color light to collect images, and using machine learning, deep learning and other methods to extract features and classify them. It can be mainly divided into the following three methods: 1) Image processing-based methods, designing different image extraction operators to extract features, and designing corresponding classifiers based on the extracted features. 2) Machine learning-based methods process images and extract features, and use features to train classifiers. 3) Data-driven deep learning methods extract features and classify at the same time. Because these image-based technologies are sensitive to light when collecting images, these technologies are usually not robust to light. The present invention uses laser radar to collect data, which can avoid the influence of light on the detection results. Summary of the invention

[0005] The technical problem to be solved by the present invention is to propose an efficient and fast binocular 3D point cloud solder defect detection method based on the effective representation of 3D point cloud for 3D objects. The present invention is different from the existing technology in that a binocular vision system is designed to quickly collect solder point clouds, and a fine-grained method is constructed to detect 3D solder point cloud defects.

[0006] In order to achieve the above object, the scheme of the present invention is as follows:

[0007] An efficient and fast binocular 3D point cloud solder joint defect detection method, the steps are as follows:

[0008] (1) Construct a binocular vision system to collect 3D point clouds of the samples to be tested;

[0009] The sample to be tested is a printed circuit board packaged in plastic material;

[0010] The binocular vision system refers to: two triangulation laser radars are located directly above the sample to be tested, and the straight line formed by the two is parallel to the plane where the printed circuit board part of the sample to be tested is located (the distance between the two is the scanning height during normal operation in the instruction manual corresponding to the laser radar); the two triangulation laser radars are placed in a mirror-symmetrical manner and there is no gap between them; when each triangulation laser radar moves along the direction of the straight line, the laser emitted by the transmitter of the triangulation laser radar vertically scans the solder joint, and then the laser is reflected by the solder joint to the receiver of the corresponding triangulation laser radar; (the test principle diagram is as shown in FIG. Figure 2 (shown)

[0011] The acquisition process is as follows: using a logic controller (i.e., PLC) to control the start scanning signals of two triangulation laser radars, and simultaneously controlling the two triangulation laser radars to move at a uniform speed on the straight line, acquiring the 3D point cloud of the sample to be tested in one direction at one time, and then transmitting the data collected by the two triangulation laser radars to the host computer via Ethernet;

[0012] During the uniform motion, the relative positions of the two triangulation laser radars remain unchanged;

[0013] The 3D point cloud of the sample to be tested is Figure 1 The 3D point cloud and view Figure 2 The 3D point cloud formed by Figure 1 The 3D point cloud is collected by a triangulation laser radar. Figure 2 The formed 3D point cloud is collected by another triangulation laser radar;

[0014] The triangulation laser radar used in the present invention has a transmitter and a receiver. Usually, the transmitter emits laser light to illuminate an object, the surface of the object reflects the laser light at a certain angle, and the receiver receives the laser light. If the reflected laser light is blocked by the object itself, such as Figure 3As shown in C in the figure, the collected point cloud will be missing. For the problem of solder joints, the top of solder joints is easy to be missing due to their variable shapes, and the missing seriously affects the detection results. Therefore, the present invention designs the triangulation laser radar as the binocular vision system, which can overcome the defect of incomplete data collection due to the triangulation laser radar itself.

[0015] (2) Use 3D template matching method to locate the welding point: first use semantic segmentation method to locate the welding point. Figure 1 The 3D point cloud and view Figure 2 The formed 3D point cloud is segmented to obtain the visual Figure 1 The 3D point cloud of the printed circuit board is denoted as Y1. Figure 2 The 3D point cloud of the printed circuit board is recorded as Y2; then Y2 is aligned with Y1 using homogeneous coordinate transformation to obtain a visual point cloud aligned with Y1. Figure 2 The 3D point cloud of the printed circuit board part is denoted as Finally, use the fast point feature histogram to convert Y1 or Register with the standard template to obtain the position information of each solder joint; because Y1 and Aligned, the solder joint position information of the two is the same, so Y1 and The point clouds belonging to each welding point are denoted as Z1 and Z2 respectively;

[0016] (3) Based on the fine-grained method, detect whether the corresponding solder joints in Z1 and Z2 in step (2) are qualified solder joints one by one;

[0017] The 3D point cloud of the welding point includes X1 and X2; wherein X1 is the 3D point cloud of the welding point in Z1, and X2 is the 3D point cloud of the welding point in Z2;

[0018] The detection process is as follows:

[0019] (3.1) First, a deep neural network is used to transform the point cloud to point features of X1 and X2 of the solder joint to obtain the point features of X1 and X2. Then, a symmetric function is used to process the point features of X1 and X2 while maintaining the permutation invariance of the input to obtain the global features of X1 and X2. The global features of X1 and X2 are expressed as follows:

[0020] f(X1)≈g(m(X1));

[0021] f(X2)≈g(m(X2));

[0022] Among them, f(X1) and f(X2) are the global features of X1 and X2 respectively, g(·) is a symmetric function, and m(·) is the transformation from point cloud to point features. “≈” means that the global features of X can be approximately obtained by the formula on the right, and after the model training is completed, they are completely equal.

[0023] The symmetric function processes the output of the shared multi-layer perceptron I to obtain the global features of X1 and X2 (the global features of X1 and X2 represent the global features of the solder joint).

[0024] (3.2) Determine the critical area of ​​the solder joint through f(X1) and f(X2), and obtain the point features of X1 and X2 in the critical area and The key area is obtained by cutting the spherical area corresponding to the top of the solder joint using an exponential function;

[0025] For each solder joint, operators usually judge the solder joint quality (i.e., whether the solder joint is qualified) by judging the top shape of the solder joint, and what is obtained in step (3.1) is the global feature of the solder joint, which includes the position information of all points on the solder joint. Therefore, the present invention obtains the detailed features of the key area of ​​the solder joint (i.e., the top area of ​​the solder joint) as the input of the classifier through the following process to improve the classification effect. In general, the key area of ​​the solder joint is estimated using global features, and the exponential function is used to cut the key area from the 3D point cloud of the solder joint to ensure that the key area can be optimized in the back propagation. In particular, the point features in the key area are obtained by masking to indirectly achieve the purpose of cutting the key area.

[0026] (3.3) Use the point features of X1 and X2 in the key area obtained in step (3.2) and As the input of the classifier multilayer perceptron, the probability of predicting that the solder joint is a qualified solder joint is predicted, and its mathematical expression is as follows:

[0027]

[0028] Among them, p(·) refers to the probability that the solder joint is a qualified solder joint, and cls(·) refers to the classifier multi-layer perceptron;

[0029] If p(X1,X2)>0.5, the solder joint is considered to be a qualified solder joint.

[0030] The method in the above step (3) of the present invention is a deep learning model.

[0031] As the preferred technical solution:

[0032] In the above-mentioned efficient and fast binocular 3D point cloud solder joint defect detection method, the start scanning signal of controlling the two triangulation laser radars means: when the laser of the triangulation laser radar can scan the sample to be tested, the start scanning signal is started, and when the laser of the triangulation laser radar no longer scans the sample to be tested, the start scanning signal is stopped. The position when the sample to be tested can be scanned and the position when the sample to be tested is no longer scanned are determined by experiments (conventional technology can be used).

[0033] As described above, an efficient and fast binocular 3D point cloud solder joint defect detection method is characterized in that the standard template refers to a 3D point cloud of a printed circuit board that is consistent with the specifications of the printed circuit board to be tested and the position distribution of the solder joints on the printed circuit board to be tested, and the standard template contains the position information of the solder joints on the printed circuit board.

[0034] As described above, an efficient and fast binocular 3D point cloud solder joint defect detection method, the transformation of the point cloud to point features is to use a shared multi-layer perceptron I to extract the point features of X1 and X2; the shared multi-layer perceptron I is a 3-layer shared perceptron with outputs of 64, 128, and 512.

[0035] As described above, an efficient and fast binocular 3D point cloud solder joint defect detection method, the specific process of step (3.2) is:

[0036] (3.2.1) Taking f(X1) and f(X2) in step (3.1) as the input of Multilayer Perceptron II, we get the spherical region, which is described as follows:

[0037] [t x ,t y ,t z ,r]=s(f(X1),f(X2));

[0038] Among them, t x ,t y ,t z They represent the coordinates of the center point of the spherical region, r represents the radius of the spherical region; s(·) represents the multilayer perceptron II;

[0039] (3.2.2) In order to ensure that the key area can be optimized in the back propagation, the spherical area obtained in step (3.2.1) is cut from X1 and X2 respectively using the exponential function to obtain the key area;

[0040] The cutting is indirectly achieved by obtaining the point features within the spherical area through a mask, and the result is as follows:

[0041]

[0042]

[0043] in, and represents the point features of X1 and X2 in the spherical region, m(·) is the transformation from point cloud to point features, ⊙ represents element-wise multiplication, M1(·) and M2(·) represent the masks of X1 and X2;

[0044] In the prior art, a step function is generally used. However, the step function is not differentiable, which will make the model unable to be optimized in back propagation, thereby reducing the detection accuracy. The exponential function is differentiable, and its differentiable property makes the optimization plane of the model smoother, helping the model to improve the detection accuracy.

[0045] In the above formula, the expressions of M1(·) and M2(·) are:

[0046] M1(·)=h(sqdist1-r 2 );

[0047] sqdist1 = sum((X1-(t x ,t y ,t z )) 2 );

[0048] M2(·)=h(sqdist2-r 2 );

[0049] spdist2=sum((X2-(t x ,t y ,t z )) 2 );

[0050] Where h(·) represents the exponential function, sqdist1 and sqdist2 represent the square of the distance from each point in X1 and X2 to the center point of the spherical region, r represents the radius of the spherical region, sum(·) represents the sum, and t x ,t y ,t z Represents the coordinates of the center point of the spherical area; the point cloud is presented in the form of a three-dimensional coordinate set at the data level; the above-mentioned calculation process can be performed.

[0051] The description of h(·) is as follows:

[0052]

[0053] Wherein, k is the exponent of the exponential function (preferably 20), and e is a natural constant.

[0054] As described above, an efficient and fast binocular 3D point cloud solder joint defect detection method, the multi-layer perceptron II is a 3-layer perceptron with outputs of 1024, 64, and 4.

[0055] As described above, an efficient and fast binocular 3D point cloud solder spot defect detection method, the classifier multi-layer perceptron is a three-layer perceptron with outputs of 512, 256, and 2.

[0056] In the above-mentioned efficient and fast binocular 3D point cloud solder spot defect detection method, the symmetry function is a maximum value function.

[0057] The principle of the present invention is:

[0058] Based on the scanning principle of triangulation laser radar, the present invention designs a binocular vision system by placing two laser radars of the same model in a mirror-symmetrical manner. The system can obtain a visual image containing complete solder joint information with only a single scan. Figure 1 And Vision Figure 2 The two 3D point cloud samples are of certain reference significance for point cloud collection in other fields. A special 3D template matching method is designed for solder joint positioning based on the samples collected by the binocular vision system. First, the general semantic segmentation model is used to obtain the visual Figure 1 And Vision Figure 2 The point cloud of the printed circuit board in the sample is removed by using the deep learning model (the point cloud around the printed circuit board affects the accuracy of the registration). However, its ability to segment small objects (solder joints in this case) is relatively weak (as is known to all). Therefore, the semantic segmentation model is not used to directly segment the solder joints. In particular, the present invention does not directly segment the visual information obtained by semantic segmentation. Figure 1 And Vision Figure 2 Instead of registering the point cloud of the printed circuit board in the image to locate the solder joint position, we first use the homogeneous coordinate transformation to Figure 1 And Vision Figure 2 Perform alignment operations. Usually, homogeneous coordinate transformation is used to describe the transformation of rotation and translation of space or plane figures. Here, for efficient preprocessing, first Figure 2 Transform to view Figure 1 In the coordinate space of Figure 1 Perform registration to obtain the visual Figure 1 The position information of the welding point can be obtained Figure 2 The location information of the solder joints reduces the visual Figure 2 The registration process is time-consuming. Figure 1 And Vision Figure 2 The position information of the solder joint can be obtained Figure 1 And Vision Figure 23D point cloud of solder joints. In the process of solder joint defect detection, the present invention uses a fine-grained method to detect solder joint defects. The specific method is as follows: 1) First, a shared multi-layer perceptron is used to extract visual Figure 1 And Vision Figure 2 The point features of the 3D point cloud of the weld point are then processed using a symmetric function (here the maximum function) Figure 1 And Vision Figure 2 Point feature acquisition view Figure 1 And Vision Figure 2 2) Using the visual Figure 1 And Vision Figure 2 The global features of the 3D point cloud of the solder joint predict the key area of ​​the solder joint (the key area here refers to the area that the operator usually observes to judge whether the solder joint is qualified), and use the exponential function (used to approximate the step function) to cut the key area so that the key area can be optimized in the back propagation. In particular, here, the point features in the key area are obtained by masking to indirectly achieve the purpose of cutting the key area. 3) Using the visual Figure 1 And Vision Figure 2 The location features in the key area are used to predict the probability that the solder joint is a qualified product through a classified multi-layer perceptron. Because the detail features of the key area have stronger characterization capabilities than the global features, the detection effect is more accurate than directly using the global features. After actual testing, for a sample, the acquisition process of the method proposed by the present invention only takes 1s, the preprocessing takes 0.3s, the detection takes only 0.37s, and the total time consumption is an average of 1.67s, reaching the standard of processing a sample in 2.5s in the actual production process, and the detection accuracy rate reaches 97%, which meets the basic requirements of the factory.

[0059] Beneficial effects:

[0060] (1) An efficient and fast binocular 3D point cloud defect detection method of the present invention, the detection method uses a binocular acquisition system to quickly and completely obtain a complete weld point cloud;

[0061] (2) The present invention predicts the key area based on the global features of the solder joint 3D point cloud, detects the solder joint defect type by extracting the detailed features of the key area, and effectively improves the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 The overall structural block diagram of the 3D point cloud solder joint defect detection method of the present invention;

[0063] Figure 2 It is a schematic diagram of the principle of binocular vision system;

[0064] Figure 3 Schematic diagram of the principle that the reflected laser is blocked by the object itself;

[0065] Figure 4 Schematic diagram of the process of locating the position of the welding spot using the 3D template matching method;

[0066] Figure 5 A schematic diagram of a process for detecting defects in solder joints;

[0067] Figure 6 These are test results based on actual data. DETAILED DESCRIPTION

[0068] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0069] An efficient and fast binocular 3D point cloud solder joint defect detection method, the overall structure diagram is as follows Figure 1 As shown, the following steps are included:

[0070] (1) Construct a binocular vision system to collect 3D point clouds of the samples to be tested;

[0071] The sample to be tested is a printed circuit board packaged in plastic material;

[0072] The binocular vision system means that: two triangulation laser radars are located directly above the sample to be tested, and the straight line formed by the two is parallel to the plane where the printed circuit board part of the sample to be tested is located (the distance between the two is the scanning height during normal operation in the instruction manual corresponding to the laser radar); the two triangulation laser radars are placed in a mirror-symmetrical manner with no gap between them; when each triangulation laser radar moves along the direction of the straight line, the laser emitted by the transmitter of the triangulation laser radar vertically scans the solder joint, and then the laser is reflected by the solder joint to the receiver of the corresponding triangulation laser radar.

[0073] The acquisition process is as follows: a logic controller (i.e., a PLC) is used to control the start scanning signals of two triangulation laser radars, and the two triangulation laser radars are simultaneously controlled to move at a uniform speed on the straight line, and the 3D point cloud of the sample to be tested is acquired in one direction at one time, and the data acquired by the two triangulation laser radars is then transmitted to the host computer via Ethernet.

[0074] The start scanning signal for controlling the two triangulation laser radars means: when the laser of the triangulation laser radar can scan the sample to be tested, the start scanning signal is started, and when the laser of the triangulation laser radar no longer scans the sample to be tested, the start scanning signal is stopped. The position when the sample to be tested can be scanned and the position when the sample to be tested is no longer scanned are both determined by experiments (known technology).

[0075] During the uniform motion, the relative positions of the two triangulation laser radars remain unchanged.

[0076] The 3D point cloud of the sample to be tested is Figure 1 The 3D point cloud and view Figure 2 The 3D point cloud formed. Figure 1 The 3D point cloud is collected by a triangulation laser radar. Figure 2 The resulting 3D point cloud is collected by another triangulation laser radar.

[0077] (2) Use 3D template matching method to locate the solder joint position (the process diagram is as follows Figure 4 shown);

[0078] (2.1) Based on the semantic segmentation method, Figure 1 And Vision Figure 2 The 3D point cloud formed is segmented to obtain the point cloud belonging to the printed circuit board part, where Figure 1 The 3D point cloud of the printed circuit board is recorded as Y1. Figure 2 The 3D point cloud of the printed circuit board is recorded as Y2. Specifically, by constructing a semantic segmentation dataset, using the general semantic segmentation model PointNet++ for training, the 3D point cloud of the sample to be tested is segmented to obtain the point cloud of the printed circuit board.

[0079] (2.2) Based on the homogeneous coordinate transformation, Y2 is aligned with Y1 to obtain a view aligned with Y1. Figure 2 The 3D point cloud of the printed circuit board part is denoted as

[0080] The process of homogeneous coordinate transformation alignment (existing technology) is as follows:

[0081] Since the relative positions of the two triangulation laser radars remain unchanged during the movement, the homogeneous coordinate transformation (R, t) from Y2 to Y1 can be obtained, Y2 is converted into homogeneous coordinates, recorded as Y2′, and the homogeneous coordinate transformation is applied to Y2′. The result is (Homogeneous coordinates are different from the three-dimensional space coordinates described in ordinary terms. To do homogeneous transformation, you must first convert them into homogeneous coordinates) as shown below:

[0082]

[0083] (2.3) Use the fast point feature histogram to convert Y1 or Acquire the position information of each solder joint by aligning with the standard template; obtain Y1 and The point cloud belonging to each solder joint (due to Y1 or are aligned 3D point clouds, therefore, the positions of the solder joints obtained by registering one of the 3D point clouds can be used to determine the positions of each solder joint in another 3D point cloud), which are denoted as Z1 and Z2 respectively;

[0084] The standard template refers to a 3D point cloud of a printed circuit board that is consistent with the specifications of the printed circuit board to be tested and the position distribution of solder joints on the printed circuit board to be tested, and the standard template contains the position information of the solder joints on the printed circuit board.

[0085] The registration process described using Y1 as an example (existing technology) is as follows:

[0086] Assume that Y1 and the standard template are and And usually N p ≠N q The transformation of point cloud P after rotation R and displacement t can be described as follows:

[0087]

[0088] Assuming that N point pairs are estimated by calculating the fast point feature histogram, the distance between point clouds P and Q is expressed as follows:

[0089]

[0090] The least squares method is used to find the transformation (R, t) that minimizes the above distance, and the transformation (R, t) is repeatedly iterated until the transformation (R, t) that meets the requirements is obtained. The position information of the weld points of Y1 after registration can be obtained based on the weld point positions predicted in advance by the standard template. The 3D point cloud belonging to each weld point is obtained from Y1 through this position information and recorded as Z1.

[0091] Because of visual Figure 2 And Vision Figure 1 Already aligned, similarly, The 3D point cloud belonging to each welding point is obtained and recorded as Z2.

[0092] (3) Based on the fine-grained method, the defects of the corresponding solder joints in Z1 and Z2 in step (2) are detected one by one (e.g. Figure 5 shown);

[0093] The 3D point cloud of the welding point includes X1 and X2; wherein X1 is the 3D point cloud of the welding point in Z1, and X2 is the 3D point cloud of the welding point in Z2;

[0094] The detection process is as follows:

[0095] (3.1) First, use a deep neural network to transform the point cloud to point features of X1 and X2 of the solder joint to obtain the point features of X1 and X2, and then use a symmetric function to maintain the permutation invariance of the input to process the point features of X1 and X2 to obtain the global features of X1 and X2;

[0096] The transformation of the point cloud to point features is to extract the point features of X1 and X2 using a shared multi-layer perceptron I; the shared multi-layer perceptron I is a 3-layer shared perceptron with outputs of 64, 128, and 512;

[0097] The global features of X1 and X2 are expressed as follows:

[0098] f(X1)≈g(m(X1));

[0099] f(X2)≈g(m(X2));

[0100] Among them, f(X1) and f(X2) are the global features of X1 and X2 respectively, g(·) is a symmetric function (maximum value function), and m(·) is the transformation from point cloud to point feature.

[0101] (3.2) Determine the critical area of ​​the solder joint;

[0102] The specific process is:

[0103] (3.2.1) Taking f(X1) and f(X2) in step (3.1) as the input of Multilayer Perceptron II, we get the spherical region, which is described as follows:

[0104] [t x ,t y ,t z ,r]=s(f(X1),f(X2));

[0105] Among them, t x ,t y ,t z They represent the coordinates of the center point of the spherical area, r represents the radius of the spherical area; s(·) represents a multilayer perceptron II; the multilayer perceptron II is a 3-layer perceptron with outputs of 1024, 64, and 4;

[0106] (3.2.2) In order to ensure that the key area can be optimized in the back propagation, the spherical area obtained in step (3.2.1) is cut from X1 and X2 respectively using the exponential function to obtain the key area;

[0107] The cutting is indirectly achieved by obtaining the point features within the spherical area through a mask, and the result is as follows:

[0108]

[0109]

[0110] in, and represents the point features of X1 and X2 in the spherical region, m(·) is the transformation from point cloud to point features, ⊙ represents element-wise multiplication, M1(·) and M2(·) represent the masks of X1 and X2;

[0111] In the above formula, the expressions of M1(·) and M2(·) are:

[0112] M1(·)=h(sqdist1-r 2 );

[0113] sqdist1 = sum((X1-(t x ,t y ,t z )) 2 );

[0114] M2(·)=h(sqdist2-r 2 );

[0115] sqdist2 = sum((X2-(t x ,t y ,t z )) 2 );

[0116] Where h(·) represents the exponential function, sqdist1 and sqdist2 represent the square of the distance from each point in X1 and X2 to the center point of the spherical region, r represents the radius of the spherical region, sum(·) represents the sum, and t x ,t y ,t z Represents the coordinates of the center point of the spherical area;

[0117] The description of h(·) is as follows:

[0118]

[0119] Wherein, k is the exponent of the exponential function (taken as 20), and e is a natural constant.

[0120] (3.3) Use the point features of X1 and X2 in the key area obtained in step (3.2) and As the input of the classifier multilayer perceptron, the probability of predicting that the solder joint is a qualified solder joint is predicted, and its mathematical expression is as follows:

[0121]

[0122] Among them, p(·) refers to the predicted probability of whether the solder joint is qualified, and cls(·) refers to the classifier multi-layer perceptron;

[0123] If p(X1,X2)>0.5, the solder joint is considered to be a qualified solder joint.

[0124] The classifier multilayer perceptron is a three-layer perceptron with outputs of 512, 256, and 2.

[0125] In order to verify the effectiveness of the present invention, the solder joint defects of the printed circuit boards in the factory were detected based on the method of the present invention. The specific process is as follows: the binocular vision system constructed in step (1) of the method of the present invention is used in the factory to collect visual images of 257 printed circuit boards packaged with plastic materials. Figure 1 And Vision Figure 2 A sample 3D point cloud of a printed circuit board containing 5 solder joints.

[0126] The 3D point cloud samples collected above are preprocessed using the 3D template matching method described in step (2) to obtain the visual Figure 1 And Vision Figure 2 3D point cloud of weld points.

[0127] In order to use the semantic segmentation model to perform semantic segmentation on 3D point cloud samples, it is necessary to first build a semantic segmentation dataset to train the semantic segmentation model. Figure 1 And Vision Figure 2 The 3D point clouds of 257 plastic-encapsulated printed circuit boards are divided into training sets and test sets, of which 200 are in the training set and 57 are in the test set, which are used to train and test the semantic segmentation model. The semantic segmentation model uses PointNet++. The training refers to taking the 3D point cloud of the plastic-encapsulated printed circuit board as input, and obtaining the point cloud belonging to the printed circuit board in the input point cloud through back propagation optimization. The test is used to measure the quality of the training process. The model stops training after it can segment the 3D point cloud samples. The test results are as follows: Figure 4 As shown in step 1.

[0128] Then respectively Figure 1 And Vision Figure 2 The point cloud belonging to the printed circuit board obtained by semantic segmentation is subjected to homogeneous transformation and point cloud registration as described in steps (2.2) and (2.3) to obtain the visual Figure 1 And Vision Figure 2 3D point cloud of weld points.

[0129] Then, the fine-grained classification model constructed in step (3) is used to detect solder joint defects. Figure 1 And Vision Figure 2 The dataset is a 3D point cloud dataset for defect classification of solder joints. The dataset has a total of 1285 samples, including 693 samples without defects and 592 samples with defects. Figure 1 And Vision Figure 2 Two 3D point clouds. The above dataset is divided into a training set and a test set, where the training set has 1000 samples and the test set has 285 samples. Then the model in step (3) is trained and tested using the constructed defect classification dataset. The training refers to the use of visual Figure 1 And Vision Figure 2 The 3D point cloud of the solder joint is used as input, and the probability that the solder joint belongs to a qualified solder joint is obtained by back propagation optimization. The test is used to measure the quality of the training process. The training parameters of the model are set as follows, where the optimizer is set to Adam, and its weight decay coefficient is 0.0001. The initial learning rate is 0.0001, and it decays to 0.7 times the original every 20 generations. The exponential function k in the model is 20. The present invention is carried out on Nvidia GeForce GTX 2080Ti GPUs, 16G memory, Ubuntu18.04, pytorch1.8 platform.

[0130] The training results of defect classification are as follows: Figure 6 As shown in the figure, the horizontal axis represents the number of training iterations, and the vertical axis represents the accuracy of training prediction. It can be seen from the figure that the test set result reaches 97%, that is, the probability that the model predicts whether the solder joint is qualified is 97%, and the average detection time of the model is only 0.37s. Its automatic detection result has certain significance for the soldering quality inspection of printed circuit boards.

Claims

1. An efficient and fast binocular 3D point cloud solder joint defect detection method, characterized by Here are the steps: (1) Build a binocular vision system to collect 3D point clouds of the samples to be tested; The sample to be tested is a printed circuit board packaged in plastic material; The binocular vision system refers to: two triangulation laser radars are located directly above the sample to be tested, and the straight line formed by the two is parallel to the plane where the printed circuit board part of the sample to be tested is located; the two triangulation laser radars are placed in a mirror-symmetrical manner; when each triangulation laser radar moves along the direction of the straight line, the laser emitted by the transmitter of the triangulation laser radar vertically scans the solder joint, and then the laser is reflected by the solder joint to the receiver of the corresponding triangulation laser radar; The acquisition process is: using a logic controller to control the start scanning signals of two triangulation laser radars, and simultaneously controlling the two triangulation laser radars to move at a uniform speed on the straight line, and acquiring the 3D point cloud of the sample to be tested in one direction at one time; During the uniform motion, the relative positions of the two triangulation laser radars remain unchanged; The 3D point cloud of the sample to be tested is composed of a 3D point cloud formed by view 1 and a 3D point cloud formed by view 2; (2) Using the 3D template matching method to locate the solder joint position: First, the semantic segmentation method is used to segment the 3D point cloud formed by view 1 and the 3D point cloud formed by view 2 to obtain the 3D point cloud of the printed circuit board part in view 1, recorded as Y1, and the 3D point cloud of the printed circuit board part in view 2, recorded as Y2; then, homogeneous coordinate transformation is used to align Y2 with Y1 to obtain the 3D point cloud of the printed circuit board part of view 2 aligned with Y1, recorded as Finally, use the fast point feature histogram to convert Y1 or Acquire the position information of each solder joint by aligning with the standard template; obtain Y1 and The point clouds belonging to each welding point are denoted as Z1 and Z2 respectively; (3) Based on the fine-grained method, detect whether the corresponding solder joints in Z1 and Z2 in step (2) are qualified solder joints one by one; The 3D point cloud of the welding point includes X1 and X2; wherein X1 is the 3D point cloud of the welding point in Z1, and X2 is the 3D point cloud of the welding point in Z2; The detection process is as follows: (3.1) First, a deep neural network is used to transform the point cloud to point features of X1 and X2 of the solder joint to obtain the point features of X1 and X2. Then, a symmetric function is used to process the point features of X1 and X2 while maintaining the permutation invariance of the input to obtain the global features of X1 and X2. The global features of X1 and X2 are expressed as follows: f(X1)≈g(m(X1)); f(X2)≈g(m(X2)); Among them, f(X1) and f(X2) are the global features of X1 and X2 respectively, g(·) is a symmetric function, and m(·) is the transformation from point cloud to point feature; (3.2) Determine the critical area of ​​the solder joint through f(X1) and f(X2), and obtain the point features of X1 and X2 in the critical area and The key area is obtained by cutting the spherical area corresponding to the top of the solder joint using an exponential function; (3.3) Use the point features of X1 and X2 in the key area obtained in step (3.2) and As the input of the classifier multilayer perceptron, the probability of predicting that the solder joint is a qualified solder joint is predicted, and its mathematical expression is as follows: Among them, p(·) refers to the probability that the solder joint is a qualified solder joint, and cls(·) refers to the classifier multi-layer perceptron; If p(X1,X2)>0.5, the solder joint is considered to be a qualified solder joint; The specific process of step (3.2) is: (3.2.1) Taking f(X1) and f(X2) in step (3.1) as the input of Multilayer Perceptron II, we get the spherical region, which is described as follows: [t x ,t y ,t z ,r]=s(f(X1),f(X2)); Among them, t x ,t y ,t z They represent the coordinates of the center point of the spherical region, r represents the radius of the spherical region; s(·) represents the multilayer perceptron II; (3.2.2) Use the exponential function to cut the spherical area obtained in step (3.2.1) from X1 and X2 respectively to obtain the key area; The cutting is indirectly achieved by obtaining the point features within the spherical area through a mask, and the result is as follows: in, and represents the point features of X1 and X2 in the spherical region, m(·) is the transformation from point cloud to point features, ⊙ represents element-wise multiplication, M1(·) and M2(·) represent the masks of X1 and X2; In the above formula, the expressions of M1(·) and M2(·) are: M1(·)=h(sqdist1-r 2 ); sqdist1=sum((X1-(t x ,t y ,t z )) 2 ); <h2 style=";text-align:left;direction:ltr">M2(·) = h(sqdist2-r<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> ); sqdist2=sum((X2-(t x ,t y ,t z )) 2 ); Where h(·) represents an exponential function, sqdist1 and sqdist2 represent the square of the distance from each point in X1 and X2 to the center point of the spherical region; The description of h(·) is as follows: Where k is the exponent of the exponential function and e is a natural constant.

2. According to the efficient and fast binocular 3D point cloud solder joint defect detection method of claim 1, it is characterized in that: The start scanning signal for controlling the two triangulation laser radars means: when the laser of the triangulation laser radar can scan the sample to be measured, the start scanning signal is started; when the laser of the triangulation laser radar no longer scans the sample to be measured, the start scanning signal is stopped.

3. According to the efficient and fast binocular 3D point cloud solder joint defect detection method of claim 1, it is characterized in that: The standard template refers to a 3D point cloud of a printed circuit board that is consistent with the specifications of the printed circuit board to be tested and the position distribution of the solder joints, and the standard template contains the position information of the solder joints on the printed circuit board.

4. The efficient and fast binocular 3D point cloud solder joint defect detection method according to claim 1 is characterized in that: The transformation of point cloud to point features uses a shared multi-layer perceptron I to extract point features of X1 and X2; the shared multi-layer perceptron I is a 3-layer shared perceptron with outputs of 64, 128, and 512.

5. The efficient and fast binocular 3D point cloud solder joint defect detection method according to claim 1 is characterized in that: The multilayer perceptron II is a 3-layer perceptron with outputs of 1024, 64, and 4.

6. The efficient and fast binocular 3D point cloud solder joint defect detection method according to claim 1 is characterized in that: The classifier multilayer perceptron is a three-layer perceptron with outputs of 512, 256, and 2.

7. The efficient and fast binocular 3D point cloud solder joint defect detection method according to claim 1 is characterized in that: The symmetric function is a maximum function.

Images