LED wire bonding point cloud classification method and device based on deep learning

Abstract

The present application relates to a kind of LED wire bonding point cloud classification methods based on deep learning, comprising the following: obtaining target LED three-dimensional point cloud;Respectively, the first feature vector is obtained by the feature extraction based on PointNet to the target LED three-dimensional point cloud, and the second feature vector is obtained by the normal vector estimation based on normal differential;The first feature vector and second feature vector are spliced, and the complete point cloud feature is obtained;Based on the complete point cloud feature, the action is carried out through fully connected network, and point cloud classification is completed.The normal differential algorithm is used to estimate the normal vector of point in the present application, and the vector is fused as the feature vector of point into the deep learning network based on PointNet, which improves the accuracy of point cloud classification.The classified and extracted wire bonding point cloud can be further used for three-dimensional defect detection, so as to realize and improve the accuracy of LED wire bonding defect detection with three-dimensional characteristics.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine vision, and particularly relates to an LED wire bonding point cloud classification method and device based on deep learning. BACKGROUND

[0002] The connecting wire between the LED chip and the wedge-shaped support is referred to as an LED wire bonding, which is a defect of the LED chip and needs to be detected and processed to ensure the product quality of the LED chip.

[0003] The existing LED wire bonding detection methods are mostly divided into two ways: the first way is to detect the LED wire bonding by traditional workers according to the experience of the workers, which is highly dependent on the work experience of the workers and has low efficiency and accuracy; the other way is to find the LED wire bonding defect by three-dimensional defect detection, which has a very complex calculation process and great difficulty in processing. SUMMARY

[0004] The present application relates to the technical field of machine vision, and particularly relates to an LED wire bonding point cloud classification method and device based on deep learning.

[0005] In order to achieve the above-mentioned purpose, the technical scheme adopted by the present application is as follows:

[0006] Specifically, an LED wire bonding point cloud classification method based on deep learning is proposed, which comprises the following:

[0007] Obtaining a target LED three-dimensional point cloud;

[0008] Respectively extracting a first feature vector based on PointNet and estimating a second feature vector based on normal differential from the target LED three-dimensional point cloud, and the loss function of the process is L=(1-μ)×L p +μ×L d , wherein L p is the loss function of the PointNet feature extraction, L d is the loss function of the normal vector estimation based on the normal differential, and μ is a balance factor and is artificially set;

[0009] Splicing the first feature vector and the second feature vector to obtain a complete point cloud feature;

[0010] Acting on the complete point cloud feature based on a full connection network to complete the point cloud classification.

[0011] Further, the feature extraction based on PointNet comprises the following process,

[0012] Assume the input is a 3D point cloud of LEDs, P = {p1, p2, ..., p...} N}, where p i This represents the total number of point clouds for the i-th point and N points;

[0013] In all the input point clouds, each point cloud is referred to as a point. The set of points enclosed by a sphere with a radius of 120μm centered on each point is taken as the dataset of the LED point cloud.

[0014] The farthest point sampling method is used to downsample the input set for each point, and the number of downsampling points is set to 512.

[0015] The grouping layer uses the KNN nearest neighbor algorithm to divide the 512 points into 32 groups, with 16 points in each group;

[0016] Then, the feature vector of that point is extracted using the PointNet network;

[0017] The above process is called a set abstraction. After three set abstractions, a 1x1021 feature vector is finally obtained.

[0018] Furthermore, specifically, the normal vector estimation based on the normal differential includes the following process:

[0019] Because of the significant dimensional variations between the LED bonding wires and the wedge-shaped support, the normal differential algorithm performs well in identifying the differences in normal vector features at different scales. Therefore, the normal differential algorithm is used to estimate the normal vector of a point to provide feature identification for that point.

[0020] For each point p in the 3D point cloud of the target LED, using a larger radius r l Estimate the normal vector Δn(p, r1) of a point p using a small radius r. s Estimate the normal vector Δn(p, r) of the same point p. s The average of the differences between the two normal vectors is the DoN feature of point p.

[0021] Furthermore, specifically, the first feature vector and the second feature vector are concatenated to obtain the complete point cloud features, including the following:

[0022] The first feature vector is a 1021-dimensional feature extracted based on PointNet, and the second feature vector is a 3-dimensional feature extracted based on normal vector estimation using normal differentiation. The 1021-dimensional feature and the 3-dimensional feature are concatenated to obtain a complete 1024-dimensional feature.

[0023] Furthermore, specifically, L p With L d The result is obtained based on the softmax function, and includes the following process.

[0024] First, the output value is normalized to the range of 0~1 by using the softmax function The formula of the softmax function is substituted into the cross-entropy loss function, that is, L p , Where x i is the i-th sample input to the softmax function, y i is the real probability distribution, and N is the number of samples.

[0025] Where n i is the estimated normal vector, and n' is the real normal vector. i

[0026] Further, the method further comprises that the training process is as follows,

[0027] Each point in the LED three-dimensional point cloud is taken as a training sample, and the network model is trained using the Adam optimizer, the initial learning rate is 0.001, the learning rate is reduced by 0.7 every 20 epochs, β1 is 0.9, β2 is 0.999, and the weight decay rate is 10 -4 .

[0028] Further, specifically, the value of the balance factor μ is 0.7.

[0029] The application also provides an LED wire bonding point cloud classification device based on deep learning, comprising the following:

[0030] A three-dimensional point cloud acquisition module is used to acquire a target LED three-dimensional point cloud.

[0031] A feature extraction module comprises,

[0032] A first feature extraction unit is used to perform PointNet-based feature extraction on the target LED three-dimensional point cloud to obtain a first feature vector,

[0033] A second feature extraction unit is used to perform normal vector estimation based on normal differential on the target LED three-dimensional point cloud to obtain a second feature vector, and the loss function of the process is L=(1-μ)×L p +μ×L d , wherein L p is the loss function of the PointNet-based feature extraction, L d is the loss function of the normal vector estimation based on normal differential, and μ is a balance factor and is artificially set.

[0034] A feature fusion module is used to splice the first feature vector and the second feature vector to obtain complete point cloud features

[0035] ​A point classification module is configured to complete point cloud classification by full connection network based on the complete point cloud features.

[0036] The present application has the following advantages:

[0037] The present application provides a LED wire bonding point cloud classification method based on deep learning, which estimates the normal vector of a point by using a normal differential algorithm, and fuses the vector as a feature vector of the point into a deep learning network based on PointNet, thereby improving the accuracy of point cloud classification. The classified and extracted wire bonding point cloud can be further used for three-dimensional defect detection, thereby realizing and improving the accuracy of LED wire bonding defect detection with three-dimensional features. The method can be used to extract the connecting lead between the chip and the wedge-shaped support in the LED production process, and then judge the quality of the connecting lead, thereby realizing three-dimensional defect detection of the LED wire bonding. BRIEF DESCRIPTION OF DRAWINGS

[0038] The above and other features of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which like reference numerals denote like or similar elements, and clearly, the drawings below are only some embodiments of the present disclosure, and other drawings can be obtained from the drawings without creative labor for those skilled in the art, and in the drawings:

[0039] Figure 1 A flowchart of the LED wire bonding point cloud classification method based on deep learning is shown.

[0040] Figure 2 A structure diagram of the point cloud classification network formed by the LED wire bonding point cloud classification method based on deep learning is shown. DETAILED DESCRIPTION

[0041] The concept, specific structure and generated technical effects of the present application will be described clearly and completely in combination with the embodiments and the drawings, so as to fully understand the purpose, scheme and effect of the present application. It should be noted that the embodiments and the features in the embodiments in the present application can be combined with each other without conflict. The same reference numerals in the drawings indicate the same or similar parts.

[0042] Reference Figure 1 And Figure 2 Embodiment 1, the present application provides a LED wire bonding point cloud classification method based on deep learning, which includes the following:

[0043] Step 110, acquiring a target LED three-dimensional point cloud;

[0044] Step 120, respectively, the target LED three-dimensional point cloud is based on the feature extraction of PointNet to obtain the first feature vector, and the normal vector estimation based on the normal differential is obtained The loss function of this process is L=(1-μ)×L p +μ×L d , wherein L p is the loss function of the PointNet feature extraction, L d is the loss function of the normal vector estimation based on the normal differential, and μ is a balance factor, which is artificially set;

[0045] Step 130, the first feature vector and the second feature vector are spliced to obtain the complete point cloud feature;

[0046] Step 140, based on the complete point cloud feature, the full connection network is used to act, and the point cloud classification is completed.

[0047] In this embodiment 1, the normal vector of the point is estimated by the normal differential algorithm, and the vector is fused into the deep learning network based on PointNet as the feature vector of the point. Because PointNet uses the symmetric function maxpooling to realize the arrangement invariance of the point cloud, the point cloud data can be directly processed, but the PointNet network can only capture the global information, and the local feature is missing. The normal vector feature of the solder wire point cloud is extracted by using the normal differential algorithm, which further supplements the problem of missing local feature information of PointNet. The cooperation of the two can make the feature collection of the point cloud data more comprehensive. So that the classified and extracted solder wire point cloud can be further used for three-dimensional defect detection, so as to realize and improve the accuracy of LED solder wire defect detection with three-dimensional characteristics.

[0048] As a preferred embodiment of the present application, specifically, the feature extraction based on PointNet includes the following processes,

[0049] Suppose the input LED three-dimensional point cloud P={p1,p2,…,p N}, wherein p i represents the i-th point, and N is the total number of point clouds;

[0050] In all input point clouds, each point cloud is called each point, and the point set surrounded by the ball radius of 120 μm is taken as the data set of the LED point cloud. Here, considering that the radius of the input point will affect the performance of the classification, too small radius will contain too little geometric information, and too large radius will lead to too much noise. The appropriate radius can be the highest overall accuracy, and after a large number of experimental demonstrations, the ball radius of 120 um is adopted in the present application;

[0051] The farthest point sampling method is used to down-sample the input set of each point, and the number of down-sampling points is set to 512;

[0052] The grouping layer is used to divide the 512 points into 32 groups by using the nearest neighbor algorithm KNN, and each group has 16 points;

[0053] Then the feature vector of the point is extracted by the PointNet network.

[0054] The above process is called one set abstraction, and after three set abstractions, the final feature vector of 1X1021 is obtained.

[0055] As a preferred embodiment of the application, the normal vector estimation based on normal differential includes the following processes,

[0056] Because the scale difference between the LED wire and the wedge-shaped support is large, the normal differential algorithm is good at identifying the difference of normal vector features under different scales, so the normal differential algorithm is used to estimate the normal vector of the point to provide feature recognition of the point,

[0057] For each point p of the target LED three-dimensional point cloud, a larger radius r l is used to estimate the normal vector of a point p, a smaller radius r s is used to estimate the normal vector of the same point p, and the average of the difference between the two normal vectors is the DoN feature of the point p. s

[0058] As a preferred embodiment of the application, specifically, the first feature vector and the second feature vector are spliced to obtain the complete point cloud feature, including the following,

[0059] The first feature vector is a 1021-dimensional feature extracted based on PointNet, and the second feature vector is a 3-dimensional feature extracted based on normal differential normal vector estimation. The 1021-dimensional feature and the 3-dimensional feature are spliced to finally obtain a complete 1024-dimensional feature.

[0060] As a preferred embodiment of the application, specifically, L p is the loss function of the softmax function, and L d is obtained based on the softmax function, including the following processes,

[0061] First, the output value is normalized to the range of 0-1 by using the softmax function The formula of the softmax function is substituted into the cross-entropy loss function, that is, L p , where x i is the i-th sample input to the softmax function, y i ​N is the number of samples; here is because the Softmax function converts the output value of the classification into a probability distribution with a range of [0, 1] and a sum of 1, obtains the classification probability of each point, and calculates the cross-entropy loss function of the class probability predicted by the model and the true value, thereby improving the effect of the model;

[0062] wherein n i is the estimated normal vector, n′ i is the real normal vector.

[0063] As a preferred embodiment of the present application, the method further comprises that the training process is as follows,

[0064] Each point in the LED three-dimensional point cloud is taken as a training sample, and the network model is trained using the Adam optimizer, the initial learning rate is 0.001, the learning rate is reduced by 0.7 every 20 epochs, β1 is 0.9, β2 is 0.999, and the weight decay rate is 10 -4 .

[0065] In the preferred embodiment, the network model is trained by the Adam optimizer, and typical parameter selection is performed, which can accelerate the convergence of the network.

[0066] As a preferred embodiment of the present application, specifically, the balance factor μ is 0.7.

[0067] The present application also provides an LED wire bonding point cloud classification device based on deep learning, comprising the following:

[0068] A three-dimensional point cloud acquisition module is used to acquire a target LED three-dimensional point cloud.

[0069] A feature extraction module comprises,

[0070] A first feature extraction unit is used to perform PointNet-based feature extraction on the target LED three-dimensional point cloud to obtain a first feature vector,

[0071] A second feature extraction unit is used to perform normal vector estimation based on normal differential on the target LED three-dimensional point cloud to obtain a second feature vector, and the loss function of the process is L=(1-μ)×L p +μ×L d , wherein L p is the loss function of the PointNet-based feature extraction, L d is the loss function of the normal vector estimation based on normal differential, and μ is a balance factor and is artificially set.

[0072] The feature fusion module is configured to splice the first feature vector and the second feature vector to obtain complete point cloud features

[0073] The point classification module is configured to perform action on the complete point cloud features through a fully connected network to complete point cloud classification.

[0074] The modules described as separate components can or can not be physically separate, and the components shown as modules can or can not be physical modules, i.e., can be located in one place or distributed to multiple network modules. Some or all of the modules can be selected to achieve the purpose of the scheme in the embodiment according to actual needs.

[0075] In addition, each functional module in each embodiment of the present application can be integrated in one processing module, or each module can be physically present alone, or two or more modules can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module.

[0076] The integrated module, if realized in the form of a software functional module and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer-readable storage medium. When the processor executes the computer program, the steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer-readable medium can include any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0077] Although the description of the present application has been quite detailed and particularly described with respect to several embodiments, it is not intended to be limited to any of these details or embodiments or any special embodiment, but should be considered as providing a broad interpretation of the prior art to the appended claims by reference, so as to effectively cover the intended scope of the present application. In addition, the present application is described above in embodiments that the inventor can foresee, and the purpose is to provide a useful description, and non-essential modifications to the present application that have not yet been foreseen can still represent equivalent modifications of the present application.

[0078] The above merely illustrates the preferred embodiments of the present application, and the present application is not limited to the above-mentioned embodiments. Any modification and change of the technical solutions and / or embodiments within the scope of the present application shall fall within the protection scope of the present application.

Claims

1. A point cloud classification method for LED bonding wires based on deep learning, characterized in that, Including the following: Obtain the 3D point cloud of the target LED; The first feature vector is obtained by performing PointNet-based feature extraction on the 3D point cloud of the target LED, and the second feature vector is obtained by performing normal vector estimation based on normal differentiation. The loss function for this process is: ,in The loss function is based on PointNet feature extraction. The loss function is based on the normal vector estimation using the normal derivative. This is a manually set balancing factor. The first feature vector and the second feature vector are concatenated to obtain the complete point cloud features; Based on the complete point cloud features, a fully connected network is used to complete point cloud classification. Specifically, PointNet-based feature extraction includes the following processes: Assuming the input LED 3D point cloud P= ,in This represents the total number of point clouds for the i-th point and N points; In the input point cloud, each point is referred to as a "point". Using each point as the center, a distance of 120 is defined. The set of points enclosed by the radius of the sphere is used as the dataset of the LED point cloud; The farthest point sampling method is used to downsample the input set for each point, and the number of downsampling points is set to 512. The grouping layer uses the KNN nearest neighbor algorithm to divide the 512 points into 32 groups, with 16 points in each group; Then, the feature vector of that point is extracted using the PointNet network; The above process is called a set abstraction. After three set abstractions, a 1x1021 feature vector is finally obtained. Specifically, normal vector estimation based on the normal differential includes the following process: Because of the significant dimensional variations between the LED bonding wires and the wedge-shaped support, the normal differential algorithm performs well in identifying the differences in normal vector features at different scales. Therefore, the normal differential algorithm is used to estimate the normal vector of a point to provide feature identification for that point. For each point p in the target LED's 3D point cloud, using a larger radius... Estimate the normal vector of a point p. Using a smaller radius Estimate the normal vector of the same point p The average of the differences between the two normal vectors is the DoN feature of point p.

2. The LED bonding wire point cloud classification method based on deep learning according to claim 1, characterized in that, Specifically, the first feature vector and the second feature vector are concatenated to obtain the complete point cloud features, including the following: The first feature vector is a 1021-dimensional feature extracted based on PointNet, and the second feature vector is a 3-dimensional feature extracted based on normal vector estimation using normal differentiation. The 1021-dimensional feature and the 3-dimensional feature are concatenated to obtain a complete 1024-dimensional feature.

3. The LED bonding wire point cloud classification method based on deep learning according to claim 1, characterized in that, Specifically, and The result is obtained based on the softmax function, and includes the following process. First, the softmax function is used to normalize the output value to the range of 0 to 1. Substituting the softmax function formula into the cross-entropy loss function yields... , ,in, For the i-th sample input to the softmax function, The true probability distribution is given, and N is the number of samples. ,in For the estimated normal vector, This is the true normal vector.

4. The LED bonding wire point cloud classification method based on deep learning according to claim 1, characterized in that, The method also includes the following training process. The sphere neighborhood corresponding to each point in the LED 3D point cloud is used as a training sample. The network model is trained using the Adam optimizer with an initial learning rate of 0.001, decreasing by 0.7 every 20 epochs. It is 0.

9. The value is 0.999, and the weight decay rate is... .

5. The LED bonding wire point cloud classification method based on deep learning according to claim 1, characterized in that, Specifically, the balance factor The value is 0.

7.

6. A point cloud classification device for LED bonding wires based on deep learning, characterized in that, The apparatus comprising the steps of the method according to any one of claims 1-5, wherein the apparatus includes the following: The 3D point cloud acquisition module is used to acquire the 3D point cloud of the target LED. The feature extraction module includes, The first feature extraction unit is used to perform PointNet-based feature extraction on the three-dimensional point cloud of the target LED to obtain a first feature vector. The second feature extraction unit is used to perform normal vector estimation based on normal differentiation on the three-dimensional point cloud of the target LED to obtain the second feature vector. The loss function of this process is: ,in The loss function is based on PointNet feature extraction. The loss function is based on the normal vector estimation using the normal derivative. This is a manually set balancing factor. The feature fusion module is used to concatenate the first feature vector and the second feature vector to obtain complete point cloud features. The point classification module is used to classify the point cloud by operating through a fully connected network based on the complete features of the point cloud.

Images