Surface defect detection method, device and storage medium based on three-dimensional point cloud

Abstract

The present invention discloses a surface defect detection method based on three-dimensional point cloud, including: obtaining the original point cloud, which is obtained by scanning the surface of the sample to be tested with a three-dimensional camera; obtaining the contribution of each point in the original point cloud through a simplified sampling algorithm, and forming the points in the original point cloud whose contribution is greater than or equal to the preset contribution into the point cloud to be tested; obtaining the standard model point cloud, aligning the point cloud to be tested with the standard model point cloud, so that the point cloud to be tested and the standard model point cloud are aligned to the same position; detecting defects by comparing the aligned point cloud to be tested with the standard model point cloud, and obtaining the defect feature quantity of the surface of the sample to be tested. The present invention can speed up the operation speed of point cloud data and improve the operation efficiency by performing simplified sampling, rough registration and fine registration on the original point cloud. And through the three-dimensional surface defect detection method of point to reconstructed surface projection, it can accurately detect the defect information of the sample surface without affecting the accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision and image processing, and in particular to a surface defect detection method, device and storage medium based on three-dimensional point cloud. Background Art

[0002] The surface defects of products mainly include differences in volume, shape, length, etc., such as bulges, depressions, holes, damage, etc. on the surface of the product. Defects on the appearance of the product surface not only affect the user experience, but may also cause losses to the user and even cause safety accidents. For example, defects on the surface of train rails will reduce their service life and pose a safety hazard.

[0003] At present, product surface defect detection is mainly divided into manual detection and automated detection. Among them, manual detection relies on human eyes to detect the appearance quality of products, but human eyes can get tired, and different people have different standards for judging defects, so there will inevitably be omissions and misjudgments. Automated detection mainly obtains product surface data through machine vision equipment and uses corresponding algorithms to process and judge the surface data. It has the advantages of high precision, intuitive effect and high detection efficiency, and is widely used in many fields such as circuit board production, electronic device appearance inspection, industrial and agricultural production, etc. Machine vision detection refers to the use of a certain optical image sensor under a certain lighting environment to collect the surface data of a product and generate a two-dimensional image or a three-dimensional point cloud in the photosensitive element inside the sensor, and extract the target information of interest through subsequent visual algorithm processing.

[0004] The existing product surface defect detection technologies include two-dimensional visual defect detection and three-dimensional visual defect detection. Among them, the two-dimensional defect detection technology uses many mature digital image processing technologies, which are fast, efficient and easy to implement. However, the two-dimensional detection based on images can only identify the grayscale, texture, contour and other information of the product, and the accuracy of defect detection depends to a large extent on the imaging quality of the product. It is also difficult to reflect the three-dimensional characteristics of the sample such as depth and height.

[0005] Three-dimensional visual defect detection uses a three-dimensional camera to scan the surface of a product to obtain point cloud data (point cloud data contains information such as the three-dimensional spatial coordinates and surface strength of the product, and can express the three-dimensional characteristics of the product). The point cloud data is processed by a series of three-dimensional point cloud algorithms (such as noise removal, streamlined sampling, three-dimensional data matching, surface reconstruction, etc.) and then compared with the point cloud model data of the standard sample to determine whether there are defects on the product surface. Compared with two-dimensional plane images, three-dimensional point cloud data can better reflect three-dimensional spatial defects such as scratches, grooves, and damage on the surface of the product. However, the original point cloud obtained by using a three-dimensional camera is of a large order of magnitude, and the distribution is complex and uneven, with a lot of noise and redundant data, which reduces the efficiency and accuracy of defect detection; three-dimensional point cloud data is discrete and difficult to process, and the original two-dimensional image processing algorithm is no longer applicable. Summary of the invention

[0006] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes a surface defect detection method based on three-dimensional point cloud, which can improve the operation efficiency without affecting the accuracy and accurately detect the defect information of the sample surface.

[0007] In a first aspect, an embodiment of the present invention provides a surface defect detection method based on a three-dimensional point cloud, comprising:

[0008] Obtaining an original point cloud, wherein the original point cloud is obtained by scanning the surface of the sample to be tested with a three-dimensional camera;

[0009] Acquire the contribution of each point in the original point cloud, and perform simplified sampling on the original point cloud according to the contribution to obtain the point cloud to be tested;

[0010] Obtain the standard model point cloud, and register the point cloud to be tested with the standard model point cloud so that the point cloud to be tested and the standard model point cloud are aligned to the same position;

[0011] Defects are detected by comparing the registered point cloud to be tested with the standard model point cloud, and the defect feature quantity of the surface of the sample to be tested is obtained.

[0012] The embodiment of the present invention can speed up the operation of point cloud data and improve the operation efficiency by simplifying the sampling of the original point cloud, introducing multiple constraints such as normal vectors, angles, and curvatures in the rough registration stage, and combining the elimination strategy of statistical distance, normal vectors, and joint point cloud feature descriptors in the fine registration stage. The present invention also improves the operation efficiency without affecting the accuracy by adopting a three-dimensional surface defect detection method of point-to-reconstructed surface projection in the defect detection stage, and accurately detects the defect information of the sample surface.

[0013] Further, obtaining the contribution of each point in the original point cloud, and performing simplified sampling on the original point cloud according to the contribution to obtain the point cloud to be tested, including:

[0014] Obtaining a standard model point cloud, and determining matching points in the standard model point cloud corresponding to each point in the original point cloud;

[0015] Obtaining a first projection distance from each point in the original point cloud to a section where a corresponding matching point in the standard model point cloud is located;

[0016] By moving the original point cloud according to a preset displacement, a second projection distance from each point of the moved original point cloud to the section where the corresponding matching point in the standard model point cloud is located is obtained;

[0017] Generate a contribution of each point in the original point cloud according to a difference between the first projection distance and the second projection distance;

[0018] Points in the original point cloud whose contribution is greater than or equal to a preset contribution are used to form a point cloud to be tested.

[0019] The present invention measures the contribution of each point in the original point cloud by the change caused by a small movement of the original point cloud, and samples the points with large contributions, thereby streamlining the original point cloud with high density and more redundant information while retaining the important information in the original point cloud to the greatest extent, thereby improving the efficiency and accuracy of subsequent defect detection.

[0020] Furthermore, the registering the point cloud to be measured with the point cloud of the standard model so that the point cloud to be measured and the point cloud of the standard model are aligned to the same position includes:

[0021] The point cloud to be tested and the point cloud of the standard model are roughly aligned through the coplanar four-point consistency algorithm;

[0022] The ICP algorithm is used to obtain all matching point pairs in the point cloud to be tested and the standard model point cloud after rough matching, and each matching point pair includes point p i and q i , the p i is a point in the point cloud to be measured, and the q i Yes i The corresponding matching points in the standard model point cloud;

[0023] The erroneous registration point pairs that appear in the registration process are eliminated through the preset elimination strategy to achieve precise registration.

[0024] Furthermore, the method of roughly registering the point cloud to be measured with the point cloud of the standard model by using a coplanar four-point consistency algorithm includes:

[0025] Selecting a plurality of coplanar four-point pairs in the point cloud to be measured to form a basic point pair set, wherein each coplanar four-point pair is in the same plane but any three points therein are not in the same straight line;

[0026] Obtaining a preset distance error, and obtaining a coplanar matching point pair set corresponding to each coplanar four-point pair in the standard model point cloud according to the preset error distance;

[0027] The rigid body transformation parameters of each point in each coplanar matching point pair are obtained through the singular value decomposition algorithm, and the four coplanar matching points corresponding to the optimal rigid body transformation parameters are selected as the optimal coplanar matching point pair for the coplanar four-point pair.

[0028] Further, each coplanar four-point pair selected in the point cloud to be measured meets the first preset condition, and each coplanar matching point pair in the coplanar matching point pair set meets the second preset condition;

[0029] The first preset condition includes: a first angle between the normal vector directions of at least one pair of points in each of the four coplanar point pairs is greater than a first preset angle threshold;

[0030] The second preset condition includes:

[0031] The angle formed by the diagonal lines connecting the four points in the coplanar four-point pair is the second angle, the diagonal lines connecting the four points of each coplanar matching point pair in the standard model point cloud form a third angle corresponding to the second angle, and the absolute value of the difference between the second angle and the third angle is less than or equal to the second angle threshold;

[0032] The average curvature between the points in the coplanar matching point pair set and the corresponding points in the coplanar four point pairs is obtained, and the absolute value of the difference between the average curvatures is less than or equal to a preset curvature threshold.

[0033] Furthermore, the preset elimination strategy includes one or more of the following: elimination strategy based on statistical distance, elimination strategy based on point cloud normal vector angle, elimination strategy based on joint feature descriptor; wherein,

[0034] The elimination strategy based on statistical distance includes: calculating the midpoint p of each matching point i To point q i The straight-line distance of the tangent plane, each matching point pair includes two points p i and q i , the p i is a point in the point cloud to be measured, and the q i Yes i corresponding matching points in the standard model point cloud; calculating the straight-line distance average and straight-line distance standard deviation of all matching point pairs according to the straight-line distance of all matching point pairs, and eliminating wrong matching point pairs according to the straight-line distance average and straight-line distance standard deviation;

[0035] The elimination strategy based on the angle of the point cloud normal vector includes: calculating the fourth angle of the normal vectors of the two points in each matching point pair; obtaining a third preset angle threshold, and eliminating the matching point pairs whose fourth angle is greater than the third preset angle threshold as incorrect matching point pairs;

[0036] The elimination strategy based on the joint feature descriptor is as follows: compare the UPF features of the two points in each matching point pair to obtain the feature difference; obtain the feature preset threshold, and when the feature difference of the matching point pair exceeds the feature preset threshold, eliminate the matching point pair as an incorrect matching point pair.

[0037] In the registration process of the point cloud to be tested and the point cloud of the standard model, the present invention introduces multiple constraints such as normal vector, angle, curvature, etc. to the coplanar four-point consistency algorithm in the coarse registration stage, thereby improving the search efficiency of the coplanar four-point set; for the wrongly registered point pairs in the fine registration stage, a removal method combining statistical distance, normal vector, and joint point cloud feature descriptor is designed to improve the success rate of removing wrongly matched point pairs. Therefore, the present invention accelerates the speed of the registration process and improves the accuracy of the registration through the "coarse first, fine later" registration method.

[0038] Further, the defect detection based on the registered measured point cloud data and the standard model point cloud includes:

[0039] Reconstruct the standard model surface triangular network by using the marching cube algorithm, and obtain the triangle with the smallest spacing corresponding to each point in the point cloud to be tested on the standard model surface triangular network as a triangular facet by using the KD-search tree;

[0040] Project each point in the point cloud to be measured onto the corresponding triangular face to obtain the direction and distance of each projection point;

[0041] According to the direction and distance of the projection point and the normal vector of the triangular facet, all the triangular facets are divided into concave defect triangular facets, convex defect triangular facets and missing facets;

[0042] The position, surface area and volume of the defects existing in the sample are obtained according to the concave defect triangle patch, the convex defect triangle patch and the missing triangle patch.

[0043] Further, according to the direction and distance of the projection point and the normal vector of the triangle, all triangles are divided into concave defect triangles, convex defect triangles and missing triangles, including:

[0044] When the distance of the projection point is greater than a preset specified threshold, and the direction of the projection point is the same as the normal vector of the triangular facet, the triangular facet is a concave defective triangular facet;

[0045] When the distance between the projection points is greater than a preset threshold, and the direction of the projection points is different from the normal vector of the triangular facet, the triangular facet is a convex defect triangular facet;

[0046] A triangle that is neither a concave defect triangle nor a convex defect triangle is a missing triangle.

[0047] During the defect detection process, the present invention reconstructs the surface triangular network of the standard model and obtains triangular facets through the marching cube algorithm, and then detects and classifies the defects of each triangular facet according to the direction and distance of the projection point of the point cloud to be tested on the triangular facet, so as to calculate whether there are defects in the original point cloud. If defects do exist, the position, surface area, volume and other characteristic quantities of the defects can also be calculated.

[0048] In a second aspect, an embodiment of the present invention provides a surface defect detection device based on a three-dimensional point cloud, comprising a processor and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, so that the processor can execute the surface defect detection method based on a three-dimensional point cloud as described above.

[0049] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions that are only executed by a computer, and the computer-executable instructions are used to enable the computer to execute the surface defect detection method based on three-dimensional point cloud as described above.

[0050] Compared with traditional two-dimensional image defect detection, the surface defect detection method based on three-dimensional point cloud of the present invention has obvious advantages in three-dimensional space defects such as scratches, grooves, and breakages. It contains richer details and is more accurate in detecting spatial information such as the volume and surface area of ​​defects. In addition, in comparison with traditional three-dimensional detection technology, the original point cloud is simplified by obtaining contribution, multiple constraints are introduced in the four-point-one-plane algorithm, multiple features are combined in eliminating erroneous matching points, and the standard model surface triangulation network is reconstructed using the moving cube algorithm in defect detection. This not only improves the accuracy of defect detection, but also has obvious speed advantages. The results show that without affecting the accuracy, the improvement of point cloud simplification and point cloud registration in this method improves the computing efficiency, and accurately detects the defect information on the sample surface, providing a quantitative basis for subsequent repair and re-inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a schematic flow chart of a specific embodiment of a surface defect detection method based on three-dimensional point cloud in an embodiment of the present invention;

[0052] Figure 2It is a schematic flow chart of a specific embodiment of obtaining the contribution of each point in the original point cloud in the surface defect detection method based on three-dimensional point cloud in an embodiment of the present invention;

[0053] Figure 3 It is a schematic flow chart of a specific embodiment of registering a point cloud to be tested with a standard model point cloud in a surface defect detection method based on three-dimensional point cloud in an embodiment of the present invention;

[0054] Figure 4 It is a schematic flow chart of a specific embodiment of roughly aligning a point cloud to be tested with a standard model point cloud in a surface defect detection method based on three-dimensional point cloud in an embodiment of the present invention;

[0055] Figure 5 It is a schematic diagram of a specific embodiment of coplanar four point pairs and coplanar matching point pairs in a surface defect detection method based on three-dimensional point cloud in an embodiment of the present invention;

[0056] Figure 6 It is a flow chart of a specific embodiment of the surface defect detection method based on three-dimensional point cloud in the embodiment of the present invention, which detects defects by comparing the registered point cloud to be tested with the standard model point cloud. DETAILED DESCRIPTION

[0057] The following will be combined with the embodiments to clearly and completely describe the concept of the present invention and the technical effects produced, so as to fully understand the purpose, characteristics and effects of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, other embodiments obtained by those skilled in the art without creative work are all within the scope of protection of the present invention.

[0058] In the description of the embodiments of the present invention, if "several" is involved, it means more than one, if "multiple" is involved, it means more than two, if "greater than", "less than" or "exceed" is involved, it should be understood as not including the number itself, if "above", "below" or "within" is involved, it should be understood as including the number itself. If "first" or "second" is involved, it should be understood as being used to distinguish technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0059] refer to Figure 1 , Figure 1 The present invention is specifically directed to a surface defect detection method based on three-dimensional point cloud, which includes the following steps.

[0060] Step S1, obtaining an original point cloud, which can be obtained by scanning the surface of a sample to be tested with a three-dimensional camera.

[0061] The original point cloud obtained by using a three-dimensional camera is of large magnitude and has a complex and uneven distribution. It contains a lot of noise and redundant data, which reduces the efficiency and accuracy of defect detection. Therefore, the original point cloud needs to be streamlined and sampled. The embodiment of the present invention uses the following steps to retain important information to the greatest extent while streamlining the data volume and denoising.

[0062] Step S2, obtaining the contribution of each point in the original point cloud, and performing simplified sampling on the original point cloud according to the contribution to obtain the point cloud to be tested.

[0063] In view of the problem that the original point cloud data has a large density and redundant information, the embodiment of the present invention proposes a method for simplifying the sampling of the original point cloud based on the registration contribution of the point pair, introducing the error function of the registration stage into the sampling stage, measuring the contribution of the point pair to the registration by the error change caused by small movement, and then sampling the points with large contribution. The specific implementation process is as follows: Figure 2 shown.

[0064] Step S21, obtaining the standard model point cloud, and determining the matching points in the standard model point cloud corresponding to each point in the original point cloud.

[0065] Specifically, first obtain the standard model point cloud of the standard sample corresponding to the sample to be tested in this embodiment. The standard model point cloud is the three-dimensional data of the surface of the standard sample collected by a three-dimensional camera. Assume that P is the original point cloud, p is a specific sampling point in the original point cloud P; Q is the standard model point cloud, and q is the corresponding matching point of p in the standard model point cloud Q.

[0066] Step S22, obtaining the first projection distance from each point in the original point cloud to the section where the corresponding matching point in the standard model point cloud is located.

[0067] In this embodiment, a point p in the original point cloud P i Match point q to the standard model point cloud Q i The projection distance of the location on the tangent plane can be calculated according to formula (1), which is as follows:

[0068]

[0069] Among them, p i is a specific point in the original point cloud P, q i is p after rigid body transformation i The matching point of the point in the standard model point cloud Q, n i for q i When the rotation matrix R is relatively small, R can be approximated by vector linearization, that is, point p i The operation performed by Rp i +t becomes [r×p i + t], where r = (rx ,r y ,r z ) represents the rotation vector, t=(t x ,t y ,t z ) represents the translation vector, and the projection distance function changes from (1) to (2):

[0070]

[0071] In the embodiment of the present invention, the first projection distance of each point in the original point cloud can be obtained by formula (2). Of course, in a specific implementation, the first projection distance can also be calculated and obtained by other methods.

[0072] Step S23, by moving the original point cloud according to a preset displacement, a second projection distance from each point of the moved original point cloud to the section where the corresponding matching point in the standard model point cloud is located is obtained.

[0073] Step S24, generating a contribution of each point in the original point cloud according to the difference between the first projection distance and the second projection distance, and then forming a point cloud to be measured by combining the points in the original point cloud whose contribution is greater than or equal to a preset contribution.

[0074] After the original point cloud is moved according to the preset displacement, the second projection distance is calculated by formula (2) and then the difference between the first projection distance and the second projection distance of each point in the original point cloud is calculated as its contribution. The embodiment of the present invention simplifies the original point cloud by selecting points with a contribution greater than or equal to a preset contribution, and places points with high contribution in the data set of the point cloud to be tested, and removes points with low contribution. Therefore, the present invention can retain the important information of the original point cloud to the greatest extent while streamlining the data volume and removing noise.

[0075] In another embodiment of the present invention, the contribution of each point in the original point cloud can be directly obtained according to the preset displacement. i Make a slight disturbance The original point cloud will be shifted according to the preset displacement If any point p in the original point cloud P is moved, i to q i The change in the distance of the tangent plane is shown in formula (3):

[0076]

[0077] If you click p i In translation space, a small translation is made, and Δd remains unchanged; or in rotation space, point p iFor small rotations, Δd remains unchanged. In both cases, the original point cloud point p i It may be located on a relatively flat surface, with a low contribution in the point cloud registration process and a high redundancy. Points with these two conditions in the original point cloud are not sampled (i.e., not added to the point cloud to be measured).

[0078] In the embodiment of the present invention, the contribution of each point in the original point cloud can also be calculated according to the following method:

[0079] Specifically, by partial differentiation of formula (2), it can be converted into a linear system Cx = b to be solved, where x is a 6×1 vector representing 6 rigid body transformation parameters, b is the registration deviation, and C is the covariance matrix obtained from each set of corresponding point pairs, as shown in formula (4). The covariance matrix C evaluates the original point cloud P when it moves slightly relative to the standard model point cloud Q. When , the degree of change in the overall registration deviation is affected, and the change ΔE of the target energy function E is shown in formula (5).

[0080]

[0081]

[0082] The SVD (singular value decomposition) method is also used to decompose the matrix C of formula (4) to calculate the six eigenvectors x 1 、x 2 , …, x 6 , and its corresponding 6 eigenvalues ​​λ 1 , 2 , …, λ 6 , each eigenvector represents a movement, consisting of a rotation component around an axis and a translation component along the axis. Assume that the six eigenvalues ​​of the matrix C satisfy λ 1 ≥λ 2 ≥λ 3 ≥λ 4 ≥λ 5 ≥λ 6 , if any eigenvalue λ k With λ 1 If the difference is small, it means that moving along any feature vector direction has a great impact on the target energy function, that is, it contributes more to point cloud registration. The sampling judgment standard of the embodiment of the present invention is c=λ 1 / λ 6 ,The sampling rule of this embodiment is to collect points whose c is as close to 1 as possible into the point cloud to be measured.

[0083] Step S3, obtaining the standard model point cloud, and registering the point cloud to be tested with the standard model point cloud so that the point cloud to be tested and the standard model point cloud are aligned to the same position.

[0084] After simplifying the original point cloud to obtain the point cloud to be measured, it is necessary to align the point cloud to be measured with the standard model point cloud. The alignment process of the embodiment of the present invention includes coarse alignment and fine alignment. For the specific implementation process, refer to Figure 3 .

[0085] Step S31 , roughly aligning the point cloud to be measured with the standard model point cloud through a coplanar four-point consistency algorithm.

[0086] The present invention is an embodiment that adopts the FPCS (Four-Point Congruent Sets) algorithm, wherein the FPCS algorithm is that the affine relationship (distance, ratio, etc.) between point pairs remains unchanged in the space rigid body transformation.

[0087] The specific steps of roughly aligning the point cloud to be measured and the point cloud of the standard model by using the coplanar four-point consistency algorithm are as follows: Figure 4 As shown, including:

[0088] Step S311, selecting a plurality of coplanar four-point pairs in the point cloud to be measured to form a basic point pair set, where each coplanar four-point pair is in the same plane but any three points therein are not in the same straight line.

[0089] For example, four points a, b, c, and d are selected from the data set P of the point cloud to be tested as a coplanar four-point pair, which satisfies the condition that they are in a plane but any three of them are not in a straight line, and their intersection point e is obtained, and two affine transformation ratios r are calculated based on the intersection point e. 1 、r 2 , these two affine scale values ​​remain unchanged during rigid body transformation.

[0090] The present invention selects some coplanar four-point pairs from the original point cloud P to form a basic point pair set B = {B 1 , B 2 ,…,B n Usually, when selecting, three points with large distance difference are first selected at different positions in the point cloud space, and at the same time, they meet the condition of not being in the same straight line. Then, a fourth point is selected that can be in the same plane with these three points within the deviation range, and any three of these four points are not in a straight line, to construct a coplanar four-point pair.

[0091] Step S312, obtaining a preset distance error, and obtaining a set of coplanar matching point pairs corresponding to each coplanar four-point pair in the standard model point cloud according to the preset error distance.

[0092] The embodiment of the present invention uses the principle that the affine scale value remains unchanged in the rigid body transformation. First, in the standard model point cloud Q, any two points q are selected. 1 ,q 2, according to the affine invariant ratio r obtained in the point cloud P to be tested 1 、r 2 , calculate the middle point e by formula (6) 1 、e 2 :

[0093]

[0094] In the standard model point cloud Q, if there are two sets of such points a′, b′ and c′, d′, one set is composed of r 1 The calculated e 1 With another group consisting of r 2 The calculated e 2 If the error between them is within the preset distance error (the preset distance error is set according to the user's requirements for the accuracy of the registration), then these two groups of points may be the corresponding matching points of the four coplanar point pairs a, b, c, d in the measured point cloud P after rigid body transformation. There may be multiple groups of points a', b' and c', d' in the standard model point cloud Q that meet the requirements of the preset distance error. All points that meet the requirements in the standard model point cloud Q constitute a coplanar matching point pair set.

[0095] For a given pair of coplanar four points B i ={a,b,c,d} and the preset distance error ε, calculate the distance d between the two segments 1 =||ab||, d 2 =||cd||, first search for the point cloud Q of the standard model with a distance of d 1 ±ε and d 2 Point pair S within ±ε 1 , S 2 , then in S 1 , S 2 In the calculation of the middle point e according to the affine invariant rule, find and B i The possible coplanar matching point pair set U = {U 1 , U 2 ,…,U n}.

[0096] Step S313, obtaining the rigid body transformation parameters of each point in each coplanar matching point pair set by singular value decomposition algorithm, and selecting four coplanar matching points corresponding to the optimal rigid body transformation parameters as the optimal coplanar matching point pair of the coplanar four point pair.

[0097] Match 4 points to each pair in U i , use SVD decomposition method to solve the rigid body transformation relationship T i, and count how many points of the standard model point cloud Q are less than the given threshold δ from the point cloud P to be tested. ANN (Approximate Nearest Neighbor) can be used to improve the judgment speed. Specifically, first select a part of the points from the standard model point cloud Q and use T i Perform rigid body transformation and count how many points in the point cloud P to be tested are within the given threshold δ. If enough points are counted, perform the same calculation on the remaining points and i Score (such as 1 / MSE), and finally select the one with the highest score as the optimal rigid body transformation parameter T i For each B in B i , the above steps are calculated until the final optimal rigid body transformation parameter T is obtained opt , the optimal rigid body transformation parameter T opt The corresponding four coplanar matching points are used as the optimal coplanar matching point pair, and then the point cloud to be tested and the point cloud of the standard model are roughly aligned according to the optimal coplanar matching point pair.

[0098] In this embodiment, there are two main factors that affect the calculation efficiency of the FPCS algorithm: one is the selection of the basic point pair set B in the point cloud to be tested P, and the other is the search for the coplanar matching point pair set U in the standard model point cloud Q. In order to select basic coplanar four-point pairs with more obvious features and stronger representativeness, and at the same time reduce the number of point pairs in the coplanar matching point pair set that may be matched in the standard model point cloud Q, and accelerate the convergence speed of the FPCS algorithm, this embodiment of the present invention proposes the following two optimization conditions: 1. Each coplanar four-point pair selected in the point cloud to be tested meets the first preset condition; 2. Each coplanar matching point pair in the coplanar matching point pair set meets the second preset condition.

[0099] Among them, the first preset condition of this embodiment includes: the first angle between the normal vector directions of at least one pair of points in each of the four coplanar point pairs is greater than a first preset angle threshold.

[0100] Specifically, the defect detection algorithm of the embodiment of the present invention introduces the constraint of the normal vector when selecting the basic point pair set B, thereby improving the representativeness of the extracted coplanar four-point pairs and reducing the number of points in the set B. For example, {p 1 ,p 2 ,p 3 ,p 4} is a set of coplanar four point pairs extracted from the point cloud P to be tested. In addition to meeting the above-mentioned requirements that any three points are not collinear and widely distributed, it is also necessary to meet the requirement that the normal vector direction angle of at least one pair of points among the four points is greater than the first preset angle threshold τ.

[0101] The second preset condition of this embodiment includes: the angle formed by the diagonal lines connecting the four points in the coplanar four-point pair is the second angle (for example, the second angle is less than 90°), the diagonal lines connecting the four points of each coplanar matching point pair in the standard model point cloud form a third angle corresponding to the second angle, and the absolute value of the difference between the second angle and the third angle is less than or equal to the second angle threshold; the average curvature between the points in the coplanar matching point pair set and the corresponding points in the coplanar four-point pair is obtained, and the absolute value of the difference in the average curvature is less than or equal to the preset curvature threshold.

[0102] Specifically, Figure 5 As shown in the figure, when obtaining possible coplanar matching point pairs in the standard model point cloud Q, in addition to considering the requirement of affine invariance, it is also necessary to introduce constraints on the degree of intersection angle and average curvature. 1 p 2 With p 3 p 4 The angle θ between 1 The corresponding matching point pair q in the standard model point cloud 1 q 2 With q 3 q 4 The angle θ between 2 To satisfy |θ 1 -θ 2 |≤ε, and each corresponding point pair {p i ,q i The average curvature between {c pi ,c qi}To meet |c pi -c qi |≤τ, where ε and τ are the second angle threshold and the preset curvature threshold predefined by the user. After introducing these two constraints, the defect detection method of the present invention will greatly reduce the number of possible points in the matching point set U, thereby improving the efficiency of verification and calculation.

[0103] Step S32, using the ICP algorithm to obtain all matching point pairs in the point cloud to be tested and the standard model point cloud after rough matching, each matching point pair includes point p i and q i , p i is a point in the point cloud to be measured, q i Yes i The corresponding matching points in the standard model point cloud.

[0104] Step S33, using a preset elimination strategy to eliminate incorrectly registered point pairs that appear during the registration process to achieve precise registration. The preset elimination strategy includes one or more of the following: a elimination strategy based on statistical distance, a elimination strategy based on the angle of point cloud normal vectors, and a elimination strategy based on joint feature descriptors.

[0105] For the elimination strategy based on statistical distance, the strategy is to calculate the midpoint p of each matching point i To point q i The straight-line distance of the tangent plane, each matching point pair includes two points p i and q i , p i is a point in the point cloud to be measured, q i Yes i The corresponding matching points in the standard model point cloud; the straight-line distance average and straight-line distance standard deviation of all matching point pairs are calculated according to the straight-line distance of all matching point pairs, and the wrong matching point pairs are eliminated according to the straight-line distance average and straight-line distance standard deviation.

[0106] For example, for each set of registration point pairs p in the point cloud to be tested P and the standard model point cloud Q i ,q i , calculate p i to q i The distance d between the point cloud tangent planes i =||p i -q i ||·n qi Assume that d i The distribution of satisfies the Gaussian distribution, and all d i The average value μ d and standard deviation σ d , the present invention removes the matching point pairs that satisfy equation (7) because their distances are too different and they are most likely mismatched point pairs.

[0107] d i >μ d +ασ d (7)

[0108] In the above formula, α is the standard deviation coefficient, and its size can be set to adjust the intensity of the elimination.

[0109] The elimination strategy based on the angle of the point cloud normal vector includes calculating the fourth angle of the normal vectors of the two points in each matching point pair; obtaining a third preset angle threshold, and eliminating the matching point pairs whose fourth angle is greater than the third preset angle threshold as incorrect matching point pairs.

[0110] For example, for each pair of matching points p in the point cloud P to be tested and the standard model point cloud Q i ,q i , calculate the angle θ(n pi , n qi). Because after the rough registration process of the point clouds, the two point clouds have reached a basically aligned position, and the normal vector directions of the corresponding matching points should be similar. If the angle between their normal vectors is too large, it is very likely that the pair of matching points is unmatched and should be removed. The present invention sets the angle degree threshold θ τ (i.e., the third preset angle threshold), remove the pi , n qi )>θ τ The matching point pair p i ,q i .

[0111] For the elimination strategy based on the joint feature descriptor, the strategy is to compare the UPF features of the two points in each matching point pair to obtain the feature difference; obtain the feature preset threshold, and when the feature difference of the matching point pair exceeds the feature preset threshold, the matching point pair is eliminated as an incorrect matching point pair.

[0112] Specifically, the defect detection method of the present invention compares each pair of matching points p in the point cloud P to be tested and the point cloud Q of the standard model. i ,q i The UPF feature of the point is used to measure its feature difference. If the difference is greater than a given threshold, it should be removed as a mismatched point pair.

[0113] The present invention can use the above three strategies at the same time to eliminate mismatched point pairs, minimize the number of mismatched point pairs, and select one or two of them in combination with actual scenarios. The defect detection method of the embodiment of the present invention aligns the streamlined original point cloud (i.e., the point cloud to be tested), and introduces multiple constraints such as normal vectors, angles, and curvature to the FPCS algorithm in the coarse registration stage under the "coarse first and fine later" registration idea, thereby improving the search efficiency of the coplanar four-point set. In addition, in order to eliminate the problem of eliminating mismatched point pairs in the fine registration stage, a elimination strategy combining statistical distance, normal vector, and joint point cloud feature descriptor is designed to improve the degree of removal of mismatched point pairs and accelerate the convergence of the registration algorithm.

[0114] Step S4, detecting defects by comparing the registered point cloud to be tested with the standard model point cloud, and obtaining defect feature quantities on the surface of the sample to be tested.

[0115] After aligning the point cloud data to be tested and the point cloud data of the standard model to the same position, they can be compared with each other to detect defects in the sample to be tested. At this stage, the defect detection method of the embodiment of the present invention first uses the MC (MarchingCubes) algorithm to reconstruct the triangular network of the surface of the standard model, and then projects the point cloud to be tested onto the triangular facet, clusters it according to the direction and distance of the projection point, and then calculates the position, surface area, volume and other characteristic quantities of the defect. The specific steps are as follows: Figure 5 shown.

[0116] Step S41, reconstructing the standard model surface triangular network by marching cube algorithm, and obtaining the triangle with the smallest spacing corresponding to each point in the point cloud to be tested on the standard model surface triangular network as a triangular facet by KD-search tree.

[0117] After obtaining the triangulated network of the standard model point cloud surface, the present invention calculates the normal vector through the plane determined by the endpoint coordinates of each triangular facet. Since the calculated normal vector direction may have positive and negative directions, different normal vector directions may affect subsequent defect detection. The embodiment of the present invention stipulates that the normal vector direction toward the outside of the model of the standard model point cloud is the positive direction, and the normal vector direction of the entire triangulated network can be adjusted by using the method of regional growth propagation. The process of determining the normal vector direction of the entire triangulated network is as follows:

[0118] Step 1: Select the triangle T with the smallest coordinate value in the Z-axis direction as the seed triangle, and adjust its normal vector direction to the correct direction n facing outward. T ; Step 2, for the three triangles T adjacent to the three sides of T 1 , T 2 , T 3 , if its normal vector n i Satisfy n T ·n i <0, indicating that cos(n T , n i )<0, then n i Reverse, otherwise keep n i The direction remains unchanged; Step 3, T 1 , T 2 , T 3 As the new seed triangle, repeat the above process and perform breadth-first traversal until the normal vector directions of all triangle faces are adjusted to the correct direction.

[0119] Since the point cloud data scanned by the three-dimensional camera in the present invention only contains the three coordinate position information of x, y, and z, it can usually detect defects such as missing or concave-convex types on the surface of the sample to be tested. The specific defect detection process of the embodiment of the present invention is as follows Figure 5 As shown:

[0120] Step S42, projecting each point in the point cloud to be measured onto a corresponding triangular facet to obtain the direction and distance of each projection point.

[0121] Specifically, for each point p in the point cloud data P to be measured i , use KD-search tree to search for the triangle patch T with the smallest spacing on the surface triangulation network of the standard model point cloud Q r ; then from q iPoint to the triangle T r Make a projection and get the coordinates of the perpendicular point q′.

[0122] Step S43, according to the direction and distance of the projection point and the normal vector of the triangle, all the triangles are divided into concave defect triangles, convex defect triangles and missing triangles.

[0123] Specifically, the user sets a preset specified threshold. When the distance of the projection point is greater than the preset specified threshold, and the direction of the projection point is the same as the normal vector of the triangle facet, the triangle facet is a concave defect triangle facet; when the distance of the projection point is greater than the preset specified threshold, and the direction of the projection point is different from the normal vector of the triangle facet, the triangle facet is a convex defect triangle facet; if it is neither a concave defect triangle facet nor a convex defect triangle facet, it is a missing triangle facet.

[0124] For example, if the length of the vector qq′||qq′|| is less than or equal to the specified threshold τ, it means that there is no defect on the surface of the sample at point q. If the length of the vector qq′||qq′|| is greater than the specified threshold τ, and the vector direction is consistent with the triangle patch T r If the normal vector direction is the same as that of the concave defect triangle, then the triangle is a concave defect triangle. r Join the concave defect queue together with q; if r , then the triangle is a convex defect triangle and is added to the convex defect queue. Then, each triangle in the surface triangle network of the standard model point cloud Q is traversed. If it is neither added to the concave defect queue nor the convex defect queue, it means that no point is projected onto this triangle. Then, the triangle is a missing triangle and is added to the missing defect queue.

[0125] Step S44, obtaining the position, surface area and volume of the defects existing in the sample according to the concave defect triangle patch, the convex defect triangle patch and the missing triangle patch.

[0126] Specifically, by traversing each triangle in the concave defect queue, convex defect queue and missing defect queue, and using the region growing propagation method, two triangles with one overlapping edge are merged and clustered, until the remaining triangles in each queue are non-adjacent and independent defective triangles.

[0127] Then the surface area of ​​each defective triangular patch is counted (that is, the areas of all triangles in each defective patch are added together), and defects with an area less than the specified threshold are filtered out, because they may be caused by noise or system errors of the 3D camera. For each defective patch after filtering, if it is a missing type defect, the surface area and position of the missing part can be obtained; if it is a concave-convex type defect, the volume is integrated using the coordinates of the projection point and the projection distance on it to obtain the volume and position of the concave-convex defect, thereby determining the position, surface area, volume and other information of various types of defects.

[0128] Compared with traditional two-dimensional image defect detection, the defect detection method of the present invention has obvious advantages in three-dimensional defects such as scratches, grooves, and breakages. It contains richer details and is more accurate in detecting spatial information such as the volume and surface area of ​​defects. Moreover, in comparison with traditional three-dimensional detection technology, it has obvious speed advantages. The improvement of traditional algorithms and defect detection ideas are verified using real-shot point cloud models. Experiments have shown that, without affecting the accuracy, the defect detection method of the present invention improves the computing efficiency by improving point cloud simplification and point cloud registration, and accurately detects the defect information on the sample surface, providing a quantitative basis for subsequent repair and re-inspection.

[0129] A surface defect detection device based on three-dimensional point cloud comprises a processor and a memory connected to the processor in communication; wherein the memory stores instructions executable by the processor, so that the processor can execute the following Figure 1-6 The surface defect detection method based on three-dimensional point cloud described in the corresponding embodiment.

[0130] The surface defect detection device in this embodiment is different from the above Figure 1-6 The surface defect detection method in the corresponding embodiment belongs to the same concept, and its specific implementation process is detailed in the corresponding method embodiment, and the technical features in the method embodiment are applicable in the device embodiment, which will not be repeated here.

[0131] A computer-readable storage medium stores instructions that can only be executed by a computer, wherein the computer-executable instructions are used to cause the computer to execute the following Figure 1-6 The surface defect detection method based on three-dimensional point cloud described in the corresponding embodiment.

[0132] The computer-readable storage medium in this embodiment is the same as the above Figure 1-6 The surface defect detection method in the corresponding embodiment belongs to the same concept, and its specific implementation process is detailed in the corresponding method embodiment, and the technical features in the method embodiment are applicable in the device embodiment, which will not be repeated here.

[0133] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0134] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed. The functional units and modules in the embodiments can be integrated into a processor, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0135] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0136] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0137] In the embodiments provided in the present application, it should be understood that the disclosed surface defect detection method, device and storage medium can be implemented in other ways. For example, the surface defect detection device embodiment described above is only illustrative.

[0138] In addition, each functional unit in each embodiment of the present application may be integrated into one processor, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0139] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or interface switching device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0140] The embodiments of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in the relevant technical field without departing from the purpose of the present invention. In addition, the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

Claims

1. A surface defect detection method based on three-dimensional point cloud, characterized in that: include: Obtaining an original point cloud, wherein the original point cloud is obtained by scanning the surface of the sample to be tested with a three-dimensional camera; The contribution of each point in the original point cloud is obtained by an error function, and the original point cloud is sampled according to the contribution to obtain a point cloud to be tested, wherein the contribution of each point in the point cloud to be tested is greater than or equal to a preset contribution; Obtain the standard model point cloud, and register the point cloud to be tested with the standard model point cloud so that the point cloud to be tested and the standard model point cloud are aligned to the same position; Defects are detected by comparing the registered point cloud to be tested with the standard model point cloud, and defect feature quantities on the surface of the sample to be tested are obtained; The registering the point cloud to be tested and the point cloud of the standard model so that the point cloud to be tested and the point cloud of the standard model are aligned to the same position includes: The point cloud to be tested and the point cloud of the standard model are roughly aligned through the coplanar four-point consistency algorithm; The ICP algorithm is used to obtain all matching point pairs in the point cloud to be tested and the standard model point cloud after rough registration. Each matching point pair includes a point and , is a point in the point cloud to be measured, yes The corresponding matching points in the standard model point cloud; Through the preset elimination strategy, the wrong registration point pairs appearing in the registration process are eliminated to achieve precise registration; The step of obtaining the contribution of each point in the original point cloud and performing simplified sampling on the original point cloud according to the contribution to obtain the point cloud to be tested includes: Obtaining a standard model point cloud, and determining matching points in the standard model point cloud corresponding to each point in the original point cloud; Obtaining a first projection distance from each point in the original point cloud to a section where a corresponding matching point in the standard model point cloud is located; By moving the original point cloud according to a preset displacement, a second projection distance from each point of the moved original point cloud to the section where the corresponding matching point in the standard model point cloud is located is obtained; Generate a contribution of each point in the original point cloud according to a difference between the first projection distance and the second projection distance; Points in the original point cloud whose contribution is greater than or equal to a preset contribution are used to form a point cloud to be tested.

2. The method according to claim 1, characterized in that The method of roughly registering the point cloud to be measured and the point cloud of the standard model by using the coplanar four-point consistency algorithm includes: Selecting a plurality of coplanar four-point pairs in the point cloud to be measured to form a basic point pair set, wherein each of the coplanar four-point pairs is in the same plane but any three points therein are not in the same straight line; Obtaining a preset distance error, and obtaining a coplanar matching point pair set corresponding to each coplanar four-point pair in the standard model point cloud according to the preset error distance; The rigid body transformation parameters of each point in each coplanar matching point pair are obtained through the singular value decomposition algorithm, and the four coplanar matching points corresponding to the optimal rigid body transformation parameters are selected as the optimal coplanar matching point pair of the coplanar four point pair.

3. The method according to claim 2, characterized in that Each coplanar four-point pair selected in the point cloud to be measured meets the first preset condition, and each coplanar matching point pair in the coplanar matching point pair set meets the second preset condition; The first preset condition includes: a first angle between the normal vector directions of at least one pair of points in each of the four coplanar point pairs is greater than a first preset angle threshold; The second preset condition includes: The angle formed by the diagonal lines connecting the four points in the coplanar four-point pair is the second angle, the diagonal lines connecting the four points of each coplanar matching point pair in the standard model point cloud form a third angle corresponding to the second angle, and the absolute value of the difference between the second angle and the third angle is less than or equal to the second angle threshold; The average curvature between the points in the coplanar matching point pair set and the corresponding points in the coplanar four point pairs is obtained, and the absolute value of the difference between the average curvatures is less than or equal to a preset curvature threshold.

4. The method according to claim 1, characterized in that The preset elimination strategy includes one or more of the following: elimination strategy based on statistical distance, elimination strategy based on point cloud normal vector angle, elimination strategy based on joint feature descriptor; wherein, The elimination strategy based on statistical distance includes: calculating the midpoint of each matching point To point The straight-line distance of the tangent plane, each matching point pair includes two points and , is a point in the point cloud to be measured, yes corresponding matching points in the standard model point cloud; calculating the straight-line distance average and straight-line distance standard deviation of all matching point pairs according to the straight-line distance of all matching point pairs, and eliminating wrong matching point pairs according to the straight-line distance average and straight-line distance standard deviation; The elimination strategy based on the angle of the point cloud normal vector includes: calculating the fourth angle of the normal vectors of the two points in each matching point pair; obtaining a third preset angle threshold, and eliminating the matching point pairs whose fourth angle is greater than the third preset angle threshold as incorrect matching point pairs; The elimination strategy based on the joint feature descriptor is as follows: compare the PPF features of the two points in each matching point pair to obtain the feature difference; obtain the feature preset threshold, and when the feature difference of the matching point pair exceeds the feature preset threshold, eliminate the matching point pair as an incorrect matching point pair.

5. The method according to claim 1, characterized in that The defect detection by comparing the registered point cloud to be tested with the standard model point cloud includes: Reconstruct the standard model surface triangular network by using the marching cube algorithm, and obtain the triangle with the smallest spacing corresponding to each point in the point cloud to be tested on the standard model surface triangular network as a triangular facet by using the KD-search tree; Project each point in the point cloud to be measured onto the corresponding triangular face to obtain the direction and distance of each projection point; According to the direction and distance of the projection point and the normal vector of the triangular facet, all the triangular facets are divided into concave defect triangular facets, convex defect triangular facets and missing facets; The position, surface area and volume of the defects existing in the sample to be tested are obtained according to the concave defect triangular facets, the convex defect triangular facets and the missing triangular facets.

6. The method according to claim 5, characterized in that According to the direction and distance of the projection point and the normal vector of the triangular facet, all triangular facets are divided into concave defect triangular facets, convex defect triangular facets and missing triangular facets, including: When the distance of the projection point is greater than a preset specified threshold, and the direction of the projection point is the same as the normal vector of the triangular facet, the triangular facet is a concave defective triangular facet; When the distance between the projection points is greater than a preset threshold, and the direction of the projection points is different from the normal vector of the triangular facet, the triangular facet is a convex defect triangular facet; A triangle that is neither a concave defect triangle nor a convex defect triangle is a missing triangle.

7. A surface defect detection device based on three-dimensional point cloud, characterized in that: It comprises a processor and a memory which is in communication with the processor; wherein the memory stores instructions which can be executed by the processor, so that the processor can execute the surface defect detection method based on three-dimensional point cloud as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the surface defect detection method based on three-dimensional point cloud according to any one of claims 1 to 6.

Images