Product surface defect detection method and related hardware

Abstract

The embodiment of the invention provides a product surface defect detection method and related hardware, and the method comprises the steps: detecting a visible light image shot on the surface of a to-be-detected product, and determining a first abnormal image region in the visible light image; determining abnormal position points in a second abnormal image area according to the depth of each position point of the second abnormal image area corresponding to the first abnormal image area; the second abnormal image area is an image area in a point cloud image shot on the surface of the to-be-detected product, and the second abnormal image area and the corresponding first abnormal image area are the same in position on the surface of the to-be-detected product; determining a third abnormal image area in the visible light image according to the position of the abnormal position point in the visible light image; and determining a defect position on the surface of the to-be-detected product according to the first abnormal image area and the third abnormal image area.

Description

Technical Field

[0001] The present invention relates to the field of automated detection technology, and in particular to a product surface defect detection method and related hardware. Background Art

[0002] During the production process of products, surface defect detection of products is mainly done manually. However, manual detection mainly relies on manual measurement and observation of the product surface to find defects on the product surface. On the one hand, manual inspection of the product surface may result in inaccurate inspection results due to objective failure of the inspectors to see defects with the naked eye or subjective negligence. On the other hand, manual inspection of the product surface consumes a lot of manpower and is inefficient. Therefore, how to use equipment automation to detect surface defects of products is a problem that needs to be solved urgently. Summary of the invention

[0003] The embodiment of the present invention provides a product surface defect detection method and related hardware, which are used to solve the problem of low accuracy of manual detection of product surfaces in the prior art.

[0004] An embodiment of the present invention provides a method for detecting surface defects of a product, comprising:

[0005] Detecting a visible light image taken of the surface of the product to be detected, and determining a first abnormal image area in the visible light image;

[0006] Determine the abnormal position point in the second abnormal image area according to the depth of each position point of the second abnormal image area corresponding to the first abnormal image area; the second abnormal image area is an image area in the point cloud image taken of the surface of the product to be detected, and the second abnormal image area and the corresponding first abnormal image area are at the same position on the surface of the product to be detected;

[0007] Determining a third abnormal image region in the visible light image according to the position of the abnormal location point in the visible light image;

[0008] The defect position on the surface of the product to be inspected is determined according to the first abnormal image area and the third abnormal image area.

[0009] Optionally, the detecting a visible light image taken from the surface of the product to be detected and determining a first abnormal image area in the visible light image includes:

[0010] A pre-trained single-shot multi-box detector (SSD) model is used to detect a visible light image taken from the surface of the product to be inspected, and a defect confidence level of each image area in the visible light image belonging to a defect image area is determined;

[0011] An image region whose defect confidence is less than a preset confidence threshold is determined as a first abnormal image region.

[0012] Further optionally, the method further comprises:

[0013] The image area whose defect confidence is greater than or equal to the preset confidence threshold value and the corresponding position on the surface of the product to be inspected are determined as the defect position.

[0014] Optionally, determining the abnormal position point in the second abnormal image area according to the depth of each position point in the second abnormal image area corresponding to the first abnormal image area includes:

[0015] Dividing the second abnormal image area into at least one sub-image area according to a preset size;

[0016] For any sub-image area, statistics are collected on the distribution of the depths of each position point in the sub-image area in each preset value interval, a reference depth range is determined according to the target preset value interval, and a position point whose depth exceeds the reference depth range is determined as the abnormal position point;

[0017] The target preset numerical range is the preset numerical range containing the largest number of depths.

[0018] Optionally, determining at least one third abnormal image region in the visible light image according to the position of the abnormal position point in the visible light image includes:

[0019] In the visible light image, the pixels corresponding to each of the abnormal position points are first eroded and then expanded to obtain a third abnormal image region.

[0020] Optionally, determining the defect position on the surface of the product to be inspected according to the first abnormal image area and the third abnormal image area includes:

[0021] For any third abnormal image region, if the third abnormal image region satisfies the following conditions, the corresponding position of the third abnormal region on the surface of the product to be inspected is determined as the defect position:

[0022] The area of ​​the third abnormal area is larger than a preset area;

[0023] The ratio of the first area to the second area is greater than a preset ratio threshold; wherein the first area is the area of ​​the overlapping area between the third abnormal image area and the target first abnormal image area, and the second area is the area of ​​the target first abnormal image area.

[0024] Based on the same inventive concept, an embodiment of the present invention further provides a product surface defect detection device, comprising:

[0025] A visual inspection module, used to inspect a visible light image captured on the surface of the product to be inspected, and determine a first abnormal image area in the visible light image;

[0026] a depth preliminary detection module, used to determine the abnormal position points in the second abnormal image area according to the depths of the position points of the second abnormal image area corresponding to the first abnormal image area; the second abnormal image area is an image area in the point cloud image taken of the surface of the product to be detected, and the second abnormal image area and the corresponding first abnormal image area are at the same position on the surface of the product to be detected;

[0027] an abnormal position point mapping module, used to determine a third abnormal image area in the visible light image according to the position of the abnormal position point in the visible light image;

[0028] A depth-vision joint detection module is used to determine the defect position on the surface of the product to be inspected based on the first abnormal image area and the third abnormal image area.

[0029] Based on the same inventive concept, an embodiment of the present invention further provides an electronic device, including:

[0030] A memory for storing program instructions;

[0031] The processor is used to call the program instructions stored in the memory and execute the product surface defect detection method according to the obtained program instructions.

[0032] Based on the same inventive concept, an embodiment of the present invention further provides a readable storage medium, wherein the readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the product surface defect detection method.

[0033] Based on the same inventive concept, an embodiment of the present invention further provides a computer program product, which includes: a computer program code, and when the computer program code runs on a computer, the computer executes the product surface defect detection method.

[0034] The beneficial effects of the present invention are as follows:

[0035] The product surface defect detection method and related hardware provided by the embodiment of the present invention first determine the first abnormal image area by detecting the visible light image, then count the depth of the second abnormal image area corresponding to the first abnormal image area in the point cloud image, screen out the abnormal position points, and then further analyze the third abnormal image area corresponding to the abnormal position points in the visible light image, so as to finally determine the location of the defect on the product surface. Compared with manual detection, the use of automatic analysis by equipment greatly improves the efficiency of product surface defect detection. And by combining the visible light image and the point cloud image taken of the surface of the product to be detected, it is possible to effectively identify small defects on the product surface that are difficult to find by visual analysis alone, thereby effectively improving the detection accuracy of product surface defects. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 One of the flow charts of the product surface defect detection method provided by an embodiment of the present invention;

[0037] Figure 2 A second flow chart of a method for detecting product surface defects provided by an embodiment of the present invention;

[0038] Figure 3 Schematic diagram of the training process of the SSD model in an embodiment of the present invention;

[0039] Figure 4 Schematic diagram of the process of calibrating the coordinate transformation relationship in an embodiment of the present invention;

[0040] Figure 5 A schematic diagram of a portion of the process flow of a product surface defect detection method provided by an embodiment of the present invention;

[0041] Figure 6 A schematic diagram of the structure of a product surface defect detection device provided by an embodiment of the present invention;

[0042] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and understandable, the present invention will be further described below with reference to the accompanying drawings and examples. However, the exemplary embodiments can be implemented in various forms and should not be understood as being limited to the embodiments described herein; on the contrary, these embodiments are provided to make the present invention more comprehensive and complete, and to fully convey the concepts of the exemplary embodiments to those skilled in the art. The same figure marks in the figures represent the same or similar structures, and thus their repeated descriptions will be omitted. The words expressing position and direction described in the present invention are all explained using the accompanying drawings as examples, but changes can be made as needed, and the changes made are all included in the scope of protection of the present invention. The drawings of the present invention are only used to illustrate the relative position relationship and do not represent the true proportions.

[0044] It should be noted that specific details are described in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in a variety of other ways different from those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The subsequent description of the specification is a preferred embodiment of the present application, but the description is for the purpose of illustrating the general principles of the present application and is not intended to limit the scope of the present application. The scope of protection of the present application shall be determined by the definition of the attached claims.

[0045] The product surface defect detection method and related hardware provided by the embodiment of the present invention are specifically described below in conjunction with the accompanying drawings.

[0046] The present invention provides a method for detecting surface defects of a product. Figure 1 and Figure 2 As shown, including:

[0047] S110, obtaining a visible light image of the surface of the product to be inspected.

[0048] In the specific implementation process, the visible light image can be a black and white image or a color image. If the visible light is a color image, then the color image can be an image in a red-green-blue (RGB) format or an image in a YUV format. The embodiments of the present invention are not limited in detail here.

[0049] S120: Detect the visible light image.

[0050] S131. Determine a first abnormal image region in a visible light image.

[0051] In the specific implementation process, a machine learning model algorithm can be used to detect visible light images, such as You Only Look Once (YOLO), RetinaNet, Region Convolutional Neural Networks (R-CNN), Single Shot Multibox Detector (SSD), etc.

[0052] As an optional implementation manner, the step S120 of detecting the visible light image specifically includes:

[0053] The pre-trained SSD model is used to detect the visible light image and determine the defect confidence that each image area in the visible light image belongs to the defect image area.

[0054] The step S131, determining the first abnormal image area in the visible light image, specifically includes:

[0055] An image region whose defect confidence is less than a preset confidence threshold is determined as a first abnormal image region.

[0056] Furthermore, if the SSD model is used to detect visible light images, then Figure 2 As shown, the method also includes:

[0057] S132: Determine the position corresponding to the image area whose defect confidence is greater than or equal to a preset confidence threshold on the surface of the product to be inspected as the defect position.

[0058] In this way, the SSD model can be used to first perform a preliminary inspection on the visible light image to find more obvious defects on the surface of the product to be inspected.

[0059] In the specific implementation process, Figure 3 As shown, the SSD model can be trained through the following steps to obtain a model that can detect visible light images:

[0060] S210, establishing a training sample set and a test sample set based on visible light images taken of the surfaces of different products to be inspected. For any training sample in the training sample set or any test sample in the test sample set, the sample feature is the visible light image, and the sample label is each image region in the visible light image and whether each image region belongs to the classification category of the defect image region.

[0061] In the specific implementation process, firstly, visible light images taken of the surfaces of different products to be inspected can be obtained, and each image area and the classification category of whether each image area belongs to the defective image area can be annotated in each visible light image by manual annotation or automatic machine annotation. Afterwards, for any visible light image, the visible light image is used as a sample feature, and each image area corresponding to the visible light image and the classification category of whether each image area belongs to the defective image area are used as sample labels, so as to establish a sample. After the sample is established, the sample can be divided by the retention method, and multiple samples can be divided into training sample sets and test sample sets according to a certain ratio (for example, the ratio of the number of training samples to test samples is 5:1). Alternatively, the sample can be divided by the cross-validation method, and the sample can be divided into multiple sample sets. In each round of training, one sample set is selected as the test sample set, and the remaining sample sets are composed of training sample sets. The test sample sets are different in different rounds of training.

[0062] In the process of establishing samples, the samples can also be preprocessed (including removing duplicate samples, deleting samples with missing sample labels, etc.) to facilitate the SSD model to process the data in the subsequent training process.

[0063] S220, for any training sample in the training sample set, the SSD model is trained with the sample features of the training sample as the input of the SSD model and the sample labels of the training sample as the output of the SSD model to obtain a training result.

[0064] S230: Use the test sample set to test the SSD model and obtain model evaluation indicators.

[0065] In the specific implementation process, at least one of the following can be used as the model evaluation index of the SSD model:

[0066] (1) Accuracy (ACC):

[0067] Accuracy refers to the ratio of correctly classified samples to the total number of samples. The calculation formula of ACC is as follows:

[0068]

[0069] Among them, TP (True Positive) represents the number of samples predicted from positive samples to positive samples, TN (True Negative) represents the number of samples predicted from negative samples to negative samples, FN (False Negative) represents the number of samples predicted from positive samples to negative samples, and FP (False Positive) represents the number of samples predicted from negative samples to positive samples. In the technical solution of the present application, the visible light images taken of the products to be inspected with defects on the surface can be used as positive samples, and the visible light images taken of the products to be inspected without defects on the surface can be used as negative samples; of course, the visible light images taken of the products to be inspected without defects on the surface can also be used as positive samples, and the visible light images taken of the products to be inspected with defects on the surface can be used as negative samples, and the embodiments of the present invention are not limited here.

[0070] (2) Precision:

[0071] Precision refers to the proportion of true positive samples in the output results. The precision P calculation formula is as follows:

[0072]

[0073] (3) Recall:

[0074] The recall rate indicates the proportion of positive samples that output correct results among all positive samples. The calculation formula of the recall rate R is as follows:

[0075]

[0076] (4) The harmonic mean of precision and recall F1:

[0077] Since precision and recall are a set of contradictory parameters, in order to take into account both precision P and recall R, F1 is used to evaluate the machine learning model in machine learning. The calculation formula of F1 is as follows:

[0078]

[0079] (5) Intersection over Union (IoU)

[0080] The intersection over union ratio represents the ratio of the intersection of the predicted image area and the actual image area to the union of the two during the image detection process. When the IoU is greater than the preset threshold, it is considered that the model divides the image area of ​​the visible light image correctly, otherwise it is considered that the model divides the image area of ​​the visible light image incorrectly.

[0081] (6) Mean Average Precision (mAP):

[0082] mAP is the average of the average precision (AP) corresponding to multiple classification categories, which is used to measure the detection performance of the model on the entire data set. The calculation formula of mAP is as follows:

[0083]

[0084] Among them, i is the serial number of the classification category, N is the total number of classification categories, AP i is the AP of the i-th classification category. AP is the lower area of ​​the Precision-Recall (PR) curve with recall as the horizontal axis and precision as the vertical axis.

[0085] S240: Determine whether the training end condition is met.

[0086] In the specific implementation process, the training end condition can be that the model evaluation index is less than the corresponding preset end threshold, that is, the model evaluation index represents whether the training end condition is met; or the model evaluation index can be further combined with the number of training times, and the training end condition is set to the model evaluation index is less than the corresponding preset end threshold or the number of training times reaches the preset number threshold, that is, the model evaluation index and the number of training times jointly represent whether the training end condition is met. The embodiments of the present invention are not too limited here.

[0087] If it is determined that the training end condition is not met, step S250 is executed; if the training end condition is met, step S260 is executed.

[0088] S250, determining the loss function value corresponding to the training result, and updating the model parameters of the SSD model according to the loss function value. Return to step S220.

[0089] S260: Use the current SSD model parameters as the final SSD model parameters.

[0090] In the specific implementation process, the loss function value L can be determined by the following method:

[0091] L=k1·L c +k2·L L

[0092] Among them, L c is the classification loss function value of the SSD model, L L It is the image area partition loss function value of the SSD model, and k1 and k2 are preset weighting coefficients.

[0093] In the specific implementation process, the classification loss function value L c The cross entropy loss function value can be used:

[0094]

[0095] Among them, i is the serial number of the classification category, N is the total number of classification categories, and y i The result of whether the model output belongs to the i-th category (if the model output belongs to the i-th category, then y i =1; otherwise y i =0), p i Output the confidence score of the model belonging to the i-th category.

[0096] In the specific implementation process, the image area division loss function value L L You can use smooth L1 loss (Smooth L1Loss):

[0097]

[0098] Here, x is the difference between the image region bounding box predicted by the model and the actual image region bounding box.

[0099] In the specific implementation process, the model parameters of the SSD model can be updated in any of the following ways:

[0100] (1) Batch Gradient Descent Algorithm:

[0101] The batch gradient descent algorithm calculates the loss function value based on the full amount of training samples, calculates the gradient of the loss function value on each model parameter, and then performs an update on the model parameters.

[0102] Specifically, for any model parameter, it is updated according to the following formula:

[0103] p t =p t-1 -η t g t

[0104] Among them, t is the version number of the model parameter p, p t is the updated model parameter, p t-1 is the model parameter before updating, η t is the learning rate, g t is the gradient vector.

[0105] (2) Stochastic Gradient Descent Algorithm:

[0106] Since the batch gradient descent algorithm needs to train all training samples once before updating the model parameters, the algorithm converges slowly when the number of training samples is large.

[0107] Stochastic gradient descent only randomly extracts one sample at a time to calculate the loss function value, and then performs an update on the parameters. Compared with the batch gradient descent algorithm, the iteration speed is fast.

[0108] Specifically, for any model parameter, the update formula used is the same as the batch gradient descent algorithm, which will not be repeated here.

[0109] (3) Mini-batch Gradient Descent Algorithm:

[0110] The mini-batch gradient descent algorithm is a compromise between the batch gradient descent algorithm and the stochastic gradient descent algorithm. All training samples are divided into multiple groups. The loss function value is calculated after each group of training samples is trained, and then the parameters are updated once. The mini-batch gradient descent algorithm inherits the advantages of the batch gradient descent algorithm, that is, the stable descent direction of the loss value curve, and the high iterative efficiency of the stochastic gradient descent algorithm.

[0111] Specifically, for any model parameter, the update formula used is the same as the batch gradient descent algorithm, which will not be repeated here.

[0112] (4) Momentum Gradient Descent Algorithm:

[0113] The Momentum algorithm is different from the previous three algorithms. It uses a momentum idea to make the direction jitter less severe when the loss function value curve decreases.

[0114] Specifically, for any model parameter, it is updated according to the following formula:

[0115]

[0116] Among them, t is the version number of the model parameter p, p t is the updated model parameter, p t-1 is the model parameter before updating, η t is the learning rate, g t is the gradient vector, γ is the momentum hyperparameter, v t is the parameter update amount for this time, v t-1 The last parameter update amount.

[0117] (5) Adaptive learning rate optimization (AdaGrad) algorithm:

[0118] In the Momentum algorithm, all model parameters share the same learning rate, while the AdaGrad algorithm solves the problem of all parameters sharing the same learning rate.

[0119] Specifically, for any model parameter, it is updated according to the following formula:

[0120]

[0121] Among them, t is the version number of the model parameter p, p t is the updated model parameter, p t-1 is the model parameter before updating, η is the learning rate, g t is the gradient vector, s t is the parameter update amount for this time, s t-1 is the last parameter update amount, ε is a preset constant, and ⊙ is the element-wise multiplication of the vector.

[0122] For each iteration, use s t The vector records all previous gradients g t Accumulation of square values ​​(second-order momentum), initial time s t is 0. Then, when calculating the update amount, the corresponding learning rate is calculated according to the squared gradient values ​​accumulated by the parameters before. The parameter with a large accumulated gradient value corresponds to a small learning rate, and vice versa.

[0123] (6) Root Mean Square Propagation (RMSProp) algorithm:

[0124] The RMSProp algorithm solves the problem of the AdaGrad algorithm's learning rate decreasing continuously and slow learning in the later stages.

[0125] Specifically, for any model parameter, it is updated according to the following formula:

[0126]

[0127] Among them, t is the version number of the model parameter p, p t is the updated model parameter, p t-1 is the model parameter before updating, η is the learning rate, g t is the gradient vector, γ is the momentum hyperparameter, s t is the parameter update amount for this time, s t-1 is the last parameter update amount, ε is a preset constant, and ⊙ is the element-wise multiplication of the vector.

[0128] The RMSPro algorithm introduces momentum into the calculation of gradient accumulation, and uses the hyperparameter γ to control the weight of the current gradient value and the historical accumulated gradient value. Each time, only the previous s are accumulated.t-1 , rather than the entire value, so as not to cause s t The problem of the learning rate decreasing is that the gradient value of a parameter is small. t The accumulated value at the corresponding position will also decrease and will not continue to increase.

[0129] (7)Adam algorithm:

[0130] The Adam algorithm takes into account both adaptive learning rate and momentum gradient, and has better results.

[0131] Specifically, for any model parameter, it is updated according to the following formula:

[0132]

[0133] Among them, t is the version number of the model parameter p, p t is the updated model parameter, p t-1 is the model parameter before updating, η is the learning rate, g t is the gradient vector, γ1 and γ2 are momentum hyperparameters, and v t is the first-order update of the parameters, v t-1 is the first-order update of the last parameter, s t is the second-order update of the parameters, s t-1 is the second-order update of the last parameter, ε is a preset constant, and ⊙ is the element-wise multiplication of the vector.

[0134] S140: Acquire a point cloud image of the surface of the product to be inspected.

[0135] In the specific implementation process, the point cloud image can be generated by time-of-flight (ToF) measurement method, structured light scanning method, stereo vision method and the like. For example, the ToF measurement method can use a laser scanning sensor to emit a laser to the surface of the product to be detected and measure the laser reflection time of each surface position, and generate a point cloud image of the surface of the product to be detected according to the laser reflection time of each surface position. Alternatively, the structured light scanning method can emit structured light to the surface of the product to be detected and capture a structured light image of the surface of the product to be detected, and generate a point cloud image of the surface of the product to be detected according to the difference between the captured structured light image and the emitted structured light pattern. Alternatively, the stereo vision method can use a multi-eye (e.g., binocular) visible light camera to capture visible light images of the surface of the product to be detected at different positions, and by comparing the differences between the visible light images captured simultaneously by the visible light cameras at different positions, a point cloud image of the surface of the product to be detected is generated using triangulation. The embodiments of the present invention are not overly limited here.

[0136] S150. For any first abnormal image region, determine a second abnormal image region in the point cloud image that is located at the same position on the surface of the product to be inspected as the first abnormal image region.

[0137] In the specific implementation process, the positions of the visible light camera used to capture the visible light image of the surface of the product to be inspected and the sensor used to generate the point cloud image of the surface of the product to be inspected are fixed, so the coordinate transformation relationship between the pixel coordinates of the visible light image and the position point coordinates of the point cloud image can be pre-calibrated, and then the second abnormal image area corresponding to the same position on the surface of the product to be inspected is determined for the first abnormal image area according to the calibrated coordinate transformation relationship. Figure 4 As shown, the coordinate conversion relationship between the pixel coordinates of the visible light image and the position point coordinates of the point cloud image can be calibrated in the following way:

[0138] S300, pre-assuming an initial coordinate transformation relationship.

[0139] S310: for any visible light image, determine the target pixel coordinates corresponding to the mark position in the visible light image, and determine the target position point coordinates corresponding to the mark position in the point cloud image corresponding to the visible light image. The mark position can be selected from important contour positions on the surface of the product to be inspected (e.g., corner positions, convex positions, concave positions, center positions of special parts, etc.).

[0140] S320: For any visible light image, use the current coordinate conversion relationship to convert the target first coordinates corresponding to each mark position into a fitting second coordinate.

[0141] S330. For any visible light image, respectively determine the fitting difference between the fitting second coordinates corresponding to the same mark position and the target second coordinates.

[0142] If the first coordinate is the pixel coordinate of the visible light image, the second coordinate is the position point coordinate of the point cloud image; if the first coordinate is the position point coordinate of the point cloud image, the second coordinate is the pixel coordinate of the visible light image.

[0143] S340: Determine whether the accuracy requirement of the coordinate transformation relationship is met based on the fitting difference of each marker position of each visible light image.

[0144] If the step S340 determines that the coordinate transformation relationship accuracy requirement is not met, step S350 is executed; if the step S340 determines that the coordinate transformation relationship accuracy requirement is met, step S360 is executed.

[0145] S350, adjusting the current coordinate transformation relationship based on the fitting differences of the positions of the marks. Return to step S320.

[0146] S360: Determine the current coordinate transformation relationship as the coordinate transformation relationship for final calibration.

[0147] In the specific implementation process, iterative optimization algorithms such as gradient descent algorithm, simulated annealing algorithm, ant colony algorithm, genetic algorithm, particle filter algorithm, particle swarm algorithm, etc. can be used to adjust the current coordinate transformation relationship based on the fitting differences of each marker position. The embodiments of the present invention do not make too many restrictions here.

[0148] S160. For any second abnormal image region, determine the abnormal position points in the second abnormal image region according to the depths of each position point in the second abnormal image region.

[0149] As an optional implementation, the qualified depth range of the position points corresponding to each position on the surface of the product to be inspected can be predetermined if there are no defects on the surface of the product to be inspected. In step S160, the depths of each position point in the second abnormal image area are compared to see whether they belong to the corresponding qualified depth range, and the position points that exceed the qualified depth range are determined as abnormal position points.

[0150] As another optional implementation, Figure 5 As shown, for any second abnormal image area, the abnormal position point can be determined in the following manner:

[0151] S161. Divide the second abnormal image area into at least one sub-image area according to a preset size.

[0152] For example, the second abnormal image region may be gridded into at least one sub-image region in a size of 10 mm×10 mm.

[0153] S162: for any sub-image region, count the distribution of the depths of each position point in the sub-image region in each preset value interval, determine a reference depth range according to a target preset value interval, and determine the position point whose depth exceeds the reference depth range as the abnormal position point. The target value interval is the preset value interval containing the largest number of depths.

[0154] For example, a sub-image area contains 10,000 position points. After pre-dividing the depth value into multiple preset numerical intervals according to the interval length of 1mm: [0, 1mm), [1mm, 2mm), [2mm, 3mm)..., the number of depths contained in each preset numerical interval is counted. If [9mm, 10mm) contains 8563 depths, which contains the largest number of depths, then [9mm, 10mm) is determined as the target numerical interval, and the reference depth range [7.5mm, 11.5mm] is determined based on the midpoint of the target numerical interval 9.5mm and the allowable floating range ±2mm. The position points corresponding to the depths that do not belong to the reference depth range [7.5mm, 11.5mm] are determined as abnormal position points.

[0155] S170: Determine a third abnormal image region in the visible light image according to the position of the abnormal position point in the visible light image.

[0156] As an optional implementation, the pixels corresponding to each abnormal position point in the visible light image may be directly determined, and the pixels corresponding to each abnormal position point may be continuously fitted and connected to obtain a third abnormal image region.

[0157] Considering that the depth of a small number of position points in the point cloud image may be abnormal due to factors such as accidental errors in the depth determined by the point cloud image for the surface of the product to be inspected, and the real surface defects of the product will appear as abnormalities in a continuous area rather than isolated position points in the image. Then as another optional implementation, in the visible light image, the pixels corresponding to each abnormal position point are first eroded and then expanded to obtain a third abnormal image area.

[0158] Erosion is the operation of "shrinking" or "thinning" objects in a binary image. Mathematically, erosion is defined as a set operation. A is eroded by B as Defined as:

[0159]

[0160] in, Represents the area covered by the structural element B when the position is z in image A. In other words, A eroded by B is the set of the origin positions of all structural elements, where the translated B does not overlap with the background of A.

[0161] Dilation is the operation of "lengthening" or "thickening" a binary image. Mathematically, dilation is defined as a set operation. The dilation of A by B is denoted as A⊕B, which is defined as:

[0162]

[0163] in, represents the area covered by the structural element B when the position is z in image A. In short, A is dilated by B and is the set of all structural element origin positions, where the mapped and translated B overlaps at least some part of A.

[0164] By performing the operation of first corroding and then dilating, isolated abnormal position points that cannot be connected to other abnormal position points can be effectively filtered out, thereby reducing the occurrence of misidentification of defect positions.

[0165] S180. Determine a defect position on the surface of the product to be inspected according to the first abnormal image region and the third abnormal image region.

[0166] Optionally, for any third abnormal image region, if the third abnormal image region satisfies at least one of the following conditions, a corresponding position of the third abnormal region on the surface of the product to be inspected is determined as a defect position:

[0167] ① The area of ​​the third abnormal region is larger than the preset area.

[0168] ② The ratio of the first area to the second area is greater than a preset ratio threshold. The first area is the area of ​​the overlapping area between the third abnormal image area and the target first abnormal image area, and the second area is the area of ​​the target first abnormal image area. That is, if the third abnormal image area overlaps with the target first abnormal image area, and the overlapping area of ​​the two accounts for a large proportion of the area of ​​the target first abnormal image area, the product surface position corresponding to the third abnormal image area is considered to be a defective position.

[0169] In order to accurately identify defects, the embodiment of the present invention preferably combines the above two conditions, that is, when the area of ​​the third abnormal area is larger than the preset area, and the ratio of the first area to the second area is greater than the preset ratio threshold, the corresponding position of the third abnormal area on the surface of the product to be inspected is determined as the defect position.

[0170] In this way, the product surface defect detection method provided by the embodiment of the present invention first determines the first abnormal image area by detecting the visible light image, then counts the depth of the second abnormal image area corresponding to the first abnormal image area in the point cloud image, screens out the abnormal position points, and then further analyzes the third abnormal image area corresponding to the abnormal position points in the visible light image, so as to finally determine the location of the defect on the product surface. Compared with manual detection, the use of automatic analysis by equipment greatly improves the efficiency of product surface defect detection. And by combining the visible light image and the point cloud image taken of the surface of the product to be detected, it is possible to effectively identify small defects on the product surface that are difficult to find by visual analysis alone, thereby effectively improving the detection accuracy of product surface defects.

[0171] Based on the same inventive concept, the embodiment of the present invention also provides a product surface defect detection device, such as Figure 6 As shown, including:

[0172] A visual inspection module M1, used to inspect a visible light image taken from the surface of the product to be inspected, and determine a first abnormal image area in the visible light image;

[0173] A depth preliminary detection module M2 is used to determine the abnormal position points in the second abnormal image area according to the depths of the position points of the second abnormal image area corresponding to the first abnormal image area; the second abnormal image area is an image area in the point cloud image taken of the surface of the product to be detected, and the second abnormal image area and the corresponding first abnormal image area are at the same position on the surface of the product to be detected;

[0174] An abnormal position point mapping module M3, used to determine a third abnormal image area in the visible light image according to the position of the abnormal position point in the visible light image;

[0175] The depth-vision joint detection module M4 is used to determine the defect position on the surface of the product to be detected based on the first abnormal image area and the third abnormal image area.

[0176] Optionally, the visual detection module M1 is specifically used for:

[0177] A pre-trained single-shot multi-box detector (SSD) model is used to detect a visible light image taken from the surface of the product to be inspected, and a defect confidence level of each image area in the visible light image belonging to a defect image area is determined;

[0178] An image region whose defect confidence is less than a preset confidence threshold is determined as a first abnormal image region.

[0179] Optionally, the visual inspection module M1 is further used for:

[0180] The image area whose defect confidence is greater than or equal to the preset confidence threshold value and the corresponding position on the surface of the product to be inspected are determined as the defect position.

[0181] Optionally, the depth preliminary detection module M2 is specifically used for:

[0182] Dividing the second abnormal image area into at least one sub-image area according to a preset size;

[0183] For any sub-image area, statistics are collected on the distribution of the depths of each position point in the sub-image area in each preset value interval, a reference depth range is determined according to the target preset value interval, and a position point whose depth exceeds the reference depth range is determined as the abnormal position point;

[0184] The target preset numerical range is the preset numerical range containing the largest number of depths.

[0185] Optionally, the abnormal location point mapping module M3 is specifically used for:

[0186] In the visible light image, the pixels corresponding to each of the abnormal position points are first eroded and then expanded to obtain a third abnormal image region.

[0187] Optionally, the depth-vision joint detection module M4 is specifically used for:

[0188] For any third abnormal image region, if the third abnormal image region satisfies the following conditions, the corresponding position of the third abnormal region on the surface of the product to be inspected is determined as the defect position:

[0189] The area of ​​the third abnormal area is larger than a preset area;

[0190] The ratio of the first area to the second area is greater than a preset ratio threshold; wherein the first area is the area of ​​the overlapping area between the third abnormal image area and the target first abnormal image area, and the second area is the area of ​​the target first abnormal image area.

[0191] It should be understood that the disclosed product surface defect detection device can be implemented in other ways. The device embodiment described above is only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the coupling or direct coupling or communication connection between each other shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms. The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, each functional module in the embodiment of the present application can be integrated in a processing module, or each module can exist physically separately, or two or more modules can be integrated in one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules. If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a readable storage medium.

[0192] Since the specific working principle of the product surface defect detection device is consistent with the product surface defect detection method described above, the corresponding implementation can be referred to the content of the product surface defect detection method described above, and will not be repeated here.

[0193] Based on the same inventive concept, an embodiment of the present application provides an electronic device, such as Figure 7 As shown, the device includes a processor 110 and a memory 120. The processor 110 is configured to execute the instructions to implement the product surface defect detection method.

[0194] In the specific implementation process, the device may have relatively large differences due to different configurations or performances, and may include one or more processors 110, memory 120, and readable storage medium 130. The memory 120 and / or readable storage medium 130 include one or more applications 131 or data 132. The memory 120 and / or readable storage medium 130 may also include one or more operating systems 133, such as Windows, Mac OS, Linux, IOS, Android, Unix, FreeBSD, etc. Among them, the memory 120 and the readable storage medium 130 may be short-term storage or persistent storage. The application 131 may include one or more modules ( Figure 7 ), each module may include a series of instruction operations. Further, the processor 110 may be configured to communicate with the readable storage medium 130, and execute a series of instruction operations in the readable storage medium 130 on the device. The device may also include one or more power supplies ( Figure 7 one or more network interfaces 140, wherein the network interface 140 includes a wired network interface 141 and / or a wireless network interface 142; and one or more input / output interfaces 143.

[0195] Based on the same inventive concept, an embodiment of the present application provides a readable storage medium, which stores a computer program. When the computer program is executed by a computer, the computer executes the product surface defect detection method.

[0196] The readable storage medium may be any available medium that can be stored by a computer or a data storage device such as a server or a data center that includes one or more available media. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a high-density digital video disc (DVD), a video compact disc (VCD)), or a semiconductor medium (such as a solid state disk (SSD)).

[0197] Since the principle of solving the problem by the above-mentioned readable storage medium is consistent with the above-mentioned product surface defect detection method, the implementation of the above-mentioned readable storage medium can refer to the implementation of the method, and the repeated parts will not be repeated.

[0198] Based on the same inventive concept, an embodiment of the present application further provides a computer program product, which includes: a computer program code, and when the computer program code runs on a computer, the computer executes the product surface defect detection method.

[0199] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a readable storage medium, or transmitted from a readable storage medium to another readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center.

[0200] Since the principle of solving the problem by the above-mentioned computer program product is consistent with the product surface defect detection method, the implementation of the above-mentioned computer program product can refer to the implementation of the method, and the repeated parts will not be repeated.

[0201] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0202] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0203] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0204] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0205] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for detecting surface defects of a product, characterized in that: include: Detecting a visible light image taken of the surface of the product to be detected, and determining a first abnormal image area in the visible light image; Determining abnormal position points in the second abnormal image area according to the depths of the position points in the second abnormal image area corresponding to the first abnormal image area; The second abnormal image region is an image region in the point cloud image taken of the surface of the product to be inspected, and the second abnormal image region and the corresponding first abnormal image region are located at the same position on the surface of the product to be inspected; Determining a third abnormal image region in the visible light image according to the position of the abnormal location point in the visible light image; The defect position on the surface of the product to be inspected is determined according to the first abnormal image area and the third abnormal image area.

2. The method according to claim 1, characterized in that The step of detecting the visible light image taken on the surface of the product to be detected and determining the first abnormal image area in the visible light image includes: A pre-trained single-shot multi-box detector (SSD) model is used to detect a visible light image taken from the surface of the product to be inspected, and a defect confidence level of each image area in the visible light image belonging to a defect image area is determined; An image region whose defect confidence is less than a preset confidence threshold is determined as a first abnormal image region.

3. The method according to claim 2, characterized in that The method further comprises: The image area whose defect confidence is greater than or equal to the preset confidence threshold value and the corresponding position on the surface of the product to be inspected are determined as the defect position.

4. The method according to claim 1, characterized in that The determining, according to the depth of each position point of the second abnormal image area corresponding to the first abnormal image area, the abnormal position point in the second abnormal image area comprises: Dividing the second abnormal image area into at least one sub-image area according to a preset size; For any sub-image area, statistics are collected on the distribution of the depths of each position point in the sub-image area in each preset value interval, a reference depth range is determined according to the target preset value interval, and a position point whose depth exceeds the reference depth range is determined as the abnormal position point; The target preset numerical range is the preset numerical range containing the largest number of depths.

5. The method according to claim 1, characterized in that The step of determining at least one third abnormal image region in the visible light image according to the position of the abnormal position point in the visible light image comprises: In the visible light image, the pixels corresponding to each of the abnormal position points are first eroded and then expanded to obtain a third abnormal image region.

6. The method according to claim 1, characterized in that The step of determining the defect position on the surface of the product to be inspected according to the first abnormal image area and the third abnormal image area comprises: For any third abnormal image region, if the third abnormal image region satisfies the following conditions, the corresponding position of the third abnormal region on the surface of the product to be inspected is determined as the defect position: The area of ​​the third abnormal area is larger than a preset area; The ratio of the first area to the second area is greater than a preset ratio threshold; wherein the first area is the area of ​​the overlapping area between the third abnormal image area and the target first abnormal image area, and the second area is the area of ​​the target first abnormal image area.

7. A product surface defect detection device, characterized in that: include: A visual inspection module, used to inspect a visible light image captured on the surface of the product to be inspected, and determine a first abnormal image area in the visible light image; a depth preliminary detection module, configured to determine an abnormal position point in the second abnormal image region according to the depth of each position point in the second abnormal image region corresponding to the first abnormal image region; The second abnormal image region is an image region in the point cloud image taken of the surface of the product to be inspected, and the second abnormal image region and the corresponding first abnormal image region are located at the same position on the surface of the product to be inspected; an abnormal position point mapping module, used to determine a third abnormal image area in the visible light image according to the position of the abnormal position point in the visible light image; A depth-vision joint detection module is used to determine the defect position on the surface of the product to be inspected based on the first abnormal image area and the third abnormal image area.

8. An electronic device, characterized in that: include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the steps included in the product surface defect detection method according to any one of claims 1-6 according to the obtained program instructions.

9. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a computer, the computer executes the product surface defect detection method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises: a computer program code, and when the computer program code is run on a computer, the computer is enabled to execute the product surface defect detection method as described in any one of claims 1 to 6.

Images