Metal plate surface detection method and device based on machine vision

Abstract

The invention belongs to the technical field of detection, and particularly discloses a metal plate surface detection method and device based on machine vision. According to the method, an RGB color space is adopted, a target and a background are modeled by using a full covariance GMM containing K Gaussian components, pixels are initialized to obtain the target and the background GMM, and iteration minimization is carried out to segment an image and smooth processing is carried out on a boundary; carrying out edge detection on the image after smoothing processing by using an improved Canny operator; and analyzing the oil stain area and shape on the surface of the metal plate according to the image edge detection result. The device comprises a rack, and a conveying mechanism, a detection mechanism and a lifting mechanism which are electrically connected with a controller, wherein the controller judges and outputs the oil stain condition on the surface of the metal plate according to a detected image. Through an innovative image processing algorithm and an automatic device design, the problems of low precision, high cost and dependence on manpower in the prior art are solved, and the method has the characteristics of high detection efficiency and precision, low cost and strong adaptability.

Description

Technical Field

[0001] The present invention belongs to the technical field of detection, and particularly relates to a method and device for detecting the surface of a metal plate based on machine vision, which have high detection efficiency and accuracy, low cost, and strong adaptability. Background Art

[0002] In industrial production, metal plates, as important raw materials and components, are widely used in fields such as automobiles, aerospace, and electrical systems. These fields have extremely high requirements for the cleanliness of the metal surface because contaminants such as oil stains may affect the performance, reliability, and safety of the products. For example, in automobile manufacturing, oil stains may cause an increase in friction between components, thereby affecting the operating efficiency and safety of the entire vehicle. Therefore, the detection of oil stains on metal plates has become a key link in ensuring product quality and performance.

[0003] For the detection of oil stains on the surface of metal plates, traditionally, the presence of oil stains is judged by visually observing the glossiness, color change of the metal surface, and whether there is deposition of viscous liquid, etc. Although the visual method is simple to operate and does not require additional equipment, it has high requirements for the experience and skills of the inspectors, and may not be able to effectively detect small or hidden oil stains, with strong subjectivity.

[0004] With the progress of technology and the development of industrial production, detection technologies such as infrared spectroscopy, electron microscope detection, and hyperspectral camera detection have emerged for the detection of oil stains on the surface of metal plates. These technologies not only improve the detection efficiency compared with the visual method but also reduce the errors caused by human factors, providing a more reliable and accurate means for the detection of oil stains on metal plates. Currently, in order to ensure the accuracy and consistency of the detection of oil stains on metal plates, countries and industry organizations have successively formulated corresponding detection standards and specifications. These standards and specifications cover aspects such as the selection of detection methods, the execution of detection steps, and the determination of detection results, providing clear guidance and basis for the detection of oil stains on metal plates. At the same time, the formulation of these standards and specifications has also promoted the further development and improvement of the detection technology for oil stains on metal plates. However, there are also problems in the existing technologies, such as the high price of the instruments for infrared spectroscopy, high requirements for the professional knowledge of the operators; the electron microscope detection method also requires professional equipment and technicians, and is complex to operate, with high detection costs, and a small detection range and low detection efficiency; the hyperspectral camera detection method has expensive equipment, complex data processing, and requires professional software and technicians for operation and analysis.

[0005] To solve the problems existing in the aforementioned technologies, visual detection technologies based on traditional image processing have also emerged in the prior art, such as the gray-scale threshold segmentation method: converting the collected image of the metal plate surface into a gray-scale image, setting a fixed threshold according to the difference in gray-scale values between the oil stain area and the background area, and pixels with gray-scale values higher than or lower than this threshold are determined as oil stain pixels, thereby extracting the oil stain area; although the algorithm is simple and the calculation speed is fast, due to the selection of the threshold relying on experience, for the situations of uneven illumination and the gray-scale of the oil stain being close to that of the background, the segmentation effect is very poor, prone to false positives and false negatives, and it is difficult to adapt to the complex and changeable industrial production environment. For this reason, the edge detection method has emerged: using edge detection operators such as Sobel and Canny to detect areas with drastic gray-scale changes in the image. Since there are usually obvious edges at the junction of the oil stain area and the clean metal surface, by identifying these edges, the outline of the oil stain is outlined, providing a basis for further analyzing parameters such as the area and shape of the oil stain in the subsequent steps; although it can reduce the dependence on experience and false positives and false negatives compared with the gray-scale threshold segmentation method, it is also sensitive to noise, and interference factors such as scratches and stains on the metal plate surface are easily misjudged as oil stain edges, and the edge detection accuracy drops significantly in low-contrast scenarios. Summary of the Invention

[0006] In view of the deficiencies of the prior art, the present invention proposes a machine vision-based metal plate surface detection method with high detection efficiency and accuracy, low cost, and strong adaptability, and also provides a machine vision-based metal plate surface detection device.

[0007] The machine vision-based metal plate surface detection method of the present invention is implemented as follows: including the steps of image segmentation, edge detection, and surface quality determination, and the content of each step is as follows: A. Image segmentation: Using the RGB color space, respectively modeling the target and the background with a full covariance GMM containing K Gaussian components, and then using the Grabcut algorithm to segment the metal plate surface image and smooth the image boundary; B. Edge detection: Using an improved Canny operator to perform edge detection on the image after the aforementioned smoothing process; C. Surface quality determination: According to the aforementioned image edge detection results, calculating the oil stain area on the metal plate surface and outputting the detection result.

[0008] Further, the full covariance GMM in step A is: Where: π is the weight of the Gaussian component, gi(x, μ, ∑) is the probability density function (PDF) of the i-th Gaussian component, πi is the weight of the i-th Gaussian component, satisfying the probability normalization condition; x is the input data point with dimension d*1; μi is the mean vector of the i-th Gaussian component; d is the data dimension, such as (pixel value dimension); ∑ is the covariance matrix, k is an additional vector, k = {k1,..., kn,..., kN}, where kn is the Gaussian component corresponding to the n-th pixel, kn ∈ {1,... K}; where K can be selected according to the color distribution or texture complexity of the image.

[0009] Further, in step A, the Grabcut algorithm is used to segment the metal plate surface image and smooth the image boundary. First, the pixel values of the target GMM and background GMM corresponding to mask in the Grabcut operator grabcut(img, mask, rect, bgdmodel, fgdmodel, iterCount, [mode]) are initialized, and then the metal plate surface image is segmented and the image boundary is smoothed by iterative minimization; the specific process of initializing the pixels is as follows: A10. An initial trimap T is obtained by directly framing the metal plate surface, that is, all pixels outside the square are used as background pixels TB, and all pixels TU inside the square are used as "possibly target" pixels; A11. For each pixel n in TB, the label α of pixel n is initialized n = 0, that is, it is a background pixel; for each pixel n in TU, the label α of pixel n is initialized n = 1, that is, it is used as a "possibly target" pixel; A12. After the above two steps, the pixels belonging to the metal plate surface are obtained, and the remaining pixels belong to the background. The target and background GMMs are estimated from the aforementioned pixels.

[0010] Further, the specific process of iterative minimization in step A is as follows: A20. By the formula each pixel is assigned a Gaussian component in the GMM; In the formula, K n is the Gaussian component index corresponding to the n-th data point, and its value range is {1, 2,..., K}, where K is the total number of Gaussian components; is to K n be optimized so that the objective function D n reaches the minimum value; D nused to measure the distance or difference between the nth data point and the K n th Gaussian component; a n is the feature vector of the nth data point; θ is the parameter set of the target and background models, including the mean and covariance matrix of the Gaussian components; z n is the latent variable or label of the nth data point; A21. For the given image data Z, estimate the optimal parameters of the GMM by minimizing the objective function U θ , that is ; In the formula: θ is the parameter set of the GMM, α is the regularization term, k is the number of Gaussian mixture components, z is the input image feature; A22. Perform segmentation estimation through the formula ; In the formula: Tu is the set of pixels / regions to be processed in the image; A23. Repeat A20 to A22 to interactively optimize the GMM model and the segmentation result until convergence, and obtain the segmentation boundary of the image; A24. Use border matting to smooth the aforementioned segmentation boundary.

[0011] Furthermore, the specific process of the B step is as follows: B10. Use bilateral filtering instead of Gaussian filtering to perform noise reduction on the aforementioned smoothed image; B20. Use the scharr operator instead of the sobel operator to calculate the pixel gradient and direction of the denoised image; B30. Use polygon approximation to optimize the edge connection part of the image segmentation boundary.

[0012] Furthermore, the specific process of the B30 sub-step is as follows: B31. Input contour: Represent the contour extracted from the image as a set of discrete point coordinates; B32. Select the starting point and the ending point: During the approximation process, select the starting point and the ending point of the contour as the endpoints of the approximation line segment; B33. Calculate the farthest point: Select the point that is the farthest from the line segment between the starting point and the ending point as the separation point; B34. Recursive segmentation: Divide the contour into two parts, and use the same method to approximate the two parts of the contour respectively; B35. Recursive merging: The approximate contours obtained after approximation are merged recursively to obtain the final polygon approximation result.

[0013] The surface detection device for metal plates based on machine vision of the present invention is implemented as follows: It includes a frame, and also includes a conveying mechanism, a detection mechanism, a lifting mechanism, and a controller. The conveying mechanism, the detection mechanism, and the lifting mechanism are respectively electrically connected to the controller. The controller judges the oil stain condition on the surface of the metal plate according to the detection image of the detection mechanism and outputs a judgment result; The conveying mechanism includes a conveyor belt, a transmission component I, and a motor I. Two conveyor belts are symmetrically arranged and rotatably arranged on both sides of the top of the frame. A number of hanging rods for hanging metal plates are supported at intervals along the moving direction on the two conveyor belts at the top of the frame. The motor I is fixedly arranged at the lower part of the frame. The transmission component I is respectively connected to the output shaft of the motor I and the conveyor belt to drive the conveyor belt to rotate and move; The lifting mechanism includes a lifting bracket, a sliding frame, a sliding plate, a clamping component, and a lifting component. The lifting brackets are respectively vertically fixed on the outer sides of the two conveyor belts at the top of the frame. Two sliding frames are arranged and symmetrically fixed on the inner sides of the two lifting brackets. The sliding plate is slidably arranged on the sliding frame. Two clamping components are symmetrically fixed on the corresponding sliding plates on both sides and can clamp the two ends of the hanging rod. The lifting component is arranged on the lifting bracket and its driving ends are respectively connected to the two sliding plates to drive the sliding plates to move up and down; The detection mechanism is respectively arranged in front of and behind the lifting mechanism in the moving direction of the conveyor belt. The camera lens of the detection mechanism faces the surface of the metal plate lifted by the lifting mechanism. The camera is electrically connected to the controller.

[0014] Further, the clamping component includes a telescopic rod and a clamping block. The telescopic rod is fixedly arranged on the sliding plate. The clamping block is fixedly arranged at the telescopic end of the telescopic rod away from the sliding plate. A clamping groove for embedding the end of the hanging rod is formed at one end of the clamping block away from the telescopic rod. A limiting block for abutting against the sliding plate is fixedly arranged at the bottom of the sliding frame; The lifting component includes a motor II, a transmission component II, a bevel gear I, a bevel gear II, and a rotating shaft II. The motor II is fixedly arranged at the top of the lifting bracket. The rotating shaft II is rotatably arranged at the top of the lifting bracket. The bevel gear I is fixedly connected to the motor shaft of the motor II. The bevel gear II is fixedly arranged at one end of the rotating shaft II and meshes with the bevel gear I. The transmission component II is respectively connected to the rotating shaft II and the sliding plate to drive the sliding plate to move up and down.

[0015] Further, the lifting mechanism also includes a support rod and a position sensor. The support rod is fixedly arranged at the lower part of the front end or the rear end of the lifting bracket. The position sensor is fixedly arranged on the support rod between the two lifting brackets. The position sensor is electrically connected to the controller.

[0016] Further, the detection mechanism further includes a light source and a detection bracket. The detection brackets are respectively arranged in front of and behind the lifting mechanism in the moving direction of the conveyor belt. The bottom ends on both sides of the detection bracket are fixedly connected to the frame outside the conveyor belt. The camera is fixedly arranged on the cross bar of the detection bracket, and the light source is fixedly arranged on another cross bar of the detection bracket above the camera. The camera and the light source are respectively electrically connected to the controller.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. The detection method of the present invention first uses the RGB color space and full covariance GMM modeling to segment the image of the metal plate surface through iterative minimization and smooth the image boundary, which can more accurately extract the oil stain area on the metal plate surface and enhance the ability to capture minute oil stains on the metal surface. Compared with the traditional gray threshold segmentation method, it is less affected by factors such as uneven illumination and similar gray levels between oil stains and the background, and can effectively reduce false positives and missed detections. Then, an improved Canny operator is used. While bilateral filtering is used to replace Gaussian filtering for noise reduction with better edge preservation effect, the scharr operator is used to replace the sobel operator to calculate pixel gradients and directions, and polygon approximation is used to optimize the edge connection part of the image segmentation boundary, improving the accuracy of edge detection, reducing the sensitivity to noise and interference factors such as scratches and stains on the metal plate surface, and enabling high precision to be maintained even in low-contrast scenarios. Therefore, the detection method of the present invention can accurately analyze the area and shape of the oil stains on the metal plate surface, thereby accurately determining the surface quality.

[0018] 2. Since the detection method of the present invention can maintain a high detection accuracy even in low-contrast scenarios, it can adapt to complex and changeable industrial production environments. Compared with traditional methods (such as visual inspection, which is difficult to detect minute or hidden oil stains and relies on the experience of inspectors), the detection method of the present invention does not rely on manual experience and has better processing capabilities for different lighting conditions, oil stain morphologies, etc., and has strong adaptability.

[0019] 3. The detection device of the present invention electrically connects the conveying mechanism, the lifting mechanism, and the detection mechanism to the controller, enabling automatic conveying, lifting, and detection of metal plates, which can improve detection efficiency and consistency. The clamping assembly of the lifting mechanism can stably hold the suspension rod, and cooperate with the lifting assembly to drive the sliding plate to move up and down, realizing the stable lifting of the metal plate. Moreover, the coordinated cooperation of the position sensor and the controller can accurately control the position of the metal plate, ensuring the consistency and stability of the detection process and reducing human operation errors. Additionally, the detection mechanism is arranged in front of and behind the lifting mechanism in the moving direction of the conveyor belt, with the camera lens facing the metal plate directly. Cooperating with the light source, clear front and back surface images of the metal plate can be obtained for subsequent detection and analysis, ultimately realizing continuous automation of the detection process. While significantly improving the detection efficiency, it also reduces the need for manual intervention, so it is particularly suitable for detection in industrial assembly line environments.

[0020] 4. Compared with the existing infrared spectroscopy method, electron microscope detection method, hyperspectral camera detection method, etc. in the prior art, the present invention not only has relatively low equipment costs, but also can achieve high detection accuracy and efficiency, with good cost-effectiveness. Moreover, automated detection reduces labor costs and time costs, improves production efficiency, and is conducive to large-scale industrial applications.

[0021] In summary, through innovative image processing algorithms and automated device designs, the present invention solves the pain points of traditional detection methods, such as low accuracy, high cost, and dependence on manual labor, realizing efficient, accurate, and economical detection of oil stains on the surface of metal plates, providing reliable technical support for improving the quality of industrial products and the level of production intelligence. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flowchart of the detection method of the present invention; Figure 2 It is a flowchart of initializing pixels in the detection method of the present invention; Figure 3 It is a flowchart of the edge connection part for optimizing the image segmentation boundary in the detection method of the present invention; Figure 4 It is a schematic diagram of the double-threshold strategy and edge connection in the comparative example of the present invention; Figure 5 It is a comparison diagram before and after processing the metal plate surface defect pictures in the embodiment of the present invention; Figure 6 Schematic structural diagram of the detection device of the present invention; Figure 7 For Figure 6 the front view; Figure 8 For Figure 6 the left view; Figure 9Structure enlarged view of the carriage and its connecting components of the present invention; Figure 10 Structure enlarged view of the clamping assembly of the present invention; Figure 11 Structure enlarged view of the lifting assembly of the present invention; In the figure: 1 - frame, 2 - conveying mechanism, 21 - conveyor belt, 22 - transmission component I, 23 - motor I, 24 - suspension rod, 25 - protrusion, 3 - detection mechanism, 31 - camera, 32 - light source, 33 - detection bracket, 4 - lifting mechanism, 41 - lifting bracket, 42 - carriage, 421 - chute, 43 - sliding plate, 431 - slider, 44 - clamping assembly, 441 - telescopic rod, 442 - clamping block, 443 - clamping groove, 444 - limiting block, 45 - lifting assembly, 451 - motor II, 452 - transmission component II, 453 - bevel gear I, 454 - bevel gear II, 455 - rotating shaft II, 46 - support rod, 47 - position sensor, 48 - connecting rod, 5 - metal plate. Detailed implementation manners

[0023] The present invention will be further described below in conjunction with the accompanying drawings and embodiments, but the present invention is not limited in any way. Any changes or improvements made based on the teachings of the present invention fall within the protection scope of the present invention.

[0024] As Figures 1 to 3 shown, the method for detecting the surface of a metal plate based on machine vision of the present invention includes steps of image segmentation, edge detection, and surface quality determination. The content of each step is as follows: A. Image segmentation: The RGB color space is adopted, and a full covariance GMM with K Gaussian components is used to model the target and the background respectively. Then, the Grabcut algorithm is used to segment the surface image of the metal plate and smooth the image boundary; B. Edge detection: An improved Canny operator is used to detect the edges of the image after the aforementioned smoothing process; C. Surface quality determination: According to the aforementioned image edge detection results, the oil stain area on the surface of the metal plate is calculated and the detection result is output.

[0025] The full covariance GMM in step A is: In the formula: π is the weight of the Gaussian component, gi(x, μ, ∑) is the probability density function (PDF) of the i-th Gaussian component, πi is the weight of the i-th Gaussian component, and satisfies the probability normalization condition; x is the input data point, with a dimension of d*1; μ i is the mean vector of the i-th Gaussian component; dis the data dimension, such as (pixel value dimension); ∑ is the covariance matrix, k is an additional vector, k = {k1,..., kn,..., kN}, where kn is the Gaussian component corresponding to the nth pixel, kn ∈ {1,... K}; where K can be selected according to the color distribution or texture complexity of the image; Through the formula , where U is the regional term, to obtain the Gibbs energy map of the entire image; E is the total energy function, which measures the segmentation quality; V is the smoothing term, ensuring that the alpha values of adjacent pixels are similar; α is the transparency vector, representing the alpha value of each pixel; k is the Gaussian component index, indicating the Gaussian distribution to which the pixel belongs; θ is the set of model parameters, describing the color distribution; z is the latent variable or label, representing the classification information of the pixel.

[0026] Taking the negative logarithm of the Gaussian model formula gives the formula:

[0027] θ are the parameters of the GMM, a total of three: π, u, ∑. Determining these three parameters and substituting them into the GMM of the target and the GMM of the background, obtaining the probabilities that the pixel belongs to the target and the background respectively, and then determining the regional energy term of the Gibbs energy, and finding the weight value of the t-link of the graph; the t-link weight value is used to construct the energy function of the graph: connecting each pixel to the source node and the sink node, and assigning the corresponding t-link weight value. By minimizing the energy function of the graph (including the regional energy term and the smoothing term), the optimal segmentation result is found.

[0028] Through the formula , where ||zm - zn|| is the difference between two pixels, γ taking 50, to obtain the weight value of the n-link. V is the smoothing term, used to ensure that the labels of adjacent pixels are similar; γ is the weight parameter of the smoothing term; ∑(m,n)∈C is the sum over all pairs of adjacent pixels; exp-β||z m -z n || 2 is the weight function based on color difference; α represents the alpha value of each pixel; z is the color vector of the pixel; β is the parameter controlling the influence of color difference.

[0029] Such as Figure 2As shown, in the A step, the Grabcut algorithm is used to segment the surface image of the metal plate and smooth the image boundary. First, the pixel values of the target GMM and background GMM corresponding to mask in the grabcut(img, mask, rect, bgdmodel, fgdmodel, iterCount, [mode]) operator are initialized, and then the surface image of the metal plate is segmented and the image boundary is smoothed by iterative minimization.

[0030] It should be noted that the GrabCut algorithm can guide the algorithm to perform segmentation by interactively selecting the foreground and background, thereby improving the accuracy and robustness of the algorithm. The API of the grabcut operator is grabcut(img, mask, rect, bgdmodel, fgdmodel, iterCount, [mode]); where img is the source image, mask is the mask image. If the mask is initialized, mask stores the mask information. Mask can also store the foreground and background set interactively when segmenting the image. Then the grabcut algorithm is introduced, and finally mask stores the result. Therefore, mask is the result of the segmentation. Since the values of mask can only be: (1) GCD_BGD (=0), which means this point is the background; (2) GCD_FGD (=1), which means this point is the foreground; when GCD_PR_BGD (=2), it means this point may be the background, and when GCD_PR_FGD (=3), it means this point may be the foreground. GCD_BGD and GCD_FGD usually need to be marked manually. If not marked, the results will only be GCD_PR_BGD (=2) and GCD_PR_FGD (=3).

[0031] rect is the segmentation range, and only the area inside the rectangle will be processed; bgdmodel and fgdmodel represent the background mode and foreground mode respectively, and must be floating-point images with only 1 row and 13*5 columns. If filled with None, the function will automatically create bgdmodel or fgdmodel internally; iterCount is the number of iterations; mode is the type of operation to be performed, which are: (1) GC_INIT_WITH_RECR (=0); (2) GC_INIT_WITH_MASK (=1); (3) GC_EVAL (=2), indicating to perform the segmentation operation. The specific operation steps indicated by mode are as follows: 1) After reading the image, display the pixel coordinates of the image through the matplotlib.pyplot library; 2) Select the background area and foreground area in the pixel coordinates, and input the initial point coordinates x, y and width and length w, h of the rectangle; 3) The Mask is a matrix that stores the segmentation result. Its four values are introduced above. It has the same shape as the source image and is of type uint8; 4) Introduce the grabcut function. bgdmodel and fgdmodel can be input as None, then the system automatically generates bgdmodel and fgdmodel, and the number of iterations is selected as 5 based on the complexity of the image; 5) When using RECT for the first time, the mode type is selected as GC_INIT_WITH_RECR (=0); 6) Through the np.where function, according to the cases where mask = 1 (foreground) and mask = 3 (possibly foreground), all foregrounds are set to 255 and the background is set to 0; 7) Use the bitewise_and function for operation. According to the np.where function, all foreground pixel values are set to 255, all background pixel values are set to 0. Performing the operation between 255 and the original image leaves the original image, and the remaining positions with pixel value 0 are black areas.

[0032] Among them, the specific process of initializing pixels is as follows: A10. Obtain an initial trimap T by directly selecting the metal plate surface by a bounding box, that is, all pixels outside the bounding box are used as background pixels TB, and all pixels TU inside the bounding box are used as "possibly target" pixels; A11. For each pixel n in TB, initialize the label α of pixel n n = 0, that is, it is a background pixel; and for each pixel n in TU, initialize the label α of pixel n n = 1, that is, it is used as a "possibly target" pixel; A12. After the above two steps, obtain the pixels belonging to the metal plate surface, and the remaining are the pixels belonging to the background. Estimate the GMM of the target and the background through the aforementioned pixels.

[0033] The specific process of iterative minimization in step A is as follows: A20. Through the formula Allocate the Gaussian components in the GMM to each pixel; In the formula, K n is the index of the Gaussian component corresponding to the nth data point, and its value range is {1, 2,..., K}, where K is the total number of Gaussian components; is to optimize K n so that the objective function D n reaches the minimum value; D n is used to measure the distance between the nth data point and the Kth nThe distance or difference between Gaussian components; a n is the feature vector of the nth data point; θ is the parameter set of the target and background models, including the mean and covariance matrix of Gaussian components; z n is the latent variable or label of the nth data point; A21. For the given image data Z, estimate the optimal parameters of the GMM by minimizing the objective function U θ , that is ; In the formula: θ is the parameter set of the GMM, α is the regularization term (penalty term to prevent overfitting), k is the number of Gaussian mixture components (i.e., the number of clusters of the GMM), z is the input image feature (pixel value); The parameter set θ of the said GMM includes: mixture weight π i (the proportion of each Gaussian distribution), mean vector μ i (the center of each Gaussian distribution), covariance matrix ∑ i (the shape of each Gaussian distribution); is to find the parameters that minimize the objective function U θ , which is usually optimized using the maximum likelihood estimation (MLE) or expectation maximization (EM) algorithm in the GMM; Objective function U ( α , k, θ, z ) In the GMM, usually refers to the negative log-likelihood function (NLL), that is: U ( α , k, θ, z ) = -logp(z| θ ); A22. Perform segmentation estimation through the formula ; In the formula: Tu is the set of pixels / regions to be processed in the image; A23. Repeat A20 to A22 to interactively optimize the GMM model and the segmentation result until convergence, and obtain the segmentation boundary of the image; A24. Use border matting to smooth the aforementioned segmentation boundary.

[0034] The A24 step first initializes the segmentation boundary: that is, obtains the initial segmentation result, determines the rough boundary between the foreground and the background, and then defines the transition region: then defines a narrow band region near the segmentation boundary as the transition region (matting region), including the inside and outside parts of the boundary; subsequently estimates the transparency (Alpha value): for each pixel in the transition region, estimates its transparency (alpha value), indicating the degree to which the pixel belongs to the foreground (0 for completely background, 1 for completely foreground); then constructs the color model: uses the known foreground and background regions to establish the color distribution models of the foreground and the background; and defines the energy function: constructs an energy function, usually including the following two parts: data term: measures the matching degree between the pixel color and the foreground / background color models; smooth term: encourages the alpha values of adjacent pixels to be similar to ensure a smooth boundary; optimizes the energy function: uses optimization algorithms (such as graph cut algorithm, gradient descent, etc.) to minimize the energy function and solve the optimal alpha value for each pixel; finally generates a smooth boundary: generates a smooth segmentation boundary according to the optimized alpha value to achieve a natural transition between the foreground and the background.

[0035] Through the above operations, the oil stain defect area with a darker color in the metal plate can be separated from the metal plate surface area with a lighter color, so as to obtain a defect area with a clearer contour. The specific effect can be seen Figure 5 , after the grabcut processing in the above steps 1)-7), the background of the metal plate picture is eliminated and only the foreground remains, that is, the defect area in the figure is separately segmented, and the background that does not need to be analyzed is removed; and problems such as uneven light and noise in the background will not affect the defect detection because the background is removed, and at the same time this operation is beneficial to subsequent pixel division and edge detection; at the same time, since there are still some noises or other influencing factors in the segmented defect area, edge detection needs to be carried out for further analysis and judgment.

[0036] In step B, the improved Canny operator is used to perform edge detection on the image after the above-mentioned smoothing process. The specific process is as follows: B10. Use bilateral filtering instead of Gaussian filtering to perform noise reduction processing on the image after the above-mentioned smoothing process; When ( i , j ) represents the position of the pixel being processed currently, ( k , l ) represents the position of the surrounding pixels, and the spatial domain Gaussian function is: The value domain Gaussian function is: Multiplying the two equations gives: With the weight coefficient w, we can obtain:

[0037] Bilateral filtering can be regarded as Gaussian filtering with an additional consideration of gray - level similarity.

[0038] B20. Use the Scharr operator instead of the Sobel operator to calculate the pixel gradient and direction of the denoised image; for example, in one embodiment, the convolution kernel is replaced with: In the formula: Gx is the horizontal convolution kernel, Gy is the vertical convolution kernel.

[0039] B30. Use polygon approximation to optimize the edge connection part of the image segmentation boundary. Polygon approximation is used to approximately represent the detected contour. In polygon approximation, by reducing the redundant points in the contour, the shape of the contour can be approximately described with fewer points, thereby realizing the simplification and optimization of the contour. Its principle is based on the Douglas - Peucker algorithm.

[0040] As Figure 3 shown, the specific process of step B30 is as follows: B31. Input contour: Represent the contour extracted from the image as a set of discrete point coordinates; B32. Select the starting point and the ending point: During the approximation process, select the starting point and the ending point of the contour as the endpoints of the approximation line segment; B33. Calculate the farthest point: Select the point farthest from the line segment between the starting point and the ending point as the dividing point; B34. Recursive segmentation: Divide the contour into two parts, and use the same method to approximate each part of the contour; B35. Recursive merging: The approximate contours obtained after approximation are merged recursively to obtain the final polygon approximation result.

[0041] The process of traditional Canny operator edge detection is as follows: (1) Denoising: For edge detection that is vulnerable to noise, noise reduction processing is generally required first, usually using Gaussian filtering.

[0042] (2) Calculate the gradient: Calculate the gradient and direction for each pixel point of the image smoothed in the previous step. Use the horizontal and vertical convolution kernels Gx and Gy of the aforementioned Sobel operator to scan the points on the pixels, finally calculate the gradient, and then calculate the direction; among them, the gradient direction is divided into four categories: horizontal, vertical, and two diagonals.

[0043] (3) Non-maximum suppression: After calculating the gradient and direction, non-edge points are erased by traversing the image. During the traversal of pixel points, observe which pixel points within the range of pixel points with the same radius in the surrounding circle have the same direction, and then find the maximum value among them. After that, retain the maximum value in the same gradient direction, while suppressing other values in this direction, so as to filter out pixel points whose gradient magnitudes are not edges, but it cannot be determined that the retained points are edges.

[0044] (4) Double-threshold strategy and edge connection: In the previous step, the pixel points with the maximum value in the same gradient direction can be obtained, but it may also cause false edges. Therefore, the issue of continuity needs to be considered. As Figure 4 shown in the figure, if the gradient value of point A in the figure is greater than the maximum threshold maxVal, then A is retained; the gradient values of B and C are greater than the minimum threshold minVal and less than the maximum threshold maxVal, and point C is connected to point A, then C and A are retained; since B is not connected, it is suppressed; the gradient value of point D is less than the minimum threshold minVal, then it is suppressed.

[0045] Conclusion: By carefully observing the effect after traditional Canny edge detection, it can be found that although the Canny operator retains the general contour well, there are many breakpoints on the edge; the higher the degree of edge information retention means the better the subsequent edge detection effect, so more consideration needs to be given to its gray-scale similarity; compared with Gaussian filtering, the bilateral filtering mentioned in B10 not only considers the spatial similarity of the image but also its gray-scale similarity, and can achieve the purpose of "preserving edges and removing noise"; moreover, the improved Canny operator has improved the edge protection ability during its filtering process, the accuracy when calculating the gradient direction, and the robustness of edge connection. Therefore, bilateral filtering is used instead of Gaussian filtering for noise reduction.

[0046] After the results of grabcut and edge detection, through binarization operation, the defective area can be expressed by calculating the proportion of white pixels, and the pixel coordinates can represent their positional relationship by completely classifying light and dark pixels into black and white categories.

[0047] As Figures 6 to 11 shown in the figure, the metal plate surface detection device based on machine vision of the present invention includes a frame 1, and also includes a conveying mechanism 2, a detection mechanism 3, a lifting mechanism 4, and a controller. The conveying mechanism 2, the detection mechanism 3, and the lifting mechanism 4 are respectively electrically connected to the controller. The controller judges the oil stain condition on the surface of the metal plate 5 according to the detection image of the detection mechanism 3 and outputs a judgment result; The conveying mechanism 2 includes a conveyor belt 21, a transmission assembly I 22, and a motor I 23. Two conveyor belts 21 are symmetrically arranged and rotatably arranged on both sides of the top of the frame 1. A plurality of suspension rods 24 for suspending the metal plates 5 are supported at intervals along the moving direction on the two conveyor belts 21 at the top of the frame 1. The motor I 23 is fixedly arranged at the lower part of the frame 1. The transmission assembly I 22 is respectively connected to the output shaft of the motor I 23 and the conveyor belt 21 to drive the conveyor belt 21 to rotate and move; The lifting mechanism 4 includes a lifting bracket 41, a sliding carriage 42, a sliding plate 43, a clamping assembly 44, and a lifting assembly 45. The lifting brackets 41 are respectively vertically fixed on the outer sides of the two conveyor belts 21 at the top of the frame 1. Two sliding carriages 42 are arranged and symmetrically fixed on the inner sides of the two lifting brackets 41. The sliding plate 43 is slidably arranged on the sliding carriage 42. The two clamping assemblies 44 are symmetrically fixed on the corresponding sliding plates 43 on both sides and can clamp the two ends of the suspension rod 24. The lifting assembly 45 is arranged on the lifting bracket 41 and its driving ends are respectively connected to the two sliding plates 43 to drive the sliding plates 43 to move up and down; The detection mechanism 3 is respectively arranged in front of and behind the lifting mechanism 4 in the moving direction of the conveyor belt 21. The lens of the camera 31 of the detection mechanism 3 faces the surface of the metal plate 5 lifted by the lifting mechanism 4. The camera 31 is electrically connected to the controller.

[0048] A plurality of regularly arranged protrusions 25 are arranged along the length direction of the conveyor belt 21. There are gaps between the protrusions 25 for supporting the suspension rod 24. The two sides of the suspension rod 24 are supported in the corresponding gaps of the two conveyor belts 21. The arrangement of the protrusions 25 can ensure that the suspension rod 24 does not shift during the movement, and ensure that the lifting mechanism 4 can accurately lift the suspension rod 24 so that the detection mechanism 3 can take pictures.

[0049] The sliding carriage 42 is provided with a chute 421, and the sliding plate 43 is provided with a slider 431. The slider 431 is slidably arranged in the chute 421.

[0050] A connecting rod 48 with one end extending to the outside of the lifting bracket 41 is fixedly arranged on the sliding plate 43. The driving ends of the lifting assembly 45 are respectively connected to the corresponding connecting rods 48 on the two sliding plates 43 to drive the sliding plates 43 to move up and down.

[0051] The clamping assembly 44 includes a telescopic rod 441 and a clamping block 442. The telescopic rod 441 is fixedly arranged on the sliding plate 43. The clamping block 442 is fixedly arranged at the telescopic end of the telescopic rod 441 away from the sliding plate 43. A clamping groove 443 for embedding the end of the suspension rod 24 is formed at the end of the clamping block 442 away from the telescopic rod 441. A limiting block 444 for abutting against the sliding plate 43 is fixedly arranged at the bottom of the sliding carriage 42; The lifting assembly 45 includes a second motor 451, a second transmission assembly 452, a first bevel gear 453, a second bevel gear 454, and a second rotating shaft 455. The second motor 451 is fixedly arranged at the top end of the lifting bracket 41. The second rotating shaft 455 is rotatably arranged at the top end of the lifting bracket 41. The first bevel gear 453 is fixedly connected to the motor shaft of the second motor 451. The second bevel gear 454 is fixedly arranged at one end of the second rotating shaft 455 and meshes with the first bevel gear 453. The second transmission assembly 452 is respectively connected to the second rotating shaft 455 and the sliding plate 43 to drive the sliding plate 43 to move up and down.

[0052] The lifting mechanism 4 further includes a support rod 46 and a position sensor 47. The support rod 46 is fixedly arranged at the lower part of the front end or the rear end of the lifting bracket 41. The position sensor 47 is fixedly arranged on the support rod 46 between the two lifting brackets 41. The position sensor 47 is electrically connected to the controller.

[0053] During operation, the metal plate 5 to be detected is hung on the suspension rod 24. The conveyor belt 21 drives the suspension rod 24 to move. When the suspension rod 24 reaches the defined position of the lifting mechanism 4, the position sensor 47 senses the suspension rod 24 and transmits a signal to the clamping assembly 44. The clamping assembly 44 clamps the suspension rod 24, and the clamping groove 443 engages with the end of the suspension rod 24, which can form an anti-slip limit for the suspension rod 24 to prevent it from falling. Then, the second motor 451 is started to drive the first bevel gear 453 to rotate and drive the second bevel gear 454 to rotate. Thus, the sliding plate 43 is driven by the second rotating shaft 455 through the second transmission assembly 452 to rise to the defined position. At this time, the detection mechanism 3 detects the metal plate 5. When the carriage 42 descends after the detection, the limit block 444 can limit its position.

[0054] The detection mechanism 3 further includes a light source 32 and a detection bracket 33. The detection brackets 33 are respectively arranged in front of and behind the lifting mechanism 4 in the moving direction of the conveyor belt 21. The two bottom ends of the detection bracket 33 are respectively fixedly connected to the frame 1 outside the conveyor belt 21. The camera 31 is fixedly arranged on the cross bar of the detection bracket 33. The light source 32 is fixedly arranged on another cross bar of the detection bracket 33 above the camera 31. The camera 31 and the light source 32 are respectively electrically connected to the controller.

[0055] The first transmission assembly 22 and the second transmission assembly 452 are respectively chain transmission pairs.

[0056] The model of the camera 31 is MV-GEF1205GC and the lens model is MV-LD-16-5M-F. The light source 32 is an annular light source and its model is MV-HLH70R / G / B / W / 90.

[0057] The controller is an industrial computer, a PC or other dedicated programmable control devices.

[0058] The above are only the preferred specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A method for detecting the surface of a metal plate based on machine vision, characterized in that: It includes steps of image segmentation, edge detection, and surface quality determination, and the content of each step is as follows: A. Image segmentation: Using the RGB color space, the full covariance GMM with K Gaussian components is used to model the target and the background respectively, and then the Grabcut algorithm is used to segment the metal plate surface image and smooth the image boundary; B. Edge detection: Using the improved Canny operator to perform edge detection on the image after the above-mentioned smoothing process; C. Surface quality determination: According to the above-mentioned image edge detection results, calculate the oil stain area on the metal plate surface and output the detection result.

2. The method for detecting the surface of a metal plate based on machine vision according to claim 1, wherein: The full covariance GMM in step A is: where: π is the weight of the Gaussian component, gi ( x, μ, ∑ ) is the probability density function (PDF) of the i -th Gaussian component, πi is the weight of the i -th Gaussian component, satisfying the probability normalization condition; x is the input data point, with dimension d*1; μi is the mean vector of the i -th Gaussian component; d is the data dimension, e.g., (pixel value dimension); ∑ is the covariance matrix, k is the additional vector, k ={k1,...,kn,...,kN}, where kn is the Gaussian component corresponding to the n -th pixel, kn ∈{1,... K}; where K can be selected according to the color distribution or texture complexity of the image.

3. The method for detecting the surface of a metal plate based on machine vision according to claim 1, wherein: In step A, when using the Grabcut algorithm to segment the metal plate surface image and smooth the image boundary, first initialize the pixel values of the target GMM and the background GMM corresponding to mask in the Grabcut operator grabcut(img, mask, rect, bgdmodel, fgdmodel, iterCount, [mode]), and then perform segmentation on the metal plate surface image and smooth the image boundary through iterative minimization; the specific process of initializing the pixels is as follows: A10. Obtain an initial trimap T by directly selecting the metal plate surface, that is, all pixels outside the square are used as background pixels TB, and all pixels inside the square TU are used as "possibly target" pixels; A11. For each pixel n in TB, initialize the label α of pixel n n = 0, which means it is a background pixel; and for each pixel n in TU, initialize the label α of pixel n n = 1, that is, a pixel that "may be the target"; A12. After the above two steps, obtain the pixels belonging to the metal plate surface, and the remaining pixels belong to the background. Use the above-mentioned pixels to estimate the GMM of the target and the background.

4. The method for detecting the surface of a metal plate based on machine vision according to claim 3, wherein: The specific process of iterative minimization in step A is as follows: A20. Assign the Gaussian components in the GMM to each pixel through the formula ; In the formula, K n is the Gaussian component index corresponding to the nth data point, and its value range is {1, 2, ……, K}, where K is the total number of Gaussian components; is to K n perform optimization to make the objective function D n reach the minimum value; D n is used to measure the distance or difference between the nth data point and the K n th Gaussian component; a n is the feature vector of the nth data point; θ is the parameter set of the target and background models, including the mean and covariance matrix of the Gaussian components; z n is the latent variable or label of the nth data point; A21. For the given image data Z, estimate the optimal parameters of the GMM by minimizing the objective function U θ , that is ; where: θ is the parameter set of the GMM, α is the regularization term, k is the number of Gaussian mixture components, z is the input image feature; A22. Perform segmentation estimation through the formula ; where: Tu is the set of pixels / regions to be processed in the image; A23. Repeat A20 to A22 to perform interactive optimization on the GMM model and the segmentation result until convergence, and obtain the segmentation boundary of the image; A24. Use border matting to smooth the above-mentioned segmentation boundary.

5. The method for detecting the surface of a metal plate based on machine vision according to any one of claims 1 to 4, characterized in that: The specific process of step B is as follows: B10. Use bilateral filtering instead of Gaussian filtering to perform noise reduction processing on the smoothed image; B20. Use the scharr operator instead of the sobel operator to calculate the pixel gradient and direction of the noise-reduced image; B30. Use polygon approximation to optimize the edge connection part of the image segmentation boundary.

6. The method for detecting the surface of a metal plate based on machine vision according to claim 5, characterized in that: The specific process of step B30 is as follows: B31. Input contour: Represent the contour extracted from the image as a set of discrete point coordinates; B32. Select the starting point and the ending point: During the approximation process, select the starting point and the ending point of the contour as the endpoints of the approximation line segment; B33. Calculate the farthest point: Select the point farthest from the line segment between the starting point and the ending point as the separation point; B34. Recursive segmentation: Divide the contour into two parts, and use the same method to approximate each part of the contour; B35. Recursive merging: The approximate contours obtained after approximation are merged in a recursive manner to obtain the final polygon approximation result.

7. A metal plate surface detection device based on machine vision, comprising a frame (1), characterized in that: It also includes a conveying mechanism (2), a detection mechanism (3), a lifting mechanism (4), and a controller. The conveying mechanism (2), the detection mechanism (3), and the lifting mechanism (4) are respectively electrically connected to the controller. The controller judges the oil stain condition on the surface of the metal plate (5) according to the detection image of the detection mechanism (3) and outputs a judgment result; The conveying mechanism (2) includes a conveyor belt (21), a transmission assembly I (22), and a motor I (23). Two conveyor belts (21) are symmetrically arranged and rotatably arranged on both sides of the top of the frame (1). A number of hanging rods (24) for hanging the metal plate (5) are supported at intervals along the moving direction on the two conveyor belts (21) at the top of the frame (1). The motor I (23) is fixedly arranged at the lower part of the frame (1). The transmission assembly I (22) is respectively connected to the output shaft of the motor I (23) and the conveyor belt (21) to drive the conveyor belt (21) to rotate and move; The lifting mechanism (4) includes a lifting bracket (41), a sliding carriage (42), a sliding plate (43), a clamping assembly (44), and a lifting assembly (45). The lifting brackets (41) are respectively vertically fixed on the outer sides of the two conveyor belts (21) at the top of the frame (1). Two sliding carriages (42) are arranged and symmetrically fixed on the inner sides of the two lifting brackets (41). The sliding plate (43) is slidably arranged on the sliding carriage (42). Two clamping assemblies (44) are symmetrically fixed on the corresponding sliding plates (43) on both sides and can clamp the two ends of the hanging rod (24). The lifting assembly (45) is arranged on the lifting bracket (41) and its driving ends are respectively connected to the two sliding plates (43) to drive the sliding plates (43) to move up and down; The detection mechanism (3) is respectively arranged in front of and behind the lifting mechanism (4) in the moving direction of the conveyor belt (21). The lens of the camera (31) of the detection mechanism (3) is directly facing the surface of the metal plate (5) lifted by the lifting mechanism (4). The camera (31) is electrically connected to the controller.

8. The metal plate surface detection device based on machine vision according to claim 7, wherein: The clamping assembly (44) includes a telescopic rod (441) and a clamping block (442). The telescopic rod (441) is fixedly arranged on the sliding plate (43). The clamping block (442) is fixedly arranged at the telescopic end of the telescopic rod (441) away from the sliding plate (43). A clamping groove (443) for embedding the end of the hanging rod (24) is formed at the end of the clamping block (442) away from the telescopic rod (441). A limiting block (444) for abutting against the sliding plate (43) is fixedly arranged at the bottom of the sliding carriage (42); The lifting component (45) includes a motor II (451), a transmission component II (452), a bevel gear I (453), a bevel gear II (454), and a rotating shaft II (455). The motor II (451) is fixedly arranged at the top of the lifting bracket (41). The rotating shaft II (455) is rotatably arranged at the top of the lifting bracket (41). The bevel gear I (453) is fixedly connected to the motor shaft of the motor II (451). The bevel gear II (454) is fixedly arranged at one end of the rotating shaft II (455) and meshes with the bevel gear I (453). The transmission component II (452) is respectively connected to the rotating shaft II (455) and the sliding plate (43) to drive the sliding plate (43) to move up and down.

9. The metal plate surface detection device based on machine vision according to claim 7, wherein: The lifting mechanism (4) further includes a support rod (46) and a position sensor (47). The support rod (46) is fixedly arranged at the lower part of the front end or the rear end of the lifting bracket (41). The position sensor (47) is fixedly arranged on the support rod (46) between the two lifting brackets (41). The position sensor (47) is electrically connected to the controller.

10. The metal plate surface detection device based on machine vision according to claim 7, characterized in that: The detection mechanism (3) further includes a light source (32) and a detection bracket (33). The detection brackets (33) are respectively arranged in front of and behind the lifting mechanism (4) in the moving direction of the conveyor belt (21). The bottom ends on both sides of the detection bracket (33) are respectively fixedly connected to the frame (1) outside the conveyor belt (21). The camera (31) is fixedly arranged on the cross bar of the detection bracket (33). The light source (32) is fixedly arranged on another cross bar of the detection bracket (33) above the camera (31). The camera (31) and the light source (32) are respectively electrically connected to the controller.

Images