Workpiece defect detection method and device, computer equipment and storage medium

Abstract

The invention relates to a workpiece defect detection method and device, computer equipment and a storage medium. The method comprises the following steps: identifying at least one defect area and the defect type of each defect area in an acquired to-be-detected workpiece image; acquiring image pixel information in each defect area, and calculating defect features of each defect area; for each defect region, determining a defect classification result of the defect region according to the defect type and the defect feature of the defect region; and determining a defect classification result of the to-be-detected workpiece image according to the defect classification result of each defect area. By adopting the method and the device, the stability of the product classification result of post-treatment in the industrial production process can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of defect detection, and in particular to a workpiece defect detection method, device, computer equipment and storage medium. Background Art

[0002] Workpiece defect detection is an important part of quality control. By using visual inspection technology, we can accurately detect defects such as spots, pits, scratches, color differences and defects on the surface of workpieces. If these surface defects are not detected and repaired, they may reduce the functionality of the workpiece and even cause safety accidents.

[0003] At present, according to the actual on-site defect acceptance standard requirements, it is necessary to manually adjust the threshold so that the deep learning model can output the result of whether it is acceptable.

[0004] However, manual adjustment of thresholds requires a lot of human resources and is easily affected by human factors, resulting in unstable detection results. At the same time, parameter thresholds are difficult to optimally adapt to on-site conditions: manual adjustment of thresholds can usually only be adjusted based on individual data, and it is difficult to ensure that global on-site data information is obtained, which may result in false detections and missed detections. Summary of the invention

[0005] Based on this, it is necessary to provide a workpiece defect detection method, device, computer equipment, computer readable storage medium and computer program product to address the above-mentioned technical problems, which can improve the stability of product classification results after post-processing in the industrial production process.

[0006] In a first aspect, the present application provides a workpiece defect detection method, comprising:

[0007] Identifying at least one defective area and the defect type of each defective area in the acquired image of the workpiece to be inspected;

[0008] Obtaining image pixel information in each defect area and calculating defect features of each defect area;

[0009] For each of the defect regions, determining a defect classification result of the defect region according to the defect type and defect characteristics of the defect region;

[0010] According to the defect classification results of each defect area, the defect classification result of the workpiece image to be inspected is determined.

[0011] In a second aspect, the present application provides a workpiece defect detection device, comprising:

[0012] An identification module, used to identify at least one defective area and the defect type of each defective area in the acquired image of the workpiece to be inspected;

[0013] A determination module, used to obtain image pixel information in each of the defect areas and calculate defect features of each of the defect areas;

[0014] A first classification module is used to determine, for each defect area, a defect classification result of the defect area according to the defect type and defect characteristics of the defect area;

[0015] The second classification module is used to determine the defect classification result of the workpiece image to be inspected according to the defect classification results of each defect area.

[0016] In a third aspect, the present application provides a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps in the above method when executing the computer program.

[0017] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps in the above method when executed by a processor.

[0018] In a fifth aspect, the present application provides a computer program product, which includes a computer program, and the computer program implements the steps in the above method when executed by a processor.

[0019] The above-mentioned workpiece defect detection method, device, computer equipment, computer-readable storage medium and computer program product can determine the image defect classification result by identifying the defect area and its defect type in the image of the workpiece to be detected, determining the defect characteristics of the defect area, and determining the defect classification result of the defect area according to the defect type and defect characteristics of the defect area. It can determine the fine-grained pixel-level feature data in the image based on all pixel information in the image, and use this to perform interception and classification judgments, which can reduce the situation of interception and classification judgments based on manually defined defect data, and can solve the problem of unstable defect detection results caused by human factors adjusting threshold judgments and manually defining defects, as well as the problem of missed detection due to difficulty in adapting to on-site acquisition of global data, which can reduce human intervention and increase the representativeness of feature data, so as to perform classification based on more comprehensive and detailed feature data, thereby improving the stability and accuracy of classification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 An application environment diagram of a workpiece defect detection method provided in an embodiment of the present application;

[0021] Figure 2 A schematic diagram of a workpiece defect detection method provided in an embodiment of the present application;

[0022] Figure 3A schematic diagram of a workpiece defect detection method provided in an embodiment of the present application;

[0023] Figure 4 A structural block diagram of a workpiece defect detection device provided in an embodiment of the present application;

[0024] Figure 5 An internal structure diagram of a computer device provided in an embodiment of the present application;

[0025] Figure 6 An internal structure diagram of another computer device provided in an embodiment of the present application;

[0026] Figure 7 An internal structure diagram of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0028] The workpiece defect detection method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through a communication network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be but is not limited to various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0029] like Figure 2 As shown, the embodiment of the present application provides a workpiece defect detection method, which is applied to Figure 1 The terminal 102 or the server 104 in the example is used for explanation. It is understandable that the computer device may include at least one of the terminal and the server. The method comprises the following steps:

[0030] S201 . Identify at least one defective region and the defect type of each defective region in an acquired image of a workpiece to be inspected.

[0031] The image of the workpiece to be detected may refer to an image obtained by capturing the workpiece. The workpiece in the image of the workpiece to be detected needs to be detected to see if it meets the standards and whether it needs to be intercepted. Usually, at the production site of the workpiece, the image of the produced workpiece is captured, and after the image of the workpiece to be detected is obtained, the produced workpiece can be intercepted and classified in real time. There is at least one defect type corresponding to the defect area. The image of the workpiece to be detected can be subjected to target detection to obtain an area where at least one defect is located, i.e., a defect area, and a defect type of each defect, i.e., a defect type corresponding to the defect area.

[0032] Usually, defects refer to visible defects on the outer surface of the workpiece. For example, if the workpiece is a mobile phone, users will check whether the mobile phone has scratches and cracks on its appearance at the mobile phone production site. For example, if the workpiece is a screen, users will check whether the screen has scratches on its appearance at the screen production site. For example, if the workpiece is a racket, users will check whether the racket has a missing outer skin defect at the racket production site.

[0033] The image of the workpiece to be inspected can be input into a pre-trained defect recognition model for processing, and the contour coordinates of at least one defect area and the defect type of each defect area can be output. The contour coordinates are used to determine the boundary of the defect area.

[0034] S202: Obtain image pixel information in each defect area, and calculate defect features of each defect area.

[0035] The image pixel information in the defective area may refer to the pixel points included in the defective area, the positions of the pixel points, the pixel values ​​and other information related to the pixel points. The defect feature may be obtained by performing statistics of at least one dimension based on the image pixel information in the defective area. The number of defect features is at least one. Exemplarily, the number of pixel points included in the defective area may be counted and the number may be used as a defect feature. In fact, the defect feature may be understood as an intuitive feature that can be directly observed from the defective area.

[0036] The contour coordinates can be mapped to the image of the workpiece to be inspected to obtain an image corresponding to the defective area, and image pixel information of the corresponding image can be obtained as the image pixel information of the defective area.

[0037] S203 . For each defect region, determine a defect classification result of the defect region according to the defect type and defect characteristics of the defect region.

[0038] The defect type is used to determine the defect classification method. The defect feature is used to represent the features of the defect area as the basis for classifying the defect area. The defect classification result may refer to the result of classifying the degree of the defect in the defect area. The defect classification results may include severe or minor. The severe degree may be considered to require interception in the workpiece inspection scenario, and the minor degree may be considered to not require interception in the workpiece inspection scenario. Accordingly, the defect classification result may include interception or pass.

[0039] Exemplarily, a defect classification result of a defect area is detected according to the defect features of the defect area in a manner corresponding to the defect type. In some embodiments, the classification conditions corresponding to different defect types are different. According to the classification conditions corresponding to the defect area, it is detected whether the defect features meet the corresponding sub-conditions in the classification conditions, and the defect classification result of the defect area is determined according to the satisfied detection results of each defect feature. Specifically, the classification thresholds in the classification conditions corresponding to different defect types are different. In some embodiments, different defect types correspond to different machine learning models, and the defect features of the defect area are input into the machine learning model for processing, and the defect classification detection result is output. Among them, the machine learning model can be a model pre-trained based on the training samples formed according to the defect features of the image area of ​​the defect type.

[0040] S204 , determining the defect classification result of the workpiece image to be inspected according to the defect classification result of each defect area.

[0041] There may be at least one defective area in the image of the workpiece to be inspected, and each defective area has a defect classification result. The detection method of the defect classification result of the image can be determined according to the production requirements of the workpiece. Specifically, according to the defect type of the defective area and the defect classification results of each defective area, the defect classification results of the same defect type and / or the number of the same defect classification results can be determined. According to the defect classification results of the same defect type and / or the number of the same defect classification results, the defect classification result of the image of the workpiece to be inspected is determined.

[0042] In some embodiments, when the number of defect classification results that are interceptions is at least one, the defect classification result of the workpiece image to be inspected is determined to be interception; when all defect classification results are passed, that is, there is no interception, the defect classification result of the workpiece image to be inspected is determined to be passed.

[0043] In some embodiments, when the number of interceptions of the defect classification result of at least one target defect type is greater than or equal to the corresponding first quantity threshold, the defect classification result of the workpiece image to be inspected is determined to be interception; when the number of interceptions of the defect classification results of each target defect type is less than the corresponding first quantity threshold, the defect classification result of the workpiece image to be inspected is determined to be passed.

[0044] It can be seen that in the embodiment of the present application, by identifying the defect area and its defect type in the image of the workpiece to be inspected, and determining the defect characteristics of the defect area, and determining the defect classification result of the defect area according to the defect type and defect characteristics of the defect area, thereby determining the image defect classification result, it is possible to determine the fine-grained pixel-level feature data in the image based on all pixel information in the image, and use this to perform interception and classification judgments, which can reduce the situation of performing interception and classification judgments based on manually defined defect data, and can solve the problem of unstable defect detection results caused by human factors adjusting threshold judgments and artificially defining defects, as well as the problem of missed detections caused by difficulty in adapting to on-site acquisition of global data, which can reduce human intervention and increase the representativeness of feature data, thereby performing classification based on more comprehensive and detailed feature data, and improving the stability and accuracy of the classification results.

[0045] In some embodiments, obtaining image pixel information in each defect area and calculating defect features of each defect area include:

[0046] Acquire image pixels included in each defect area;

[0047] Calculating the size feature and position feature of each defect area according to the position of the image pixels included in each defect area;

[0048] Calculating pixel value features of each defect area according to pixel values ​​of image pixels included in each defect area;

[0049] Defect features of each defect area are determined based on the size features, position features and pixel value features of each defect area.

[0050] Get the position of the defect area mapped to the image of the workpiece to be inspected, and determine the image pixels included in the defect area. The size feature is used to describe the size of the defect. The position feature is used to describe the location of the defect. The pixel value feature is used to describe the pixel value of the defect.

[0051] For the size feature, the absolute size can be determined based on the number of image pixels included in the defect area and used as the size feature, or the relative size can be determined based on the ratio of the number of image pixels included in the defect area to the number of image pixels included in the image and used as the size feature.

[0052] For the position feature, the position of the key pixel can be determined according to the position of each image pixel and used as the position feature. The minimum circumscribed rectangle of the defect area can also be determined and used as the position feature.

[0053] For defect features, statistics may be performed according to at least one dimension based on the grayscale values ​​of image pixels included in the defect area, such as grayscale mean and grayscale variance, and the statistical results may be used as defect features.

[0054] The size feature, position feature and pixel value feature of the defect area are fused to obtain the defect feature. The fusion method may be to splice the size feature, position feature and pixel value feature.

[0055] It can be seen that in this embodiment, size features, position features and pixel value features are determined according to the positions and grayscale values ​​of the image pixels included in the defect area, feature data are calculated from the three dimensions of size, position and pixel value, and the defect features are obtained by fusion. Multi-dimensional features can be obtained, and the defect classification results of the defect area can be determined based on more comprehensive feature data. Richer data can be obtained and missed detections can be reduced.

[0056] In some embodiments, the size feature and position feature of each defect region are calculated according to the position of the image pixels included in each defect region, including:

[0057] Calculate the number of image pixels in each defect area according to the image pixels included in each defect area;

[0058] Determine the size and position of the minimum circumscribed rectangle of each defect area according to the positions of the image pixels included in each defect area;

[0059] Extracting skeleton pixels of each defect area according to the image pixels included in each defect area, and counting the number of skeleton pixels of each defect area;

[0060] Determine the size characteristics of each defect area according to the number of image pixels, the size of the minimum circumscribed rectangle and the number of skeleton pixels of each defect area;

[0061] The position feature of each defect area is determined according to the position of the minimum circumscribed rectangle of each defect area.

[0062] For each defective area, the number of image pixels included in the defective area is counted and determined as the number of image pixel points of the defective area. The number of image pixel points is used to describe the actual size of the defective area.

[0063] In some embodiments, assuming that the size of the input workpiece image to be inspected is W*H, the coordinates of each pixel point are (i, j), and after the defect detection model, a defect is detected, and the contour coordinate value of the defect is Contours a .

[0064] (1) Defect area

[0065] Based on the coordinate values ​​of the defect area contour, define a function X c as follows:

[0066]

[0067] Then the defect area a The calculation method is as follows:

[0068]

[0069] Area a The actual representation of the defect area includes the sum of the number of pixels, so Area a is the number of image pixels.

[0070] For each defective area, based on the positions of the image pixels included in the defective area, determine a rectangle whose rectangular range includes all image pixels and whose rectangular area is the smallest, and determine the rectangle as the minimum bounding rectangle of the defective area. Calculate the size of the minimum bounding rectangle, and determine the position of the minimum bounding rectangle based on the positions of the image pixels included in the minimum bounding rectangle. For example, the four vertices of the minimum bounding rectangle can be used as the position of the minimum bounding rectangle. For another example, the center point of the minimum bounding rectangle can be used as the position of the minimum bounding rectangle, where the center point can be the intersection of the diagonals of the minimum bounding rectangle. The size of the minimum bounding rectangle is used to describe the size of the boundary area corresponding to the defective area.

[0071] In some embodiments, the size of the minimum bounding rectangle may be the length and width of the minimum bounding rectangle. a , you can use the minAreaRect function to calculate Contours a The coordinate value (x, y) of the upper left corner of the defect's minimum bounding rectangle and the length and width of the rectangle. The length and width of the rectangle are the length and width W of the defect's minimum bounding rectangle. a and H a .

[0072] For each defective area, the skeleton pixels of the defective area are determined according to the positions of the image pixels included in the defective area. The number of skeleton pixels of the defective area is counted and determined as the number of skeleton pixel points of the defective area. The number of skeleton pixel points is used to describe the size of the main structure of the defective area.

[0073] Convert the defect area into a binary image, or convert the entire workpiece image to be inspected into a binary image, and define the pixel value with coordinate (i, j) in the image as S ij , the skeletonize function can be used to calculate the skeleton pixels, then Contours a The internal skeleton pixels are:

[0074] S(i, j) = skeletonize(X c (i, j))

[0075] The skeleton length LLL can be expressed as the sum of all pixels in the skeleton image, that is, the number of skeleton pixels Skeleton a for:

[0076]

[0077] For each defect area, the number of image pixels, the size of the minimum circumscribed rectangle and the number of skeleton pixels of the defect area are determined as size features. The size of the defect area can be described from three dimensions: the actual shape of the defect area, the approximate regular shape and the main structure.

[0078] For each defect area, the position of the minimum bounding rectangle is determined as the position feature. The position of the minimum bounding rectangle is used to describe the position of the boundary area corresponding to the defect area. The position of the minimum bounding rectangle is determined as the position feature of the defect area. The minimum bounding rectangle is actually a rectangle that approximates the defect. The minimum bounding rectangle can effectively represent the distribution position of the defect. In fact, the shapes of defects are diverse. Using rectangles to approximate defects can reduce the error introduced by the detection position due to different shapes of defects.

[0079] It can be seen that in this embodiment, by determining the size feature by the number of image pixels in the defect area, the size of the minimum enclosing rectangle and the number of skeleton pixels, the content of the size feature can be enriched. From details to the whole, from special defect types to the enclosing rectangle of the same standard, the content of the size feature can be increased in a diversified and multi-angle manner. Moreover, by determining the position of the minimum enclosing rectangle as the position feature of the defect area, the position of the defect area can be accurately described. At the same time, the interference introduced by defects of various shapes can be reduced, the representativeness of the position feature can be improved, and the redundant components in the position feature can be reduced, thereby improving the accuracy of the classification detection results.

[0080] In some embodiments, the grayscale values ​​of the image pixels included in each defect area are obtained, and at least one grayscale feature is calculated.

[0081] Grayscale features may include grayscale maximum value, grayscale minimum value, grayscale mean value, grayscale variance, etc. The image of the workpiece to be inspected may be converted into a grayscale image or only each defect area may be converted into a grayscale image.

[0082] Define the pixel value of the image with coordinates (i, j) as I ij :

[0083] GrayMax: maximum gray value of the image a :

[0084]

[0085] GrayMin: minimum gray value of the image a:

[0086]

[0087] Image gray mean GrayMean a :

[0088]

[0089] Image grayscale variance GrayVar a :

[0090]

[0091] In some embodiments, based on the defect features obtained above, a feature sequence may be constructed as follows:

[0092] Feature a =(Area a , W d , H a , GrayMax a , GrayMin a , GrayMean a , GrayVar a , Skeleton a )

[0093] Therefore, for any defect a in the image, a set of feature sequences Feature can be obtained according to the above feature calculation method. a .

[0094] In some embodiments, determining a defect classification result of the defect area according to the defect type and defect characteristics of the defect area includes:

[0095] Obtain a defect classification model corresponding to the defect type of the defect area;

[0096] The defect features of the defect area are input into the defect classification model corresponding to the defect type for processing, and the defect classification result of the defect area is output.

[0097] The defect classification model may refer to a pre-trained machine learning model. The defect classification model is used to process the defect features of the defect area to obtain the defect classification result of the defect area. Exemplarily, different defect types correspond to different defect classification models. The defect classification model is used to process the defect features of the defect area of ​​a certain defect type to obtain the defect classification result of the defect area of ​​the defect type.

[0098] It can be seen that in this embodiment, by using the defect classification model corresponding to the defect type to process the defect area of ​​the defect type, a corresponding defect classification model can be configured for each defect type, and each defect classification model can only classify the defect area of ​​the corresponding defect type, thereby improving the accuracy of the classification detection results of each defect type. At the same time, defect features are used as input to the defect classification model for processing to obtain classification results, which can increase the portion of valid data in the input data and increase the interpretability of the defect classification model, thereby increasing the accuracy of the classification results.

[0099] In some embodiments, determining the defect classification result of the workpiece image to be inspected according to the defect classification result of each defect area includes:

[0100] In the case where there is a defect area where the defect classification result is interception, determining that the defect classification result of the image of the workpiece to be inspected is interception; or,

[0101] When the defect classification results of each defect area are all passed, it is determined that the defect classification result of the workpiece image to be inspected is passed.

[0102] Intercept means that the defect in the workpiece requires interception. Pass means that the defect in the workpiece does not need to be intercepted, so that the workpiece can pass the inspection. Specifically, in industrial scenarios, NG and OK are used to represent defect classification results. Intercept NG means that the defect is a defect that needs to be intercepted, and Pass OK means that the defect does not need to be intercepted and can be ignored.

[0103] In an industrial production environment, interception indicates that there are problems with the product quality of the workpiece, and the product needs to be intercepted to prevent the product from being sold in the future. By indicating that there are no problems with the product quality of the workpiece and that it meets the acceptance standards, the product can be sold in the future. In actual industrial production scenarios, as long as the product has a little defect or flaw, it does not meet the acceptance standards. Accordingly, as long as there is a defect area, the defect classification result is intercepted, indicating that the product has at least one defect that does not meet the acceptance standards, affecting the quality of the product and requiring interception. If the product has no defects or every defect in the product meets the acceptance standards, the defect classification results of all defect areas are passed, indicating that the product does not have defects that affect the quality and does not need to be intercepted.

[0104] It can be seen that in this embodiment, when the defect classification result of any defective area is set to intercept, the defect classification result of the workpiece image to be inspected is determined to be intercept; only when the defect classification results of all defective areas are passed, the defect classification result of the workpiece image to be inspected is determined to be passed. This can adapt to actual production scenarios, determine defect classification results based on product quality acceptance standards, meet actual business needs, and reasonably configure the interception and passing classification results of products, thereby improving product generation quality.

[0105] In some embodiments, the defect classification model corresponding to the defect type is trained in the following manner:

[0106] Acquire images of industrial workpieces at industrial sites;

[0107] Identifying at least one defective region and a defect type of each defective region in an industrial workpiece image;

[0108] Obtaining image pixel information in each defect area and calculating defect features of each defect area;

[0109] Obtain the labeling and classification results of each defect area;

[0110] According to the defect types of the defect areas, the defect areas are combined to obtain feature groups corresponding to the defect types;

[0111] For the feature group corresponding to each defect type, the labeled classification results and defect features of each defect area in the feature group corresponding to the defect type are used as the training sample set corresponding to the defect type;

[0112] For each defect type, a defect classification model corresponding to the defect type is trained based on a training sample set corresponding to the defect type.

[0113] The industrial site may refer to the site where workpieces are produced. The images of the produced products, that is, the produced workpieces, are collected to obtain images of the workpieces to be inspected. Before using the defect classification model to classify the images of the workpieces to be inspected, it is necessary to pre-train the defect classification model.

[0114] Industrial workpiece images are collected images of workpieces in industrial sites. Industrial workpiece images are used as training samples for defect classification models. In the same way as when the defect classification model is applied, target detection is performed in the industrial workpiece image to obtain defect areas in the industrial workpiece image and defect types in each defect area. According to the image pixels included in the defect area in the industrial workpiece image, the number of image pixels in each defect area is calculated; according to the position of the image pixels included in each defect area, the size and position of the minimum circumscribed rectangle of each defect area are determined; the grayscale value of the image pixels included in each defect area is obtained, and at least one grayscale feature is calculated; according to the image pixels included in each defect area, the skeleton pixels of each defect area are extracted, and the number of skeleton pixels of each defect area is counted; according to the number of image pixels in each defect area, the size of the minimum circumscribed rectangle and the number of skeleton pixels of each defect area, the size feature of each defect area is determined; according to the position of the minimum circumscribed rectangle of each defect area, the position feature of each defect area is determined. According to the size feature, position feature and pixel value feature of the defect area, the defect feature of the defect area is determined.

[0115] The labeled classification result is the standard classification result of the defect area labeled by the user. The labeled classification result is used as the true value classification result. The defect features of the defect area are combined with the labeled classification result of the defect area to obtain a training sample.

[0116] Defective areas of the same defect type are placed in the same feature group, and defective areas of different feature groups have different defect types.

[0117] A large number of industrial workpiece images of the same workpiece can be collected, or a large number of industrial workpiece images of different workpieces can be collected to generate a large number of defect areas and put them into feature groups corresponding to corresponding defect types. Each defect area in a feature group can form a corresponding training sample, thereby obtaining a training sample corresponding to the feature group, and adding the training sample formed by each defect area included in the feature group to the training sample set corresponding to the defect type corresponding to the feature group, thereby obtaining a training sample set corresponding to each defect type. Models with the same number of feature groups can be set, and the same model can be trained using a training sample set corresponding to the same defect type to obtain a defect classification model corresponding to the defect type. Different models are trained with training sample sets corresponding to different feature groups to obtain defect classification models corresponding to different defect types.

[0118] It can be seen that in this embodiment, industrial workpiece images are acquired from the industrial site, and defect areas and defect types corresponding to each defect area are identified accordingly. Corresponding annotation classification results are acquired for each defect area to obtain training samples, and they are divided according to defect types to obtain training sample sets corresponding to multiple defect types. A defect classification model is trained using the training sample set corresponding to each defect type, and a defect classification model for the defect type can be obtained. An adapted defect classification model can be trained for each defect type, and defect classification can be focused on each defect type, thereby improving the classification accuracy corresponding to each defect type.

[0119] In some embodiments, the defect classification model is a machine learning model; after training a defect classification model corresponding to a defect type according to a training sample set corresponding to the defect type, the method further includes:

[0120] Acquire a new industrial workpiece image, identify at least one defect region in the new industrial workpiece image and the defect type of each defect region, and determine the defect characteristics of each defect region;

[0121] Obtain the labeling and classification results of each defect area;

[0122] Obtain new annotation and classification results of defect areas in historical industrial workpiece images;

[0123] According to the defect type of each defect area, each defect area of ​​the new industrial workpiece image and each defect area in the historical industrial workpiece image are combined to obtain a new group;

[0124] For the new group corresponding to each defect type, the labeled classification results and defect features of each defect area in the new group corresponding to the defect type are used as the new sample set corresponding to the defect type;

[0125] For each defect type, a defect classification model corresponding to the defect type is retrained based on a new sample set corresponding to the defect type.

[0126] The size of the machine learning model is smaller than that of the deep learning model. Using the machine learning model as a defect classification model can reduce the learning time and improve the efficiency of defect classification. In addition, the machine learning model is more suitable for training with a small and medium number of sample sets. In the scenario of industrial sites, it is necessary to quickly conduct quality inspections on products. Samples are usually generated on site, and the number of samples is small. In order to quickly achieve defect classification, a shorter learning time is usually required. The machine learning model has a small number of parameters and a short training time, which can meet the defect classification needs of industrial sites. The cost of the machine learning model is lower than that of the deep learning model, which can reduce the cost of defect classification. Exemplarily, the machine learning model can be a model formed by a random forest algorithm.

[0127] Actual business needs will keep changing, and may even be completely different from the previous labeled classification results. At this time, a new defect classification model needs to be retrained. However, machine learning models have low costs, small training parameters, fast learning speed, and short training time. By retraining the defect classification model with a machine learning model, a defect classification model that adapts to the latest classification needs can be quickly obtained.

[0128] The new industrial workpiece image is used to train a new defect classification model. The new annotation classification result may refer to the defect classification result obtained by annotation according to the new classification requirements. The new annotation classification result may be different from the annotation classification result. In some embodiments, for the same defect area, the historical annotation classification result is intercepted, and the new annotation classification result is passed.

[0129] The new sample set is a sample set formed according to the new classification requirements. Based on the new sample set corresponding to the defect type, the new machine learning model is trained to obtain the defect classification model corresponding to the defect type.

[0130] It can be seen that in this embodiment, in view of the ever-changing actual business needs and acceptance standards in industrial projects, new industrial workpiece images are obtained, and a new sample set is formed. The defect classification model corresponding to each defect type is retrained, which can flexibly adapt to the real-time changing acceptance standards and quickly generate a defect classification model that meets the new defect classification needs, thereby improving the implementation efficiency of defect classification and taking into account the defect classification accuracy of new needs.

[0131] In the industrial defect detection scenario, for the defects given at the business site, we will use the defect data to customize the development of deep learning algorithms to detect the defects. However, there are the following differences between algorithm development and actual business needs:

[0132] During the algorithm development process, defects are defined according to their actual representation in the image, such as the texture, shape, color and other information of the defects. In actual business needs, users also define defects based on information such as the size, area and location of the defects. However, this information cannot be directly learned by the algorithm model through model training. Therefore, based on the algorithm detection, the results of the algorithm detection need to be filtered through post-processing to ensure that the defects finally output are defects that meet the needs of actual business scenarios.

[0133] In actual business scenarios, users will modify the acceptance criteria of defects according to the actual business needs on site. For the modified acceptance criteria, the site needs to re-adjust the post-processing parameter information based on the detection results of the algorithm to ensure that the defects finally output are defects that meet the needs of the actual business scenarios.

[0134] Based on the above two problems, the current solution is usually to manually adjust the post-processing parameters for the defective images and normal images defined by the on-site users to ensure that the user's defect detection requirements are met. However, manual adjustment of parameters has the following problems:

[0135] 1. Manual post-processing consumes a lot of manpower: Manual adjustment of thresholds requires a lot of human resources and is easily affected by human factors, resulting in unstable detection results.

[0136] 2. Parameter thresholds are difficult to optimally adapt to on-site conditions: Manually adjusted thresholds can usually only be adjusted based on individual data. It is difficult to ensure that global on-site data information is obtained, and it is impossible to obtain the threshold that best balances the false detection rate and missed detection rate indicators.

[0137] In response to the above problems, Figure 3 As shown, the embodiment of the present application provides an optional example.

[0138] The embodiments of the present application provide some defect features, which are used as the basis for defect classification.

[0139] For an input workpiece image to be inspected, the real defect and workpiece area can be browsed in the workpiece image to be inspected. The real defect is Figure 3 The middle red part, the workpiece area is Figure 3 The blue octagon. The image of the workpiece to be inspected is input into the deep defect detection model, which is a pre-trained deep learning model for detecting defects. The deep defect detection model will output a series of defect types and defect locations. The defect location is represented by the location of the defect area where the defect is located. The area of ​​the defect location is cropped to obtain the defect area. For the image information in each defect area, this application designs the following features based on actual industrial business scenarios to characterize the feature information of the defect area for subsequent post-processing judgment.

[0140] The size of the workpiece image to be inspected is W*H, and the coordinates of each pixel are (i, j). After the defect detection model, defect a is detected, and the contour coordinate value of the defect is Contours a The following defect features can be constructed based on the defect contour area within the contour range. The features can be calculated by referring to the above example.

[0141] The calculated features are concatenated to obtain a feature sequence, which is the defect feature.

[0142] Forming a feature sequence:

[0143] Feature a ={Area, W a , H a、GrayMax a 、GrayMin a 、GrayMean a GrayVar a and

[0144] Skeleton a}

[0145] Construct a defect dimension decision algorithm based on machine learning algorithm: For any defect a in the image, the feature sequence Feature can be calculated a For each defect, the user will manually review it on site and give a label of NG or OK as the result of the annotation classification. In fact, these annotation classification results will be configured according to the product project and the user, and the user's defect identification dimensions usually include scratches, discoloration and location, which are mapped to the defect image as the size, pixel value and location of the defect.

[0146] Assuming that the current workpiece may have N types of defects, several categories of defects a, b, c, d, e, f... will be detected on the image. At the same time, each defect will get a feature sequence and corresponding label information. Here, machine learning models will be constructed for N categories of defects to handle these defects.

[0147] During the training phase, a large number of industrial workpiece images will be collected on site, and target detection will be performed in batches. The defect area and defect type of each image will be output. For each type of defect, the defect area is processed to obtain defect features, and the user will give the label of the defect on site as the annotation classification result. The defect features and labels are combined to form training samples. For the same type of defect, the training samples formed by the defect type are used to train the random forest model to obtain a defect classification model that meets the requirements of the site. Each type of defect is trained to obtain the corresponding defect classification model, and finally the post-processing defect classification model corresponding to N defect types is obtained.

[0148] In the post-launch reasoning phase after the defect classification model training is completed, the defect detection algorithm obtains at least one defect area and defect type for the input workpiece image to be inspected. The defect features are calculated for each defect area and input into the defect classification model corresponding to the corresponding defect type to obtain the defect classification result of NG or OK for the defect area.

[0149] Construct the final judgment result logic based on the post-processing algorithm results:

[0150] For each type of defect on each image, several NG and OK results will be obtained. Considering the actual application scenario, due to the inconsistency of defect prediction types and quantities on each image, it is impossible to perform feature alignment to construct algorithm training, so logical processing is used.

[0151] Usually in industrial scenarios, it is necessary to ensure the interception rate of defective samples first. Therefore, for the NG and OK results of several defects output in the image, as long as there is one defect that is NG, the final result of the current image is determined to be an NG sample and needs to be intercepted. Otherwise, it is an OK sample and can pass.

[0152] The embodiment of the present application designs defect features suitable for various defect scenarios by targeting defect features in the industrial field, including defect area, length and width of the maximum circumscribed rectangle of the defect, maximum grayscale difference of the defect grayscale, defect skeleton size and other feature types; a machine learning algorithm can also be designed to fit the feature according to the defect feature type, train the machine learning model according to the manually given label, and output the final judgment result of the defect type; and a judgment method can be designed for the output result of the machine learning model corresponding to each defect, and the final result of the current picture can be given. This can improve industrial production efficiency: in an industrial production environment, efficient and accurate defect detection is crucial to ensuring product quality and improving production efficiency. The application of new technology solutions is expected to reduce manual intervention in post-processing in production. The machine learning model can be automatically trained by applying the defect judgment label required by the user to determine the defect category; achieve better generalization performance: because the actual business needs and acceptance standards in industrial projects may change continuously, by applying this solution, business data can be continuously accumulated, and the post-processing algorithm can be trained through the training of the full amount of historical data to ensure that the reasonable configuration of the threshold can be achieved to the greatest extent, and sufficient generalization performance can be guaranteed; and costs can be reduced: to a certain extent, the automation of the system and higher detection accuracy help reduce product losses caused by defects, thereby reducing quality control and production costs.

[0153] It should be understood that, although the steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indications of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0154] Based on the same inventive concept, the embodiment of the present application also provides a workpiece defect detection device. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more workpiece defect detection device embodiments provided below can refer to the limitations of the workpiece defect detection method above, and will not be repeated here.

[0155] like Figure 4 As shown, the embodiment of the present application provides a workpiece defect detection device 400, comprising:

[0156] An identification module 401 is used to identify at least one defective area and the defect type of each defective area in the acquired image of the workpiece to be inspected;

[0157] A determination module 402 is used to obtain image pixel information in each defect area and calculate defect features of each defect area;

[0158] The first classification module 403 is used to determine the defect classification result of each defect area according to the defect type and defect characteristics of the defect area;

[0159] The second classification module 404 is used to determine the defect classification result of the workpiece image to be inspected according to the defect classification result of each defect area.

[0160] In some embodiments, in terms of obtaining image pixel information in each defect area and calculating defect features of each defect area, the determination module 402 is specifically used to:

[0161] Acquire image pixels included in each defect area;

[0162] Calculating the size feature and position feature of each defect area according to the position of the image pixels included in each defect area;

[0163] Calculating pixel value features of each defect area according to pixel values ​​of image pixels included in each defect area;

[0164] Defect features of each defect area are determined based on the size features, position features and pixel value features of each defect area.

[0165] In some embodiments, in calculating the size feature and position feature of each defect region according to the position of the image pixels included in each defect region, the determination module 402 is specifically used to:

[0166] Calculate the number of image pixels in each defect area according to the image pixels included in each defect area;

[0167] Determine the size and position of the minimum circumscribed rectangle of each defect area according to the positions of the image pixels included in each defect area;

[0168] Extracting skeleton pixels of each defect area according to the image pixels included in each defect area, and counting the number of skeleton pixels of each defect area;

[0169] Determine the size characteristics of each defect area according to the number of image pixels, the size of the minimum circumscribed rectangle and the number of skeleton pixels of each defect area;

[0170] According to the position of the minimum circumscribed rectangle of each defect area, the position feature of each defect area is determined.

[0171] In some embodiments, in determining the defect classification result of the defect area according to the defect type and defect characteristics of the defect area, the first classification module 403 is specifically used to:

[0172] Obtain a defect classification model corresponding to the defect type of the defect area;

[0173] The defect features of the defect area are input into the defect classification model corresponding to the defect type for processing, and the defect classification result of the defect area is output.

[0174] In some embodiments, in determining the defect classification result of the workpiece image to be inspected according to the defect classification result of each defect area, the second classification module 404 is specifically used to:

[0175] In the case where there is a defect area where the defect classification result is interception, determining that the defect classification result of the image of the workpiece to be inspected is interception; or,

[0176] When the defect classification results of each defect area are all passed, it is determined that the defect classification result of the workpiece image to be inspected is passed.

[0177] In some embodiments, the workpiece defect detection device also includes a model training module, which is used to: obtain an image of an industrial workpiece at an industrial site; identify at least one defect area and the defect type of each defect area in the industrial workpiece image; obtain image pixel information in each defect area, and calculate the defect features of each defect area; obtain the annotation and classification results of each defect area; combine each defect area according to the defect type of each defect area to obtain a feature group corresponding to each defect type; for the feature group corresponding to each defect type, use the annotation and classification results and defect features of each defect area in the feature group corresponding to the defect type as the training sample set corresponding to the defect type; for each defect type, train a defect classification model corresponding to the defect type based on the training sample set corresponding to the defect type.

[0178] In some embodiments, the defect classification model is a machine learning model; the workpiece defect detection device also includes a model updating module, which is used to: obtain a new industrial workpiece image, and identify at least one defect area and the defect type of each defect area in the new industrial workpiece image, and determine the defect characteristics of each defect area; obtain the annotation classification results of each defect area; obtain new annotation classification results of the defect areas in the historical industrial workpiece image; for the defect type of each defect area, combine the defect areas of the new industrial workpiece image and the defect areas in the historical industrial workpiece image to obtain a new group; for the new group corresponding to each defect type, use the annotation classification results and defect characteristics of each defect area in the new group corresponding to the defect type as the new sample set corresponding to the defect type; for each defect type, retrain according to the new sample set corresponding to the defect type to obtain the defect classification model corresponding to the defect type.

[0179] Each module in the above workpiece defect detection device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0180] In some embodiments, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data related to workpiece defect detection. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in the above-mentioned workpiece defect detection method are implemented.

[0181] In some embodiments, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, the steps in the above-mentioned workpiece defect detection method are implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen; the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0182] Those skilled in the art will understand that Figure 5 or Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0183] In some embodiments, a computer device is provided. The computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps in the above method embodiments are implemented.

[0184] In some embodiments, Figure 7 The figure shows an internal structure diagram of a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0185] In some embodiments, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0186] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0187] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0188] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0189] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A workpiece defect detection method, characterized in that: include: Identifying at least one defective area and the defect type of each defective area in the acquired image of the workpiece to be inspected; Acquire image pixel information in each of the defect areas, and calculate defect features of each of the defect areas; For each of the defect regions, determining a defect classification result of the defect region according to the defect type and defect characteristics of the defect region; The defect classification result of the workpiece image to be inspected is determined according to the defect classification result of each defect area.

2. The method according to claim 1, characterized in that The obtaining of image pixel information in each defect area and calculating defect features of each defect area includes: Acquire image pixels included in each of the defective areas; Calculating the size feature and position feature of each defect area according to the position of the image pixels included in each defect area; Calculating pixel value features of each defect area according to pixel values ​​of image pixels included in each defect area; Determine the defect features of each defect area according to the size features, position features and pixel value features of each defect area.

3. The method according to claim 2, characterized in that The calculating the size feature and the position feature of each defect area according to the position of the image pixels included in each defect area comprises: Calculating the number of image pixels in each defect area according to the image pixels included in each defect area; Determining the size and position of the minimum circumscribed rectangle of each defect area according to the positions of the image pixels included in each defect area; Extracting skeleton pixels of each defect area according to image pixels included in each defect area, and counting the number of skeleton pixels of each defect area; Determining the size characteristics of each defect area according to the number of image pixels, the size of the minimum circumscribed rectangle and the number of skeleton pixels of each defect area; The position feature of each defect area is determined according to the position of the minimum circumscribed rectangle of each defect area.

4. The method according to claim 1, characterized in that Determining the defect classification result of the defect area according to the defect type and defect characteristics of the defect area includes: Obtaining a defect classification model corresponding to the defect type of the defect area; The defect features of the defect area are input into a defect classification model corresponding to the defect type for processing, and a defect classification result of the defect area is output.

5. The method according to claim 1, characterized in that Determining the defect classification result of the workpiece image to be inspected according to the defect classification result of each defect area includes: In the case where there is a defect area where the defect classification result is interception, determining that the defect classification result of the image of the workpiece to be inspected is interception; or, When the defect classification results of each of the defect areas are all passed, it is determined that the defect classification result of the to-be-detected workpiece image is passed.

6. The method according to claim 4, characterized in that The defect classification model corresponding to the defect type is trained in the following way: Acquire images of industrial workpieces at industrial sites; Identifying at least one defective region and a defect type of each defective region in the industrial workpiece image; Acquire image pixel information in each of the defect areas, and calculate defect features of each of the defect areas; Obtaining the labeling and classification results of each defect area; According to the defect types of the defect areas, the defect areas are combined to obtain feature groups corresponding to the defect types; For the feature groups corresponding to the defect types, the labeled classification results and defect features of the defect areas in the feature groups corresponding to the defect types are used as the training sample sets corresponding to the defect types; For each of the defect types, a defect classification model corresponding to the defect type is trained based on a training sample set corresponding to the defect type.

7. The method according to claim 6, characterized in that The defect classification model is a machine learning model; After training a defect classification model corresponding to the defect type according to the training sample set corresponding to the defect type, the method further includes: Acquire a new industrial workpiece image, identify at least one defective region in the new industrial workpiece image and the defect type of each defective region, and determine the defect characteristics of each defective region; Obtaining the labeling and classification results of each defect area; Obtain new annotation and classification results of defect areas in historical industrial workpiece images; According to the defect type of each defect area, each defect area in the new industrial workpiece image and each defect area in the historical industrial workpiece image are combined to obtain a new group; For each new group corresponding to the defect type, the labeled classification results and defect features of each defect area in the new group corresponding to the defect type are used as a new sample set corresponding to the defect type; For each defect type, a defect classification model corresponding to the defect type is retrained according to a new sample set corresponding to the defect type.

8. A workpiece defect detection device, characterized in that: include: An identification module, used to identify at least one defective area and the defect type of each defective area in the acquired image of the workpiece to be inspected; A determination module, used to obtain image pixel information in each of the defect areas and calculate defect features of each of the defect areas; A first classification module is used to determine, for each defect area, a defect classification result of the defect area according to the defect type and defect characteristics of the defect area; The second classification module is used to determine the defect classification result of the workpiece image to be inspected according to the defect classification results of each defect area.

9. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the workpiece defect detection method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the workpiece defect detection method according to any one of claims 1 to 7 are implemented.

Images