Defect classification method based on multi-model integration

Through the multi-model integration method, the complex sample images are coordinated and the defects and interference characteristics are identified, which solves the confusion between real defects and pseudo-defects in visual defect detection, and improves the accuracy and classification effect of defect classification.

CN119027732APending Publication Date: 2024-11-26EASY THINKING HANGZHOU TECH CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202411124259.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-16
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In the field of visual defect detection, in the image classification of complex samples, the confusion between real defects and pseudo defects and the confusion of different defect categories lead to poor mis-checking and classification effects, affecting product quality analysis and quality control.

Method used

The defect classification method based on multi-model integration is adopted, and complex sample images are processed through three deep learning models in a coordinated manner, and defects and interference characteristics are identified respectively to improve classification accuracy. Specific steps include: Model 1 is used to identify defect existence, Model 2 is used to identify defect types, and Model 3 is used to supplement the identification of defects and interference feature types.

Benefits of technology

It effectively improves the accuracy of defect classification results, reduces missed and misidentified recognition, and improves the ability to identify complex samples.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027732A_ABST
    Figure CN119027732A_ABST
Patent Text Reader

Abstract

The invention discloses a defect classification method based on multi-model integration, and the method comprises the steps: inputting a sample into a trained deep learning model, firstly inputting the sample into a first model, and enabling the first model to be used for recognizing whether the surface of an object to be detected has defects or not; if the output result of the first model is that the defect exists, the sample is input into a second model, and the second model is used for identifying the type of the defect contained in the surface of the object to be detected; otherwise, inputting the sample into a model III which is used for identifying the type of the interference feature and is also used for complementally identifying whether the surface of the to-be-detected object has defects or not; if the output result of the model 3 is no defect, the model 3 identifies the interference feature type included in the surface of the object to be detected; otherwise, inputting the sample into a model II, and identifying the type of the defect contained in the surface of the object to be detected by the model II; according to the method, three models are used for cooperatively processing a complex sample image, the accuracy of a defect classification result is effectively improved, and missed recognition and wrong recognition are avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and in particular to a defect classification method based on multi-model integration. Background Art

[0002] In the field of visual defect detection, accurate defect classification directly affects product quality analysis and quality control. Due to the wide variety of test objects, complex testing environments, and uncontrollable interference factors, the quality of images collected at factory inspection stations varies greatly, which greatly troubles the identification and classification of defects. Interference information in certain image backgrounds is very likely to be mistakenly identified as defects, leading to false detections. For example, in the classification and recognition of automotive paint defects, the coupling of defects and noise and the complexity of samples will lead to reduced defect classification accuracy. Specifically, the following three situations may exist in the collected sample images:

[0003] 1) There are real defects, such as dirty spots, fibers, shrinkage holes, etc.

[0004] 2) The presence of interference features (pseudo-defects), such as the edge of the object under test, random noise, water stains, etc.

[0005] 3) Contains both real defects and interference features, such as dirty point samples with random noise in the background;

[0006] In the field of defect recognition, such samples are called complex samples. When complex samples are input into a neural network model for classification, the following problems are likely to occur:

[0007] ① Confusion between true defect samples and false defect samples, leading to false detection or missed detection;

[0008] ② Confusion between true defect categories leads to poor classification results. Summary of the Invention

[0009] To address the above technical issues, the present invention provides a defect classification method based on multi-model integration. This method utilizes three models to collaboratively process complex sample images, effectively improving the accuracy of defect classification results and avoiding missed or misidentified defects.

[0010] The technical solution is as follows:

[0011] A defect classification method based on multi-model integration, which obtains an image of the surface of the object to be tested and generates samples based on the image; the samples are input into a trained deep learning model, which includes model 1, model 2, and model 3;

[0012] The sample is first input into Model 1, which is used to identify whether the surface of the object to be tested has defects;

[0013] If the output result of model 1 is defective, the sample is input into model 2, and model 2 is used to identify the type of defect contained in the surface of the object to be tested;

[0014] Otherwise, the sample is input into Model 3, which is used to identify the type of interference features and also to supplement the identification of whether the surface of the object to be tested has defects. If the output result of Model 3 is that there are no defects, Model 3 identifies the type of interference features contained in the surface of the object to be tested. Otherwise, the sample is input into Model 2, which identifies the type of defects contained in the surface of the object to be tested.

[0015] Furthermore, the samples include: multi-frequency fringe images, three-dimensional point cloud images, grayscale distribution images, defect area binary images and defect depth values;

[0016] The multi-frequency fringe image is a set of fringe images of the surface of the object to be tested obtained at a single detection posture;

[0017] A three-dimensional point cloud image, a grayscale distribution image, a defect area binary image and a defect depth value are obtained using the multi-frequency fringe image;

[0018] The three-dimensional point cloud image contains the depth value of each point; the gray value of each point in the gray distribution image is the gray mean of the corresponding point in each sinusoidal fringe image; the defect area binary image is a three-dimensional point cloud image and / or gray distribution image. Figure 2 The defect depth value is the difference between the depth mean of each point in the defect area and the depth mean of each point in the non-defect area.

[0019] Furthermore, the grayscale distribution map is obtained as follows:

[0020] Denoise each sinusoidal fringe image separately, and use the processed images to synthesize a grayscale distribution map. The grayscale value F of a single pixel (u, v) in the grayscale distribution map is:

[0021]

[0022] Where i = 1, 2...n, n is the total number of sinusoidal fringe patterns; D i is the grayscale value of the image point (u, v) in the i-th denoised sinusoidal fringe image.

[0023] Furthermore, the binary image of the defect area is obtained by the first method, the second method, or the third method:

[0024] Method 1: Set the grayscale threshold and perform threshold segmentation on the grayscale distribution map, and record the segmented binary image A as the defect area binary map;

[0025] Method 2: Set a depth threshold and perform threshold segmentation on the 3D point cloud image, and record the segmented binary image B as the defect area binary image;

[0026] Method 3: First, obtain the binary image A and the binary image B through method 1 and method 2 respectively, then perform a union operation on the binary image A and the binary image B, and record the image obtained by the operation as the binary image of the defect area.

[0027] Furthermore, the defect depth value is obtained as follows:

[0028] In the defect area binary image, count the points in the defect area and the points in the non-defect area, take the average of the depth values ​​of the points in the defect area and the points in the non-defect area in the three-dimensional point cloud image, make a difference between the two means, and record the difference as the defect depth value.

[0029] Preferably, at least two images among the multi-frequency fringe image, the grayscale distribution image, the three-dimensional point cloud image, and the defect area binary image are input into the first model. The output channel number of the model is 1, and the output result is: the probability of the defect. The presence of the defect is determined based on the size of the probability value.

[0030] At least two types of information, including the multi-frequency fringe image, the three-dimensional point cloud image, the grayscale distribution image, the defect area binary image, and the defect depth value, are input into the second model. The model has m output channels and the output result is: the probability of the existence of m types of defects. The presence of a certain type of defect is determined based on the size of the probability value.

[0031] At least two images from the multi-frequency fringe image, grayscale distribution image, three-dimensional point cloud image, and defect area binary image are input into model three. The model has n+1 output channels, and the output results are: the probabilities corresponding to n types of interference features and the probability of defects. The presence of defects or a certain type of interference feature is determined based on the size of the probability value.

[0032] Preferably, the input information in the three models is not exactly the same.

[0033] Preferably, if the probability value of defect output by model 1 is greater than threshold I, it is considered that a defect exists; if the probability value of defect output by model 3 is greater than threshold II, it is considered that a defect exists; wherein threshold I and threshold II are not equal.

[0034] Preferably, the network architectures of Model 1 and Model 3 are different.

[0035] Furthermore, the training process of the deep learning model is as follows:

[0036] Collect images of different surfaces of the object to be tested and generate multiple sets of samples accordingly;

[0037] Label each sample with a composite label to form a training sample;

[0038] The composite label includes label one, label two and label three;

[0039] The label 1 includes an element representing whether there is a defect;

[0040] Label 2 includes m elements, each element represents a defect category;

[0041] Label 3 consists of n+1 elements, where n elements represent n interference features and one element is the element in label 1.

[0042] Input the corresponding image and label 1 in the training sample into model 1 for training, input the corresponding image and label 2 in the training sample into model 2 for training, and input the corresponding image and label 3 in the training sample into model 3 for training;

[0043] Multiple groups of samples participate in the training, and the model parameters are updated iteratively. When the recognition accuracy of the deep learning model meets the requirements, the training ends and the model parameters are stored.

[0044] Furthermore, the object to be measured is a mirror or a mirror-like object;

[0045] The defects include dirty spots, shrinkage holes, fibers, scratches, impurities, convex spots, pits, sheet metal deformation, and sagging;

[0046] The interference features include edges of the object to be tested, random noise, orange peel, and water stains.

[0047] This method has the following characteristics:

[0048] ① Three deep learning models were trained, each with different recognition capabilities. The three models work together to process complex sample data, improve classification accuracy, and reduce the misleading effects of interference features.

[0049] ② Since different types of defects have different degrees of prominence in different feature images, this method designs samples in such a way that they contain a variety of information, including: multi-frequency fringe images (such as three-frequency four-term sinusoidal fringes), three-dimensional point cloud images, grayscale distribution images, defect area binary images, and defect depth value information. In addition, for different models, a variety of information in the samples will be input in a targeted manner to improve the recognition accuracy.

[0050] ③ During the model training process, this method annotates the samples with composite labels (label one, label two, and label three). For different models, different sample information and labels are input, and targeted training is performed, which improves the recognition accuracy of the model, enables the model to converge quickly, and speeds up the training. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1It is a schematic diagram of the classification process flow in a specific implementation manner;

[0052] Figure 2 This is a schematic diagram for comparing information of five samples;

[0053] Figure 3 Schematic diagram of the classification process of five samples. DETAILED DESCRIPTION

[0054] The technical solution of the present invention is described in detail below in conjunction with specific implementation methods.

[0055] A defect classification method based on multi-model integration acquires an image of the surface of the object to be tested and generates samples based on the image; the samples are input into a trained deep learning model, which includes model 1, model 2, and model 3;

[0056] like Figure 1 As shown, the sample is first input into Model 1, which is used to identify whether there are defects on the surface of the object to be tested;

[0057] If the output result of model one is defective, the sample is input into model two, which is used to identify the type of defects contained in the surface of the object to be tested;

[0058] Otherwise, the sample is input into Model 3, which is used to identify the type of interference features and also to supplement the identification of whether the surface of the object to be tested has defects. If the output result of Model 3 is no defects, Model 3 identifies the type of interference features contained in the surface of the object to be tested. Otherwise, the sample is input into Model 2, which identifies the type of defects contained in the surface of the object to be tested.

[0059] When the object to be measured is a mirror or mirror-like surface, the image of the surface of the object to be measured is a multi-frequency fringe pattern;

[0060] At this time, the samples include: multi-frequency fringe image, three-dimensional point cloud image, grayscale distribution image, defect area binary image and defect depth value;

[0061] A multi-frequency fringe pattern is a set of fringe images obtained on the surface of the object under test at a single detection posture. For example, by projecting three-frequency four-phase sinusoidal fringes, 12 fringe patterns are obtained on the surface of the object under test at a single detection posture.

[0062] The three-dimensional point cloud image, grayscale distribution image, defect area binary image and defect depth value are obtained using multi-frequency fringe image;

[0063] The 3D point cloud image contains the depth value of each point; the grayscale value of each point in the grayscale distribution image is the grayscale mean of the corresponding point in each sinusoidal fringe image; the binary image of the defect area is the 3D point cloud image and / or grayscale distribution Figure 2The image is formed after quantization; the defect depth value is the difference between the depth mean of each point in the defect area and the depth mean of each point in the non-defect area.

[0064] In order to improve the accuracy of the classification results, it is preferably implemented as follows:

[0065] Input at least two images from among the multi-frequency fringe image, grayscale distribution image, three-dimensional point cloud image, and defect area binary image into model 1. The output channel number of the model is 1, and the output result is: the probability of a defect. The existence of a defect is determined based on the size of the probability value.

[0066] At least two types of information, including the multi-frequency fringe image, the three-dimensional point cloud image, the grayscale distribution image, the defect area binary image, and the defect depth value, are input into the second model. The model has m output channels and the output result is: the probability of the existence of m types of defects. The presence of a certain type of defect is determined based on the size of the probability value.

[0067] At least two images from the multi-frequency fringe image, grayscale distribution image, three-dimensional point cloud image, and defect area binary image are input into model three. The model has n+1 output channels, and the output results are: the probabilities corresponding to n types of interference features and the probability of defects. The presence of defects or a certain type of interference feature is determined based on the size of the probability value.

[0068] A more preferred implementation is that the input information in the three models is not completely the same.

[0069] Furthermore, if the defect probability value output by model 1 is greater than threshold I, a defect is considered to exist; if the defect probability value output by model 3 is greater than threshold II, a defect is considered to exist; thresholds I and II are not equal. Models 1 and 3 have different network architectures.

[0070] Among them, the training process of the deep learning model is as follows:

[0071] Collect images of different surfaces of the object to be tested and generate multiple sets of samples accordingly;

[0072] Label each sample with a composite label to form a training sample;

[0073] The composite label includes label one, label two, and label three;

[0074] Label one includes an element representing whether there is a defect;

[0075] Label 2 includes m elements, each element represents a defect category;

[0076] Label 3 consists of n+1 elements, where n elements represent n interference features and one element is the element in label 1.

[0077] Input the corresponding image and label 1 in the training sample into model 1 for training, input the corresponding image and label 2 in the training sample into model 2 for training, and input the corresponding image and label 3 in the training sample into model 3 for training;

[0078] Multiple groups of samples participate in the training, and the model parameters are updated iteratively. When the recognition accuracy of the deep learning model meets the requirements, the training ends and the model parameters are stored.

[0079] The following is an example explanation:

[0080] In this embodiment, the object to be tested is a mirror or a mirror-like object (such as a sheet metal part after painting);

[0081] Defects include: dirty spots, shrinkage holes, fibers, scratches, etc.

[0082] Interference features include: edges of the object under test, random noise, orange peel, water stains, etc.

[0083] First, the model training process is carried out. In this embodiment, three deep learning models are set:

[0084] Model 1 is used to identify whether the surface of the object to be tested has defects; the output channel number of this model is 1, and the output result is: the probability of having defects;

[0085] Model 2 is used to identify the types of defects on the surface of the object being tested. The number of output channels of this model is m, and the output result is: the probability of the existence of m types of defects respectively;

[0086] Model 3 is used to identify the type of interference features and also to supplement the identification of whether the surface of the object to be tested has defects. The number of output channels of this model is n+1, and the output results are: the probabilities corresponding to n types of interference features and the probability of defects.

[0087] This method does not restrict the type of deep learning model. When implementing it, you can choose classic model architectures that perform well in image classification tasks, such as ViT, VOLO, CoAtNet, EfficientNet, etc., or networks embedded with SE (Squeeze and Excitation) and CBAM (Convolutional Block Attention Module) modules; or customize the model architecture based on the characteristics of the target task.

[0088] In order to identify defects from a more comprehensive perspective, it is preferably implemented as follows: the network architectures of the three models are not completely the same.

[0089] Specifically, to better assist Model 1 in identifying defects and prevent missed detections, Model 1 and Model 3 have different network architectures. For example, Model 1 uses the VOLO model, Model 2 uses the EfficientNet model, and Model 3 uses the CBAM model. More preferably, when the probability of a defect output by Model 1 is greater than Threshold I, the defect is considered present; and when the probability of a defect output by Model 3 is greater than Threshold II, the defect is considered present; where Threshold I and Threshold II are not equal.

[0090] After the model is selected, the following training process is performed:

[0091] 1) Collect images of different surfaces of the object to be tested and generate multiple sets of samples accordingly;

[0092] Specifically, the image of the surface of the object to be tested is a multi-frequency fringe image, and a three-dimensional point cloud image, a grayscale distribution image, a binary image of the defect area and a defect depth value are obtained based on the multi-frequency fringe image; the three-dimensional point cloud image contains the depth value of each point, the grayscale value of each point in the grayscale distribution image is the grayscale mean of the corresponding point in each sinusoidal fringe image, and the binary image of the defect area is a three-dimensional point cloud image and / or grayscale distribution image. Figure 2 The image formed after quantization has a defect depth value that is the difference between the depth mean of each point in the defect area and the depth mean of each point in the non-defect area.

[0093] The multi-frequency fringe image, the three-dimensional point cloud image, the grayscale distribution image, the defect area binary image and the defect depth value are stored as a set of samples.

[0094] like Figure 2 Shown are the multi-frequency fringe images (only one of the sinusoidal fringe images is shown), three-dimensional point cloud images, grayscale distribution images, and defect area binary images corresponding to five samples containing different types of defects and interference features.

[0095] Label each sample with a composite label to form a training sample;

[0096] The composite label includes label one, label two, and label three;

[0097] Label 1 includes an element, which represents a defect;

[0098] label1=[label1[0]]

[0099] label1[0]=0 means no defect, label1[0]=1 means defect;

[0100] Label 2 contains m elements, each of which represents a defect category.

[0101] label2=[label2[0], label2[1], label2[2],…, label2[m-1]]

[0102] Among them, label2[0] represents a dirty spot, label2[1] represents a shrinkage hole, label2[2] represents a fiber, etc.; label2[0] = 1 means there is a dirty spot, label2[0] = 0 means there is no dirty spot, and so on...

[0103] Label 3 consists of n+1 elements, where n elements represent n interference features and the other element is the element in label 1.

[0104] label3=[label3[0],label3[1],label3[2],…,label3[n-1],label1[0]]

[0105] Among them, label3[0] represents the edge of the object to be tested, label3[1] represents random noise, label3[2] represents water stains, etc.; label3[0]=1 means there is an edge of the object to be tested, label3[0]=0 means there is no edge texture, and so on...

[0106] As shown in the following table, Figure 2 The five samples in the image correspond to three labels. Label2 contains four elements, representing dirt, shrinkage, scratches, and fibers. Label3 contains 4+1 elements, the first four representing edge, orange peel, random noise, and water stains, respectively; the last element represents the element in label1.

[0107]

[0108]

[0109] During training, the three models can each have their own training sets and be trained separately, or they can be trained together. There are no special restrictions on this.

[0110] The corresponding image and label in the training sample are input into the model for training. In specific implementation, at least two images of the multi-frequency fringe image, grayscale distribution image, three-dimensional point cloud image, and defect area binary image can be selected and input into the model according to the actual situation, such as:

[0111] Example 1: Input a multi-frequency fringe image, a grayscale distribution image, and a 3D point cloud image;

[0112] Example 2: Input grayscale distribution map, 3D point cloud map, and defect area binary map;

[0113] Example 3: Input grayscale distribution map and 3D point cloud map;

[0114] Since defects will have different representations in different images, they will be clear in some images and unclear in others. For example, pit defects are more obvious in the three-dimensional point cloud image, and different-colored dirty spots are more obvious in the grayscale distribution image. Therefore, it is preferred to implement: at least input the grayscale distribution image and the three-dimensional point cloud image into model one to improve the accuracy of defect search.

[0115] When training model 2:

[0116] The corresponding image and label 2 in the training sample are input into the model 2 for training. In specific implementation, at least two types of information from the sample: multi-frequency fringe image, three-dimensional point cloud image, grayscale distribution image, defect area binary image and defect depth value can be selected and input into the model 2 according to the actual situation, such as:

[0117] Example 1: Input grayscale distribution map, defect area binary map, and defect depth value;

[0118] Example 2: Input grayscale distribution map, 3D point cloud map, and defect depth value;

[0119] Example 3: Input grayscale distribution map and defect depth value; further explanation: The three-dimensional point cloud can characterize the morphological features of the surface of the object to be tested, thereby assisting Model 2 in identifying three-dimensional morphology and geometric features, enhancing the classification and generalization capabilities of Model 2, and reducing the probability of overfitting; in addition, defects can be divided into three categories: concave defects, convex defects, and oscillation defects; if the defect depth value is greater than a certain positive number, the sample contains convex defects, and if it is less than a certain negative number, the sample contains concave defects. For this reason, the defect depth value can intuitively judge concave defects / convex defects, thereby improving classification accuracy. Therefore, in Model 2, it is preferably implemented as follows: at least the three-dimensional point cloud map and the defect depth value are input into Model 2 to improve the accuracy of Model 2 in distinguishing defect categories.

[0120] Training model three:

[0121] The corresponding image and label three in the training sample are input into the model three for training. In specific implementation, at least two images of the multi-frequency fringe image, grayscale distribution image, three-dimensional point cloud image, and defect area binary image can be selected and input into the model three according to the actual situation, such as:

[0122] Example 1: Input grayscale distribution map, 3D point cloud map, and defect area binary map;

[0123] Example 2: Input grayscale distribution map and 3D point cloud map;

[0124] Example 3: Input a multi-frequency fringe image and a binary image of a defect area;

[0125] Further explanation: Introducing the binary defect area map is equivalent to using prior information to add an attention mechanism to the model. This helps Model 3 identify areas of focus, helping it determine the authenticity of defects. If Model 1 misses a detection, Model 3 can promptly address any gaps, improving the effectiveness of the detection results. To this end, the preferred implementation is to input the binary defect area map into Model 3.

[0126] A more preferred implementation is that, when designing the input information of models one, two, and three, in order to improve the robustness of the three, it is preferably implemented that the input information in the three models is not exactly the same.

[0127] Multiple groups of samples participate in model training, and the model parameters are iteratively updated. When the recognition accuracy of the deep learning model meets the requirements, the training ends and the model parameters are stored.

[0128] In order to ensure that the model's generalization ability and accuracy meet the requirements, the number of training samples is generally no less than 500, and the more preferred number is 5,000 to 20,000.

[0129] After the model training is completed, the following is the formal use process of the model:

[0130] In specific implementation, after obtaining the image of the surface of the object to be tested, a preliminary screening can be performed first, and the images with abnormalities can be input into the deep learning model for classification and recognition;

[0131] Specifically, an image of the surface of the object to be tested is obtained and a sample is generated based on the image; as described above, in this embodiment, the image is a multi-frequency fringe image, and the sample includes a multi-frequency fringe image, a three-dimensional point cloud image, a grayscale distribution image, a binary image of the defect area, and a defect depth value;

[0132] In more detail, in this embodiment, an exemplary method for obtaining the grayscale distribution map, the defect area binary map, and the defect depth value is given:

[0133] I. The grayscale distribution map is obtained as follows:

[0134] Denoise each sinusoidal fringe image separately, and use the processed images to synthesize a grayscale distribution map. The grayscale value F of a single pixel (u, v) in the grayscale distribution map is:

[0135]

[0136] Where i = 1, 2...n, n is the total number of sinusoidal fringe patterns; D i is the grayscale value of the image point (u, v) in the i-th denoised sinusoidal fringe image.

[0137] II. The binary image of the defect area is obtained by method 1, method 2 or method 3:

[0138] Method 1: Set the grayscale threshold and perform threshold segmentation on the grayscale distribution map, and record the segmented binary image A as the defect area binary map;

[0139] Method 2: Set a depth threshold and perform threshold segmentation on the 3D point cloud image, and record the segmented binary image B as the defect area binary image;

[0140] Method 3: First, obtain the binary image A and the binary image B through method 1 and method 2 respectively, then perform a union operation on images A and B, and record the image obtained by the operation as the binary image of the defect area.

[0141] III. The method for obtaining the defect depth value is:

[0142] In the defect area binary image, count the points in the defect area and the points in the non-defect area, take the average of the depth values ​​of the points in the defect area and the points in the non-defect area in the three-dimensional point cloud image, make a difference between the two means, and record the difference as the defect depth value.

[0143] Next:

[0144] Input the sample into the trained deep learning model, which includes model 1, model 2, and model 3;

[0145] The sample is first input into Model 1, which is used to identify whether the surface of the object under test has defects. Specifically, at least two images (consistent with the input information during model training) from the multi-frequency fringe image, grayscale distribution image, 3D point cloud image, and defect area binary image are input into Model 1. The output channel number of this model is 1, and the output result is: the probability of a defect. The existence of a defect is determined based on the size of the probability value.

[0146] If the output of model 1 is defective (the probability of defective is greater than the preset threshold):

[0147] The sample is then input into Model 2, which is used to identify the types of defects on the surface of the object being tested. Specifically, at least two types of information from the sample (consistent with the input information during model training) are input into Model 2. The model has m output channels and outputs the probability of the existence of m types of defects. The presence of a certain type of defect is determined based on the probability value. For example, if the probability of a defect exceeds a preset value, it means that the defect exists; otherwise, it does not exist.

[0148] Otherwise, the sample is input into Model 3, which is used to identify the type of interference feature and also to supplement the identification of defects on the surface of the object under test. Specifically, at least two images (consistent with the input information during model training) are input into Model 3: a multi-frequency fringe image, a grayscale distribution image, a 3D point cloud image, and a binary image of the defect area. This model has n+1 output channels and outputs the probabilities corresponding to n types of interference features and the probability of a defect. The presence of a defect or a particular type of interference feature is determined based on the probability value.

[0149] If the output result of model three is no defect, model three directly outputs the probability of each type of interference feature existing; otherwise, the sample is input into model two, and model two outputs the probability of each type of defect existing.

[0150] For example, Figure 3 As shown, if Figure 2 The five samples in are input into the trained deep learning model respectively, and the corresponding output results are:

[0151] For sample ①, the probability value of 0.998 output by model 1 is greater than the preset threshold of 0.7. Therefore, sample ① is transferred to model 2, which outputs the probabilities of dirt spots, shrinkage holes, scratches, and fiber defects. Among them, the probability of dirt spots is the highest, and the probability value is greater than the preset value, indicating that there are dirt spots in sample ①.

[0152] For sample ②, the probability value of defect output in model 1 is 0.118, which is less than the preset threshold value of 0.7 (indicating no defect). Therefore, sample ② is passed to model 3. Model 3 outputs four interference features of the object under test: edge, orange peel, random noise, and water stains, as well as the probability of defect existence. Among them, the probability value of the orange peel interference feature is greater than the preset value, indicating that orange peel (pseudo-defect) exists in sample ②.

[0153] For sample ③, the probability value of defect output in model 1 is 0.021, which is less than the preset threshold value of 0.7 (indicating no defect). Therefore, sample ③ is passed to model 3. Model 3 outputs four interference features of the object to be tested: edge, orange peel, random noise, and water stains, as well as the probability of defect existence. Among them, the probability value of the edge interference feature is greater than the preset value, indicating that there is an edge (pseudo-defect) in sample ③.

[0154] In model 1, the probability value of sample ④ outputting defects is 0.986, which is greater than the preset threshold value of 0.7 (indicating a defect). Sample ④ is then transferred to model 2, which outputs the probabilities of the existence of dirt spots, shrinkage holes, scratches, and fiber defects. Among them, the probability of scratches is the highest, and the probability value is greater than the preset value, indicating that there are dirt spots in sample ④.

[0155] Sample ⑤ outputs a defective probability value of 0.689 in Model 1, which is less than the preset threshold value of 0.7 (indicating no defects). Sample ⑤ is then transferred to Model 3. Model 3 outputs four interference features of the object to be tested, namely, edge, orange peel, random noise, and water stains, as well as the probability of the existence of defects. Among them, the probability value of the existence of defects is greater than the preset value, indicating that there are defects in sample ⑤. Sample ⑤ is then transferred to Model 2. Model 2 outputs the probability of the existence of dirt spots, shrinkage holes, scratches, and fiber defects. Among them, the probability value of the existence of scratches is greater than the preset value, indicating that there are dirt spots in sample ⑤.

[0156] To prevent unnecessary missed detections, this solution incorporates a dual-model (Model 1 + Model 3) mechanism for defect identification. More preferably, a defect is considered present when the probability of a defect output by Model 1 is greater than Threshold I; a defect is considered present when the probability of a defect output by Model 3 is greater than Threshold II. Thresholds I and II are not equal. For example, a defect is considered present when the probability of a defect output by Model 1 is greater than 70%, and a defect is considered present when the probability of a defect output by Model 3 is greater than 80%.

[0157] The foregoing descriptions of specific exemplary embodiments of the present invention have been presented for purposes of illustration and description. The foregoing descriptions are not intended to be exhaustive or to limit the invention to the precise forms disclosed, and it is apparent that many variations and modifications are possible in light of the foregoing teachings. The exemplary embodiments have been chosen and described in order to explain the specific principles of the invention and their practical application, thereby enabling others skilled in the art to make and utilize the various exemplary embodiments of the invention and various alternatives and modifications thereof. The scope of the invention is intended to be defined by the appended claims and their equivalents.

Claims

1. A defect classification method based on multi-model integration, characterized in that: Acquire an image of the surface of the object to be tested and generate a sample based on the image; input the sample into a trained deep learning model, wherein the deep learning model includes model one, model two, and model three; The sample is first input into model 1, which is used to identify whether the surface of the object to be tested has defects; If the output result of model one is defective, the sample is input into model two, and model two is used to identify the type of defect contained in the surface of the object to be tested; Otherwise, the sample is input into Model 3, which is used to identify the type of interference features and also to supplement the identification of whether the surface of the object to be tested has defects; if the output result of Model 3 is that there is no defect, Model 3 identifies the type of interference features contained in the surface of the object to be tested; Otherwise, the sample is input into the second model, and the second model identifies the type of defects contained in the surface of the object to be tested.

2. The defect classification method based on multi-model integration as claimed in claim 1, characterized in that: The samples include: a multi-frequency fringe image, a three-dimensional point cloud image, a grayscale distribution image, a defect area binary image and a defect depth value; The multi-frequency fringe image is a group of fringe images obtained on the surface of the object to be tested at a single detection posture; A three-dimensional point cloud image, a grayscale distribution image, a defect area binary image and a defect depth value are obtained using the multi-frequency fringe image; The three-dimensional point cloud image contains the depth value of each point; the grayscale value of each point in the grayscale distribution image is the grayscale mean of the corresponding point of the point in each sinusoidal fringe image; the defect area binary image is an image formed by binarizing the three-dimensional point cloud image and / or the grayscale distribution image; the defect depth value is the difference between the depth mean of each point in the defect area and the depth mean of each point in the non-defect area.

3. The defect classification method based on multi-model integration as claimed in claim 2, characterized in that: The grayscale distribution map is obtained as follows: Each sinusoidal fringe image is denoised separately, and the processed images are used to synthesize a grayscale distribution map. The grayscale value F of a single pixel (u, v) in the grayscale distribution map is: Wherein, i=1, 2...n, n is the total number of sinusoidal fringe patterns; D i is the gray value of the image point (u, v) in the i-th denoised sinusoidal fringe image.

4. The defect classification method based on multi-model integration as claimed in claim 2, characterized in that: The binary image of the defect area is obtained by method 1, method 2 or method 3: Method 1: Set the grayscale threshold and perform threshold segmentation on the grayscale distribution map, and record the segmented binary image A as the defect area binary map; Method 2: Set a depth threshold and perform threshold segmentation on the 3D point cloud image, and record the segmented binary image B as a binary image of the defect area; Method 3: First, obtain the binary image A and the binary image B through method 1 and method 2 respectively, then perform a union operation on the binary image A and the binary image B, and the image obtained by the operation is recorded as a binary image of the defect area.

5. The defect classification method based on multi-model integration as claimed in claim 2, characterized in that: The defect depth value is obtained as follows: In the defect area binary image, count the points in the defect area and the points in the non-defect area, take the average of the depth values ​​of the points in the defect area and the points in the non-defect area in the three-dimensional point cloud image, make a difference between the two means, and record the difference as the defect depth value.

6. The defect classification method based on multi-model integration as claimed in claim 2, characterized in that: At least two images among the multi-frequency fringe image, grayscale distribution image, three-dimensional point cloud image, and defect area binary image are input into model 1. The output channel number of the model is 1, and the output result is: the probability of defect. Whether the defect exists is determined according to the size of the probability value. At least two types of information among the multi-frequency fringe image, the three-dimensional point cloud image, the grayscale distribution image, the defect area binary image and the defect depth value are input into the second model. The number of output channels of the model is m, and the output result is: the probability of existence of m types of defects respectively, and whether a certain type of defect exists is judged according to the size of the probability value; At least two images among the multi-frequency fringe image, grayscale distribution image, three-dimensional point cloud image, and defect area binary image are input into model three. The number of output channels of this model is n+1, and the output results are: the probabilities corresponding to n types of interference features and the probability of defects. The presence of defects and a certain type of interference feature is judged according to the size of the probability value.

7. The defect classification method based on multi-model integration as claimed in claim 6, characterized in that: The input information in the three models is not exactly the same.

8. The defect classification method based on multi-model integration as claimed in claim 6, characterized in that: If the defect probability value output by model one is greater than threshold value I, it is considered that a defect exists; if the defect probability value output by model three is greater than threshold value II, it is considered that a defect exists; wherein threshold value I is not equal to threshold value II.

9. The defect classification method based on multi-model integration as claimed in claim 1 or 6, characterized in that: The network architectures of Model 1 and Model 3 are different.

10. The defect classification method based on multi-model integration according to claim 6 or 7, characterized in that: The training process of the deep learning model is as follows: Collect images of different surfaces of objects to be tested and generate multiple groups of samples accordingly; Label each sample with a composite label to form a training sample; The composite label includes label one, label two and label three; The label 1 includes an element representing whether there is a defect; Label 2 includes m elements, each element represents a defect category; Label three includes n+1 elements, where n elements represent n interference features and the other element is the element in label one; The corresponding image and label one in the training sample are input into model one for training, the corresponding image and label two in the training sample are input into model two for training, and the corresponding image and label three in the training sample are input into model three for training; Multiple groups of samples participate in the training, and the model parameters are updated iteratively. When the recognition accuracy of the deep learning model meets the requirements, the training ends and the model parameters are stored.

Citation Information

Patent Citations

  • Image defect detection model training method and device, image defect detection model testing method and device and storage medium

    CN110796637A

  • PCB defect detection system and PCB defect detection method

    CN114820598A

  • Product surface detection method

    CN115393594A

  • Panel defect detection method, system and device and medium

    CN115661160A

  • Wafer surface defect classification method based on deep learning network

    CN116778235A