Industrial surface defect image classification method based on mixed query strategy active learning

Abstract

The invention discloses an industrial surface defect image classification method based on hybrid query strategy active learning, and belongs to the technical field of computer image processing and machine learning. The invention aims to solve the technical problems of high cost, long period and low rare defect recognition rate caused by category imbalance due to dependence on large-scale manual labeling. The core of the method is to execute a hybrid query strategy in an iterative loop: firstly, screening out a candidate sample set of model cognitive ambiguity through uncertainty measurement of prediction entropy; secondly, in the candidate set, a core set and hierarchical thought diversity sampling method is adopted to select a final to-be-labeled sample with both characteristic representativeness and category balance. And the query batch is manually annotated and then is used for carrying out iterative updating and optimization on the model. According to the method, the recognition precision and generalization ability of the classification model on various defects can be remarkably improved with extremely low manual labeling cost, and the model development period is greatly shortened.

Description

Technical Field

[0001] This invention belongs to the field of computer image processing and machine learning, specifically to a high-efficiency image classification method based on active learning using a hybrid query strategy for surface quality control of industrial products. Applicable fields include, but are not limited to, metalworking, semiconductor manufacturing, textiles, printing, and glass and plastic products. Background Art

[0002] In modern industrial automation, automated identification and classification of surface defects is crucial for ensuring product quality. Surface defects, such as scratches, dents, stains, color differences, cracks, and foreign matter, not only affect the product's appearance but also severely impact its performance and safety.

[0003] Currently, automated quality control technologies based on deep learning algorithms, particularly convolutional neural networks (CNNs), are rapidly replacing traditional manual visual inspection and becoming the mainstream solution for industrial defect identification. The core of this technology lies in building highly accurate classification models through supervised learning on massive amounts of precisely labeled defect images. While this technology has achieved significant success in recognition accuracy, the "large-scale data annotation" model upon which it relies also poses a significant bottleneck to its flexible deployment and rapid iteration in the industrial sector. This is reflected in the following common challenges: (1) High labeling costs and reliance on experts: Training a high-performance deep learning model requires a massive amount of training images accurately labeled by professional quality inspection engineers. In industrial scenarios, different defect categories may have high visual similarities, or the defects themselves may be extremely small. This requires labelers to have deep domain knowledge, resulting in extremely high manpower and time costs. (2) Common class imbalance: In a normal production process, the number of qualified products often far exceeds the total number of defective products. Furthermore, the frequencies of different defect types vary widely. This inherent data imbalance causes the model to be heavily biased towards the most numerous classes during training, resulting in poor recognition of rare but critical defect classes and a high rate of missed detection. (3) Poor adaptability to new patterns: Adjustments to production processes, replacement of raw materials, or changes in the environment may give rise to new defect patterns that have never appeared in historical data. Traditional supervised learning models have limited generalization capabilities when faced with these unknown patterns, requiring companies to re-collect expensive data and retrain models, resulting in a long response cycle. Summary of the Invention

[0004] To address the above technical issues, this paper proposes an active learning method based on a hybrid query strategy for classifying industrial surface defect images. This hybrid query strategy first uses the model's predicted probabilities for unlabeled samples to calculate the uncertainty of each sample and select a set of the most uncertain candidate samples. Within this set of uncertain candidates, a core set approach is then used to select the most widely distributed and representative samples in the feature space, ensuring diversity in the newly labeled samples. Stratified sampling is then used to cluster the samples by category, ensuring that each category is sampled.

[0005] In order to achieve the above object, the technical solution of the present invention is as follows:

[0006] A method for industrial surface defect image classification based on hybrid query active learning includes the following steps: Step 1: Collection and processing of industrial defect datasets; Step 2: Preprocess the industrial defect dataset input into the neural network; Step 3: Build an initialized model; Step 4: Extract features from unlabeled image data; Step 5: Select image data using a hybrid strategy active learning method; Step 6: Retrain the deep learning model; Step 7: Loop iteration has reached maximum performance.

[0007] The details of step 1 are as follows: Using industrial-grade high-resolution cameras and a carefully configured lighting system, we capture large-scale, standardized images of product surfaces on the actual production line. During this process, we ensure consistent acquisition conditions to reduce noise introduced by factors such as varying lighting and camera angles, while also capturing all known defect types and conforming parts of various shapes.

[0008] The details of step 2 are as follows: The glass image is subjected to noise reduction and enhancement using image preprocessing techniques. First, grayscale conversion is used to convert the color image into a grayscale image, reducing data complexity while effectively preserving key structural features. This preprocessing method can significantly improve the recognition accuracy and classification performance of the deep learning model for glass surface defects.

[0009] The details of step three are as follows: This process builds a "baseline model" with preliminary judgment capabilities, providing a basis for subsequent intelligent sample selection. This step utilizes a small, manually labeled, and balanced sample set reserved during the data processing phase. Using a ResNet18 network structure with pretrained weights, and using a cross-entropy loss function and optimizers like Adam, the model develops a basic understanding of various defect types.

[0010] The specific situation in the said step 4 is as follows: utilizing the powerful automatic learning ability of the deep convolutional neural network, automatically learning and extracting deep and fine features from low-level textures to high-level semantics from the preprocessed image layer by layer and end to end, which can distinguish different defect categories.

[0011] The specific situation in step five is as follows: In order to achieve more comprehensive and efficient sample selection, this method further adopts a hybrid learning strategy that integrates the two dimensions of "uncertainty" and "diversity" to ensure that the selected samples can both resolve the current cognitive ambiguity of the model. This hybrid query strategy first uses the model's predicted probability for unlabeled samples to calculate the uncertainty of each sample and screen out a batch of the most uncertain candidate samples. In the uncertainty candidate set, the core set method is further used to select the samples with the widest distribution and strongest representativeness in the feature space to ensure the diversity of the newly labeled samples. At this time, the stratified sampling method is used to cluster by category to ensure that each category can be sampled to improve the accuracy and robustness of the classification.

[0012] The specific situation in step six is ​​as follows: a small batch of high-value samples are screened out by the hybrid active learning strategy, and then labeled by human experts and added to the existing training set to retrain the deep learning model. The model can extract more discriminative features, thereby improving the classification performance of industrial defects.

[0013] The details of step seven are as follows: the entire method is driven by a closed-loop iterative cycle of "query-labeling-retraining". In each round of the cycle, the model self-optimizes based on the newly labeled data until it reaches the preset performance indicator or the labeling budget upper limit, thereby improving the classification performance of the model.

[0014] Compared with the prior art, the beneficial effects of the technical solution of the present invention are: (1) Extremely high labeling efficiency and cost-effectiveness: By intelligently balancing the selection of "difficult" samples and "novel" samples, the value of each manual labeling is maximized. The model performance can be achieved with a much smaller amount of labeling than that required by traditional methods, thereby directly reducing the manpower and time costs of the project. (2) Stronger generalization and robustness: The introduction of diversity metrics encourages the model to learn from a wider data distribution rather than focusing solely on difficult decision boundaries. This makes the final trained model more robust and has better generalization and recognition capabilities for unseen defect images that are somewhat different from the training data. (3) Rapid adaptation to new defect patterns: When a new defect type appears on the production line, its image is assigned a high score based on the diversity metric because it is far away from all known categories in the feature space, and is therefore prioritized for labeling. This establishes a closed loop of rapid response and learning of new patterns, greatly enhancing the adaptability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is the overall flow chart of the present invention; Figure 2 It is the active learning structure of the hybrid query strategy of the present invention. DETAILED DESCRIPTION

[0016] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention. The present invention provides an industrial surface defect image classification method based on hybrid query active learning, which significantly improves the efficiency and cost-effectiveness of data annotation, enhances the generalization ability and robustness of the model, and also improves the model's ability to quickly adapt to new defects.

[0017] Figure 1The flowchart of the method of the present invention fully describes a systematic, closed-loop intelligent classification method. The method begins with standardized image acquisition and grayscale conversion of industrial product surfaces. Using a small amount of manually annotated seed data, an initial baseline model based on ResNet18 is trained to extract basic features. This baseline model is then subjected to a core iterative optimization process. Within this process, the method employs an innovative two-stage hybrid query strategy: First, an uncertainty metric is used to filter candidate samples that confuse the model the most. Then, a diversity approach combining core sets and stratified sampling is used to select a query batch with the highest information value, which is both representative and considers the minority class. After expert annotation, this query batch is used to iteratively retrain the deep learning model, continuously enhancing its ability to automatically extract discriminative features. This iterative "query-annotation-retraining" cycle continues until the model classification performance reaches a preset benchmark or the annotation budget is exhausted. Ultimately, efficient and accurate industrial defect image classification is achieved at the most economical annotation cost.

[0018] The specific implementation steps are: Step 1.1: By deploying industrial-grade high-resolution cameras on actual industrial production lines, combined with a carefully configured lighting system that provides uniform, shadowless illumination, large-scale, standardized, continuous image capture of product surfaces is performed. Step 1.2 We strictly maintain the consistency of acquisition conditions such as camera focal length, shooting angle, and light intensity to minimize image noise caused by changes in environmental factors and ensure data stability; The collection process in Step 1.3 intentionally covers all known defect types and qualified product samples of various normal forms, laying a solid foundation for the subsequent construction of a comprehensive and unbiased dataset.

[0019] Step 2.1 uses grayscale conversion technology to uniformly convert the collected three-channel (RGB) color images into single-channel grayscale images. This operation can effectively preserve key structural features such as the defect's outline and texture while significantly reducing the data dimension and computational complexity of subsequent network model processing; Step 2.2: Further image enhancement processing, such as histogram equalization, is performed on the grayscale image to increase image contrast and make minor defects easier to identify. All pre-processed images are resized to a standard size to serve as standard input for the neural network model.

[0020] Step 3.1 From the preprocessed dataset, select a small set of samples that have been accurately labeled and are as balanced as possible as possible as the initial training set. Step3.2 Select a deep convolutional neural network with pre-trained weights, such as ResNet18, as the basic network structure. Using its pre-trained weights can speed up model convergence and improve classification performance; Step3.3 In the training process, use the cross-entropy loss function as the optimization target for the classification task, and use the Adam optimizer to efficiently update the network parameters due to its adaptive learning rate adjustment characteristics. Finally, an initial classification model with basic cognitive ability for various defects is trained.

[0021] Step4.1 Deep convolutional neural networks (such as ResNet18) can automatically learn and extract features for distinguishing different defect categories from input preprocessed images through layer-by-layer and end-to-end learning. Step4.2 The first few layers of the network tend to learn low-level texture features such as edges and corners. As the number of layers increases, the network gradually combines these low-level features into more complex and high-level semantic information.

[0022] Step5.1 Use the current latest trained classification model to predict all unlabeled image samples. Then, according to the prediction probability output by the model, use the prediction entropy measurement criterion to calculate the uncertainty score of each sample. According to the score from high to low, select a certain proportion of samples with high uncertainty to form a candidate sample set. This aims to narrow down the subsequent selection range to the samples that the model is most confused about; Step5.2 In the high uncertainty candidate set generated in Step5.1, further select the final samples to be labeled by combining the core set and hierarchical sampling ideas to ensure the diversity and representativeness of the new labeled samples.

[0023] Step6.1 The small batch of high-value samples selected by the mixed learning strategy in Step5.2 are handed over to human experts for accurate classification labeling; Step6.2 Add this batch of newly labeled samples to the existing training dataset to form an enhanced training set with more information; Step6.3 Use this enhanced dataset to retrain the deep learning model, optimize the network weights, and enable the model to learn more discriminative features, thereby improving its overall classification performance for industrial defects.

[0024] Step 7.1: After a model retraining cycle is complete, the system automatically returns to Steps 5.1 and 5.2 with the updated, more powerful model to begin a new round of hybrid queries. This cycle repeats itself, and in each cycle, the model optimizes and evolves based on the newly annotated key data. This iterative cycle does not continue indefinitely, but instead terminates when one or more of the following pre-defined termination conditions are met:

[0025] Figure 2 The active learning structure of the hybrid query strategy of the present invention uses uncertainty metrics to screen candidate samples that are most confusing to the model. Then, using a diversity method combining core sets and stratified sampling, a batch of queries with the highest information value is selected that is both representative and considers minority classes. This process, the core innovation of the present invention, aims to efficiently and unbiasedly select the most valuable samples from the unlabeled data pool. This is achieved through a two-stage screening process, ensuring that the final selected samples have both "high uncertainty" and "high diversity."

[0026] The specific implementation steps are: Step 1.1 The first stage: candidate set screening based on uncertainty. First, the present invention screens a candidate set of moderate size that contains the most confusing samples for the model from the unlabeled data pool. . Prediction Entropy is used as the core metric of uncertainty. Specifically, for each unlabeled sample , using the model Predict each defect category Probability The predicted entropy of this sample is It can be calculated by the following formula: in: Represents the total number of all preset categories (including qualified products), Representative model judgment sample Belong to category The probability of . Entropy value It measures the degree of disorder in the probability distribution of the model's predictions. A higher entropy value means a more uniform probability distribution, indicating that the model cannot clearly attribute the sample to any one category, that is, the uncertainty is higher. We calculate the predicted entropy of all unlabeled samples and select the one with the highest entropy value. samples, forming a high uncertainty candidate set . Step 1.2 The second stage: perform stratified diversity sampling within the candidate set. Then, the goal of this method is to select a number of The final query batch that maximizes information coverage in the feature space To directly address the class imbalance problem and ensure sample diversity, we adopted the idea of ​​combining core set and stratified sampling. (1) Pre-classification and stratification: First, use the current model Candidate set All samples in the Divide non-intersecting layers ,in Contains all predicted categories Sample. (2) Intra-layer core set selection: Then, in each layer Internally, core set selection is performed independently. The purpose of core set selection is to select a small subset that can best represent the feature distribution of the entire sample set. This process aims to solve the following optimization problem, which is to select the best subset from the layer Select a size Batch (in is the number of samples allocated to the layer), so that any sample in the layer To its batch The maximum value of the distance between the nearest neighbor samples in is minimized. Its objective function can be expressed as: in: Represent samples and Through the current model The extracted high-dimensional feature vector, Represents the Euclidean distance between two eigenvectors. Step 1.3 Final batch combination: from each layer The core set selected Merge to form the final query batch: This stratified sampling mechanism ensures that diverse samples can be selected from minority classes, thereby directly and effectively solving the sampling bias problem caused by data imbalance in traditional diversity sampling methods.

Claims

1. A method for industrial surface defect image classification based on active learning of hybrid query strategy, characterized by The following steps are involved: Step 1: Collection and processing of industrial defect datasets; Step 2: Preprocess the industrial defect dataset input into the neural network; Step 3: Build an initialized model; Step 4: Extract features from unlabeled image data; Step 5: Use hybrid strategy active learning method to select image data; Step 6: Retrain the deep learning model; Step 7: The loop iteration has reached the highest performance.

2. The industrial surface defect image classification method based on active learning of a hybrid query strategy according to claim 1 is characterized by: The specific process in Step 1 is as follows: Using industrial-grade high-resolution cameras and a carefully configured lighting system, we capture large-scale, standardized images of product surfaces on the actual production line. During this process, we ensure consistent acquisition conditions to reduce noise caused by factors such as varying lighting and shooting angles, while also capturing all known defect types and conforming parts of various shapes.

3. The industrial surface defect image classification method based on active learning using a hybrid query strategy according to claim 1 is characterized by: The specific process in Step 2 is as follows: Image preprocessing techniques are used to reduce noise and enhance the glass image. First, grayscale conversion is used to convert the color image into a grayscale image, reducing data complexity while effectively preserving key structural features. This preprocessing method can significantly improve the recognition accuracy and classification performance of the deep learning model for glass surface defects.

4. The industrial surface defect image classification method based on active learning using a hybrid query strategy according to claim 1 is characterized by: The specific process in Step 3 is as follows: This process builds a "baseline model" with preliminary judgment capabilities, providing a basis for subsequent intelligent sample selection. This step uses a small set of accurately labeled, well-balanced samples reserved during the data processing phase. Using a ResNet18 network structure with pretrained weights, and using a cross-entropy loss function and optimizers such as Adam, the model develops a basic understanding of various defect types.

5. The industrial surface defect image classification method based on active learning using a hybrid query strategy according to claim 1 is characterized by: The specific process in Step 4 is as follows: utilizing the powerful automatic learning capability of the deep convolutional neural network, automatically learning and extracting deep and fine features from low-level textures to high-level semantics from the preprocessed image layer by layer and end to end, which can distinguish different defect categories.

6. The industrial surface defect image classification method based on active learning using a hybrid query strategy according to claim 1 is characterized by: The specific process in Step 5 is as follows: In order to achieve more comprehensive and efficient sample selection, this method further adopts a hybrid learning strategy that integrates the two dimensions of "uncertainty" and "diversity" to ensure that the screened samples can solve the current cognitive ambiguity of the model. The hybrid query strategy first uses the model's predicted probability for unlabeled samples to calculate the uncertainty of each sample and screen out a batch of the most uncertain candidate samples. In the uncertainty candidate set, the core set method is further used to select the samples with the widest distribution and strongest representativeness in the feature space to ensure the diversity of the newly labeled samples. At this time, the stratified sampling method is used to cluster by category to ensure that each category can be sampled to improve the accuracy of classification. and robustness.

7. The industrial surface defect image classification method based on active learning using a hybrid query strategy according to claim 1 is characterized by: The specific process in Step 6 is as follows: a small batch of high-value samples are screened by the hybrid active learning strategy, annotated by human experts, and added to the existing training set to retrain the deep learning model. The model can extract more discriminative features, thereby improving the classification performance of industrial defects.

8. The industrial surface defect image classification method based on active learning using a hybrid query strategy according to claim 1 is characterized by: The specific process in Step 7 is as follows: The entire method is driven by a closed-loop iterative cycle of "query-labeling-retraining". In each cycle, the model self-optimizes based on the newly labeled data until it reaches the preset performance indicator or the upper limit of the labeling budget, thereby improving the model's classification performance.

Images