An industrial image defect detection method based on a multi-head unbalanced semi-supervised network

Abstract

The present application relates to the technical field of industrial image defect detection, and in particular to an industrial image defect detection method based on a multi-head unbalanced semi-supervised network.The industrial image defect detection method based on the multi-head unbalanced semi-supervised network provided by the present application faces a more efficient end-to-end architecture in the class unbalanced semi-supervised defect detection, uses multi-classifier heads to balance and accuracy of classification while taking into account effectiveness of representation, integrates the class unbalanced semi-supervised framework into the currently popular target detection framework, can fully cope with problems such as serious imbalance of positive and negative samples, lack of label data and the like in industrial image anomaly detection, thereby improving accuracy of an actual defect detection task, and has higher detection precision.

Description

Technical Field

[0001] This invention relates to the field of industrial image defect detection technology, specifically to an industrial image defect detection method based on a multi-head unbalanced semi-supervised network. Background Technology

[0002] Industrial image defect and anomaly detection is generally regarded as a specific application of image target detection and classification tasks in the industrial field. Currently available deep learning-based industrial image anomaly detection algorithms are mainly divided into supervised learning, unsupervised learning, and semi-supervised learning algorithms.

[0003] In industrial image defect detection based on traditional supervised learning, training set labels are indispensable, and all samples must have complete labels. Representative works include anomaly detection methods based on Siamese Network, ShuffleNet, Faster R-CNN, and U-Net. However, identifying and effectively labeling defect regions in industrial images is extremely laborious, and collecting a large number of defect samples is often impractical. Therefore, supervised learning-based anomaly detection is not applicable to all areas of industrial image quality inspection.

[0004] Unsupervised learning-based industrial image defect detection primarily utilizes reconstruction loss for model training and anomaly detection. Representative works include: anomaly detection frameworks based on Autoencoders; anomaly detection methods based on GANs; and anomaly detection frameworks based on feature learning. Compared to supervised learning-based anomaly detection, unsupervised learning eliminates the reliance on large amounts of labeled data during training, reducing human intervention. However, due to the lack of crucial supervisory information, the classification accuracy and robustness of unsupervised models still need improvement.

[0005] Semi-supervised learning combines the characteristics of supervised and unsupervised learning. Under conditions of only a few labeled samples and a large number of unlabeled samples, its training performance approaches or even surpasses that of supervised learning. Representative works include semi-supervised methods based on a small number of labeled samples, semi-supervised methods based on weak labels, and the latest semi-supervised methods based on artificially generated anomalies. Furthermore, the closest works to this invention include the unbiased teacher method and semi-supervised pseudo-labeled object detection methods. However, none of these methods fully consider the serious threat that class imbalance poses to the accuracy of semi-supervised object detection in practical applications. Simply resampling or reweighting within the same classifier has been shown to affect feature extraction, inevitably leading to poor characterization of anomalies and consequently impacting detection accuracy.

[0006] In recent research on imbalanced semi-supervised learning, researchers have introduced an auxiliary balancing classifier layer attached to existing semi-supervised algorithms to address the feature extraction degradation caused by simple resampling or reweighting on the same classifier head. This effectively balances high-quality feature representations and classification accuracy during training. Subsequently, a three-parallel module collaborative learning framework called CoSSL was proposed for efficient pseudo-label-based imbalanced semi-supervised methods, featuring decoupled representation and classifier learning. Similarly, a three-stage label propagation and redistribution framework for imbalanced semi-supervised learning problems has been proposed, comprising three stages, with an additional offline label propagation stage in the second stage to obtain more stable pseudo-labels. However, current cutting-edge research on imbalanced semi-supervised learning typically focuses on balancing pseudo-label strategies during training, without giving much consideration to the inefficient label propagation problem. Meanwhile, the effective integration of imbalanced semi-supervised learning algorithms into object (anomaly) detection is still in its early exploratory stages. Summary of the Invention

[0007] To address the shortcomings of the existing technology, the present invention aims to provide an industrial image defect detection method based on a multi-head unbalanced semi-supervised network, solving the following problems:

[0008] 1. How to integrate class-imbalanced semi-supervised learning methods into object detection models to fully address the problems of severe imbalance between positive and negative samples and lack of labeled data in industrial image anomaly detection, thereby improving the accuracy of defect detection;

[0009] 2. How to achieve reliable propagation of pseudo-labels and reasonable screening of unlabeled samples under imbalanced semi-supervised conditions;

[0010] 3. How to obtain a target detection model that balances representation effectiveness and classification balance accuracy under class imbalance conditions, thereby effectively improving the detection accuracy of small categories (outlier data).

[0011] The technical solution adopted in this invention is as follows: An industrial image defect detection method based on a multi-head unbalanced semi-supervised network, comprising the following steps:

[0012] S1. Create the model and initialize the model parameters;

[0013] S2. Construct a dataset of industrial images;

[0014] S3. Create the optimization target and optimizer;

[0015] S4. Process the data from S2 using model S1;

[0016] S5. Determine whether the current model has been improved based on whether the data processed in S4 meets the requirements;

[0017] S6. Based on the judgment result of S5, if there is an improvement, the process ends; otherwise, repeat steps S4 and S5 until the previous model has improved or the maximum number of repetitions of step S4 is reached.

[0018] The step S1 of creating a model and initializing model parameters includes supervised branch head models, semi-supervised branch head models, and unsupervised branch head models.

[0019] Furthermore, step S4 includes the following steps:

[0020] A1. Randomly select data from S2;

[0021] A2. Preprocess the data and then iterate it into the model to calculate the loss;

[0022] A3. Update the model parameters using the optimizer in S3.

[0023] Furthermore, step S6 includes the following steps:

[0024] B1. Determine whether the model parameters in step A3 are improved compared to the current model. If so, after updating and saving the model parameters, determine whether the maximum number of repetitions in step S4 has been reached. Otherwise, directly determine whether the maximum number of repetitions in step S4 has been reached.

[0025] B2. Based on the judgment in step B1, determine whether the maximum number of repetitions in step S4 has been reached. If yes, end the process; otherwise, repeat steps 4 and 5.

[0026] Furthermore, the data in step S2 includes labeled data and unlabeled data.

[0027] It is worth noting that the supervised branch head model, semi-supervised branch head model, and unsupervised branch head model created and initialized in step S1 are described below. When creating the model and initializing the model parameters, to balance information-rich feature extraction with accurate balanced classification results, a three-branch head architecture is adopted: a supervised branch head model, a semi-supervised branch head model, and an unsupervised branch head model. Simultaneously, to effectively integrate class-imbalanced semi-supervised learning methods into the object detection model, this invention utilizes pseudo-labels as explicit supervision for RPN and ROIhead, thereby alleviating the overfitting problem of RPN and ROIhead to samples with very few labels. Secondly, to minimize disruption to the existing structure of Faster-RCNN object detection and simplify the semi-supervised network architecture, only the classification layer of ROIhead is modified. Specifically, three network branches are grafted before the classification layer of ROIHead: a supervised branch head model, a semi-supervised branch head model, and an unsupervised branch head model. The three branch head models are described below:

[0028] For supervised branch head models, in order to effectively utilize high-confidence pseudo-label data for supervised training and minimize the threat posed by mislabels to model stability, this invention proposes using Generalized PageRank (GPR) to capture the correlation between sample features, thereby providing reliable pseudo-labels. Furthermore, the supervised loss function employs a cross-entropy loss function with a reweighting strategy, denoted as loss. sup .

[0029] This invention constructs a dynamic mini-image of candidate targets within a batch (batch size = 4, RTX 3090 GPU) to fully adapt to the original mini-batch SGD training process without significantly increasing the model's runtime. First, the depth features of the image corresponding to the candidate bounding box of interest are obtained through the shared feature extractor g(·), and denoted as... d m is the extracted feature dimension, and B is the number of candidate bounding boxes of interest in this batch. In this invention, for ease of description, the Faster-RCNN part connected before the g(·) fully connected layer is also included in g(·). Then, a normalized similarity matrix with self-looping is calculated. I represents the identity matrix. Next, in the graph network information propagation section, the invention constructs a structure with the following structure: [d] m -dropout(0.5)-ReLU-d m A fully connected network of [×2-ReLU-C] generates the hidden layer features H0 = s(E) for each node. A), where C is the number of classes. Subsequent inventions employed GPR to capture the correlations between vertices. Therefore, the graph-based classifier head h(·) is represented as:

[0030]

[0031] Where ω0 and ω1 are learnable weights controlling the first two propagation steps. As can be seen from the above formula, unlike the linear classifier head, this invention feeds a batch of data into a graph network used for information propagation, further obtaining more reliable label propagation results. It is important to emphasize here that experiments have shown that the pseudo-label accuracy of unlabeled training data in the training samples after using GPR propagation is more than 5% higher than that of a fully connected linear classifier.

[0032] Next, in a batch update, this invention defines loss. sup As shown in Formula 2:

[0033]

[0034] Where Y = {y1, y2, ... y B} represents the labels for all data in a batch of detected objects. For labeled data, its value is the one-hot true value. For unlabeled data, the pseudo-label y is... i Its value is determined by the data u i The weakly enhanced (random horizontal flip) version The maximum value is obtained after processing the network, i.e. To enhance the model's generalization ability, E is used for unlabeled data. A Acquired using a heavily enhanced version (color dithering, Gaussian blur, and random cropping). After passing through the front layer of Faster-CRNN and being input into g(·), the result is obtained.

[0035] W(Y) is a reweighting function, which is applied to the data x. i The corresponding label y i In this regard, its weight is defined as:

[0036]

[0037] Where M C This is the number of samples belonging to the largest category C. For belonging to category y i The number of samples. This shows that for categories with a small number of samples, the reweighted weight W(y) is... i The larger the number of samples, the smaller the reweighted weights for the category with more samples. Therefore, reweighting can be used to train a balanced classifier at the head of the supervised branch.

[0038] In addition, in Formula 2 This is a discrimination function; it returns 1 if the internal condition is met, and 0 otherwise. It is used in conjunction with the hyperparameter threshold. Reliable samples can be selected for supervised training, and in this invention, this value is uniformly set to 0.7. For the sake of brevity, this condition is consistently satisfied for labeled data. CE(·,·) is the standard cross-entropy loss function. Importantly, to address the problem of predicting repeated bounding boxes, a threshold is used... Before screening, duplicate predictions are removed by applying non-maximum suppression (NMS) by class.

[0039] For semi-supervised branch head models, in order to comprehensively utilize both labeled and unlabeled data to obtain reliable data representation, popular semi-supervised network heads and loss functions, such as ReMixMatch, FixMatch, FlexMatch, and FreeMatch, can be used in this part. Preferably, the more general and simpler FixMatch loss function form is adopted, denoted as loss. semi . Specifically:

[0040] The semi-supervised loss in this part consists of supervised and unsupervised losses, and the network structure follows the classification head structure of Faster R-CNN. Similarly, for an update of a batch, the loss... semi Defined as:

[0041]

[0042] A comparison of Equations 2 and 4 reveals their similar forms, the only difference being the absence of a reweighting component. This is because reweighting has been shown to negatively impact feature extraction efficiency. Therefore, to balance the balance of the supervised model's classification head with the effectiveness of feature extraction, the semi-supervised portion here does not undergo any adjustments to adapt to class imbalance. Since Equation 4 describes existing technology, and this invention merely utilizes it to obtain more effective features under class-imbalanced semi-supervised object detection conditions, the key innovation lies in the multi-branch network architecture, which will not be elaborated upon here.

[0043] It is worth noting here that l(·) does not use the graph network branch head h(·) constructed in (1). This is because heterogeneous networks are often more likely to learn data representations with different focuses. In the case of class imbalance semi-supervised problems, which have the inherent small sample size, the anti-overfitting effect of heterogeneous classification heads is more obvious.

[0044] For the unsupervised branch head model, to further enhance the unbiasedness of feature extraction, an unsupervised branch head is established, and all data are treated as unlabeled data for self-supervised training. This minimizes the bias impact of the perceived instability of unlabeled data on training bias while fully utilizing the unlabeled data to extract effective features. The loss function is denoted as loss. unsup . Specifically:

[0045] Considering computational cost and experimental results, this invention preferably utilizes simple consistency regularization for self-supervised training. Therefore, the loss function for unsupervised branch heads... unsup Represented as:

[0046]

[0047] in and For two different weakly augmented versions of unsupervised samples, `consist(·,·)` is the consistency penalty function. In this invention, the L2-norm loss function is chosen, and `f(·)` is a single-layer fully connected network with the structure [d m -C]. It is worth noting that, typically, to effectively utilize all the data, unlabeled samples will include labeled samples, but their label values ​​are not used during training. As can be seen from Equation 5, unsupervised loss drives the model to achieve smooth and consistent representations around all samples, thereby enhancing the generalization ability of the model's representation on the one hand, and alleviating the feature bias caused by label embedding during class-imbalanced training on the other.

[0048] Furthermore, to prevent the separation training of labeled / unlabeled data from causing a "feature shift" in the model output and thus worsening the confirmation bias, a feature matching loss is proposed to apply unbiased constraints to the output features of supervised and unsupervised branches, denoted as loss. FM :

[0049] loss FM =mean(|mean(h(x)) l ),0)-mean(f(s u ),0)|) (6)

[0050] Where f(·) represents the unsupervised branch head output, and h(·) is the supervised branch head output; mean(h(·),0) represents the averaging operation on the 0th dimension of the h(·) output. As can be seen from Equation 6, the model encourages the feature distribution of labeled samples to be consistent with that of unlabeled samples, thereby further reducing overfitting of the model to labeled samples and incorrect classification of unlabeled samples.

[0051] In summary, the total loss function of this invention is defined by Formula 7:

[0052] loss = loss sup +loss semi +loss unsup +loss FM (7)

[0053] Finally, after the model training is complete, the invention outputs the final target detection task based on the prediction results of the supervisory head.

[0054] Beneficial effects:

[0055] This invention provides an industrial image defect detection method based on a multi-head imbalanced semi-supervised network. Aiming for a more efficient end-to-end architecture in imbalanced semi-supervised defect detection, it uses multiple classifier heads to balance representation effectiveness with classification balance and accuracy. By integrating the imbalanced semi-supervised framework into currently popular object detection frameworks, it effectively addresses issues such as severe imbalance between positive and negative samples and lack of labeled data in industrial image anomaly detection, thereby improving the accuracy of practical defect detection tasks and achieving higher detection precision. This invention uses generalized PageRank as an information propagation scheme for unlabeled data, enabling reliable propagation and balanced screening of pseudo-labels under imbalanced semi-supervised conditions. Simultaneously, this invention integrates unsupervised learning, semi-supervised learning, and supervised learning to construct a three-head network architecture, achieving an object detection model that balances representation effectiveness and classification balance accuracy. As shown in the following two tables, the method employed in this invention achieves higher accuracy and precision.

[0056] Table 1. Performance of Nut Surface Defect Detection (%)

[0057] method accuracy Accuracy Recall rate Faster-RCNN 79.70 28.24 83.30 Semi-supervised pseudo-label target detection method 86.67 39.71 90.00 Method of the present invention 89.09 45.00 90.00

[0058] Table 2. AP_50 values ​​(%) for surface defect detection of strip steel at Northeastern University

[0059] Attached Figure Description

[0060] Figure 1 This is a flowchart of an industrial image defect detection method based on a multi-head unbalanced semi-supervised network according to the present invention.

[0061] Figure 2 This is an architecture diagram for establishing initial model parameters in an industrial image defect detection method based on a multi-head unbalanced semi-supervised network according to the present invention.

[0062] Figure 3 The image shows the test results of using the industrial image defect detection method based on a multi-head unbalanced semi-supervised network of the present invention to detect surface defects in nuts. The red box represents the location of the detected nut defects.

[0063] Figure 4 The image shows the test results of using the industrial image defect detection method based on a multi-head unbalanced semi-supervised network of the present invention to detect defects on the surface of strip steel. The colored boxes represent the locations of the detected defects. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the protection scope of the present invention.

[0065] like Figure 1 , Figure 2 The method for industrial image defect detection based on a multi-head unbalanced semi-supervised network, as shown, includes the following steps:

[0066] S1. Create the model and initialize the model parameters;

[0067] S2. Construct a dataset of industrial images;

[0068] S3. Create the optimization target and optimizer;

[0069] S4. Process the data from S2 using model S1;

[0070] S5. Determine whether the current model has been improved based on whether the data processed in S4 meets the requirements;

[0071] S6. Based on the judgment result of S5, if there is an improvement, the process ends; otherwise, repeat steps S4 and S5 until the previous model has improved or the maximum number of repetitions of step S4 is reached.

[0072] The step S1 of creating a model and initializing model parameters includes supervised branch head models, semi-supervised branch head models, and unsupervised branch head models.

[0073] Furthermore, step S4 includes the following steps:

[0074] A1. Randomly select data from S2;

[0075] A2. Preprocess the data and then iterate it into the model to calculate the loss;

[0076] A3. Update the model parameters using the optimizer in S3.

[0077] Furthermore, step S6 includes the following steps:

[0078] B1. Determine whether the model parameters in step A3 are improved compared to the current model. If so, after updating and saving the model parameters, determine whether the maximum number of repetitions in step S4 has been reached. Otherwise, directly determine whether the maximum number of repetitions in step S4 has been reached.

[0079] B2. Based on the judgment in step B1, determine whether the maximum number of repetitions in step S4 has been reached. If yes, end the process; otherwise, repeat steps 4 and 5.

[0080] Furthermore, the data in step S2 includes labeled data and unlabeled data.

[0081] It is worth noting that the supervised branch head model, semi-supervised branch head model, and unsupervised branch head model created and initialized in step S1 are described below. When creating the model and initializing the model parameters, to balance information-rich feature extraction with accurate balanced classification results, a three-branch head architecture is adopted: a supervised branch head model, a semi-supervised branch head model, and an unsupervised branch head model. Simultaneously, to effectively integrate class-imbalanced semi-supervised learning methods into the object detection model, this invention utilizes pseudo-labels as explicit supervision for RPN and ROIhead, thereby alleviating the overfitting problem of RPN and ROIhead to samples with very few labels. Secondly, to minimize disruption to the existing structure of Faster-RCNN object detection and simplify the semi-supervised network architecture, only the classification layer of ROIhead is modified. Specifically, three network branches are grafted before the classification layer of ROIHead: a supervised branch head model, a semi-supervised branch head model, and an unsupervised branch head model. The three branch head models are described below:

[0082] For supervised branch head models, in order to effectively utilize high-confidence pseudo-label data for supervised training and minimize the threat posed by mislabels to model stability, this invention proposes using Generalized PageRank (GPR) to capture the correlation between sample features, thereby providing reliable pseudo-labels. Furthermore, the supervised loss function employs a cross-entropy loss function with a reweighting strategy, denoted as loss. sup .

[0083] This invention constructs a dynamic mini-image of candidate targets within a batch (batch size = 4, RTX 3090 GPU) to fully adapt to the original mini-batch SGD training process without significantly increasing the model's runtime. First, the depth features of the image corresponding to the candidate bounding box of interest are obtained through the shared feature extractor g(·), and denoted as... d mis the extracted feature dimension, and B is the number of candidate bounding boxes of interest in this batch. In this invention, for ease of description, the Faster-RCNN part connected before the g(·) fully connected layer is also included in g(·). Then, a normalized similarity matrix with self-looping is calculated. I represents the identity matrix. Next, in the graph network information propagation section, the invention constructs a structure with the following structure: [d] m -dropout(0.5)-ReLU-d m A fully connected network of [×2-ReLU-C] generates the hidden layer features H0 = s(E) for each node. A ), where C is the number of classes. Subsequent inventions employed GPR to capture the correlations between vertices. Therefore, the graph-based classifier head h(·) is represented as:

[0084]

[0085] Where ω0 and ω1 are learnable weights controlling the first two propagation steps. As can be seen from the above formula, unlike the linear classifier head, this invention feeds a batch of data into a graph network used for information propagation, further obtaining more reliable label propagation results. It is important to emphasize here that experiments have shown that the pseudo-label accuracy of unlabeled training data in the training samples after using GPR propagation is more than 5% higher than that of a fully connected linear classifier.

[0086] Next, in a batch update, this invention defines loss. sup As shown in Formula 2:

[0087]

[0088] Where Y = {y1, y2, ... y B} represents the labels for all data in a batch of detected objects. For labeled data, its value is the one-hot true value. For unlabeled data, the pseudo-label y is... i Its value is determined by the data u i The weakly enhanced (random horizontal flip) version The maximum value is obtained after processing the network, i.e. To enhance the model's generalization ability, E is used for unlabeled data. A Acquired using a heavily enhanced version (color dithering, Gaussian blur, and random cropping). After passing through the front layer of Faster-CRNN and being input into g(·), the result is obtained.

[0089] W(Y) is a reweighting function, which is applied to the data x. i The corresponding label y i In this regard, its weight is defined as:

[0090]

[0091] Where M C This is the number of samples belonging to the largest category C. For belonging to category y i The number of samples. This shows that for categories with a small number of samples, the reweighted weight W(y) is... i The larger the number of samples, the smaller the reweighted weights for the category with more samples. Therefore, reweighting can be used to train a balanced classifier at the head of the supervised branch.

[0092] In addition, in Formula 2 This is a discrimination function; it returns 1 if the internal condition is met, and 0 otherwise. It is used in conjunction with the hyperparameter threshold. Reliable samples can be selected for supervised training, and in this invention, this value is uniformly set to 0.7. For the sake of brevity, this condition is consistently satisfied for labeled data. CE(·,·) is the standard cross-entropy loss function. Importantly, to address the problem of predicting repeated bounding boxes, a threshold is used... Before screening, duplicate predictions are removed by applying non-maximum suppression (NMS) by class.

[0093] For semi-supervised branch head models, in order to comprehensively utilize both labeled and unlabeled data to obtain reliable data representation, popular semi-supervised network heads and loss functions, such as ReMixMatch, FixMatch, FlexMatch, and FreeMatch, can be used in this part. Preferably, the more general and simpler FixMatch loss function form is adopted, denoted as loss. semi . Specifically:

[0094] The semi-supervised loss in this part consists of supervised and unsupervised losses, and the network structure follows the classification head structure of Faster R-CNN. Similarly, for an update of a batch, the loss... semi Defined as:

[0095]

[0096] A comparison of Equations 2 and 4 reveals their similar forms, the only difference being the absence of a reweighting component. This is because reweighting has been shown to negatively impact feature extraction efficiency. Therefore, to balance the balance of the supervised model's classification head with the effectiveness of feature extraction, the semi-supervised portion here does not undergo any adjustments to adapt to class imbalance. Since Equation 4 describes existing technology, and this invention merely utilizes it to obtain more effective features under class-imbalanced semi-supervised object detection conditions, the key innovation lies in the multi-branch network architecture, which will not be elaborated upon here.

[0097] It is worth noting here that l(·) does not use the graph network branch head h(·) constructed in (1). This is because heterogeneous networks are often more likely to learn data representations with different focuses. In the case of class imbalance semi-supervised problems, which have the inherent small sample size, the anti-overfitting effect of heterogeneous classification heads is more obvious.

[0098] For the unsupervised branch head model, to further enhance the unbiasedness of feature extraction, an unsupervised branch head is established, and all data are treated as unlabeled data for self-supervised training. This minimizes the bias impact of the perceived instability of unlabeled data on training bias while fully utilizing the unlabeled data to extract effective features. The loss function is denoted as loss. unsup . Specifically:

[0099] Considering computational cost and experimental results, this invention preferably utilizes simple consistency regularization for self-supervised training. Therefore, the loss function for unsupervised branch heads... unsup Represented as:

[0100]

[0101] in and For two different weakly augmented versions of unsupervised samples, `consist(·,·)` is the consistency penalty function. In this invention, the L2-norm loss function is chosen, and `f(·)` is a single-layer fully connected network with the structure [d m -C]. It is worth noting that, typically, to effectively utilize all the data, unlabeled samples will include labeled samples, but their label values ​​are not used during training. As can be seen from Equation 5, unsupervised loss drives the model to achieve smooth and consistent representations around all samples, thereby enhancing the generalization ability of the model's representation on the one hand, and alleviating the feature bias caused by label embedding during class-imbalanced training on the other.

[0102] Furthermore, to prevent the separation training of labeled / unlabeled data from causing a "feature shift" in the model output and thus worsening the confirmation bias, a feature matching loss is proposed to apply unbiased constraints to the output features of supervised and unsupervised branches, denoted as loss. FM :

[0103] loss FM =mean(|mean(h(x)) l ),0)-mean(f(x u ),0)|) (6)

[0104] Where f(·) represents the unsupervised branch head output, and h(·) is the supervised branch head output; mean(h(·),0) represents the averaging operation on the 0th dimension of the h(·) output. As can be seen from Equation 6, the model encourages the feature distribution of labeled samples to be consistent with that of unlabeled samples, thereby further reducing overfitting of the model to labeled samples and incorrect classification of unlabeled samples.

[0105] In summary, the total loss function of this invention is defined by Formula 7:

[0106] loss = loss sup +loss semi +loss unsup +loss FM (7)

[0107] Finally, after the model training is complete, the invention outputs the final target detection task based on the prediction results of the supervisory head.

[0108] When implementing this industrial image defect detection method based on multi-head unbalanced semi-supervised networks, such as... Figure 1 As shown, the implementation steps are as follows:

[0109] 1. Faster-RCNN and ResNet-50 are used as the base networks for object detection, and a pre-trained model on COCO is used. To address the characteristics of imbalanced semi-supervised learning, a multi-head imbalanced semi-supervised learning-based object detection system is constructed.

[0110] 2. Construct labeled training data and unlabeled training dataset. This part depends on the specific application. The labeled data uses Labelme and is stored as a .xml format label box.

[0111] 3. Establish an optimization objective within the imbalanced semi-supervised object detection framework. Details of this setup are provided in the technical solution. The initial learning rate is 1e-3, the minimum learning rate is 1e-6, and cosine decay is used. The optimizer employed is Adam.

[0112] 4. Randomly sample unlabeled / labeled images. Each batch contains 2 labeled and 2 unlabeled images, and an epoch is defined as one round of training with all labeled data completed.

[0113] 5. Perform data preprocessing, unify the size of training images to 600×600, feed them into the constructed class-imbalanced semi-supervised object detection network, and calculate the loss function.

[0114] 6. Construct the deep learning model update process and train the model using the optimization objective.

[0115] 7. Repeat steps (4) to (6) until the specified number of epochs is reached, and save the trained network model for testing.

[0116] 8. During the testing phase, the test images are standardized to 600×600 pixels and fed into the trained network model to output the detection results.

[0117] In summary, the method proposed in this invention is a more efficient end-to-end architecture for class-imbalanced semi-supervised defect detection. It uses multiple classifier heads to balance the effectiveness of representation and the balance and accuracy of classification, and integrates the class-imbalanced semi-supervised framework into the currently popular object detection framework.

[0118] It is worth noting that during the testing phase, you only need to load the last saved model, scale the test image to 600×600 and send it into the model to output the detection results.

[0119] like Figure 3 As shown, in a real-world nut anomaly detection task (where only about 30% of the training data used labels to annotate defect locations, and some duplicate or very similar images were removed, resulting in a normal / abnormal ratio of approximately 10:1), with an IoU of 0.5 considered a correct anomaly detection, compared to the traditional Faster-RCNN which only uses labeled samples for training, this invention improves the recall rate of anomaly detection by 6.7% and achieves an overall accuracy of over 89%, significantly improving detection precision and reducing manpower consumption. Compared to the semi-supervised pseudo-labeled target detection method proposed by Li et al., the anomaly detection precision is also improved by at least 5.2%.

[0120] like Figure 4As shown, in the publicly available Northeastern University strip steel surface defect detection, 20% of the data was randomly selected as the test set. For the remaining training set, the number of samples for six types of defects (cracking, inclusions, patches, pitting corrosion, rolling oxide scale, and scratches) was exponentially reduced from 240 to 24 (240, 150, 96, 60, 38, 24), and the label ratio was set to 50%. The proposed class-imbalanced semi-supervised object detection algorithm achieved the same performance as Faster-RCNN using 100% labels, which fully demonstrates the effectiveness of the proposed method for semi-supervised object detection. More importantly, due to the reduced label ratio, the proposed method only requires half the manual annotation effort to achieve the detection effect of full annotation. Figure 4 In the table, the first row shows the ground truth labels for the training set, the second row shows the detection results of Faster-RCNN using 100% labels, and the third row shows the detection results of this invention. Different colors represent different surface defects. It can be seen that, under the condition of using only 50% labels, the detection performance of the method proposed in this invention is very similar to that of the fully labeled Faster-RCNN method. To highlight the detection performance of the algorithm of this invention for small classes under class imbalance conditions, AP_50 is used as the metric. Figure 3 Based on the demonstrated results, the algorithm of this invention has high detection accuracy.

[0121] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for industrial image defect detection based on a multi-head unbalanced semi-supervised network, characterized in that, Includes the following steps: S1. Create the model and initialize the model parameters; S2. Construct a dataset of industrial images; S3. Create the optimization target and optimizer; S4. Process the data from S2 using model S1; S5. Determine whether the current model has been improved based on whether the data processed in S4 meets the requirements; S6. Based on the judgment result of S5, if there is an improvement, the process ends; otherwise, repeat steps S4 and S5 until the previous model has improved or the maximum number of repetitions of step S4 is reached. The step S1 of creating the model and initializing the model parameters includes supervised branch head model, semi-supervised branch head model and unsupervised branch head model; The data in step S2 includes labeled data and unlabeled data; Step S4 includes the following steps: A1. Randomly select data from S2; A2. Preprocess the data and then iterate it into the model to calculate the loss; A3. Update the model parameters using the optimizer in S3.

2. The industrial image defect detection method based on a multi-head unbalanced semi-supervised network according to claim 1, characterized in that: Step S6 includes the following steps: B1. Determine whether the model parameters in step A3 are improved compared to the current model. If so, after updating and saving the model parameters, determine whether the maximum number of repetitions in step S4 has been reached. Otherwise, directly determine whether the maximum number of repetitions in step S4 has been reached. B2. Based on the judgment in step B1, determine whether the maximum number of repetitions in step S4 has been reached. If yes, end the process; otherwise, repeat steps 4 and 5.

Images