Supervised and unsupervised defect detection methods, model building methods and devices

Abstract

This invention provides a method, model building method, and apparatus for supervised and unsupervised defect detection. The model building method includes: establishing an initial defect detection model, including a shared feature extraction module based on the YOLO backbone network, a supervised learning branch based on the YOLO architecture, and an unsupervised learning branch based on an anomaly detection architecture; freezing the unsupervised learning branch, and training the shared feature extraction module and the supervised learning branch based on labeled samples; freezing the shared feature extraction module and the supervised learning branch after training, and unfreezing the unsupervised learning branch, and training the unsupervised learning branch based on positive samples; unfreezing the supervised learning branch, and jointly training the shared feature extraction module, the supervised learning branch, and the unsupervised learning branch using labeled samples and positive samples, and obtaining a defect detection model after training. This method can improve the accuracy of defect detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a supervised and unsupervised defect detection method, a model establishment method and device. BACKGROUND

[0002] In the industrial production process, product quality detection is a key link, which directly affects production efficiency and product quality. In recent years, defect detection methods based on deep learning have been widely used in industrial quality inspection tasks, greatly improving detection efficiency. The current mainstream deep learning defect detection methods are divided into supervised and unsupervised forms.

[0003] Patent text CN117351288A proposes a PCB defect detection method based on an improved YOLOv5 algorithm, including the following steps: S1, collecting PCB defect image data to be detected to form a sample image data set; S2, preprocessing the sample image data set to obtain a first image data set; the first image data set includes a training set and a test set; S3, using an EfficientViT feature network to build a PCB defect detection network model; S4, using the training set to train the PCB defect detection network model to obtain a trained PCB defect detection network model; S5, using the trained PCB defect detection network model to test the test set to obtain the defect detection result of the PCB image. This method is based on a supervised learning method, and through a certain scale of supervised feature learning, it can accurately identify known defects. However, in actual industrial scenarios, there are often problems such as difficulty in collecting defect samples and high randomness of defect types, and it is difficult to cover all defect types relying only on supervised methods, with a high risk of defect omission. To solve the above problems, researchers introduce unsupervised defect detection methods, such as patent text CN115761359A, which provides a photovoltaic image defect classification method based on transfer learning and unsupervised learning. This method performs transfer learning through a pre-trained model obtained on a general image data set to establish an initial defect classification model; then a large number of unlabeled images are mapped to a feature space through the initial model, and the corresponding labels are obtained according to the L2 distance in the feature space, so that the initial model is continuously retrained, thereby improving the accuracy of the photovoltaic image defect classification model. This method only requires unlabeled samples to complete training and has good generalization ability for unknown defects. However, most unsupervised methods have shortcomings in detection accuracy and inference speed, and have poor detection ability for slight defects. SUMMARY

[0004] The present application provides a supervised and unsupervised defect detection method, a model establishment method and device, which can effectively improve the accuracy of defect detection.

[0005] A supervised and unsupervised defect detection model establishment method, comprising:

[0006] An initial defect detection model is established, which comprises a shared feature extraction module based on a YOLO backbone network, a supervised learning branch based on a YOLO architecture, and an unsupervised learning branch based on an anomaly detection architecture, the shared feature extraction module being connected to the supervised learning branch and the unsupervised learning branch;

[0007] The unsupervised learning branch is frozen, and the shared feature extraction module and the supervised learning branch are trained based on labeled samples;

[0008] After the training of the shared feature extraction module and the supervised learning branch is completed, the shared feature extraction module and the supervised learning branch are frozen, and the unsupervised learning branch is unfrozen, and the unsupervised learning branch is trained based on positive samples;

[0009] The supervised learning branch is unfrozen, and the shared feature extraction module, the supervised learning branch, and the unsupervised learning branch are jointly trained using labeled samples and positive samples, and a defect detection model is obtained after training.

[0010] Further, the shared feature extraction module is a backbone network of a YOLO network, the supervised learning branch comprises a neck network module and a detection head module of the YOLO network, and the unsupervised learning branch comprises a feature compression embedding module, a feature library construction module, a search module, and an output module.

[0011] Further, training the shared feature extraction module and the supervised learning branch based on labeled samples comprises:

[0012] A classification loss function and a bounding box regression loss function are established;

[0013] The following steps are repeatedly executed until a stop condition is met:

[0014] The labeled samples are input into the shared feature extraction module to extract first multi-scale feature maps;

[0015] The first multi-scale feature maps are input into the neck network module for feature fusion, and multi-scale fused features are output;

[0016] The detection head module outputs detection results based on the multi-scale fused features, calculates a classification loss and a bounding box regression loss based on the detection results, and updates the parameters of the shared feature extraction module and the supervised learning branch according to the classification loss and the bounding box regression loss.

[0017] Further, a shape-aware midline point distance and a width-height penalty term are introduced into the bounding box regression loss function; the detection result comprises a prediction box;

[0018] The bounding box regression loss function is 1 minus the intersection over union of the prediction box and the labeled true box, plus the shape-aware midline point distance and the width-height penalty term under the corresponding coefficient.

[0019] Further, the unsupervised learning branch is trained based on positive samples, comprising:

[0020] The total loss function of the unsupervised learning branch is constructed, and the total loss function of the unsupervised learning branch is a weighted sum of alignment loss and consistency loss;

[0021] The feature library construction module establishes an initialized feature library;

[0022] The following steps are repeatedly executed until the stopping condition is met:

[0023] The positive sample is input into the shared feature extraction module to extract a second multi-scale feature map and input into the unsupervised learning branch, the feature compression embedding module of the unsupervised learning branch is used to reduce the dimension of the second multi-scale feature map, extract local features, the search module calculates the Euclidean distance between different positions in the local features and the corresponding features in the feature library and calculates the distance mean, and the anomaly detection heat map is obtained by upsampling according to the distance mean, and output by the output module;

[0024] The alignment loss and the consistency loss are calculated according to the Euclidean distance, the total loss of the unsupervised learning branch is calculated, the parameters of the unsupervised learning branch are updated according to the total loss of the unsupervised learning branch;

[0025] When the preset number of rounds is reached, the feature library construction module updates the feature library.

[0026] Further, the shared feature extraction module, the supervised learning branch and the unsupervised learning branch are jointly trained using labeled samples and positive samples, comprising:

[0027] The following steps are repeatedly executed until the stopping condition is met:

[0028] The labeled sample and the positive sample are input into the shared feature extraction module for feature extraction to obtain a multi-scale feature map;

[0029] The multi-scale feature map obtained by the labeled sample is input into the supervised learning branch for training, and the total loss of the supervised learning branch is calculated;

[0030] Input the multi-scale feature maps obtained through the positive samples into the unsupervised learning branch for training, and calculate the total loss of the unsupervised learning branch;

[0031] Determine the channel selection mask of the output channel of the shared feature extraction module according to the total loss of the supervised learning branch and the total loss of the unsupervised learning branch;

[0032] Calculate the comprehensive loss according to the total loss of the supervised learning branch and the total loss of the unsupervised learning branch, and update the parameters of the shared feature extraction module, the supervised learning branch and the unsupervised learning branch according to the comprehensive loss.

[0033] Further, determining the channel selection mask of the output channel of the shared feature extraction module according to the total loss of the supervised learning branch and the total loss of the unsupervised learning branch comprises:

[0034] Calculate the first gradient influence score of the total loss of the supervised learning branch on each output channel of the shared feature extraction module;

[0035] Calculate the second gradient influence score of the total loss of the unsupervised learning branch on each output channel of the shared feature extraction module;

[0036] According to the first gradient influence score and the second gradient influence score, calculate the final influence score of each output channel, and determine the channel selection mask of the output channel according to the final influence score.

[0037] A supervised and unsupervised defect detection method based on the defect detection model obtained by the above method, the defect detection method comprising:

[0038] Input the to-be-detected data into the defect detection model to obtain the prediction box output by the supervised learning branch and the abnormal multi-scale heat map output by the unsupervised learning branch, and the prediction box is located in the abnormal multi-scale heat map;

[0039] Traverse the prediction box, calculate the heat mean value in each prediction box, and calculate the target defect confidence according to the calculated prediction box heat mean value;

[0040] The prediction box with a target defect confidence higher than a preset confidence threshold is taken as a first defect detection result.

[0041] Further, after obtaining the defect detection result, further comprising:

[0042] Remove the prediction box in the abnormal multi-scale heat map, calculate the heat mean value of each region in the remaining abnormal multi-scale heat map after removing the prediction box, and take the region with a heat mean value greater than a preset threshold as a second defect detection result;

[0043] Integrate and output the first defect detection result and the second defect detection result. A defect detection model establishment device based on supervised and unsupervised learning comprises:

[0044] An initial model establishment module is configured to establish an initial defect detection model, wherein the initial defect detection model comprises a shared feature extraction module based on a YOLO backbone network, a supervised learning branch based on a YOLO architecture, and an unsupervised learning branch based on an anomaly detection architecture, and the shared feature extraction module is connected to the supervised learning branch and the unsupervised learning branch.

[0045] A supervised training module is configured to freeze the unsupervised learning branch, and train the shared feature extraction module and the supervised learning branch based on labeled samples.

[0046] An unsupervised training module is configured to freeze the shared feature extraction module and the supervised learning branch after the training of the shared feature extraction module and the supervised learning branch is completed, and unfreeze the unsupervised learning branch, and train the unsupervised learning branch based on positive samples.

[0047] A comprehensive training module is configured to unfreeze the supervised learning branch, and jointly train the shared feature extraction module, the supervised learning branch, and the unsupervised learning branch based on labeled samples and positive samples, and obtain a defect detection model after the training is completed.

[0048] Further, the shared feature extraction module is a backbone network of a YOLO network, the supervised learning branch comprises a neck network module and a detection head module of the YOLO network, and the unsupervised learning branch comprises a feature compression embedding module, a feature library construction module, a search module, and an output module.

[0049] Further, the supervised training module trains the shared feature extraction module and the supervised learning branch based on labeled samples, comprising:

[0050] Establish a classification loss function and a bounding box regression loss function.

[0051] Repeat the following steps until the stop condition is met:

[0052] Input the labeled samples into the shared feature extraction module to extract first multi-scale feature maps.

[0053] Input the first multi-scale feature maps into the neck network module for feature fusion to output multi-scale fused features.

[0054] The detection head module outputs a detection result based on the multi-scale fusion feature, calculates a classification loss and a bounding box regression loss based on the detection result, and updates parameters of the shared feature extraction module and the supervised learning branch according to the reverse propagation of the classification loss and the bounding box regression loss.

[0055] Further, a shape-aware midline point distance and a width-height penalty term are introduced into the bounding box regression loss function; the detection result includes a predicted bounding box;

[0056] The bounding box regression loss function is 1 minus the intersection over union of the predicted bounding box and the labeled true bounding box, plus the shape-aware midline point distance and the width-height penalty term under the corresponding coefficient.

[0057] Further, the unsupervised training module trains the unsupervised learning branch based on positive samples, including:

[0058] The total loss function of the unsupervised learning branch is constructed, and the total loss function of the unsupervised learning branch is the weighted sum of the alignment loss and the consistency loss;

[0059] The feature library construction module establishes an initialized feature library;

[0060] Repeat the following steps until the stopping condition is met:

[0061] The positive sample is input into the shared feature extraction module, the second multi-scale feature map is extracted and input into the unsupervised learning branch, the feature compression embedding module of the unsupervised learning branch performs dimension reduction processing on the second multi-scale feature map, extracts local features, the search module calculates the Euclidean distance between the features at different positions in the local features and the corresponding features in the feature library and calculates the distance mean, and the anomaly detection heat map is obtained by upsampling according to the distance mean, and output by the output module;

[0062] According to the Euclidean distance, the alignment loss and the consistency loss are calculated, the total loss of the unsupervised learning branch is calculated, the parameters of the unsupervised learning branch are updated according to the reverse propagation of the total loss of the unsupervised learning branch;

[0063] When the preset number of rounds is reached, the feature library construction module updates the feature library.

[0064] Further, the comprehensive training module jointly trains the shared feature extraction module, the supervised learning branch and the unsupervised learning branch using labeled samples and positive samples, including:

[0065] Repeat the following steps until the stopping condition is met:

[0066] The labeled sample and the positive sample are input into the shared feature extraction module for feature extraction, and a multi-scale feature map is obtained;

[0067] The multi-scale feature map obtained by the labeled sample is input into the supervised learning branch for training, and the total loss of the supervised learning branch is calculated;

[0068] The multi-scale feature map obtained by the positive sample is input into the unsupervised learning branch for training, and the total loss of the unsupervised learning branch is calculated;

[0069] According to the total loss of the supervised learning branch and the total loss of the unsupervised learning branch, the channel selection mask of the output channel of the shared feature extraction module is determined;

[0070] According to the total loss of the supervised learning branch and the total loss of the unsupervised learning branch, a comprehensive loss is calculated, and the parameters of the shared feature extraction module, the supervised learning branch and the unsupervised learning branch are updated according to the comprehensive loss.

[0071] Further, the comprehensive training module determines the channel selection mask of the output channel of the shared feature extraction module according to the total loss of the supervised learning branch and the total loss of the unsupervised learning branch, comprising:

[0072] Calculate the first gradient influence score of the total loss of the supervised learning branch on each output channel of the shared feature extraction module;

[0073] Calculate the second gradient influence score of the total loss of the unsupervised learning branch on each output channel of the shared feature extraction module;

[0074] According to the first gradient influence score and the second gradient influence score, the final influence score of each output channel is calculated, and the channel selection mask of the output channel is determined according to the final influence score.

[0075] A supervised and unsupervised defect detection method based on the defect detection model obtained by the above method, the defect detection method comprising:

[0076] The data to be detected is input into the defect detection model, and the prediction box output by the supervised learning branch and the abnormal multi-scale heat map output by the unsupervised learning branch are obtained, and the prediction box is located in the abnormal multi-scale heat map;

[0077] Traverse the prediction box, calculate the heat mean value in each prediction box, and calculate the target defect confidence according to the prediction box heat mean value obtained by calculation;

[0078] The prediction box with a target defect confidence higher than the pre-set confidence threshold is taken as the first defect detection result.

[0079] Further, after obtaining the defect detection result, further comprising:

[0080] Removing the prediction box in the abnormal multi-scale heat map, calculating the heat mean value of each region in the remaining abnormal multi-scale heat map after removing the prediction box, and taking the region with a heat mean value greater than a preset threshold as a second defect detection result;

[0081] Integrating and outputting the first defect detection result and the second defect detection result. A supervised and unsupervised defect detection device is applied to the above detection method, and the device comprises:

[0082] A prediction module is configured to input the to-be-detected data into the defect detection model to obtain a prediction box output by a supervised learning branch and an abnormal multi-scale heat map output by an unsupervised learning branch, and the prediction box is located in the abnormal multi-scale heat map;

[0083] A confidence calculation module is configured to traverse the prediction boxes, calculate a heat mean value in each prediction box, and calculate a target defect confidence according to the calculated prediction box heat mean value;

[0084] A judgment module is configured to take a prediction box with a target defect confidence higher than a preset confidence threshold as a first defect detection result.

[0085] Further, the judgment module is further configured to:

[0086] Remove the prediction box in the abnormal multi-scale heat map, calculate the heat mean value of each region in the remaining abnormal multi-scale heat map after removing the prediction box, and take the region with a heat mean value greater than a preset threshold as a second defect detection result;

[0087] Integrate and output the first defect detection result and the second defect detection result. The supervised and unsupervised defect detection method, model establishment method and device provided by the present application at least have the following beneficial effects:

[0088] (1) By constructing a supervised learning branch based on the YOLO architecture and an unsupervised learning branch based on the anomaly detection architecture, the supervised and unsupervised tasks in the industrial defect detection are cooperatively perceived. The shared feature extraction module based on the YOLO backbone network is used to realize feature extraction, and the supervised learning branch and the unsupervised learning branch are used to realize positioning detection and abnormality discrimination, which significantly improves the consistency and generalization ability of the industrial defect detection system, and improves the precision and accuracy of the defect detection;

[0089] (2) The model uses dynamic channel selection to automatically select shared channels during the training process, constructs a locally decoupled structure of supervised learning branches and unsupervised learning branches, avoids interference between the detection task of the supervised learning branch and the anomaly task of the unsupervised learning branch in the feature space, and balances the detection accuracy of the supervised learning branch and the anomaly detection ability of the unsupervised learning branch. A three-stage hierarchical strategy is adopted in the training stage, and the step-by-step training effectively prevents task interference and gradient competition, thereby improving the stability and robustness of the model;

[0090] (3) The unsupervised learning branch adopts channel dimension reduction processing, which significantly reduces the feature library size without losing accuracy, and significantly reduces the feature library storage and inference delay. The loss part uses a joint optimization loss function to strengthen the normal sample feature aggregation, thereby improving the quality of the feature storage library and effectively reducing the false detection caused by local interference.

[0091] (4) In the defect detection, a unified defect detection result is obtained based on the fusion strategy of the heat map and the prediction box, the modeling ability of the model for different anomalies in scale and morphology is strengthened, and the recognition rate of small defects and defects with inconspicuous features is effectively improved.

[0092] (5) The modular design provides a flexible deployment method, and a single model can meet the detection needs of pure supervised, pure unsupervised and combined supervised and unsupervised, and can configure feature scales, channel numbers and dimension reduction ratios as needed to adapt to high-speed industrial detection needs under different resource conditions. BRIEF DESCRIPTION OF DRAWINGS

[0093] Figure 1 A flowchart of an embodiment of the method for establishing a supervised and unsupervised defect detection model provided by the present application is provided.

[0094] Figure 2 A structural schematic diagram of an embodiment of the defect detection model in the method for establishing a supervised and unsupervised defect detection model provided by the present application is provided.

[0095] Figure 3 A flowchart of an embodiment of the method for establishing a supervised and unsupervised defect detection model provided by the present application is provided.

[0096] Figure 4 A structural schematic diagram of an embodiment of the device for establishing a supervised and unsupervised defect detection model provided by the present application is provided.

[0097] Figure 5 A structural schematic diagram of an embodiment of the device for establishing a supervised and unsupervised defect detection model provided by the present application is provided. DETAILED DESCRIPTION

[0098] For better understanding of the above technical solutions, the above technical solutions will be described in detail below in combination with the drawings of the specification and specific embodiments.

[0099] Reference Figure 1 In some embodiments, a supervised and unsupervised defect detection model establishment method is provided, comprising:

[0100] S1, an initial defect detection model is established, which includes a shared feature extraction module based on a YOLO backbone network, a supervised learning branch based on a YOLO architecture, and an unsupervised learning branch based on an anomaly detection architecture, the shared feature extraction module is connected with the supervised learning branch and the unsupervised learning branch;

[0101] S2, freeze the unsupervised learning branch, train the shared feature extraction module and the supervised learning branch based on the labeled samples;

[0102] S3, after the training of the shared feature extraction module and the supervised learning branch is completed, freeze the shared feature extraction module and the supervised learning branch, and unfreeze the unsupervised learning branch, train the unsupervised learning branch based on the positive samples;

[0103] S4, unfreeze the supervised learning branch, and jointly train the shared feature extraction module, the supervised learning branch and the unsupervised learning branch using labeled samples and positive samples, and obtain a defect detection model after training.

[0104] Specifically, referring to Figure 2 , in step S1, the shared feature extraction module is the backbone network 1 of the YOLO network, the supervised learning branch includes the neck network module 2 and the detection head module 3 of the YOLO network, and the unsupervised learning branch adopts a feature embedding model, including a feature compression embedding module 4, a feature library construction module 5, a search module 6 and an output module 7.

[0105] Further, in step S2, the unsupervised learning branch is frozen, specifically the parameters of the unsupervised learning branch are frozen. Train the shared feature extraction module and the supervised learning branch based on the labeled samples, including:

[0106] S21, establish a classification loss function and a bounding box regression loss function;

[0107] S22, repeat the following steps until the stop condition is met:

[0108] input the labeled samples into the shared feature extraction module to extract a first multi-scale feature map;

[0109] The first multi-scale feature map is input into the neck network module for feature fusion, and a multi-scale fusion feature is output.

[0110] The detection head module outputs a detection result based on the multi-scale fusion feature, calculates a classification loss and a bounding box loss based on the detection result, and updates parameters of the shared feature extraction module and the supervised learning branch according to the reverse propagation of the classification loss and the bounding box loss.

[0111] In step S21, a shape-aware midline point distance and a width-height penalty term are introduced into the bounding box regression loss function; and the detection result includes a predicted bounding box.

[0112] The bounding box regression loss function is 1 minus an intersection over union of the predicted bounding box and a labeled real bounding box, plus the shape-aware midline point distance and a width-height penalty term under a corresponding coefficient.

[0113] Specifically, the loss of the supervised branch includes a classification loss and a bounding box regression loss. The classification loss calculates a binary classification loss for each class using sigmoid and BEC Loss, which is more stable and more parallelized in image edges, occlusion and blur scenes. The bounding box regression loss uses an improved bounding box regression loss to replace the original CIoU loss, which introduces the change of defect shape or local boundary difference on the basis of the intersection over union, and can more truly reflect the consistency of the predicted and real defect regions. The definition of the bounding box regression loss is as follows:

[0114] (1)

[0115] wherein L shape represents the bounding box regression loss, IoU represents the intersection over union of the predicted bounding box and the real bounding box, distance shape represents the shape-aware center point distance, v shape represents the width-height penalty term, and the specific definitions of the parts are as follows:

[0116] (2)

[0117] wherein B pre represents the predicted bounding box, B gt represents the real bounding box.

[0118] (3)

[0119] wherein c represents the length of the diagonal of the minimum bounding rectangle of the predicted bounding box and the real bounding box, scale is an adjustable parameter and is set to 1 by default, h gt represents the height of the real label, w gt represents the width of the real label, x c represents the horizontal coordinate of the center point of the predicted bounding box, and yc This represents the ordinate of the center point of the prediction box. This represents the x-coordinate of the center point of the actual label. The vertical coordinate of the center point of the actual label.

[0120] (4)

[0121] in, This indicates an excessive penalty to suppress minor differences. θ defaults to 4 and is used to amplify the effects of significant shape deviations. w represents the width of the labeled sample, h represents the height of the labeled sample, and ω represents the size difference weight.

[0122] In some embodiments, the total loss function of the supervised learning branch is the sum of the classification loss and the bounding box regression loss.

[0123] Further, in step S22, the labeled samples are input into the shared feature extraction module to extract a first multi-scale feature map, which includes a small-scale feature map, a medium-scale feature map, and a large-scale feature map.

[0124] The first multi-scale feature map is input into the neck network module for feature fusion, and the multi-scale fused features are output. The detection head module uses 3x3 and 1x1 convolutions to perform feature aggregation and dimensionality reduction based on the multi-scale fused features, and outputs the detection results.

[0125] Based on the detection results, calculate the classification loss and bounding box regression loss, and then perform backpropagation based on these losses to update the parameters of the shared feature extraction module and the supervised learning branch. Repeat the above steps until the stopping condition is met.

[0126] Furthermore, in step S3, after the shared feature extraction module and the supervised learning branch have been trained, the shared feature extraction module and the supervised learning branch are frozen. Specifically, the parameters of the shared feature extraction module and the supervised learning branch are frozen, while the shared feature extraction module still serves as the input module for positive samples, but its parameters are not updated.

[0127] Training the unsupervised learning branch based on positive samples includes:

[0128] S31. Construct the total loss function of the unsupervised learning branch, wherein the total loss function of the unsupervised learning branch is a weighted sum of the alignment loss and the consistency loss;

[0129] S32. The feature library construction module establishes an initialized feature library;

[0130] S33. Repeat the following steps until the stopping condition is met:

[0131] inputting the positive sample into the shared feature extraction module to extract a second multi-scale feature map;

[0132] inputting the second multi-scale feature map into the unsupervised learning branch, a feature compression embedding module of the unsupervised learning branch being configured to perform dimension reduction processing on the second multi-scale feature map to extract local features, the search module being configured to calculate the Euclidean distances between different positions of the local features and corresponding features in the feature library, select Top-K smallest Euclidean distances to calculate a distance mean value, perform up-sampling on the distance mean value to obtain an anomaly detection heat map, and output the anomaly detection heat map through the output module; wherein Top-K is a hyperparameter and is obtained through training;

[0133] calculating an alignment loss and a consistency loss according to the Euclidean distances, obtaining a total loss of the unsupervised learning branch, performing back propagation according to the total loss of the unsupervised learning branch, and updating parameters of the unsupervised learning branch;

[0134] The feature library construction module updates the feature library when a preset number of rounds is reached.

[0135] Specifically, in step S31, the alignment loss is the average value of the Euclidean distances between the local features and the closest features in the feature library; and the consistency loss is the average value of the Euclidean distances between the local features and the mean value of all features in the feature library.

[0136] The total loss of the unsupervised learning branch is designed as the weighted sum of the alignment loss (Align Loss) and the consistency loss (Compactness Loss), and is only applied to positive sample data. The role is to enable the model to learn the feature aggregation of normal samples under the condition of no abnormal annotation, so that the positive sample features are as aggregated as possible, and the field adaptation is completed to improve the accuracy of anomaly detection in the subsequent steps.

[0137] The alignment loss is calculated by calculating the Euclidean distance between the local features and the closest features in the feature library, and the Euclidean distance is used to supervise the normal image to constrain the model output to make the Euclidean distance as low as possible. The formula can be expressed as:

[0138] ; (5)

[0139] wherein L align represents the alignment loss, N represents the number of all local features, represents the current local feature, represents the closest feature searched in the feature library according to the current local feature.

[0140] Specifically, searching the nearest neighbor of the current local feature in the feature library comprises: constructing a FAISS index using the feature library, extracting and normalizing the local features, and performing nearest neighbor search on each local feature using the FAISS index to obtain the nearest neighbor of the local feature.

[0141] The consistency loss refers to calculating the Euclidean distance based on the current local feature and the mean of all features in the feature storage, and constraining the model by the calculated Euclidean distance to force the distance between the output of the model and the center of the feature library to be the smallest. Its definition can be described as:

[0142] ; (6)

[0143] wherein, L comp represents the consistency loss, N represents the number of all local features, represents the current local feature, M represents the number of features in the feature library, represents the jth feature in the feature library.

[0144] The total loss of the final unsupervised learning branch is the weighted fusion result of the alignment loss and the consistency loss, which can be represented as:

[0145] ; (7)

[0146] wherein, a and β represent weights, which are directly passed in by hyperparameters, L AD represents the total loss of the unsupervised learning branch.

[0147] Further, in step S32, the feature library construction module based on the second multi-scale feature map extracted when the first input positive sample is connected to the 1x1 convolution for channel dimension reduction and uniform feature dimension; After flattening processing in the width and height dimensions of the feature map, the feature vector is input into the full connection layer to obtain the initialized feature library based on the feature vector of all local features and sparse sampling.

[0148] Further, in step S4, the labeled sample and the positive sample are used to jointly train the shared feature extraction module, the supervised learning branch and the unsupervised learning branch, comprising:

[0149] Repeat the following steps until the stop condition is met:

[0150] S41, input the labeled sample and the positive sample into the shared feature extraction module for feature extraction to obtain a multi-scale feature map;

[0151] S42, input the multi-scale feature map obtained by the labeled sample into the supervised learning branch for training, and calculate the total loss of the supervised learning branch;

[0152] S43, input the multi-scale feature map obtained through the positive sample to the unsupervised learning branch for training, and calculate the total loss of the unsupervised learning branch;

[0153] S44, determine the channel selection mask of the output channel of the shared feature extraction module according to the total loss of the supervised learning branch and the total loss of the unsupervised learning branch;

[0154] S45, calculate the comprehensive loss according to the total loss of the supervised learning branch and the total loss of the unsupervised learning branch, and update the parameters of the shared feature extraction module, the supervised learning branch and the unsupervised learning branch according to the comprehensive loss.

[0155] Specifically, task joint training, fine-tuning shared features. Thaw the shared feature extraction module and the supervised learning branch, and use data containing labeled samples and positive samples for full-parameter training.

[0156] In step S44, the channel selection mask of the output channel of the shared feature extraction module is determined according to the total loss of the supervised learning branch and the total loss of the unsupervised learning branch, comprising:

[0157] Calculate the first gradient influence score of the total loss of the supervised learning branch on each output channel of the shared feature extraction module;

[0158] Calculate the second gradient influence score of the total loss of the unsupervised learning branch on each output channel of the shared feature extraction module;

[0159] According to the first gradient influence score and the second gradient influence score, the final influence score of each output channel is calculated, and the channel selection mask of the output channel is determined according to the final influence score.

[0160] Through gradient statistics of the supervised task training process, the response intensity of each output channel on each branch is dynamically evaluated on the small-scale feature map, the medium-scale feature map and the large-scale feature map in the multi-scale feature map, respectively, and the multi-task shared channel is selected using a mixed channel selection strategy.

[0161] Wherein, the first gradient influence score of the total loss of the supervised learning branch on each output channel of the shared feature extraction module is calculated, denoted as:

[0162] ; (8)

[0163] Wherein, The first gradient influence score of the shared feature extraction module in the supervised learning is denoted as Bc, B represents the batch dimension, H represents the height dimension, W represents the width dimension, L YOLOTotal loss of supervised learning branch, which is the sum of classification loss and bounding box regression loss, F1 is the first multi-scale feature map.

[0164] Calculate the second gradient influence score of the total loss of the unsupervised learning branch on each output channel of the shared feature extraction module, denoted as:

[0165] ; (9)

[0166] Wherein, The second gradient influence score of the cth channel of the shared feature extraction module in unsupervised learning is denoted as B represents the batch dimension, H represents the height dimension, W represents the width dimension, and L AD Total loss of unsupervised learning branch, F2 is the second multi-scale feature map.

[0167] Integrate the scores of the two task channels to obtain the final influence score:

[0168] ; (10)

[0169] Wherein, Norm represents normalization to [0, 1], and γ is an adjustable weight coefficient, The final influence score of the cth channel is denoted as

[0170] According to the final influence score, a channel selection mask of the output channel is constructed.

[0171] When the input data is a labeled sample, it can be determined to be a labeled sample by identifying the label, and the supervised loss is calculated by the loss function of the supervised branch. When the input data is a positive sample, the unsupervised loss is calculated by the loss function of the unsupervised branch, and finally the comprehensive loss is calculated according to the supervised loss and the unsupervised loss and back propagation. For the unsupervised learning branch, the channel selection mask of the output channel and the feature library (hyperparameter setting) are dynamically updated in the training phase, and the channel selection mask of the output channel and the final feature storage library are saved for inference at the end of training.

[0172] Wherein, the comprehensive loss is the weighted sum of the supervised loss and the unsupervised loss.

[0173] Reference Figure 3 In some embodiments, a supervised and unsupervised defect detection method based on the defect detection model obtained by the above method is also provided, which comprises:

[0174] S101, input the data to be detected into the defect detection model, and obtain the prediction box output by the supervised learning branch and the abnormal multi-scale heat map output by the unsupervised learning branch, the prediction box is located in the abnormal multi-scale heat map;

[0175] S102, traversing the prediction box, calculating the thermal mean value in each prediction box, and calculating the target defect confidence according to the calculated prediction box thermal mean value;

[0176] S103, taking the prediction box with a target defect confidence higher than a preset confidence threshold as a first defect detection result.

[0177] Specifically, in step S101, the to-be-detected data is input into the defect detection model, the supervised learning branch analyzes it, and outputs the prediction box. The unsupervised learning branch analyzes the to-be-detected data and outputs the abnormal multi-scale thermal map. The size of the abnormal multi-scale thermal map is the same as that of the original to-be-detected data. Figure 1 Therefore, the prediction box is combined with the abnormal multi-scale thermal map to obtain an abnormal multi-scale thermal map with a prediction box.

[0178] Further, in step S102, the thermal mean value is calculated in each prediction box, and the target defect confidence is calculated according to the calculated prediction box thermal mean value.

[0179] ; (11)

[0180] wherein, is the original confidence of the supervised learning branch, is the thermal mean value of the prediction box, is a fusion weight, which is 0.8 by default, and s is the target defect confidence.

[0181] In step S103, the prediction box with a target defect confidence higher than a preset confidence threshold is taken as a defect detection result.

[0182] In some embodiments, after obtaining the first defect detection result, the method further includes:

[0183] Removing the prediction box in the abnormal multi-scale thermal map, calculating the thermal mean value of each region in the remaining abnormal multi-scale thermal map after removing the prediction box, and taking the region with a thermal mean value greater than a preset threshold as a second defect detection result.

[0184] The first defect detection result and the second defect detection result are integrated and output. Reference Figure 4 In some embodiments, a defect detection model establishment device based on supervised and unsupervised learning is provided, which includes:

[0185] The initial model establishing module 201 is configured to establish an initial defect detection model, which comprises a shared feature extraction module based on a YOLO backbone network, a supervised learning branch based on a YOLO architecture, and an unsupervised learning branch based on an anomaly detection architecture, and the shared feature extraction module is connected with the supervised learning branch and the unsupervised learning branch;

[0186] The supervised training module 202 is configured to freeze the unsupervised learning branch, and train the shared feature extraction module and the supervised learning branch based on labeled samples;

[0187] The unsupervised training module 203 is configured to freeze the shared feature extraction module and the supervised learning branch after the training of the shared feature extraction module and the supervised learning branch is completed, and unfreeze the unsupervised learning branch, and train the unsupervised learning branch based on positive samples;

[0188] The comprehensive training module 204 is configured to unfreeze the supervised learning branch, and jointly train the shared feature extraction module, the supervised learning branch and the unsupervised learning branch based on labeled samples and positive samples, and obtain a defect detection model after the training is completed.

[0189] Further, the shared feature extraction module is a backbone network of a YOLO network, the supervised learning branch comprises a neck network module and a detection head module of the YOLO network, and the unsupervised learning branch comprises a feature compression embedding module, a feature library construction module, a search module and an output module.

[0190] Further, the supervised training module 202 trains the shared feature extraction module and the supervised learning branch based on labeled samples, comprising:

[0191] establishing a classification loss function and a bounding box regression loss function;

[0192] repeating the following steps until a stop condition is met:

[0193] inputting the labeled samples into the shared feature extraction module to extract first multi-scale feature maps;

[0194] inputting the first multi-scale feature maps into the neck network module for feature fusion to output multi-scale fused features;

[0195] the detection head module outputs detection results based on the multi-scale fused features, calculates a classification loss and a bounding box regression loss based on the detection results, and updates parameters of the shared feature extraction module and the supervised learning branch according to the classification loss and the bounding box regression loss.

[0196] Further, a shape-aware midline point distance and a width-height penalty term are introduced into the bounding box regression loss function; the detection result comprises a prediction box;

[0197] The bounding box regression loss function is 1 minus the intersection over union of the prediction box and the labeled true box, plus the shape-aware midline point distance and the width-height penalty term under the corresponding coefficient.

[0198] Further, the unsupervised training module 203 trains the unsupervised learning branch based on the positive sample, comprising:

[0199] The total loss function of the unsupervised learning branch is constructed, which is the weighted sum of the alignment loss and the consistency loss;

[0200] The feature library construction module establishes an initialized feature library;

[0201] Repeat the following steps until the stopping condition is met:

[0202] The positive sample is input into the shared feature extraction module to extract a second multi-scale feature map;

[0203] The second multi-scale feature map is input into the unsupervised learning branch, and the feature compression embedding module of the unsupervised learning branch is used to perform dimension reduction processing on the second multi-scale feature map to extract local features. The search module calculates the Euclidean distance between the features at different positions in the local features and the corresponding features in the feature library, selects the Top-K smallest Euclidean distances to calculate the distance mean, and obtains an anomaly detection heat map by upsampling according to the distance mean, and outputs through the output module; wherein Top-K is a hyperparameter;

[0204] The alignment loss and the consistency loss are calculated according to the Euclidean distance, the total loss of the unsupervised learning branch is calculated, the parameters of the unsupervised learning branch are updated according to the total loss of the unsupervised learning branch, and the parameters of the unsupervised learning branch are updated according to the total loss of the unsupervised learning branch.

[0205] When the preset number of rounds is reached, the feature library construction module updates the feature library.

[0206] Further, the comprehensive training module 204 jointly trains the shared feature extraction module, the supervised learning branch and the unsupervised learning branch using labeled samples and positive samples, comprising:

[0207] Repeat the following steps until the stopping condition is met:

[0208] The labeled sample and the positive sample are input into the shared feature extraction module for feature extraction to obtain a multi-scale feature map;

[0209] The multi-scale feature maps obtained through the labeled samples are input into the supervised learning branch for training, and the total loss of the supervised learning branch is calculated;

[0210] The multi-scale feature maps obtained through the positive samples are input into the unsupervised learning branch for training, and the total loss of the unsupervised learning branch is calculated;

[0211] The channel selection mask of the output channel of the shared feature extraction module is determined according to the total loss of the supervised learning branch and the total loss of the unsupervised learning branch;

[0212] The comprehensive loss is calculated according to the total loss of the supervised learning branch and the total loss of the unsupervised learning branch, and the parameters of the shared feature extraction module, the supervised learning branch and the unsupervised learning branch are updated according to the comprehensive loss.

[0213] Further, the comprehensive training module 204 determines the channel selection mask of the output channel of the shared feature extraction module according to the total loss of the supervised learning branch and the total loss of the unsupervised learning branch, comprising:

[0214] The first gradient influence score of the total loss of the supervised learning branch on each output channel of the shared feature extraction module is calculated;

[0215] The second gradient influence score of the total loss of the unsupervised learning branch on each output channel of the shared feature extraction module is calculated;

[0216] The final influence score of each output channel is calculated according to the first gradient influence score and the second gradient influence score, and the channel selection mask of the output channel is determined according to the final influence score.

[0217] Reference Figure 5 In some embodiments, a supervised and unsupervised defect detection device based on the above-mentioned detection method is also provided, and the device comprises:

[0218] The prediction module 301 is configured to input the to-be-detected data into the defect detection model to obtain a prediction box output by the supervised learning branch and an abnormal multi-scale heat map output by the unsupervised learning branch, and the prediction box is located in the abnormal multi-scale heat map;

[0219] The confidence calculation module 302 is configured to traverse the prediction boxes, calculate the heat mean value in each prediction box, and calculate the target defect confidence according to the calculated prediction box heat mean value;

[0220] The judgment module 303 is configured to take the prediction box with a target defect confidence higher than a preset confidence threshold as a first defect detection result.

[0221] Further, the judging module 303 is further configured to:

[0222] remove the prediction box in the abnormal multi-scale heat map, calculate the heat mean of each region in the remaining abnormal multi-scale heat map after removing the prediction box, and take the region with the heat mean greater than a preset threshold as a second defect detection result;

[0223] integrate the first defect detection result and the second defect detection result and output.

[0224] The supervised and unsupervised defect detection method, model establishment method and device provided by the above embodiments at least have the following beneficial effects:

[0225] (1) By constructing a supervised learning branch based on the YOLO architecture and an unsupervised learning branch based on the anomaly detection architecture, the supervised and unsupervised tasks in industrial defect detection are realized. The shared feature extraction module based on the YOLO backbone network is used to realize feature extraction, and the supervised learning branch and the unsupervised learning branch are used to realize positioning detection and anomaly discrimination, which significantly improves the consistency and generalization ability of the industrial defect detection system, and improves the precision and accuracy of defect detection;

[0226] (2) The model uses dynamic channel selection to automatically select shared channels during training, and constructs a local decoupling structure of the supervised learning branch and the unsupervised learning branch, avoiding the interference between the detection task of the supervised learning branch and the anomaly task of the unsupervised learning branch in the feature space, and realizing the balance between the detection accuracy of the supervised learning branch and the anomaly detection ability of the unsupervised learning branch. In the training stage, a three-stage hierarchical strategy is adopted, and the step-by-step training effectively prevents task interference and gradient competition, improving the stability and robustness of the model;

[0227] (3) The unsupervised learning branch uses channel dimension reduction processing, which significantly reduces the feature library size without losing accuracy, and significantly reduces the feature library storage and inference delay. The loss part uses a joint optimization loss function to strengthen the normal sample feature aggregation, improving the quality of the feature storage library and effectively reducing the false detection caused by local interference.

[0228] (4) The fusion strategy based on heat map and prediction box in defect detection obtains unified defect detection results, strengthens the modeling ability of the model for different scale and morphology anomalies, and effectively improves the recognition rate of small defects and defects with unclear features.

[0229] (5) The modular design provides a flexible deployment method, and a single model can meet the detection needs of pure supervised, pure unsupervised and supervised unsupervised combination. The feature scale, channel number and dimension reduction ratio can be configured as needed to adapt to the high-speed industrial detection needs under different resource conditions.

[0230] While preferred embodiments of the application have been described, those skilled in the art will appreciate that other modifications and variations to the preferred embodiments are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the application, modifications and variations of the preferred embodiments can be practiced by those of ordinary skill. Similarly, any one or more features of one embodiment can be combined with any one or more features of another embodiment, or with any one or more features of the application. Also, it should be understood that the term "application" as used herein covers all modifications and variations of the preferred embodiments and includes all the features and combinations thereof.

Claims

1. A method for establishing a supervised and unsupervised defect detection model based on, characterized by, The method comprises the following steps: An initial defect detection model is established, which comprises a shared feature extraction module based on a YOLO backbone network, a supervised learning branch based on a YOLO architecture, and an unsupervised learning branch based on an anomaly detection architecture, the shared feature extraction module being connected with the supervised learning branch and the unsupervised learning branch; the shared feature extraction module is a backbone network of a YOLO network, the supervised learning branch comprises a neck network module and a detection head module of the YOLO network, and the unsupervised learning branch comprises a feature compression embedding module, a feature library construction module, a search module, and an output module; The unsupervised learning branch is frozen, and the shared feature extraction module and the supervised learning branch are trained based on labeled samples; After the training of the shared feature extraction module and the supervised learning branch is completed, the shared feature extraction module and the supervised learning branch are frozen, and the unsupervised learning branch is unfrozen, and the unsupervised learning branch is trained based on positive samples: a total loss function of the unsupervised learning branch is constructed, the total loss function of the unsupervised learning branch being a weighted sum of alignment loss and consistency loss; the feature library construction module establishes an initialized feature library; the following steps are repeatedly executed until a stop condition is met: the positive samples are input into the shared feature extraction module, second multi-scale feature maps are extracted and input into the unsupervised learning branch, the feature compression embedding module of the unsupervised learning branch performs dimension reduction processing on the second multi-scale feature maps, local features are extracted, the search module calculates the Euclidean distances between features at different positions in the local features and corresponding features in the feature library and calculates a distance mean value, an anomaly detection heat map is obtained by upsampling according to the distance mean value, and output is performed through the output module; the alignment loss and the consistency loss are calculated according to the Euclidean distances, the total loss of the unsupervised learning branch is calculated, the parameters of the unsupervised learning branch are updated through back propagation according to the total loss of the unsupervised learning branch; when a preset number of rounds is reached, the feature library construction module updates the feature library; The shared feature extraction module and the supervised learning branch are unfrozen, and the shared feature extraction module, the supervised learning branch, and the unsupervised learning branch are jointly trained based on labeled samples and positive samples, and a defect detection model is obtained after the training is completed.

2. The method of claim 1, wherein, The shared feature extraction module and the supervised learning branch are trained based on labeled samples, which comprises the following steps: A classification loss function and a bounding box regression loss function are established; The following steps are repeatedly executed until a stop condition is met: The labeled samples are input into the shared feature extraction module, and first multi-scale feature maps are extracted; The first multi-scale feature maps are input into the neck network module for feature fusion, and multi-scale fused features are output; The detection head module outputs a detection result based on the multi-scale fused features, calculates a classification loss and a bounding box regression loss based on the detection result, and updates the parameters of the shared feature extraction module and the supervised learning branch through back propagation according to the classification loss and the bounding box regression loss.

3. The method of claim 2, wherein, The shape-aware center point distance and the width-height penalty term are introduced into the bounding box regression loss function; and the detection result includes a prediction box. The bounding box regression loss function is 1 minus the intersection over union of the prediction box and a labeled true box, plus the shape-aware center point distance and the width-height penalty term under the corresponding coefficient.

4. The method of claim 1, wherein, The labeled samples and the positive samples are used to jointly train the shared feature extraction module, the supervised learning branch and the unsupervised learning branch, including: The following steps are repeatedly performed until a stop condition is met: The labeled samples and the positive samples are input into the shared feature extraction module for feature extraction to obtain multi-scale feature maps; The multi-scale feature maps obtained through the labeled samples are input into the supervised learning branch for training, and the total loss of the supervised learning branch is calculated; The multi-scale feature maps obtained through the positive samples are input into the unsupervised learning branch for training, and the total loss of the unsupervised learning branch is calculated; The channel selection mask of the output channel of the shared feature extraction module is determined according to the total loss of the supervised learning branch and the total loss of the unsupervised learning branch; The comprehensive loss is calculated according to the total loss of the supervised learning branch and the total loss of the unsupervised learning branch, and the parameters of the shared feature extraction module, the supervised learning branch and the unsupervised learning branch are updated according to the comprehensive loss.

5. The method of claim 4, wherein, The channel selection mask of the output channel of the shared feature extraction module is determined according to the total loss of the supervised learning branch and the total loss of the unsupervised learning branch, including: The first gradient influence score of the total loss of the supervised learning branch on each output channel of the shared feature extraction module is calculated; The second gradient influence score of the total loss of the unsupervised learning branch on each output channel of the shared feature extraction module is calculated; The final influence score of each output channel is calculated according to the first gradient influence score and the second gradient influence score, and the channel selection mask of the output channel is determined according to the final influence score.

6. A supervised and unsupervised based defect detection method, characterized in that, A defect detection model obtained by the method of any one of claims 1-5 is used for defect detection, and the defect detection method includes: The data to be detected is input into the defect detection model to obtain a prediction box output by the supervised learning branch and an abnormal multi-scale heat map output by the unsupervised learning branch, and the prediction box is located in the abnormal multi-scale heat map; The heat mean value in each prediction box is calculated by traversing the prediction box, and the target defect confidence is calculated according to the calculated heat mean value of the prediction box; The prediction box with a target defect confidence higher than a preset confidence threshold is taken as a first defect detection result.

7. The method of claim 6, wherein, After obtaining the first defect detection result, further including: The prediction box in the abnormal multi-scale heat map is removed, the heat mean value of each region in the remaining abnormal multi-scale heat map after removing the prediction box is calculated, and the region with a heat mean value greater than a preset threshold is taken as a second defect detection result; The first defect detection result and the second defect detection result are integrated and output.

8. A device for establishing a defect detection model based on supervised and unsupervised learning, applied to the method according to any one of claims 1 to 5, characterized in that, including: An initial model establishing module is configured to establish an initial defect detection model, which includes a shared feature extraction module based on a YOLO backbone network, a supervised learning branch based on a YOLO architecture, and an unsupervised learning branch based on an anomaly detection architecture, the shared feature extraction module being connected to the supervised learning branch and the unsupervised learning branch; A supervised training module is configured to freeze the unsupervised learning branch, train the shared feature extraction module and the supervised learning branch based on labeled samples; An unsupervised training module is configured to freeze the shared feature extraction module and the supervised learning branch after the training of the shared feature extraction module and the supervised learning branch is completed, and unfreeze the unsupervised learning branch, train the unsupervised learning branch based on positive samples; A comprehensive training module is configured to unfreeze the shared feature extraction module and the supervised learning branch, jointly train the shared feature extraction module, the supervised learning branch and the unsupervised learning branch based on labeled samples and positive samples, and obtain a defect detection model after the training is completed.

Images