Deep learning based few-shot fine-grained defect detection method

Abstract

The application discloses a few-shot fine-grained defect detection method based on deep learning, which comprises the following steps: generating sample slices according to image features; training a one-class classifier using normal samples, training the model using a loss function of deep one-class classification anomaly detection, and generating pseudo-labels for unlabeled samples; constructing a new data set using the obtained pseudo-label defect samples and real normal samples, inputting the data set into a feature mapping embedding encoder after image preprocessing, exploring the multi-scale features and internal relations of the fine-grained images, calculating the feature mapping of the images, and balancing the class distribution through a balanced cross-entropy function. The application expands the data scale through a slice generator, constructs a model of the overall texture information from the local features, introduces a feature mapping embedding encoder to model the multi-scale features, enhances the learning of the sample on the key feature information of the image, and thus obtains higher accuracy of the few-shot fine-grained defect detection.

Description

Technical Field

[0001] This invention belongs to the field of few-sample fine-grained defect detection, specifically relating to a few-sample fine-grained defect detection method based on deep learning. Background Technology

[0002] Fine-grained defect detection is a fundamental research topic in the field of pattern recognition. However, the high cost of acquiring datasets and manual annotation limits their size, posing a challenge to industrial defect detection tasks. Current defect detection research suffers from imbalanced data distribution, meaning the number of defect samples is typically far less than the number of normal samples, and the samples exhibit complex and diverse intra-class textures with small inter-class differences. Fine-grained image recognition primarily focuses on class recognition tasks that construct different fine-grained defect subclasses within a unified category. Few-shot learning, a popular direction in deep learning, aims to address the issues of limited sample numbers and high data annotation costs. Therefore, deep learning-based few-shot fine-grained defect detection methods can effectively handle the learning and detection of defect features across different dimensions in practical applications.

[0003] Current fine-grained defect detection methods have not fully utilized the characteristics of datasets and the fusion of multi-scale information. Although they have achieved good results in fine-grained defect detection with few samples, they still lack the ability to explore the intrinsic relationships between samples. The method in this patent fills this gap. Summary of the Invention

[0004] The purpose of this invention is to provide a few-sample fine-grained defect detection method based on deep learning, which reduces computational complexity while improving defect detection accuracy.

[0005] The technical solution for achieving the objective of this invention is as follows:

[0006] A deep learning-based method for fine-grained defect detection with few samples, which performs defect detection through a constructed defect detection network model, characterized in that the defect detection network model adopts a feature mapping embedded encoder network structure, and its training steps include:

[0007] Step 1: Preprocess the acquired images and generate sample slices based on the characteristics of the dataset using a slice generator.

[0008] Step 2: Introduce a pseudo-label mechanism. Input the sample slices into the feature map embedding encoder for training. During training, use the loss function of deep single-class classification anomaly detection to enhance the model's ability to represent samples and the accuracy of pseudo-label annotation. In this training stage, the feature map embedding encoder acts as a classifier to generate pseudo-label defect samples.

[0009] Step 3: Construct a new dataset from the obtained pseudo-label defect samples and real normal samples. Generate new sample slices through image preprocessing in Step 1. Input the new sample slices into the feature map embedding encoder. Considering that there may be a class imbalance problem between the number of obtained pseudo-label defect samples and real normal sample data, a balanced cross-entropy function is used for training to obtain a defect detection network model that meets the requirements.

[0010] Step 1, generating sample slices using the sample slice generator, specifically includes:

[0011] The sample is input into the slice generator, and slices are obtained by setting the parameter k, resulting in slice clusters. If n≥2, the generated slices will be flipped to align with the image texture.

[0012] Furthermore, the feature mapping embedding encoder network structure includes ResNet, convolutional layers, multi-scale feature stacks, and a Transformer encoder. ResNet extracts multi-scale features from sample slices, convolutional layers obtain multi-scale features at different resolutions, and the multi-scale feature stack flattens and embeds the multi-scale features at different resolutions, resulting in a feature vector that is then input into the Transformer encoder for prediction.

[0013] Furthermore, the method of obtaining multi-scale features at different resolutions through convolutional layers specifically includes:

[0014] Let b be the size of each training batch, then the total number of slices is bn. For N samples, there are a total of... For each batch of processing, let c be the number of channels in each image, and let H and W be the height and width of the image, then the slice size of each batch is (bn, c, H, W).

[0015] Extract multi-scale feature maps from the feature maps output from layers C3 to C5 of the network. The output image sizes of layers C3, C4, and C5 are respectively... and Using a 1×1 convolution with a stride of 1, obtain respectively and Feature maps C1, C2, and C3; and the output image of layer C3 is obtained by performing a 3×3 convolution with a stride of 2. Feature map C4.

[0016] Furthermore, the step of flattening and embedding multi-scale features at different resolutions using a multi-scale feature stack and then inputting the resulting feature vector into the Transformer encoder specifically includes:

[0017] Feature maps C1, C2, C3, and C4 are flattened using a multi-scale feature stack and embedded into a generated vector C = C j , C∈R 1×1×D For each image, j = 1, 2, ..., n, all the vectors obtained are: The size of the obtained processing batch is (b, n, D), where b is the number of images in a processing batch, n represents the number of slices in an image, and D is the dimension of the vector C after multi-scale feature extraction, which is defined as:

[0018]

[0019] For each batch of slices x j Construct location-encoded information, generate a token and prepend a token-like element to obtain a set of image patches of size (b, n+1, D) embedded with location information, denoted as [b, n+1, D]. Input it into the Transformer encoder.

[0020] Furthermore, the Transformer encoder obtains class token information P∈(b, 1, D) after computation, where each P corresponds to an x. i For i∈N, a fully connected network consisting of one input layer, one hidden layer, and one output layer is used to perform a multilayer perceptron classification for P. The dimensions of the input layer, hidden layer, and output layer are:

[0021]

[0022] The output layer is y c This refers to the output of the Cth neuron. This network is a binary classification network, C=2. The probability distribution of y is calculated as follows:

[0023]

[0024] Calculate the confidence score for each sample using the decision function:

[0025]

[0026] Will Convert the vector to a one-hot vector as the predicted label for the image, and denote the true label of the image as y0.

[0027] Furthermore, the loss function training for deep single-class anomaly detection involves using a support vector machine to map and constrain sample feature points to a hypersphere in high-dimensional space, and then solving for the minimum constraint volume of this hypersphere; specifically:

[0028] For a sample x∈χ, assume the input sample space is Training is performed using a loss function for classifying anomaly detection:

[0029]

[0030] in, This represents the feature mapping function of a neural network, and the corresponding weights. c is the center of the hypersphere, F is the output space, λ is the penalty factor, and λ≥0;

[0031] For a test sample point x'∈χ, the anomaly score can be defined as the distance from the sample point to the center of the hypersphere in the high-dimensional space of the output:

[0032]

[0033] in, Indicates the weights of the trained model;

[0034] Calculate the average distance from all real, normal samples to the center of the sphere:

[0035]

[0036] Where N is the number of real normal samples, and δ is the threshold of the decision boundary;

[0037] The relationship between δ and s(x') is used to determine whether test sample points can be classified as outliers:

[0038]

[0039] Pseudo-labeled defect samples are obtained by converting the judgment type values ​​of abnormal boundaries into one-hot vectors.

[0040] Furthermore, δ is optimized using the stochastic gradient descent algorithm.

[0041] Furthermore, the balanced cross-entropy function adopts the focus loss function, and a weighting factor is added to the loss function.

[0042] Furthermore, the focus loss function is:

[0043]

[0044] make but:

[0045]

[0046] Where γ is the focusing parameter, and γ≥0.

[0047] Compared with the prior art, the significant advantages of this invention are: (1) This method adopts the slicing method, which enhances the convergence of the model while expanding the sample; (2) The loss function of deep class-aspect anomaly detection is used to enhance the model's ability to represent samples and the accuracy of pseudo-label annotation; (3) For the feature mapping embedding encoder, feature information at different scales is used to capture global information and the dependence of the context in the image; at the same time, by modeling the feature information at different scales of the input, the fusion representation of the sample is obtained, which enriches the receptive field and enhances the understanding of the context; (4) The balanced cross-entropy function is used to train and solve the class imbalance problem that may exist between the number of pseudo-label defective samples and real normal sample data; (5) Compared with other pseudo-label mechanisms based on semi-supervised learning, the method of this patent has lower complexity, is easier to train and has higher recognition performance. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the few-sample fine-grained defect detection method based on deep learning according to the present invention.

[0049] Figure 2 This is a schematic diagram illustrating how to obtain multi-scale features at different resolutions using convolutional layers. Detailed Implementation

[0050] To address the task of learning and detecting defect features across different dimensions, this patent utilizes the CODED dataset from six different circular workpiece production lines, comprising training and testing sets. The training set includes labeled normal samples and unlabeled samples, while the testing set includes labeled normal samples and labeled defective samples. A carefully designed network for anomaly detection is used to learn from the training set, aiming to detect and identify unknown defects with a very limited number of normal samples. The model needs to learn how to accurately identify defects when the differences between defective and normal samples are small. To overcome the challenges of detecting defects with few samples and fine-grained defects, this paper proposes a semi-supervised learning-based Symmetry Slice Network method. This method leverages the characteristics of the dataset by slicing samples to enhance the network's learning ability. Simultaneously, feature maps extracted from images at different scales are embedded into an encoder to learn and detect defect types at different scales. Through training on the expanded slice samples, the model can obtain global features from the fusion of local features at different scales and uses pseudo-labeling techniques to improve the utilization of unlabeled data. By learning from a symmetric slice network in a semi-supervised paradigm, the few-shot problem and the fine-grained defect detection problem are addressed separately, improving the model's performance in detecting few-shot fine-grained defects in practical applications.

[0051] Combination Figure 1 A deep learning-based method for fine-grained defect detection with few samples includes the following steps:

[0052] Step 1: Preprocess the images and generate sample slices based on the characteristics of the dataset.

[0053] First, the sample is input into the slice generator, and slices are obtained by setting the parameter k, resulting in slice clusters. If n≥2, the generated slices will be flipped to align with the image texture.

[0054] Step 2, Slice clusters are input as mini-batches into the feature map embedding encoder. Let b be the size of each training batch, then the total number of slices is bn, and the total number of slices is N samples - the total number of slices is... There are 10 processing batches. Let c be the number of channels for each image, and H and W be the height and width of the image, respectively. Then the slice size of each batch is (bn, c, H, W).

[0055] Residual network ResNet and Transformer encoder are used as the backbone network for feature extraction, combined with Figure 2Extract multi-scale feature maps from the feature maps output from layers C3 to C5 of the network. The output image sizes of layers c3, c4, and c5 are respectively... and Using a 1×1 convolution with a stride of 1, we obtain... and Feature maps C1, C2, and C3 are obtained. Furthermore, a 3×3 convolution with a stride of 2 is applied to the output image of layer C3 to obtain... Feature map C4.

[0056] C1, C2, C3, and C4 are flattened using a multi-scale feature stack and embedded to generate the vector C = C j , C∈R 1 ×1×D For each image, j = 1, 2, ..., n, all the vectors obtained are: The size of the obtained processing batch is (b, n, D), where b is the number of images in a processing batch, n represents the number of slices in an image, and D is the dimension of the vector C after multi-scale feature extraction, which is defined as:

[0057]

[0058] For each batch of slices x j Construct location encoding information, generate a token, and prepend it with a class token. This yields a set of image patches of size (b, n+1, D) embedded with location information, denoted as... This is input into the Transformer encoder. The output is token information P∈(b, 1, D), where each P corresponds to a single x. i (i∈N). A fully connected network consisting of one input layer, one hidden layer, and one output layer is used to perform a multilayer perceptron (MLP) classification on P. The dimensions of each layer are as follows:

[0059]

[0060] The output layer is y c This refers to the output of the C-th neuron. In this case, it's a binary classification network, and C=2. Calculate the probability distribution of y:

[0061]

[0062] Then, the confidence score for each sample is calculated, and the decision function is as follows:

[0063]

[0064] Will The image is converted into a one-hot vector as the predicted labels, and the ground truth labels are denoted as y0.

[0065] When using Support Vector Machines (SVM) to solve in high-dimensional space, sample feature points are mapped and constrained to a hypersphere, and the minimum constrained volume of this hypersphere is calculated. The model is trained using the loss function of one-class Deep SVDD anomaly detection.

[0066] For a sample x∈χ, assume the input sample space is The model is trained using the loss function of deep one-class classification anomaly detection:

[0067]

[0068] in, This represents the feature mapping function of a neural network, and the corresponding weights. c is the center of the hypersphere, F is the output space, λ is the penalty factor, and λ≥0.

[0069] For a test sample point x'∈χ, the anomaly score can be defined as the distance from the sample point to the center of the hypersphere in the high-dimensional space of the output:

[0070]

[0071] in, This represents the weights of the model after training.

[0072] Calculate the average distance from all real, normal samples to the center of the sphere:

[0073]

[0074] Where N is the number of real normal samples, and δ is the threshold of the decision boundary.

[0075] The relationship between δ and s(x') is used to determine whether test sample points can be classified as outliers:

[0076]

[0077] The stochastic gradient descent (SGD) algorithm is used to optimize δ to improve the model performance, and pseudo-labeled defect samples are obtained by converting the judgment type values ​​of the anomaly boundary into one-hot vectors.

[0078] Step 3: Construct a new dataset using the pseudo-labeled defect samples and real normal samples obtained in Step 2. After image preprocessing, input the dataset into the feature mapping embedding encoder to calculate the image's feature maps. Considering the potential class imbalance between the number of pseudo-labeled defect samples and real normal samples, a balanced cross-entropy function is used to train the anomaly detection classifier network. To prevent the loss function from becoming skewed due to sample imbalance, a weighting factor is added to the loss function to increase the weight of minority classes and balance the distribution of the loss function.

[0079] Meanwhile, the focal loss function is used to address the sample imbalance problem:

[0080]

[0081] make but:

[0082]

[0083] Where γ is the focusing parameter, γ≥0, p r γ It is the modulation factor.

[0084] This invention expands the data scale by using a slice generator and constructs a model of overall texture information from local features. It introduces a feature mapping embedding encoder to model multi-scale features, enhancing the learning of key image feature information by samples, thereby achieving a higher accuracy in detecting fine-grained defects with fewer samples.

[0085] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A few-shot fine-grained defect detection method based on deep learning, which performs defect detection through a constructed defect detection network model, characterized in that... The defect detection network model adopts a feature mapping embedded encoder network structure. Its training steps include: Step 1: Preprocess the acquired images and generate sample slices using a sample slice generator based on the characteristics of the dataset. Step 2: Introduce a pseudo-label mechanism. Input the sample slices into the feature mapping embedding encoder for training. Use the loss function of deep single-class classification anomaly detection for training. The feature mapping embedding encoder in the training phase is used as a class classifier to generate pseudo-label defect samples. Step 3: Construct a new dataset using the obtained pseudo-label defect samples and real normal samples. Generate new sample slices through image preprocessing in Step 1. Input the new sample slices into the feature map embedding encoder and train it using the balanced cross-entropy function to obtain a defect detection network model that meets the requirements.

2. The deep learning-based few-sample fine-grained defect detection method according to claim 1, characterized in that, Step 1, generating sample slices using the sample slice generator, specifically includes: Input the sample into the slice generator and set the parameters. Obtain slices, and get slice clusters. , ,like The generated slices will then be flipped to align with the image texture.

3. The deep learning-based few-sample fine-grained defect detection method according to claim 2, characterized in that, The feature mapping embedding encoder network structure includes ResNet, convolutional layers, multi-scale feature stacks, and a Transformer encoder. ResNet extracts multi-scale features from sample slices, convolutional layers obtain multi-scale features at different resolutions, and the multi-scale feature stack flattens and embeds the multi-scale features at different resolutions, resulting in a feature vector that is then input into the Transformer encoder for prediction.

4. The deep learning-based few-sample fine-grained defect detection method according to claim 3, characterized in that, The specific steps of obtaining multi-scale features at different resolutions through convolutional layers include: The size of each training batch is set as denoted as The total number of corresponding slices is , There are a total of samples. Each processing batch is recorded as having a channel count for each image. The height and width of the image are denoted as follows: and Then the slice size of each batch is ; From the network Layer to Extracting multi-scale feature maps from the feature maps output by the layer ,in , and The output image sizes of the layers are respectively , and Using a step size of 1 1 convolution to obtain , and Feature map , and ; and for The output image of the layer uses a stride of 2 for 3 3 convolutions yield Feature map .

5. The deep learning-based few-sample fine-grained defect detection method according to claim 4, characterized in that, The specific steps of flattening and embedding multi-scale features at different resolutions using a multi-scale feature stack and then inputting the resulting feature vector into the Transformer encoder include: feature map , , and Flattening multi-scale feature stacks and embedding generated vectors , All vectors obtained from an image are The size of the obtained processing batch is ,in The number of images in a processing batch. This indicates the number of slices in an image. The vector after multi-scale feature extraction The dimension of is defined as: For slices in each batch Construct location encoding information, generate a token and prepend a token-like element, resulting in a token of size. A set of image patches with embedded location information, denoted as Then input it into the Transformer encoder.

6. The deep learning-based few-sample fine-grained defect detection method according to claim 5, characterized in that, The Transformer encoder obtains class token information after computation. Each Corresponding to one A fully connected network consisting of one input layer, one hidden layer, and one output layer is used. To perform a multilayer perceptron classification, the dimensions of the input layer, hidden layer, and output layer are: The output layer is , It refers to the first The output of each neuron forms a binary classification network. calculate The probability distribution is: Calculate the confidence score for each sample using the decision function: Will Convert to a one-hot vector as the predicted label of the image, and denote the image's true label as... .

7. The deep learning-based few-sample fine-grained defect detection method according to claim 3, characterized in that, The loss function training for deep single-class anomaly detection involves using a support vector machine (SVM) to map and constrain sample feature points to a hypersphere in high-dimensional space, and then solving for the minimum constraint volume of this hypersphere. Specifically: For the sample Assume the input sample space is The system is trained using a loss function for classifying anomaly detection. in, This represents the feature mapping function of a neural network, and the corresponding weights. , Center of the hypersphere It is the Frobenius norm. As a penalty factor, , L represents the batch size for each training iteration, and L represents the number of layers in the neural network. For test sample points The anomaly score is defined as the distance from a sample point in the high-dimensional output space to the center of the hypersphere: in, Indicates the weights of the model after training; Calculate the average distance from all real, normal samples to the center of the sphere: in, It represents the number of real, normal samples. It is the threshold of the decision boundary; according to and The size relationship is used to determine whether test sample points can be classified as outliers: Pseudo-labeled defect samples are obtained by converting the judgment type values ​​of abnormal boundaries into one-hot vectors.

8. The deep learning-based few-sample fine-grained defect detection method according to claim 7, characterized in that, The Optimize using the stochastic gradient descent algorithm.

9. The deep learning-based few-sample fine-grained defect detection method according to claim 3, characterized in that, The balanced cross-entropy function uses a focal loss function and adds a weighting factor to the loss function.

10. The deep learning-based few-sample fine-grained defect detection method according to claim 9, characterized in that, The focus loss function is: make ,but: in, These are the focusing parameters. , This represents the probability distribution of the output layer.

Images