Plate and strip steel surface defect classification system and method based on small sample self-similar embedding inference

Abstract

The present invention discloses a plate and strip steel surface defect classification system and method based on small sample self-similar embedding inference. The system includes a feature extraction module and a classification module. The classification module consists of a self-similar representation SSR module and a graph embedding label inference GELE module; wherein the self-similar representation SSR module includes two self-similar calculation submodules SS_s and SS_q, two self-similar representation learning submodules R_s and R_q for extracting structural patterns, and two feature aggregation submodules FA_s and FA_q; the graph embedding label inference GELE module includes a graph embedding submodule GE, a label inference submodule LE and a classifier; it is intended to use a self-similar method, that is, measuring the similarity of local patches in adjacent areas to enhance the embedding expression, and a non-parametric regularizer, that is, expanding the decision boundary in a popular smoothing manner to reduce the distribution drift problem caused by class noise. The system and method of the present invention further improve the performance of small sample surface defect classification.

Description

Technical Field

[0001] The present invention relates to the technical field of defect image classification, and in particular to a plate and strip steel surface defect classification system and method based on small sample self-similar embedding inference. Background Art

[0002] The classification of surface defects of industrial products is an important part of the surface quality control process and a key method to ensure the quality of industrial products. Traditional surface defect classification methods use manually designed features (such as SIFT features and HOG features). Not only is the manual feature design process complicated, but the classification performance is also unsatisfactory. With the development of deep learning technology, a large number of methods have proposed the use of deep convolutional neural networks to achieve surface defect classification tasks. However, these methods usually require a large amount of labeled data for training. Classification is challenging in scenarios such as plate and strip surface defect classification where there is little data. In the actual industrial production process of strip steel, production workshops usually continuously optimize the production process, so there are few defective samples, and it becomes more difficult to collect defective samples. In addition, as time goes by, new defect types will also appear in the industrial production process, so it will be impossible to collect labeled training samples for possible future defect types in advance. Finally, manual labeling of all collected data still requires a lot of manpower and funds. The above situations bring huge challenges to training a highly robust fully supervised learning system. This has led to the emergence of deep learning methods based on transfer learning, few-shot learning, and semi-supervised learning, which address the challenges of learning with limited data.

[0003] Few-shot image classification aims to train a deep learning model to enable it to recognize unknown categories from a small number of samples. The task is defined as taking a given query image as the target class, training a learner with a few support images, and using the learned learner to classify the query set images, where the support set and the query set are of different categories. Currently, most few-shot learning algorithms focus on the recognition of natural images, and no few-shot learning algorithms have been applied to the surface defect classification task.

[0004] A key challenge in few-shot image classification is how to generalize the trained model well to unseen classes. This requires the feature representation to be robust to small changes in the data distribution. Some regularization techniques such as dropout, batch normalization, and manifold mixup are used outside the few-shot learning literature to solve this problem. Currently, most few-shot classification algorithms also adopt a meta-learning model, that is, the training data is converted into episodes of few-shot learning in order to better adapt to the situation of few samples. In this direction, based on converting features into an embedding space through a meta-learner, the distance between images in the embedding space conforms to their semantic distance, and then they are classified using a metric classifier.

[0005] However, compared with natural image datasets, defect sample datasets have similarity within data classes, large differences between classes, and less data. Therefore, there are still some problems in solving the classification of surface defects with few samples based on these methods: ① The decision boundary of the regularization method is relatively fuzzy and cannot accurately divide the categories; and class noise causes distribution drift, resulting in poor classification results; ② The learned embedding function often overfits irrelevant features and cannot be transferred to new classes that were not observed in training. Although deep neural features provide rich semantic information, it is still challenging to learn generalized embeddings without being disturbed by false features. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a plate and strip steel surface defect classification system and method based on small sample self-similar embedding inference in view of the deficiencies of the above-mentioned prior art.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is: a plate and strip steel surface defect classification system based on small sample self-similar embedding inference, the system comprises: a feature extraction module and a classification module;

[0008] The feature extraction module is used to extract the basic features of the plate and strip steel defect image;

[0009] The classification module consists of a self-similar representation SSR (Self-Similar Representation) module and a graph embedding and label extrapolate GELE (Graph Embedding and Label Extrapolate) module, and classifies unlabeled plate and strip defect images through a classifier according to the basic features of the plate and strip defect images.

[0010] Furthermore, the self-similar representation SSR module includes two self-similar calculation submodules SS_s and SS_q, two self-similar representation learning submodules R_s and R_q for extracting structural patterns, and two feature aggregation submodules (FeatureAggregation) FA_s and FA_q;

[0011] The self-similarity calculation submodule SS_s processes the basic features Fs obtained by the support set image Is after the feature extraction module, and calculates the inner product of the c-dimensional vector at a certain position x in Fs and the c-dimensional vector at its neighboring position to obtain a similar tensor representation of the c-dimensional vector at position x. This method is used to traverse all position points in Fs, and the output result obtained is the self-similar representation of the basic representation of the support set defect image by the SS_s module, that is, the self-similarity tensor Ts;

[0012] The self-similarity calculation submodule SS_q processes the basic feature Fq obtained after the feature extraction module of the query set image Iq, and calculates the inner product of the c-dimensional vector at a certain position x in Fq and the c-dimensional vector at its neighboring position to obtain a similar tensor representation of the c-dimensional vector at position x. In this way, all position points in Fq are traversed, and the output result obtained is the self-similar representation of the basic representation of the defect image of the query set by the SS_q module, that is, the self-similarity tensor Tq;

[0013] The self-similar representation learning submodule R_s stores the self-similar tensor Ts in R_s, and passes it into a series of 2D convolution blocks for self-similar representation learning. The output result obtained is the similar feature representation of the self-similar tensor Ts of the support set defect image by the R_s module, and the feature Rs rich in structural information is obtained;

[0014] The self-similar representation learning submodule R_q stores the self-similar tensor Tq in R_q, and passes it into a series of 2D convolution blocks for self-similar representation learning. The output result obtained is the similar feature representation of the self-similar tensor Ts of the defect image of the query set by the R_q module, and the feature Rq rich in structural information is obtained;

[0015] The feature aggregation submodule FA_s aggregates the feature Rs rich in structural information with the basic feature Fs from the support set image Is of the feature extractor to obtain the aggregated feature Zs;

[0016] The feature aggregation submodule FA_q performs feature aggregation on the feature Rq rich in structural information and the basic feature Fq from the query set image Iq of the feature extractor to obtain the aggregated feature Zq.

[0017] Furthermore, the series of 2D convolution blocks include: a 1×1 convolution block for reducing the number of tensor channels, two 3×3 convolution blocks for conversion operations, and a 1×1 convolution block for restoring the number of tensor channels; the convolution blocks all aggregate vectors without padding to obtain local correlation patterns, thereby reducing their spatial dimensions;

[0018] In order to avoid overfitting and the influence of interference features, regularization and relu operations are used in the middle of the convolution block.

[0019] Furthermore, the graph embedding label inference GELE module comprises a graph embedding submodule GE, a label inference submodule LE and a classifier;

[0020] The graph embedding submodule GE uses the aggregated features Zs and Zq to perform further feature transformation through a neural network based on a similarity graph, treating all aggregated features as a feature graph, with each feature data as a node, and calculating the Euclidean distance and adjacency matrix of each pair of features to accurately represent the feature distribution of the support set and the query set;

[0021] The label inference submodule LE uses the feature distribution obtained by the graph embedding submodule GE to calculate the propagation matrix according to the similarity value of the feature distribution, showing the possibility of conversion between any aggregated feature and its surrounding neighbor feature labels. The feature outputs the inference interpolation matrix between the features by weighted summation with its surrounding neighbor features.

[0022] The classifier includes a linear classifier for label prediction and a classifier for label inference, both of which are composed of softmax and cross entropy loss functions; the linear classifier directly predicts the corresponding label of the query set image through the aggregated features of the support set and the query set input images; the label inference classifier obtains the support set with the most similar feature distribution to the unlabeled query set through the inference interpolation matrix between the features obtained by module LE, and infers the known label information of the support set as the label of the query set.

[0023] On the other hand, the present invention also provides a method for classifying surface defects of plate and strip steel by using the above-mentioned small sample self-similar embedding inference plate and strip steel surface defect classification system, comprising the following steps:

[0024] Step 1: Input a dataset of surface defect images of steel strips with different defect categories, and divide the dataset into a training set and a test set;

[0025] Step 2: Select n defect categories from the training set and the test set, sample them evenly and without replacement in the form of episodes, and then separate the query set query and the support set support from the sampled n category data, denoted as Iq and Is respectively;

[0026] Step 3: Input Iq and Is into the feature extraction module respectively, and the basic features extracted from Iq and Is are recorded as Fq and Fs respectively;

[0027] Furthermore, the feature extraction module uses ResNet12 as the skeleton of the deep learning network to extract the basic features of the image.

[0028] Step 4: Input Fq and Fs into the self-similar representation SSR module, associate each position with its neighboring pixels to encode rich semantic structures, and perform representation learning and aggregation on this basis to obtain aggregated features Zq and Zs rich in structural information. The specific process is as follows:

[0029] Step 4.1: Input the basic features Fq and Fs into the self-similarity calculation submodules SS_q and SS_s respectively, take out the c-dimensional vector of each position x in Fq and Fs respectively, find the vector of the neighborhood of the position through a convolutional layer, and then calculate the Hadamard product of the two to obtain the self-similarity tensors Tq and Ts, simplifying a pair of feature vectors into a scalar correlation, retaining the rich semantics of the feature vectors for classification;

[0030] Step 4.2: Input the self-similarity tensors Tq and Ts into the self-similar representation learning submodules R_q and R_s respectively, and use 2D convolution for self-similar learning, that is, use a convolution block to reduce the number of tensor channels; use 2 convolution blocks for conversion operations; and use another convolution block to restore the number of tensor channels to obtain features Rq and Rs rich in structural information;

[0031] Step 4.3: Input the structural information-rich features Rq and Rs and the basic features Fq and Fs into the feature aggregation submodules FA_q and FA_s respectively, and combine the above two features to generate aggregated features Zq and Zs.

[0032] Furthermore, when the self-similarity tensors Tq and Ts are input into the self-similar representation learning submodules R_q and R_s as feature maps for convolution, the edges are zero-padded to facilitate sampling of edge information and avoid information loss.

[0033] Step 5: The aggregate features Zq and Zs are passed into the graph embedding label inference GELE module, the distance between the aggregate features is calculated, and the inferred interpolation matrix between the output features is transformed through a series of transformations to reduce the noise of the category representation. The specific process is as follows:

[0034] Step 5.1: Pass the aggregate features Zq and Zs into the graph embedding submodule GE. Treat all aggregate features as a graph, and each feature data is a node. For each pair of features (i, j), calculate the Euclidean distance d between the features.ij and the adjacency matrix A ij , let A ii =0;

[0035]

[0036]

[0037] Among them, z i 、z j are the i-th feature and the j-th feature respectively, σ 2 is the scale factor, when Stable training;

[0038] Step 5.2: Calculate the Laplace transform L of the adjacency matrix:

[0039]

[0040] Among them, D ii is the value of the i-th row and i-th column of the adjacency matrix D;

[0041] Step 5.3: Obtain the inference matrix P based on the Laplace transform L of the above adjacency matrix:

[0042] P=(I-αL) -1

[0043] in, is the scaling factor, I is the identity matrix;

[0044] Step 5.4: Treat the inference matrix P as a weight matrix and calculate the inference interpolation matrix

[0045]

[0046] in, The inferred interpolation matrix The i-th vector in .

[0047] Step 6: Use two classifiers: the first linear classifier directly predicts the label information of the unlabeled sample images in the query set based on the aggregated features, and the other classifier based on label inference predicts the label by inferring the interpolation matrix. The category information of the unlabeled sample images in the query set is obtained according to the category information of the sample images with known labels in the support set. The two classifiers are used together to complete the classification.

[0048] Furthermore, the plate and strip steel surface defect classification system based on the small sample self-similar embedding inference uses the episodes form during training, and the specific process of step 6 is as follows:

[0049] Step 6.1: The first linear classifier directly predicts the class label of the input image, optimizes by minimizing the cross entropy loss, applies the softmax logic provided by the neural network to obtain the probability, and calculates the linear classification error as follows:

[0050]

[0051] in, For all known labels of the support set data, L C is the linear classification error, x i is the input image, y i is the true label of the input image, W l is the parameter of the linear classifier, and θ is the parameter of the feature extractor. At the same time, the parameters of the feature extractor are optimized by back propagation to make the extracted features more useful.

[0052] Step 6.2: The second classifier based on label inference predicts labels by inferring the interpolation matrix. It also seeks optimization by minimizing the cross entropy loss, applies the softmax logic provided by the neural network to obtain the probability, and calculates the label inference classification error as follows:

[0053]

[0054] Among them, L L is the label inference classification error, is the inferred interpolation matrix, The inferred interpolation matrix The i-th vector in Y S One-hot encoding of image labels is supported;

[0055] Step 6.3: The integrated minimized loss function is:

[0056]

[0057] Among them, Q is the query set and S is the support set;

[0058] Step 6.4: During the training process, each episode minimizes the loss function and finds a model state with the minimum loss through gradient descent; all parameter information of the model in this state is saved; during the testing process, the support set calls this model for label inference and further fine-tunes the parameters. Finally, the query set uses the fine-tuned model and takes the classification result of the label inference classifier as the final classification result.

[0059] The beneficial effects of adopting the above technical solution are:

[0060] 1. The system and method provided by the present invention adopts small samples, that is, inputting the data set into the classification network in batches to classify the surface defects of plate and strip steel, thereby achieving the purpose of achieving good classification effect under unfavorable scenarios such as small number of samples and high similarity between defect classes.

[0061] 2. The present invention proposes to use the autocorrelation representation module through SSR to make full use of similar information within the class, that is, to extract transferable structural patterns from the image itself and make full use of the semantic information correlation in the defect sample. This allows the network to pay more attention to the relevant area, providing a more robust input for the next module and improving the classification performance of the entire classification network.

[0062] 3. The present invention realizes inter-class label inference through the graph embedding inference GELE module, that is, regularizes the data information in an unsupervised manner, expands the decision boundary, and reduces the distribution drift problem caused by class noise. It uses embedded interpolation to capture higher-order feature interactions, transfers labeled data information to unlabeled data to achieve classification purposes, and at the same time realizes the most advanced transduction setting of small sample classification results.

[0063] 4. The present invention has created a new surface defect dataset for plate and strip steel industry, which includes 15 types of plate and strip steel surface defect images and 1 type of plate and strip steel surface normal image. It is also the first time that normal plate and strip steel images are added to the test of the classification network, which also proves the stability of the system designed by the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a schematic diagram of the structure of the plate and strip steel surface defect classification system based on small sample self-similar embedding inference in an embodiment of the present invention;

[0065] Figure 2 This is a schematic diagram of the operation flow of the SSR module in an embodiment of the present invention;

[0066] Figure 3 This is a schematic diagram of a plate and strip steel defect data set in an embodiment of the present invention; DETAILED DESCRIPTION

[0067] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0068] In this embodiment, a new plate and strip steel defect dataset is established, which includes 15 types of surface defect images and 1 type of normal image. Figure 3As shown. Each type of defect contains 50 images with a resolution of 224*224. In order to detect the stability of the classification system, we consider two data set detection methods. The first is that we randomly select 10 types of defect images from 15 types as training sets, and the other 5 types of defects as test sets to verify the classification effect of the classification system; the second method takes into account that the defect samples on the actual strip processing line are much fewer than the normal samples, and it is inevitable to mix the normal samples into the test during the screening process. Therefore, based on the first experiment, 1 type of normal samples is added to the test set to verify whether the classification system can achieve a higher classification accuracy in the presence of interference.

[0069] like Figure 1 As shown, the plate and strip steel surface defect classification system based on small sample self-similar embedding inference in this embodiment is described as follows.

[0070] The system includes a feature extraction module and a classification module;

[0071] The feature extraction module is used to extract the basic features of the plate and strip steel defect image;

[0072] In specific implementation, the existing classification networks VGG and ResNet can be used in the feature extraction module to extract the basic features of the input query and support grayscale images Iq and Is respectively. In this embodiment, ResNet12 is used as the skeleton of the network, and Iq and Is are input into ResNet12 at the same time. It learns the shallow features of the image better through multiple 1x1 convolutional layers and 3x3 convolutional layers in the network, processing with different numbers of channels, and residual blocks, and obtains the corresponding basic features Fq and Fs.

[0073] Furthermore, in order to reduce the number of parameters and improve classification efficiency, the defective images of the query set query and the support set support are adjusted from 224*224*3 to 84*84*3. They are input into the feature extractor network Resnet12 to extract their respective basic features Fq and Fs, and 640 basic feature maps with a size of 5*5 are obtained.

[0074] The classification module consists of a self-similar representation SSR (Self-Similar Representation) module and a graph embedding and label extrapolate GELE (Graph Embedding and Label Extrapolate) module, and classifies unlabeled plate and strip defect images through a classifier according to the basic features of the plate and strip defect images.

[0075] Furthermore, the self-similar representation SSR module includes two self-similar calculation submodules SS_s and SS_q, two self-similar representation learning submodules R_s and R_q for extracting structural patterns, and two feature aggregation submodules (FeatureAggregation) FA_s and FA_q;

[0076] The self-similarity calculation submodule SS_s processes the basic features Fs obtained by the support set image Is after the feature extraction module, and calculates the inner product of the c-dimensional vector at a certain position x in Fs and the c-dimensional vector at its neighboring position to obtain a similar tensor representation of the c-dimensional vector at position x. This method is used to traverse all position points in Fs, and the output result obtained is the self-similar representation of the basic representation of the support set defect image by the SS_s module, that is, the self-similarity tensor Ts;

[0077] The self-similarity calculation submodule SS_q processes the basic feature Fq obtained after the feature extraction module of the query set image Iq, and calculates the inner product of the c-dimensional vector at a certain position x in Fq and the c-dimensional vector at its neighboring position to obtain a similar tensor representation of the c-dimensional vector at position x. In this way, all position points in Fq are traversed, and the output result obtained is the self-similar representation of the basic representation of the defect image of the query set by the SS_q module, that is, the self-similarity tensor Tq;

[0078] In this embodiment, the 5*5*640 basic features Fs and Fq are respectively input into the SS_s and SS_q modules. Through a window of size 5*5 with a filling value of 2, the two-dimensional feature tensors are organized into one-dimensional self-similar representation tensors Ts and Tq with a size of 16000*25, so that all points in the basic features and their neighborhood areas establish a self-similar calculation relationship.

[0079] The self-similar representation learning submodule R_s stores the self-similar tensor Ts in R_s, and passes it into a series of 2D convolution blocks for self-similar representation learning. The output result obtained is the similar feature representation of the self-similar tensor Ts of the support set defect image by the R_s module, and the feature Rs rich in structural information is obtained;

[0080] The self-similar representation learning submodule R_q stores the self-similar tensor Tq in R_q, and passes it into a series of 2D convolution blocks for self-similar representation learning. The output result obtained is the similar feature representation of the self-similar tensor Ts of the defect image of the query set by the R_q module, and the feature Rq rich in structural information is obtained;

[0081] Furthermore, the series of 2D convolution blocks include: a 1×1 convolution block for reducing the number of tensor channels, two 3×3 convolution blocks for conversion operations, and a 1×1 convolution block for restoring the number of tensor channels; the convolution blocks all aggregate vectors without padding to obtain local correlation patterns, thereby reducing their spatial dimensions;

[0082] In order to avoid overfitting and the influence of interference features, regularization and relu operations are used in the middle of the convolution block.

[0083] In this embodiment, the Ts and Tq tensors of size 16000*25 are input into the self-similar representation learning modules R_s and R_q respectively, and self-similar representation learning is performed through a series of 2D convolutions. First, the size of the tensor is adjusted to 640 feature maps of size 25*25. In order to reduce the amount of network parameters, a convolution block with a size of 1*1 and 64 channels is first used for convolution dimensionality reduction operation to obtain 64 feature maps of size 25*25 for the tensor, that is, the number of channels is reduced from 640 to 64; next, two convolution blocks with a size of 1*3*3 and 64 channels are used for conversion operation to obtain 64 feature maps containing 25 feature maps of size 1*1, that is, similar local structural features on the feature map represented by the tensor, because what is extracted at this time is the high-level features compressed after step-by-step convolution operations, so it actually represents an area on the original defect image; finally, a convolution block with a size of 1*1 and 640 channels is used for convolution dimensionality recovery operation, 640 feature maps containing 5*5 feature maps of size 1*1, that is, the number of channels is restored from 64 to 640.

[0084] Furthermore, 640 feature map tensors Rs and Rq containing 5*5 and 1*1 size are input into the convolution block g(·) respectively. Because the convolution operation is performed without padding, the spatial dimension is reduced and the tensor size is adjusted to 640 5*5 feature maps Rs and Rq. The purpose of this operation is to make the features with similar structures and the basic features of the image the same size for easy aggregation.

[0085] The feature aggregation submodule FA_s aggregates the feature Rs rich in structural information with the basic feature Fs from the support set image Is of the feature extractor to obtain the aggregated feature Zs;

[0086] The feature aggregation submodule FA_q performs feature aggregation on the feature Rq rich in structural information and the basic feature Fq from the query set image Iq of the feature extractor to obtain the aggregated feature Zq.

[0087] The aggregated features Zs and Zq include not only the basic features of the defective image, but also the self-similar structural information within the image, which separates the defective area from the background information as a whole, allowing the subsequent network to focus on these self-similar areas and better classify the defects.

[0088] Furthermore, the graph embedding label inference GELE module comprises a graph embedding submodule GE, a label inference submodule LE and a classifier;

[0089] The graph embedding submodule GE uses the aggregated features Zs and Zq to perform further feature transformation through a neural network based on a similarity graph, treating all aggregated features as a feature graph, with each feature data as a node, and calculating the Euclidean distance and adjacency matrix of each pair of features to accurately represent the feature distribution of the support set and the query set;

[0090] The label inference submodule LE uses the feature distribution obtained by the graph embedding submodule GE to calculate the propagation matrix according to the similarity value of the feature distribution, showing the possibility of conversion between any aggregated feature and its surrounding neighbor feature labels. The feature outputs the inference interpolation matrix between the features by weighted summation with its surrounding neighbor features.

[0091] The classifier includes a linear classifier for label prediction and a classifier for label inference, both of which are composed of softmax and cross entropy loss functions; the linear classifier directly predicts the corresponding label of the query set image through the aggregated features of the support set and the query set input images; the label inference classifier obtains the support set with the most similar feature distribution to the unlabeled query set through the inference interpolation matrix between the features obtained by module LE, and infers the known label information of the support set as the label of the query set.

[0092] On the other hand, the present invention also provides a method for classifying surface defects of plate and strip steel by using the above-mentioned small sample self-similar embedding inference plate and strip steel surface defect classification system, comprising the following steps:

[0093] Step 1: Input a dataset of surface defect images of steel strips with different defect categories, and divide the dataset into a training set and a test set;

[0094] Step 2: Select n defect categories from the training set and the test set, sample them evenly and without replacement in the form of episodes, and then separate the query set query and the support set support from the sampled n category data, denoted as Iq and Is respectively;

[0095] Step 3: Input Iq and Is into the feature extraction module respectively, and the basic features extracted from Iq and Is are recorded as Fq and Fs respectively;

[0096] Furthermore, the feature extraction module uses ResNet12 as the skeleton of the deep learning network to extract the basic features of the image.

[0097] Step 4: Input Fq and Fs into the self-similar representation SSR module, associate each position with its neighboring pixels to encode rich semantic structure, and perform representation learning and aggregation on this basis to obtain aggregate features Zq and Zs rich in structural information. The specific process is as follows: Figure 2 As shown, the process is as follows:

[0098] Step 4.1: Input the basic features Fq and Fs into the self-similarity calculation submodules SS_q and SS_s respectively, take out the c-dimensional vector of each position x in Fq and Fs respectively, find the vector of the neighborhood of the position through a convolutional layer, and then calculate the Hadamard product of the two to obtain the self-similarity tensors Tq and Ts, simplifying a pair of feature vectors into a scalar correlation, retaining the rich semantics of the feature vectors for classification;

[0099] Step 4.2: Input the self-similarity tensors Tq and Ts into the self-similar representation learning submodules R_q and R_s respectively, and use 2D convolution for self-similar learning, that is, use a convolution block to reduce the number of tensor channels; use 2 convolution blocks for conversion operations; and use another convolution block to restore the number of tensor channels to obtain features Rq and Rs rich in structural information;

[0100] Step 4.3: Input the structural information-rich features Rq and Rs and the basic features Fq and Fs into the feature aggregation submodules FA_q and FA_s respectively, and combine the above two features to generate aggregated features Zq and Zs.

[0101] Furthermore, when the self-similarity tensors Tq and Ts are input into the self-similar representation learning submodules R_q and R_s as feature maps for convolution, the edges are zero-padded to facilitate sampling of edge information and avoid information loss.

[0102] In this embodiment, the basic features of the support image are first Input into the SS_s module, take out the c-dimensional vector of each position x∈[1,h]×[1,w] (that is, each pixel) in the feature map, find the vector of the neighborhood of the position through a convolutional layer with a receptive field of f×e, and then calculate the Hadamard product of the two and save it in the self-similar tensor Then, the basic features of the query image are Input into the SS_q module, take out the c-dimensional vector of each position x∈[1,h]×[1,w] (that is, each pixel) in the feature map, find the vector of the neighborhood of the position through a convolutional layer with the same receptive field f×e, and then calculate the Hadamard of the two and save them in the self-similar tensor middle.

[0103]

[0104] Among them, p∈[-d f ,d f ]×[-d e ,d e ] corresponds to the relative position in the neighborhood window, d f is the size of the neighborhood window expanding to the left and right, d e is the size of the neighborhood window extending upward and downward, so that 2d f +1 = f and 2d e +1=e, including the center position. Note that the edges of the feature map are padded with zeros so that the edge information can also be sampled to avoid information loss.

[0105] Different from the existing methods, the present invention uses channel correlation to simplify a pair of feature vectors into a scalar correlation, thereby retaining the rich semantics of the feature vectors for classification.

[0106] In this embodiment, the self-similar tensor Ts and To analyze the self-similar patterns in Ts and Tq, the present invention applies a series of two-dimensional convolutions along the f×e dimensions. Figure 2 As shown in the figure, in order to improve computational efficiency, the convolution block uses a 1×1 convolution (i.e., a bottleneck structure). Self-similar representation learning includes a point-by-point 1×1 convolution layer for reducing the channel size, two 3×3 convolution layers for transformation, and another point-by-point 1×1 convolution layer for channel size recovery. Batch normalization and ReLU are inserted between the convolution layers. Rq and Rs are obtained from this step, and their feature vector width, length, and number of channels are the same as the basic features Fq and Fs extracted by the feature extractor, that is, This process of analyzing the structural pattern is a supplement to the appearance pattern of the basic representations Fq and Fs. It increases the similarity relationship of the foreground information and makes the network pay more attention to similar areas. Therefore, this application combines these two representations through feature aggregation modules FA_q and FA_s to generate aggregated features Z S ,

[0107] The above aggregated features strengthen the basic features with relational features and help learners with small samples better understand "what to observe". Experiments show that SSR is robust to intra-class changes and helps it generalize to unseen target categories.

[0108] Step 5: The aggregate features Zq and Zs are passed into the graph embedding label inference GELE module, the distance between the aggregate features is calculated, and the inferred interpolation matrix between the output features is transformed through a series of transformations to reduce the noise of the category representation. The specific process is as follows:

[0109] Step 5.1: Pass the aggregate features Zq and Zs into the graph embedding submodule GE. Treat all aggregate features as a graph, and each feature data is a node. For each pair of features (i, j), calculate the Euclidean distance d between the features. ij and the adjacency matrix A ij , let A ii =0;

[0110]

[0111]

[0112] Among them, z i 、z j are the i-th feature and the j-th feature respectively, σ 2 is the scale factor, when Stable training;

[0113] Step 5.2: Calculate the Laplace transform L of the adjacency matrix:

[0114]

[0115] Among them, D ii is the value of the i-th row and i-th column of the adjacency matrix D;

[0116] Step 5.3: Obtain the inference matrix P based on the Laplace transform L of the above adjacency matrix:

[0117] P=(I-αL) -1

[0118] in, is the scaling factor, I is the identity matrix;

[0119] Step 5.4: Treat the inference matrix P as a weight matrix and calculate the inference interpolation matrix

[0120]

[0121] in, The inferred interpolation matrix The i-th vector in .

[0122] because is now a weighted sum of its neighbors, and the embedding inferred has the effect of removing unwanted noise from the feature vector. The computational complexity of is negligible for small sample datasets because the datasets are small.

[0123] Step 6: Use two classifiers: the first linear classifier directly predicts the label information of the unlabeled sample images in the query set based on the aggregated features, and the other classifier based on label inference predicts the label by inferring the interpolation matrix. The category information of the unlabeled sample images in the query set is obtained according to the category information of the sample images with known labels in the support set. The two classifiers are used together to complete the classification.

[0124] Furthermore, the plate and strip steel surface defect classification system based on the small sample self-similar embedding inference uses the episodes form during training, and the specific process of step 6 is as follows:

[0125] Step 6.1: The first linear classifier directly predicts the class label of the input image, optimizes by minimizing the cross entropy loss, applies the softmax logic provided by the neural network to obtain the probability, and calculates the linear classification error as follows:

[0126]

[0127] in, For all known labels of the support set data, L C is the linear classification error, x i is the input image, y i is the true label of the input image, W l is the parameter of the linear classifier, and θ is the parameter of the feature extractor. At the same time, the parameters of the feature extractor are optimized by back propagation to make the extracted features more useful.

[0128] Step 6.2: The second classifier based on label inference predicts labels by inferring the interpolation matrix. It also seeks optimization by minimizing the cross entropy loss, applies the softmax logic provided by the neural network to obtain the probability, and calculates the label inference classification error as follows:

[0129]

[0130] Among them, L L is the label inference classification error, is the inferred interpolation matrix, The inferred interpolation matrix The i-th vector in Y SOne-hot encoding of image labels is supported;

[0131] Step 6.3: The integrated minimized loss function is:

[0132]

[0133] Among them, Q is the query set and S is the support set;

[0134] Step 6.4: During the training process, each episode minimizes the loss function and finds a model state with the minimum loss through gradient descent; all parameter information of the model in this state is saved; during the testing process, the support set calls this model for label inference and further fine-tunes the parameters. Finally, the query set uses the fine-tuned model and takes the classification result of the label inference classifier as the final classification result.

[0135] Compared with the prior art, the system and method of the present invention achieve good classification results in complex scenarios such as a small number of defect samples. In order to verify the plate and strip steel defect classification performance of the system and method of the present invention, the system and method proposed by the present invention and the existing classification methods are tested on the plate and strip steel surface dataset proposed by us. All methods are uniformly trained and tested on the same equipment, and the parameter settings of each method are consistent.

[0136] In order to intuitively show the quantitative performance index comparison results of the method of the present invention and the existing classification methods, two standard 5way-1shot and 5way-5shot evaluation indicators are used in this embodiment to compare with the EPNet method, TPMAP method, ICI method, TIM-GD method, TIM-AMD method, and S2SM_R method to verify the performance of the system and method of the present invention. The comparison results are shown in Table 1 below.

[0137] Table 1 Comparison results with other small sample classification methods on the plate and strip steel dataset

[0138]

[0139] The higher the value of the evaluation index in Table 1, the better. From the comparison results of the quantitative indicators shown in Table 1, it can be seen that the system and method of the present invention have great advantages in extraction accuracy compared with other existing methods, and can obtain better classification performance. Among them, on the plate and strip steel surface defect data set that does not contain normal samples, 5way 1shot and 5way 1shot achieved accuracies of 81.3% and 91.7% respectively, and on the plate and strip steel surface defect data set that contains normal samples, 5way1shot and 5way 1shot achieved accuracies of 80.4% and 90.5% respectively.

Claims

1. A strip steel surface defect classification system based on small sample self-similar embedding inference, characterized in that: The system includes a feature extraction module and a classification module; The feature extraction module is used to extract basic features of the plate and strip steel defect image, and the basic features include basic features Fs and basic features Fq; The classification module is composed of a self-similar representation SSR module and a graph embedding label inference GELE module, and classifies the unlabeled plate and strip defect images through a classifier according to the basic features of the plate and strip defect images; The self-similar representation SSR module is used to process the basic features Fs and the basic features Fq respectively to obtain the corresponding aggregated features Zs and Zq rich in structural information; The graph embedding label inference GELE module comprises a graph embedding submodule GE, a label inference submodule LE and a classifier; The graph embedding submodule GE uses the aggregated features Zs and Zq rich in structural information to perform further feature transformation through a neural network based on a similarity graph, treating all aggregated features as a feature graph, with each feature data as a node, and calculating the Euclidean distance and adjacency matrix of each pair of features to accurately represent the feature distribution of the support set and the query set; The label inference submodule LE uses the feature distribution obtained by the graph embedding submodule GE to calculate the propagation matrix according to the similarity value of the feature distribution, showing the possibility of conversion between any aggregated feature and its surrounding neighbor feature labels. The feature outputs the inference interpolation matrix between the features by weighted summation with its surrounding neighbor features. The classifier includes a linear classifier for label prediction and a classifier for label inference, both of which are composed of softmax and cross entropy loss functions; the linear classifier directly predicts the corresponding label of the query set image through the aggregated features of the support set and the query set input images; the label inference classifier obtains the support set with the most similar feature distribution to the unlabeled query set through the inference interpolation matrix between the features obtained by module LE, and infers the known label information of the support set as the label of the query set.

2. The plate and strip steel surface defect classification system based on small sample self-similar embedding inference according to claim 1 is characterized in that: The self-similar representation SSR module includes two self-similar calculation submodules SS_s and SS_q, two self-similar representation learning submodules R_s and R_q for extracting structural patterns, and two feature aggregation submodules FA_s and FA_q; The self-similarity calculation submodule SS_s processes the basic features Fs obtained by the support set image Is after the feature extraction module, and calculates the inner product of the c-dimensional vector at a certain position x in Fs and the c-dimensional vector at its neighboring position to obtain a similar tensor representation of the c-dimensional vector at position x. This method is used to traverse all position points in Fs, and the output result obtained is the self-similar representation of the basic representation of the support set defect image by the SS_s module, that is, the self-similarity tensor Ts; The self-similarity calculation submodule SS_q processes the basic feature Fq obtained after the feature extraction module of the query set image Iq, and calculates the inner product of the c-dimensional vector at a certain position x in Fq and the c-dimensional vector at its neighboring position to obtain a similar tensor representation of the c-dimensional vector at position x. In this way, all position points in Fq are traversed, and the output result obtained is the self-similar representation of the basic representation of the defect image of the query set by the SS_q module, that is, the self-similarity tensor Tq; The self-similar representation learning submodule R_s stores the self-similar tensor Ts in R_s, and passes it into a series of 2D convolution blocks for self-similar representation learning. The output result obtained is the similar feature representation of the self-similar tensor Ts of the support set defect image by the R_s module, and the feature Rs rich in structural information is obtained; The self-similar representation learning submodule R_q stores the self-similar tensor Tq in R_q, and passes it into a series of 2D convolution blocks for self-similar representation learning. The output result obtained is the similar feature representation of the self-similar tensor Ts of the defect image of the query set by the R_q module, and the feature Rq rich in structural information is obtained; The feature aggregation submodule FA_s aggregates the feature Rs rich in structural information with the basic feature Fs of the support set image Is from the feature extraction module to obtain the aggregated feature Zs; The feature aggregation submodule FA_q performs feature aggregation on the feature Rq rich in structural information and the basic feature Fq of the query set image Iq from the feature extraction module to obtain the aggregated feature Zq.

3. The plate and strip steel surface defect classification system based on small sample self-similar embedding inference according to claim 2 is characterized in that: The series of 2D convolution blocks include: a 1×1 convolution block for reducing the number of tensor channels, two 3×3 convolution blocks for conversion operations, and a 1×1 convolution block for restoring the number of tensor channels; the convolution blocks all aggregate vectors without padding to obtain local correlation patterns, thereby reducing their spatial dimensions; In order to avoid overfitting and the influence of interference features, regularization and relu operations are used in the middle of the convolution block.

4. A method for classifying surface defects of plate and strip steel using the plate and strip steel surface defect classification system based on small sample self-similar embedding inference as described in any one of claims 1 to 3, characterized in that: The steps include: Step 1: Input a dataset of surface defect images of steel strips with different defect categories, and divide the dataset into a training set and a test set; Step 2: Select n defect categories from the training set and the test set, sample them evenly and without replacement in the form of episodes, and then separate the query set query and the support set support from the sampled n category data, denoted as Iq and Is respectively; Step 3: Input Iq and Is into the feature extraction module respectively, and the basic features extracted from Iq and Is are recorded as Fq and Fs respectively; Step 4: Input Fq and Fs into the self-similar representation SSR module, associate each position with its neighboring pixels to encode rich semantic structures, and perform representation learning and aggregation on this basis to obtain aggregated features Zq and Zs rich in structural information; Step 5: The aggregate features Zq and Zs are passed into the graph embedding label inference GELE module, the distance between the aggregate features is calculated, and the inferred interpolation matrix between the output features is transformed through a series of transformations, thus reducing the noise of the category representation; Step 6: Use two classifiers: the first linear classifier directly predicts the label information of the unlabeled sample images in the query set based on the aggregated features, and the other classifier based on label inference predicts the label by inferring the interpolation matrix. The category information of the unlabeled sample images in the query set is obtained according to the category information of the sample images with known labels in the support set. The two classifiers are used together to complete the classification.

5. The method for classifying surface defects of plate and strip steel according to claim 4, characterized in that: The feature extraction module uses ResNet12 as the skeleton of the deep learning network to extract the basic features of the image.

6. The method for classifying surface defects of plate and strip steel according to claim 4, characterized in that: The specific process of step 4 is as follows: Step 4.1: Input the basic features Fq and Fs into the self-similarity calculation submodules SS_q and SS_s respectively, take out the c-dimensional vector of each position x in Fq and Fs respectively, find the vector of the neighborhood of the position through a convolutional layer, and then calculate the Hadamard product of the two to obtain the self-similarity tensors Tq and Ts, simplifying a pair of feature vectors into a scalar correlation, retaining the rich semantics of the feature vectors for classification; Step 4.2: Input the self-similarity tensors Tq and Ts into the self-similar representation learning submodules R_q and R_s respectively, and use 2D convolution for self-similar learning, that is, use a convolution block to reduce the number of tensor channels; use 2 convolution blocks for conversion operations; and use another convolution block to restore the number of tensor channels to obtain features Rq and Rs rich in structural information; Step 4.3: Input the structural information-rich features Rq and Rs and the basic features Fq and Fs into the feature aggregation submodules FA_q and FA_s respectively, and combine the above two features to generate aggregated features Zq and Zs.

7. The method for classifying surface defects of plate and strip steel according to claim 6, characterized in that: The self-similarity tensors Tq and Ts described in step 4.2 are input into the self-similar representation learning submodules R_q and R_s as feature maps, and the edges are zero-padded during the convolution process to facilitate sampling of edge information and avoid information loss.

8. The method for classifying surface defects of plate and strip steel according to claim 4, characterized in that: The process of step 5 is as follows: Step 5.1: Pass the aggregated features Zq and Zs into the graph embedding submodule GE. Treat all aggregated features as a graph, and each feature data is a node. For each pair of features (i, j), calculate the Euclidean distance d between the features. ij and the adjacency matrix A ij , let A ii =0; Among them, z i 、z j are the i-th feature and the j-th feature respectively, σ 2 is the scale factor, when Stable training; Step 5.2: Calculate the Laplace transform L of the adjacency matrix: Among them, D ii is the value of the i-th row and i-th column of the adjacency matrix D; Step 5.3: Obtain the inference matrix P based on the Laplace transform L of the above adjacency matrix: P=(I-αL) -1 in, is the scaling factor, I is the identity matrix; Step 5.4: Treat the inference matrix P as a weight matrix and calculate the inference interpolation matrix in, The inferred interpolation matrix The i-th vector in .

9. The method for classifying surface defects of plate and strip steel according to claim 4, characterized in that: The plate and strip steel surface defect classification system based on small sample self-similar embedding inference uses the episodes form during training. The specific process of step 6 is as follows: Step 6.1: The first linear classifier directly predicts the class label of the input image, optimizes by minimizing the cross entropy loss, applies the softmax logic provided by the neural network to obtain the probability, and calculates the linear classification error as follows: in, For all known labels of the support set data, L C is the linear classification error, x i is the input image, y i is the true label of the input image, W l is the parameter of the linear classifier, and θ is the parameter of the feature extractor. At the same time, the parameters of the feature extractor are optimized by back propagation to make the extracted features more useful. Step 6.2: The second classifier based on label inference predicts labels by inferring the interpolation matrix. It also seeks optimization by minimizing the cross entropy loss, applies the softmax logic provided by the neural network to obtain the probability, and calculates the label inference classification error as follows: Among them, L L is the label inference classification error, is the inferred interpolation matrix, The inferred interpolation matrix The i-th vector in Y S One-hot encoding of image labels is supported; Step 6.3: The integrated minimized loss function is: Among them, Q is the query set and S is the support set; Step 6.4: During the training process, each episode minimizes the loss function and finds a model state with the minimum loss through gradient descent; all parameter information of the model in this state is saved; during the testing process, the support set calls this model for label inference and further fine-tunes the parameters. Finally, the query set uses the fine-tuned model and takes the classification result of the label inference classifier as the final classification result.

Images