Metal surface defect detection method based on adversarial and residual network

Abstract

The invention provides a metal surface defect detection method based on an adversarial and residual network, which comprises the following steps: firstly, acquiring a metal surface image data set through camera equipment, and carrying out preprocessing such as data cleaning, label coding and verification to obtain an optimized feature result graph; then, an adversarial generative network combined with an attention mechanism is used, attention of the model to different features is enhanced, and a generator network continuously improves adversarial training to generate a high-quality defect image expansion training data set; in model training, a ResNet-50 model is combined with a multi-loss function mapping method to improve performance and generalization ability. According to the method, the problem that metal surface defect samples are limited is solved, the prediction precision of a small sample data set is improved, the method can be used for surface defects of different materials, such as paper and stone surfaces, and manpower and algorithm development cost in production are saved.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision defect detection, and in particular to a metal surface defect detection method based on adversarial and residual networks. Background Art

[0002] Machine vision technology is increasingly used in the industrial field, especially in quality control and defect detection. Among them, the detection of metal surface defects is a crucial link in the production process of metal products. Metal defects may cause product quality to decline and even affect its performance and durability.

[0003] Traditional metal defect detection methods usually rely on manual visual inspection, which has problems such as high cost, low efficiency and subjectivity. Therefore, automated machine vision systems have become one of the key technologies to solve this problem.

[0004] In recent years, the rapid development of deep learning technology has provided new possibilities for metal defect detection. Among them, the introduction of deep learning architectures such as Generative Adversarial Networks (GANs) and Residual Networks (ResNet) has significantly improved the accuracy and efficiency of metal defect detection.

[0005] Generative Adversarial Networks (GANs) are an architecture consisting of a generator and a discriminator, which compete with each other in a game to generate realistic images or data. In metal defect detection, the generator is responsible for generating images of simulated defects, while the discriminator determines whether these images are real defect images. Through mutual competition, the generator is encouraged to continuously optimize the output. This adversarial training mechanism helps to improve the accuracy of metal defect detection systems with small samples.

[0006] On the other hand, the residual network (ResNet) is a deep convolutional neural network that solves the gradient vanishing problem in deep network training by introducing skip connections. In metal defect detection, ResNet can effectively capture the features of the metal surface and has a strong feature reuse capability, thereby improving the detection performance.

[0007] The research and development of metal defect detection methods is not only of great significance in industrial production, but also provides strong support for improving product quality, reducing production costs and improving production efficiency. The application of methods based on adversarial generative networks and residual networks in this field has brought a higher level of automation and accuracy to metal defect detection, and is expected to continue to promote the advancement of machine vision technology in the future. Summary of the invention

[0008] The technical problem to be solved by the present invention is to overcome the deficiencies in the prior art and provide a metal surface defect detection method based on adversarial and residual networks, especially to solve the problem of lack of diversity in data set samples.

[0009] The present invention is specifically implemented by the following technical solution: a metal surface defect detection method based on adversarial and residual network, specifically comprising the following steps:

[0010] Step 1.1, data collection and preparation: Use camera equipment to collect image data sets. The collected raw data sets will go through a series of data cleaning, label encoding and label verification;

[0011] Step 1.2, image preprocessing: preprocess the uploaded image, the preprocessing process includes graying, normalizing and performing image enhancement operations;

[0012] Step 1.3, image generation: In this stage, a generative adversarial network (GANs) combined with an attention mechanism is used. The input signal of the generator is a random noise vector, and the output is an image simulating metal surface defects. In this process, the GANs network is continuously iterated to optimize its network parameters.

[0013] Step 1.4, Dataset Generation and Segmentation: The iteratively tuned generator is used to generate defect images. These generated images will be merged with the original dataset, and the expanded dataset will be divided into training set, validation set and test set;

[0014] Step 1.5, model selection and training: ResNet-50 residual network was selected as the basic model, and a mapping method combining multiple loss functions was adopted to improve accuracy and robustness, and the test set was used for accuracy testing.

[0015] Furthermore, the step 1.3 specifically includes the following:

[0016] A generative adversarial network with a dual attention mechanism of channel and space is adopted. The generator and discriminator are initialized for adversarial training, and an attention mechanism is introduced to focus on the key areas of the metal surface. At the same time, a loss function is defined and the network parameters are iteratively optimized to minimize the loss and improve the quality of the generated images.

[0017] Further, the so-called Generative Adversarial Networks (GANs) combined with the attention mechanism, first we need to calculate the similarity score between each word in the text description and different sub-regions in the image. The word attention allows the model to focus on the image areas related to different words in the text description when generating images, and it is done in the following way:

[0018] First, we need to calculate the similarity scores between the words in each text description and the different sub-regions in the image. For each word i, calculate the similarity score e i , which is expressed as the similarity between the word and different sub-regions of the image. Here, the cosine similarity function is used to measure it. The cosine similarity function expression is as follows:

[0019]

[0020] a·b represents the dot product (inner product) of vector a and vector b, |a| and |b| represent the norm (length) of vector a and vector b respectively;

[0021] We then use the similarity score to calculate the attention weight α for each word i , the attention weight indicates that the model pays attention to the importance of each word in the text description for generating the image. Here, the softmax function is used to normalize the similarity score. The softmax function expression is as follows:

[0022]

[0023] n represents the number of words in the text description;

[0024] Then, compression-excitation attention is an attention mechanism for capturing nonlinear interactions between features. It is used to process image features and map the vector to a new vector z through a fully connected layer and a ReLU activation function, as shown in formula (3):

[0025] z=ReLU(W1x+b1) (12)

[0026] W1 is the compressed weight matrix, ReLU is the activation function, and b1 is the bias term;

[0027] Next, we perform an activation operation on the compressed representation z to capture the nonlinear relationship between different channels. This is achieved through matrix multiplication and nonlinear activation function, as shown in formula (4):

[0028] y=σ(W2f(z)) (13)

[0029] Here, W2 is the weight matrix of the excitation, f is the excitation function, and σ is the normalization function;

[0030] Word attention calculates the similarity score between text words and image regions and calculates the attention weight through the softmax function. The compression-excitation attention captures the nonlinear relationship between image features through compression and excitation operations. These attention mechanisms can work together to generate more realistic images.

[0031] Furthermore, the residual network specifically includes the following steps:

[0032] Step 1.5.1, feature extraction: In this stage, the data and images are processed through convolution and pooling layers to extract features and generate corresponding feature maps;

[0033] Step 1.5.2, Stacking Residual Blocks: This method introduces multiple residual blocks, each of which contains two convolutional layers, one for extracting features and the other for processing and transmitting the extracted features;

[0034] Step 1.5.3, Global Average Pooling: In this stage, the feature map is converted into a vector representation with a fixed size through a global average pooling operation;

[0035] Step 1.5.4, Fully Connected Layer Mapping: Through the fully connected layer, the extracted features are mapped to the final category prediction, and a multi-combination loss function is used to generate a category probability distribution to more accurately represent the predictions of different categories;

[0036] Step 1.5.5: In the training phase of this method, the training set is used to train the model, the validation set is used for hyperparameter adjustment and model overfitting check, and finally, the test set is used to comprehensively evaluate the model performance.

[0037] Furthermore, the so-called multi-combination loss function combines the Softmax loss function with the Center loss function and the Triplet loss function;

[0038] First, the training set data is input into the ResNet network for training. Here we assume that the training set contains N samples, each sample is represented by X i , where i = 1, 2, ..., N;

[0039] Calculate the Softmax loss function. The Softmax loss function is usually used for classification tasks of deep networks. For each sample X i , the Softmax expression is as follows:

[0040]

[0041] Among them, W j is the weight parameter of the network, C is the number of classification categories, y i is a sample X iThe true category label of

[0042] Calculate the Center loss function: The goal of the Triplet loss function is to cluster feature vectors of the same category around a central point. For each sample X i , Center loss expression is as follows (6):

[0043]

[0044] Among them, c yi is the sample x i Category i The center point of

[0045] Calculate the Triplet loss function: The goal of the Triplet loss function is to increase the distance between samples of different categories and reduce the distance between samples in the same category. For each sample x i , select a positive sample x i+ and a negative sample x i- , the triplet loss expression is as follows:

[0046]

[0047] Among them, f(x) represents the feature vector of sample x in the network, and α is a positive constant that controls the boundary;

[0048] Calculate the total loss: Combine Softmax loss, Center loss and Triplet loss. The total loss function expression is as follows:

[0049] L=L softmax +λ1·L center +λ2·L triplet (8)

[0050] Among them, λ1 and λ2 are hyperparameters that control the relative importance of each loss function. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is the overall flow chart of the present invention.

[0052] Figure 2 This is a metal defect image captured by the camera of the present invention.

[0053] Figure 3 This is the image after preprocessing by the present invention.

[0054] Figure 4 A random noise image input to the generator of the present invention.

[0055] Figure 5 It is a false image generated by multiple iterations of the present invention. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0057] like Figure 1 As shown, a metal surface defect detection method based on adversarial and residual network in an embodiment of the present invention includes the following steps:

[0058] Step 1.1, data collection and preparation: Use camera equipment to collect image data sets. The collected raw data sets will undergo a series of data cleaning, label encoding and label verification. In order to verify the image processing effect of the model of the present invention, firstly, real metal surface defect images are obtained, and then the adversarial network is trained and learned on the real images. In the embodiment of the present invention, real metal surface defects are taken as an example. The obtained raw data sets are as follows: Figure 2 As shown;

[0059] Step 1.2, Image preprocessing: Perform image preprocessing, grayscale converts the color value of each pixel into the corresponding grayscale level, gradient from black to white, and uniformly sets the size to 200*200, making subsequent processing more intuitive and effective. Normalization ensures that all input images have a similar brightness range, thereby improving the stability of subsequent algorithms and facilitating subsequent analysis and classification, such as Figure 3 As shown, the specific steps include:

[0060] Step 1.2.1, grayscale the acquired image;

[0061] Step 1.2.2: Unify the image size and normalize the brightness range of all input images;

[0062] Step 1.2.3: Image enhancement improves the color, brightness and contrast of the image.

[0063] Step 1.3, image generation: In this stage, a generative adversarial network with channel and spatial dual attention mechanism is adopted. The generator and discriminator are initialized for adversarial training, and the attention mechanism is introduced to focus on the key areas of the metal surface. At the same time, the loss function is defined and the network parameters are iteratively optimized to minimize the loss and improve the quality of the generated image.

[0064] The attention mechanism is a method used in deep learning models that mimics human attention patterns. In image generation tasks, the attention mechanism allows the model to better focus on specific areas or features related to the generated image, thereby improving the quality and accuracy of the generated image. The Generative Adversarial Network is a deep learning framework for generating data. GAN continuously optimizes the quality of generated defective images through adversarial training of the generator and the discriminator.

[0065] In the quality inspection process of actual industrial products, the noise and interference of the target background are complex, and the features such as plaques, inclusions, defects, etc. on the metal surface are very variable. In order to adapt to different features, the present invention selects a generative adversarial network (GANs) model combined with an attention mechanism, combines the advantages of the attention mechanism generation network, calculates the similarity score between each word in the text description and different sub-regions in the image, and the word attention enables the model to focus on the image area related to different words in the text description when generating the image, and is completed in the following way:

[0066] First, we need to calculate the similarity scores between the words in each text description and the different sub-regions in the image. For each word i, calculate the similarity score e i , which is expressed as the similarity between the word and different sub-regions of the image. Here, the cosine similarity function is used to measure it. The cosine similarity function expression is as follows:

[0067]

[0068] a·b represents the dot product (inner product) of vector a and vector b, |a| and |b| represent the norm (length) of vector a and vector b respectively;

[0069] We then use the similarity score to calculate the attention weight α for each word i , the attention weight indicates that the model pays attention to the importance of each word in the text description for generating the image. Here, the softmax function is used to normalize the similarity score. The softmax function expression is as follows:

[0070]

[0071] n represents the number of words in the text description;

[0072] Then, compression-excitation attention is an attention mechanism for capturing nonlinear interactions between features. It is used to process image features and map the vector to a new vector z through a fully connected layer and a ReLU activation function, as shown in formula (3):

[0073] z=ReLU(W1x+b1) (3)

[0074] W1 is the compressed weight matrix, ReLU is the activation function, and b1 is the bias term;

[0075] Next, we perform an activation operation on the compressed representation z to capture the nonlinear relationship between different channels. This is achieved through matrix multiplication and nonlinear activation function, as shown in formula (4):

[0076] y=σ(W2f(z)) (4)

[0077] Here, W2 is the weight matrix of the excitation, f is the excitation function, and σ is the normalization function;

[0078] Step 1.4, Dataset Generation and Segmentation: After the GANs generator has been iterated for many times and has reached the predetermined number of times and performance indicators, it can start generating defect images. The generator takes random noise as input, such as Figure 4 As shown, and under appropriate conditions, high-quality, realistic defect images are generated, such as Figure 5 As shown; these images will become part of the augmented dataset, providing more training samples for our metal defect detection model, and the generated images are merged with the original dataset to ensure the diversity and coverage of the training data. By mixing the generated defect images with the real defect images, we can better train the model to have the ability to recognize a wider range of defect features, including the following steps:

[0079] Step 1.4.1. Label assignment: In order to verify the generated images, the correct labels need to be assigned by ensuring that the generator generates images under the correct conditions so that they can be correctly labeled as “generated” rather than real defective images;

[0080] Step 1.4.2, Sample quality assessment: Before merging the generated images and the original dataset, the generated images must be evaluated for quality; this includes checking the image clarity, realism, and whether it has features similar to real defects; low-quality generated images will have a negative impact on the training of the model, so they need to be screened in the dataset;

[0081] Step 1.4.3. Data balance: Ensure that the number of generated images and real defect images is balanced in the dataset to prevent the model from being biased towards any one category.

[0082] Step 1.4.4, data division: In order to make full use of the data and reduce overfitting and underfitting, the original data set is divided into training set, validation set and test set;

[0083] Step 1.5, model selection and training: Select the ResNet-50 model for training and testing on the expanded data set, and use a mapping method combining multiple loss functions to improve the performance and generalization ability of the model, so as to better apply it to actual scenarios. The step 1.5 specifically includes the following contents:

[0084] Step 1.5.1, feature extraction: The input image passes through a series of convolutional layers and pooling layers, gradually reducing the size and increasing the number of channels. These convolutional layers are used to extract features from the input image to capture low-level and high-level features of the image.

[0085] Step 1.5.2, Stacking Residual Blocks: The core idea of ​​ResNet is to solve the gradient vanishing problem in deep neural networks by introducing residual blocks. Each residual block consists of two main branches: an identity mapping branch and a main branch containing a series of convolutional layers; the output of the main branch is added to the output of the identity mapping branch instead of replacing it; ResNet usually contains multiple such residual blocks for effectively stacking features;

[0086] Step 1.5.3, Global Average Pooling: In traditional convolutional neural networks, the Flatten operation is usually used before the fully connected layer to flatten the features. However, in ResNet, a global average pooling layer is used instead of Flatten; this means that in global average pooling, all elements on each feature map are averaged, thereby converting the entire feature map into a single value; this helps reduce the number of parameters, reduce the risk of overfitting, and improve the generalization ability of the model;

[0087] Step 1.5.4, Fully Connected Layer Mapping: Finally, a mapping method combining multiple loss functions is used to improve the generalization and robustness of the model, so that the GANs network can generate more realistic pictures, and this is done in the following way:

[0088] First, the training set data is input into the ResNet network for training. Here we assume that the training set contains N samples, each sample is represented by X i , where i = 1, 2, ..., N;

[0089] Calculate the Softmax loss function. The Softmax loss function is usually used for classification tasks of deep networks. For each sample X i , the Softmax expression is as follows:

[0090]

[0091] Among them, W j is the weight parameter of the network, C is the number of classification categories, y i is a sample X i The true category label of

[0092] Calculate the Center loss function: The goal of the Center loss function is to cluster feature vectors of the same category around a central point, where is the sample x i Category i The center point of each sample X i , Center loss expression is as follows (6):

[0093]

[0094] Calculate the Triplet loss function: The goal of the Triplet loss function is to increase the distance between samples of different categories and reduce the distance between samples in the same category. For each sample x i , select a positive sample x i+ and a negative sample x i- , the triplet loss expression is as follows:

[0095]

[0096] f(x) represents the feature vector of sample x in the network, and α is a positive constant that controls the boundary;

[0097] The total loss is calculated by combining Softmax loss, Center loss and Triplet loss. The total loss function expression is as follows:

[0098] L=L softmax +λ1·L center +λ2·L triplet (8)

[0099] Among them, λ1 and λ2 are hyperparameters that control the relative importance of each loss function. By minimizing this total loss function, the model can be constrained by these three losses at the same time, so as to better learn the characteristics of metal surface defects during training and improve classification accuracy and robustness.

[0100] Step 1.5.5. The improved model was tested on the test data set, and the accuracy achieved was over 85%, indicating that our improved ResNet model can still accurately perform classification, prediction, detection and other tasks when processing small-scale data; through the performance on the test set, we further verified the generalization ability of the model, and the model can process diverse data samples, not just examples in the training set, which shows that our model has made some progress in the overfitting problem and has good potential in practical applications. The test set prediction results are shown in Table 1.

[0101] Table 1 Prediction results of 100 real samples

[0102]

Claims

1. A metal surface defect detection method based on adversarial and residual networks, characterized in that: The specific steps include: Step 1.1, data collection and preparation: Use camera equipment to collect image data sets. The collected raw data sets will go through a series of data cleaning, label encoding and label verification; Step 1.2, image preprocessing: preprocess the uploaded image, the preprocessing process includes graying, normalizing and performing image enhancement operations; Step 1.3, image generation: In this stage, a generative adversarial network (GANs) combined with an attention mechanism is used. The input signal of the generator is a random noise vector, and the output is an image simulating metal surface defects. In this process, the GANs network is continuously iterated to optimize its network parameters. Step 1.4, Dataset Generation and Segmentation: The generator after iterative parameter adjustment is used to generate defect images. These generated images will be merged with the original dataset, and the expanded dataset will be divided into training set, validation set and test set; Step 1.5, model selection and training: ResNet-50 residual network was selected as the basic model, and a mapping method combining multiple loss functions was adopted to improve accuracy and robustness, and the test set was used for accuracy testing.

2. The metal surface defect detection method based on adversarial and residual network according to claim 1 is characterized in that: The step 1.3 combines the attention mechanism with the generative adversarial network (GANs), specifically including the following: A generative adversarial network with a dual attention mechanism of channel and space is adopted. The generator and discriminator are initialized for adversarial training, and an attention mechanism is introduced to focus on the key areas of the metal surface. At the same time, a loss function is defined and the network parameters are iteratively optimized to minimize the loss and improve the quality of the generated images.

3. The metal surface defect detection method based on adversarial and residual network according to claim 2 is characterized in that: The so-called Generative Adversarial Networks (GANs) combined with the attention mechanism, first we need to calculate the similarity score between each word in the text description and different sub-regions in the image. The word attention allows the model to focus on the image areas related to different words in the text description when generating images, and it is done in the following way: First, we need to calculate the similarity scores between the words in each text description and the different sub-regions in the image. For each word i, calculate the similarity score e i , which represents the similarity between the word and different sub-regions of the image. Here, the cosine similarity function is used to measure it. The cosine similarity function expression is as shown in formula (1); a·b represents the dot product (inner product) of vector a and vector b, |a| and |b| represent the norm (length) of vector a and vector b respectively; We then use the similarity score to calculate the attention weight α for each word i , the attention weight indicates that the model pays attention to the importance of each word in the text description for generating the image. Here, the softmax function is used to normalize the similarity score. The softmax function expression is as shown in formula (2); n represents the number of words in the text description; Then, compression-excitation attention is an attention mechanism for capturing nonlinear interactions between features. It is used to process image features and maps the vector s to a new vector z through a fully connected layer and a ReLU activation function, as shown in formula (3): z=ReLU(W1x+b1) (3) W1 is the compressed weight matrix, ReLU is the activation function, and b1 is the bias term; Next, we perform an activation operation on the compressed representation z to capture the nonlinear relationship between different channels; this is achieved through matrix multiplication and nonlinear activation function, as shown in formula (4): y=σ(W2f(z))(4) Here, W2 is the weight matrix of the excitation, f is the excitation function, and σ is the normalization function; Word attention calculates the similarity score between text words and image regions and calculates the attention weight through the softmax function. The compression-excitation attention captures the nonlinear relationship between image features through compression and excitation operations. These attention mechanisms can work together to generate more realistic images.

4. The metal surface defect detection method based on adversarial and residual network according to claim 1 is characterized in that: The residual network specifically includes the following contents: Step 1.5.1, feature extraction: In this stage, the data and images are processed through convolution and pooling layers to extract features and generate corresponding feature maps; Step 1.5.2, Stacking Residual Blocks: This method introduces multiple residual blocks, each of which contains two convolutional layers, one for extracting features and the other for processing and transmitting the extracted features; Step 1.5.3, Global Average Pooling: In this stage, the feature map is converted into a vector representation with a fixed size through a global average pooling operation; Step 1.5.4, Fully Connected Layer Mapping: Through the fully connected layer, the extracted features are mapped to the final category prediction, and a multi-combination loss function is used to generate a category probability distribution to more accurately represent the predictions of different categories; Step 1.5.5: In the training phase of this method, the training set is used to train the model, the validation set is used for hyperparameter adjustment and model overfitting check, and finally, the test set is used to comprehensively evaluate the model performance.

5. The metal surface defect detection method based on adversarial and residual network according to claim 4 is characterized in that: The so-called multi-combination loss function combines the Softmax loss function with the Centerloss function and the Tripletloss function. First, the training set data is input into the ResNet network for training. Here we assume that the training set contains N samples, each sample is represented by X i , where i = 1, 2, ..., N; Calculate the Softmax loss function. The Softmax loss function is usually used for classification tasks of deep networks. For each sample X i , the Softmax expression is as follows: Among them, W j is the weight parameter of the network, C is the number of classification categories, y i is a sample X i The true category label of Calculate the Centerloss function: The goal of the Centerloss function is to gather feature vectors of the same category around a central point, where is the sample x i Category i The center point of each sample X i , Centerloss expression is as follows: Calculate the Tripletloss function: The goal of the Tripletloss function is to increase the distance between samples of different categories and reduce the distance between samples in the same category. For each sample x i , select a positive sample x i+ and a negative sample x i- , Tripletloss expression is as follows (7): f(x) represents the feature vector of sample x in the network, and α is a positive constant that controls the boundary; The total loss is calculated by combining Softmax loss, Centerloss and Tripletloss. The total loss function expression is as follows: L=L softmax +λ1·L center +λ2·L triplet (9) Among them, λ1 and λ2 are hyperparameters that control the relative importance of each loss function.

Images