A strip steel defect detection method based on deep learning

Abstract

The present invention discloses a strip defect detection method based on deep learning. The encoder is based on U‑Net. The encoder extracts features in the image through repeated convolution operations, and performs five downsampling operations to extract features of different scales. The decoder part is symmetrical with the encoder. It receives the output of the encoder and performs five upsampling operations, and the result obtained by each upsampling will be spliced ​​with the feature map of the same scale of the encoder. A Gaussian low-pass filter is used before downsampling to remove high-frequency information therein to prevent aliasing due to the high original frequency of the feature map, which leads to deviation in the feature map. The removed high-frequency information contains detailed information of the original image, which is crucial to the detection of small defects in the strip, and directly discarding it will affect the effect. This method further adds these high-frequency information back to the feature map of the model decoder to produce a more refined segmentation effect.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology and defect detection, and mainly involves some digital image processing, image enhancement, image semantic segmentation methods, deep learning image segmentation methods, etc. Background Art

[0002] Since the Industrial Revolution, the production capacity of steel materials has begun to increase rapidly, and has gradually become one of the most widely used materials in human society. In modern times, the strength of a country's steel industry is the basic material guarantee for its rapid development in the 21st century. Steel materials play an indispensable role in many aspects such as national industrial needs, housing needs, and transportation needs. Strip steel is one of the important output items of steel companies. Strip steel is a narrow and long steel plate that is widely used in automobiles, aerospace, machinery manufacturing, construction, daily hardware, etc. It is a steel material with large output, wide uses, and many types. The quality of strip steel has a great impact on the performance and quality of the final product. With the development of the times, the quality requirements of various merchants for strip steel are also constantly increasing, and the competition in the industry is becoming increasingly fierce. For steel companies, in order to ensure the production quality of steel, it is essential to improve the level of quality inspection.

[0003] Among the traditional strip defect detection methods, manual visual inspection is an effective method, but it is labor-intensive and the production environment of the strip is harsh, which can easily cause great physical and psychological harm to the workers. Among the current automated detection methods, eddy current detection is a very common method. The defect detection method of this method mainly uses a high-frequency induction coil to generate an induced current in the strip. As long as the surface temperature change is detected, the surface defects of the strip can be detected. However, this method has high requirements on the surface state of the object being tested, and the detection effect on some types of defects is not good.

[0004] In addition, with the development of image acquisition technology, computer vision-based methods have also been put into use. This method mainly scans the surface of the strip steel sample into a digital image for further inspection. In the field of computer digital image processing, the problem of strip steel defect detection can be regarded as a semantic segmentation problem of the image, that is, each pixel in the image is classified into a category and the defect area is segmented. The traditional image semantic segmentation method is mainly based on the material, texture, color and other features of the image, combined with professional knowledge, to manually design the image feature extraction method, and use support vector machine (SVM), random forest, AdaBoost and other classification algorithms for classification. Obviously, for complex image scenes, the effect of manually designed feature extraction methods is not ideal.

[0005] With the continuous improvement of computer computing power and the continuous accumulation of machine learning algorithms, the relevant technologies of deep learning have been greatly developed. There have been related attempts and applications of deep learning in the field of images a long time ago. In 1998, Lecun et al. proposed a neural network combined with convolution operations, which was called a convolutional neural network (CNN). At first, CNN achieved good results in simple small-size image scenes, but it was difficult to process large images. It was not until 2012 that Krizhevsky et al. constructed a deeper network structure AlexNet, which made great progress on the famous ImageNet dataset, which made deep learning research in the field of image recognition move forward significantly. In 2015, He et al. proposed a deep residual network (ResNet). ResNet introduced a skip connection structure, which allowed the module input to be directly added to the module output, avoiding the degradation of the model. The network depth has therefore reached an unprecedented 152 layers. This skip connection structure has basically become the basic unit of CNN design in the future. In the field of semantic segmentation, Long et al. designed and processed the fully convolutional neural network (FCN) in 2014. This model can directly complete end-to-end image semantic segmentation, greatly simplifying the complexity of semantic segmentation-related problems and making breakthrough progress. In the following years, other semantic segmentation models such as SegNet, U-Net, DeepLab, PSPNet, etc. were successively proposed and shined in the field of semantic segmentation.

[0006] At present, there are still some problems in using deep learning methods for strip defect detection. The defect samples of strip steel are distributed unevenly and in different shapes. Even defects of the same category can vary greatly in material and texture. This may be because the production line of strip steel is complex, the environment is changeable, and the collection cycle is long, and the defects of different batches of products vary greatly. In addition, the size of the defects may vary greatly in scale, which requires the model to have the ability to capture tiny details and the ability to obtain overall spatial information. These problems have brought great challenges to high-precision detection. Summary of the invention

[0007] Based on the above analysis, the present invention mainly designs a semantic segmentation method based on deep learning to detect strip defects.

[0008] The main architecture of the semantic segmentation model of this method is based on the encoder-decoder structure of U-Net. The encoder extracts features from the image through repeated convolution operations, and performs five downsampling operations to extract features of different scales. The decoder part is symmetrical with the encoder. It receives the output of the encoder and performs five upsampling operations. The result of each upsampling will be spliced ​​with the feature map of the same scale of the encoder. In this method, in order to make the proposed semantic segmentation model as accurate as possible after each downsampling, a Gaussian low-pass filter will be used before downsampling to remove the high-frequency information in it, so as to prevent aliasing due to the high original frequency of the feature map, which will cause deviation in the feature map. At the same time, the removed high-frequency information contains the detailed information of the original image, which is crucial for the detection of small defects in strip steel. Directly discarding it will affect the effect. Therefore, this method further adds these high-frequency information back to the feature map of the model decoder to produce a more refined segmentation effect.

[0009] In this method, the data used for model training and testing comes from the open-source strip steel image data of Severstal, a steel company. Before training the model, the data set is first divided into a training set and a test set. The training set is used to train the model during training. In order to enhance the robustness of the model, the training set images input during training will be data augmented. In order to alleviate the problem of the imbalance in the number of various defects in the data set, the loss function of the training model is the FocalLoss loss function. Finally, the accuracy of the model is tested with the test set to detect the training effect of the model.

[0010] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme: In order to realize the method of strip defect detection, Python language is preferred for writing. The implementation and training of the model are mainly implemented using the machine learning library PyTorch. The encoder of the model adopts ResNet50 pre-trained on ImageNet. The decoder part consists of 5 consecutive decoder modules stacked together. The strip images in the data set are long strips with a resolution of 256×1600. The size of the direct input model training is too large, so it will be cropped into 4 256×400 images before training. OpenCV is used for data enhancement during training to achieve random flipping, rotation, adding Gaussian noise, histogram equalization and image sharpening of the image.

[0011] A strip defect detection method based on deep learning mainly includes:

[0012] Step 1: Collect the image dataset of the strip steel and clean the data in the dataset.

[0013] Step 2: Randomly divide the strip steel images in the dataset into training set and test set, and use the variance and mean of the ImageNet dataset for regularization.

[0014] Step 3: Build a model based on the U-Net architecture. The encoder uses the ResNet50 network pre-trained on the ImageNet dataset and inserts a Gaussian low-pass filter before each downsampling. The decoder is composed of 5 decoder modules stacked together.

[0015] Step 4: Train the model constructed in step 3. During training, the input data will be randomly enhanced. After training, test the model on the test set to check whether the training results meet the requirements.

[0016] Preferably, step 2 takes the following steps:

[0017] Step 2.1, randomly divide the data set into training set and test set;

[0018] Step 2.2, divide each strip steel image with a resolution of 256×1600 into four small images of 256×400;

[0019] Preferably, step 3 specifically comprises the following steps:

[0020] Step 3.1. Build the encoder part. Use ResNet50 trained on the ImageNet dataset as the model decoder. Insert a Gaussian low-pass filter module before each downsampling operation of ResNet50. In this module, first use a Gaussian low-pass filter to reduce the frequency of the feature map to make the downsampled feature map clearer. Then make a difference between the feature map before and after the Gaussian low-pass filter processing, calculate the lost high-frequency information, and prepare for the input of the decoder part.

[0021] Step 3.2, build the decoder part. The decoder uses the U-Net decoder structure as the main structure and consists of 5 decoder modules. These 5 decoder modules will continuously upsample the feature map output by the encoder until the feature map is restored to the original image size, and the final output result is the defect recognition result. The input of the first 4 decoder modules will be spliced ​​with the output feature map of the encoder, and the high-level and low-level semantic information will be comprehensively utilized to obtain better segmentation results. On the basis of the original U-Net decoder, the input of the first four decoder modules will also be added with the high-frequency information removed by the corresponding Gaussian low-pass filter in the encoder to avoid the loss of detail information. The last decoder is the same as the first 4 decoder modules except that it does not accept the input of the encoder.

[0022] Preferably, step 4 specifically comprises the following steps:

[0023] Step 4.1: Train the constructed model using the standard Adam optimization algorithm. The model is trained for a total of 100 epochs with a batch size of 10. The initial learning rate of the training is 0.01, which decays linearly with the epochs of training. The loss function used in the training is the FocalLoss loss function to alleviate the problem of imbalanced data sets.

[0024] Step 4.2: During training, data enhancement is performed on the input data to increase data diversity. Data enhancement is performed using one of the following strategies: no enhancement, random vertical or horizontal flipping, random rotation within 5° clockwise or counterclockwise, random Gaussian noise, histogram equalization, and image sharpening. The image is then standardized using the mean and standard deviation of the dataset.

[0025] Step 4.3: After training for 80 epochs, the model accuracy will be tested using the 5-fold cross-validation method after each epoch is trained. The model with the highest accuracy will be selected as the result.

[0026] Step 4.4: After training, verify whether the model is effective on the test set.

[0027] Compared with the prior art, the present invention has the following advantages:

[0028] The present invention transforms the traditional U-Net architecture and adds a Gaussian low-pass filter before the downsampling operation in the encoder to ensure that the downsampling result of the model is more refined and improves the accuracy of the image feature extraction by the convolution layer. At the same time, in the decoder, in addition to splicing with the feature map of the encoder, the high-frequency information removed by the Gaussian low-pass filter is added back to the feature map to prevent the loss of tiny details at different scales of the strip image. In general, the method of the present invention can detect defects in strip images more efficiently and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is the main structure diagram of the model proposed by the present invention;

[0030] Figure 2 It is a structural diagram of the encoder in the present invention;

[0031] Figure 3 The Gaussian low-pass filter module structure designed for the present invention;

[0032] Figure 4 The decoder module structure designed for the present invention; DETAILED DESCRIPTION

[0033] The present invention will be further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0034] The present invention provides a strip steel defect detection method based on deep learning, which specifically comprises the following steps:

[0035] The hardware equipment used in the present invention includes a PC and a GTX1080 graphics card;

[0036] Step 1: Collect the image dataset of the strip steel and clean the data in the dataset.

[0037] Step 2: Randomly divide the strip steel images in the dataset into training set and test set, and use the variance and mean of the ImageNet dataset for regularization.

[0038] Step 2.1, randomly divide the data set into training set and test set;

[0039] Step 2.2, divide each strip steel image with a resolution of 256×1600 into four small images of 256×400;

[0040] Step 3: Figure 1 As shown in Figure 1, the model structure is constructed, where the encoder uses the ResNet50 network pre-trained on the ImageNet dataset and inserts a Gaussian low-pass filter before each downsampling. The decoder is composed of 5 decoder modules stacked together.

[0041] Step 3.1: Build the encoder part. Use the ResNet50 trained on the ImageNet dataset as the model decoder. Figure 2 As shown in Figure 2, a Gaussian low-pass filter module is then inserted before each downsampling operation of ResNet50. The structure of the Gaussian low-pass filter module is as follows: Figure 3 As shown in the figure. This module first reduces the frequency of the feature map through a Gaussian low-pass filter to make the downsampled feature map clearer. Then, the feature map before and after the Gaussian low-pass filter is processed is subtracted to calculate the lost high-frequency information to prepare for the input of the decoder part.

[0042] Step 3.2: Construct the decoder part. The decoder uses the U-Net decoder design as the main structure, which is composed of 5 stacked decoder modules. These 5 decoder modules will continuously upsample the feature map output by the encoder until the feature map is restored to the original size. The final output result is the defect recognition result. The structure of the first 4 decoder modules is as follows: Figure 4As shown in the figure, the inputs of these four decoders will be concatenated with the feature maps output by the encoder, and the high-level and low-level semantic information will be used to obtain better segmentation results. Based on the original U-Net decoder, the inputs of the first four decoder modules will also be added with the high-frequency information removed by the corresponding Gaussian low-pass filter in the encoder to avoid the loss of detail information. The last decoder is the same as the first four decoder modules except that it does not accept the input of the encoder.

[0043] Step 4: Train the model constructed in step 3. During training, the input data will be randomly enhanced. After training, test the model on the test set to check whether the training results meet the requirements.

[0044] Step 4.1: Train the constructed model using the standard Adam optimization algorithm. The model is trained for a total of 100 epochs with a batch size of 10. The initial learning rate for training is 0.01, which decays linearly with the epochs of training. The loss function used in training is the Focal Loss loss function to alleviate the problem of imbalanced data sets.

[0045] Step 4.2: During training, data enhancement is performed on the input data to increase data diversity. Data enhancement is performed using one of the following strategies: no enhancement, random vertical or horizontal flipping, random rotation within 5° clockwise or counterclockwise, random Gaussian noise, histogram equalization, and image sharpening. The image is then standardized using the mean and standard deviation of the dataset.

[0046] Step 4.3: After training for 80 epochs, the model accuracy will be tested using the 5-fold cross-validation method after each epoch is trained. The model with the highest accuracy will be selected as the result.

[0047] Step 4.4: After training, verify whether the model is effective on the test set.

[0048] The above embodiments are only exemplary embodiments of the present invention and are not intended to limit the present invention. The protection scope of the present invention is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present invention within the essence and protection scope of the present invention, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present invention.

Claims

1. A strip defect detection method based on deep learning, characterized in that: The steps include: Step 1: Collect image data sets of steel strips, and clean the data of steel strip images in the image data sets; Step 2: Randomly divide the strip steel images in the dataset into training set and test set, and use the variance and mean of the ImageNet dataset for regularization; Step 3: Build a model based on the U-Net architecture; the encoder uses the ResNet50 network pre-trained on the ImageNet dataset and inserts a Gaussian low-pass filter before each downsampling; The decoder is composed of 5 decoder modules stacked together; Step 4: Train the model constructed in step 3; during training, the input strip image data will be randomly enhanced; After the training is completed, the model is tested on the test set to check whether the training results meet the requirements; Step 3 specifically includes the following steps: Step 3.

1. Build the encoder part; use the ResNet50 trained on the ImageNet dataset as the model decoder; insert a Gaussian low-pass filter module before each downsampling operation of ResNet50; in this module, first use the Gaussian low-pass filter to reduce the frequency of the feature map to make the downsampled feature map clearer; then make the difference between the feature map before and after the Gaussian low-pass filter processing, calculate the lost high-frequency information, and prepare for the input of the decoder part; Step 3.2, construct the decoder part; the decoder uses the U-Net decoder structure as the main structure and consists of five decoder modules; these five decoder modules will continuously upsample the feature map output by the encoder until the feature map is restored to the original image size, and the final output result is the defect recognition result; the input of the first four decoder modules will be spliced ​​with the output feature map of the encoder, and the high-level and low-level semantic information will be comprehensively utilized to obtain better segmentation results; on the basis of the original U-Net decoder, the input of the first four decoder modules will also be added with the high-frequency information removed by the corresponding Gaussian low-pass filter in the encoder to avoid the loss of detail information; the last decoder does not accept the input of the encoder.

2. The strip steel defect detection method based on deep learning according to claim 1, characterized in that: Step 2 Take the following steps: Step 2.1, randomly divide the data set into training set and test set; Step 2.2: Divide each strip image with a resolution of 256×1600 into four small images of 256×400.

3. The strip steel defect detection method based on deep learning according to claim 1, characterized in that: Step 4 specifically includes the following steps: Step 4.1: Train the constructed model using the standard Adam optimization algorithm. The model is trained for a total of 100 epochs with a batch size of 10. The initial learning rate of the training is 0.01, which decays linearly with the epochs of the training. The loss function used in the training is the FocalLoss loss function to alleviate the problem of imbalanced data set categories. Step 4.2: During training, data enhancement is performed on the input data to increase data diversity. Data enhancement is performed randomly using one of the following strategies: no enhancement, random vertical or horizontal flipping, random rotation within 5° clockwise or counterclockwise, random Gaussian noise, histogram equalization, and image sharpening. The image is then standardized using the mean and standard deviation of the data set. Step 4.3: After training for 80 epochs, the model accuracy will be tested using the 5-fold cross-validation method after each epoch is trained. The model with the highest accuracy will be selected as the result. Step 4.4: After training, verify whether the model is effective on the test set.

Images