A steel surface defect detection method based on deep learning

Abstract

The present invention discloses a steel surface defect detection method based on deep learning. The method first obtains a strip surface defect dataset and performs preprocessing. Secondly, a deep feature extraction network is constructed to perform feature extraction to obtain deep features, and a cascade feature fusion module is constructed to fuse deep features. Then, based on the output features of the corresponding stage of the deep feature extraction network, the feature attention is enhanced through the convolution attention module to obtain recalibrated features. Finally, based on the recalibrated features and the fused deep features, edge guidance and significant target fusion are performed to output the detection results. The present invention can accurately perform defect detection when processing complex defect scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and specifically relates to a steel surface defect detection method based on deep learning. Background Art

[0002] Surface defect detection refers to the process of inspecting product surfaces to identify and locate defects during the manufacturing and production process. These defects may include cracks, dents, scratches, color unevenness, bubbles, and more, significantly impacting product quality and appearance. Research in surface defect detection typically involves fields such as computer vision, image processing, and machine learning to develop automated detection systems.

[0003] Taking steel strip as an example, these defects not only affect the overall appearance of the strip, but may also affect the stability and safety of the product during long-term use, such as impairing the strip's high temperature resistance and corrosion resistance. Therefore, strict defect detection of the strip surface is crucial.

[0004] In recent years, manual defect detection, the mainstream method, has become increasingly inadequate to meet the needs of rapidly developing businesses. It is time-consuming and susceptible to subjective evaluation metrics. Currently, most businesses urgently need to adopt accurate and efficient detection methods. Automated detection methods based on machine vision are gaining increasing attention, as they effectively address the labor-intensive, inefficient, and subjective nature of manual inspection. Furthermore, the human visual attention mechanism reveals that the human visual system is capable of automatically focusing on areas of interest in a scene. Based on this, researchers have proposed numerous salient object detection models, which capture important visual information in an image for further processing and filter out redundant background interference. Inspired by this, many researchers have attempted to apply visual salient object detection technology to defect detection.

[0005] Currently, salient object detection methods are primarily categorized into two categories, based on feature extraction methods: traditional model-based methods and deep learning-based methods. The former fully exploits the inherent properties of handcrafted visual features and detects salient objects based on various assumptions or heuristic priors, such as sparse coding, manifold sorting, low-rank matrix recovery, and prior knowledge. However, these handcrafted features struggle to capture features in complex backgrounds. Therefore, deep learning-based salient object detection methods have been introduced. These methods automatically learn deep-level image features through neural networks, making them applicable to feature detection in a wide range of complex environments.

[0006] Although deep learning based saliency models have promoted the research of saliency detection, they still have shortcomings when dealing with complex defect scenarios, especially with cluttered backgrounds and noise interference. Summary of the Invention

[0007] In view of the shortcomings of the existing technology, the present invention provides a steel surface defect detection method based on deep learning

[0008] The method of the present invention comprises the following steps:

[0009] Step (1). Obtain a strip surface defect dataset and preprocess it by adding noise, flipping and normalizing.

[0010] Step (2). Construct a deep feature extraction network, extract features from the data set, and obtain deep features. The specific method is as follows:

[0011] First, a deep feature extraction network is constructed, using ResNet-34 to build three high / medium / low resolution convolutional branches. For the high-resolution convolutional branch, the 7×7 convolutional layer in ResNet-34's "conv1" is replaced with a 3×3 convolutional layer. At the same time, the final max pooling layer of "conv1" is discarded. Then, combined with "conv2_x," a new convolutional block, Conv-E1, is constructed. Following this, Conv-E2, Conv-E3, and Conv-E4 respectively employ the three residual learning blocks of ResNet-34 ("conv3_x," "conv4_x," and "conv5_x"). Furthermore, to expand the network's receptive field, a max pooling layer and three basic residual blocks (Basic Blocks) are deployed after Conv-E4. These layers constitute Conv-E5 in the encoder convolutional block. For the medium-resolution convolutional branch, a max pooling layer is first added before the first convolutional block, Conv-E1, of ResNet-34. For the low-resolution convolution branch, a maximum pooling layer is added before the first convolution block Conv-E1 of ResNet-34, and the rest of the structure is the same as the high-resolution convolution branch.

[0012] According to the three convolution branches, three deep semantic features are obtained respectively. and Afterwards, in order to aggregate these three deep semantic features, a cascaded feature fusion module was specially designed.

[0013] Step (3). Construct a cascade feature fusion module to fuse deep features. The specific method is as follows:

[0014] First, the deep features Pass it to a convolution block Conv to get enhanced depth features Right now:

[0015]

[0016] Among them, each convolution layer in Conv is followed by a batch normalization (BN) layer and a ReLU layer.

[0017] Afterwards, the enhanced depth features are transformed into Adjust to the same level by upsampling The same size. Then, these two deep features are combined and passed to a convolution block Conv. The process can be defined as:

[0018]

[0019] Among them up ×2 (.) indicates a 2x upsampling operation through bilinear interpolation, and "+" is an element-wise summation operation. Finally, the deep semantic features are And two enhanced depth features and Also passed to a convolution block Conv to obtain enhanced depth features The process can be defined as:

[0020]

[0021] Among them up ×4 (.) refers to a 4x upsampling operation using bilinear interpolation.

[0022] In this way, three high-level semantic deep features can be aggregated and Generating deep features Finally, in order to expand the receptive field of the encoder, a convolutional block ConvB (denoted by hB) is further deployed, which contains three extended convolutional layers, three batch normalization layers (BN) and three ReLU layers. In this way, the fused rich high-level deep semantic features F D Passed to the decoder. The process can be defined as:

[0023]

[0024] Step (4). Based on the output features of the corresponding stage of the deep feature extraction network, the convolutional attention CBAM module is used to enhance the feature attention and obtain the recalibrated features. The specific method is as follows:

[0025] The output features of the corresponding stage of the high-resolution convolution branch of the encoder are passed through a convolutional block attention module (CBAM) before being fused with the decoder information (concat). The input features are filtered by the channel attention module and the spatial attention module in sequence, and finally the recalibrated features are obtained. The purpose is to emphasize important features and compress unimportant features. This process can be defined as:

[0026]

[0027] Where CBAM represents the convolutional attention module, FC i Represents the output of the i-th CBAM module.

[0028] Step (5). Based on the recalibrated features and the fused deep features, edge guidance and salient target fusion are performed to output the detection results. The specific method is as follows:

[0029] For the decoder network, there are 5 convolutional blocks Each convolutional block consists of three convolutional layers, followed by a batch normalization layer and a ReLU layer. The input of each stage is the concatenation of the upsampled output of the previous stage and the output of the corresponding stage in the encoder. In this way, we can get the input of the i-th decoder block Conv-Di. The process can be defined as:

[0030]

[0031] where f i D Denotes the i-th decoder block Conv-Di, F i D is the output of Conv-Di.

[0032] Then, the edge extraction branch and the salient object prediction branch are deployed in the side path of each decoder block. Specifically, these two operations are to extract the deep features. Processing, namely:

[0033]

[0034] Among them, f s is the function of the salient target prediction branch, A i is the output of the ith salient target prediction branch, i.e., the attention map, f e is the function of the edge extraction branch, E iis the output of the i-th edge extraction branch, i.e., the edge information. Here, both the salient object prediction branch and the edge extraction branch consist of a 3×3 convolutional layer. In addition, to provide deep supervision for the salient object prediction branch and the edge extraction branch, we deploy upsampling operations after these branches.

[0035] Then the side outputs of all salient object prediction branches are merged, namely the salient object map At the same time, the first edge information E1 is selected to participate in the salient target fusion process. In addition, it should be noted that in the salient target fusion, both the attention map and the edge information are not processed by the Sigmoid activation function. In addition, in order to concatenate the attention map and the edge information, the attention maps A2, A3, A4, and A5 are first adjusted to the same size through an upsampling operation. Finally, under the guidance of the edge information E1, the salient target fusion can be defined as

[0036] S=f[E1,A1,up ×2 (A2), up ×4 (A3), up ×8 (A4), up ×16 (A5)]

[0037] Where S is the final salient object map, [,] represents the concat operation, and f represents the convolution operation and the sigmoid activation function. This produces a high-quality salient object map with good boundary details, effectively highlighting defective areas on the strip surface.

[0038] The beneficial effects of the present invention are as follows:

[0039] The main advantages of the method of the present invention are three aspects:

[0040] 1. In the encoder part, a multi-scale strategy is adopted to extract feature information.

[0041] 2. Introduce convolutional attention module to emphasize important features.

[0042] 3. In the side path of each decoder block, an edge supervision branch is deployed to optimize edge extraction.

[0043] The entire network has a U-shaped encoder-decoder structure. First, the encoder deploys three convolutional branches (with a ResNet-34 backbone) to extract multi-scale features: low, medium, and high. A cascaded feature fusion module then generates deep semantic features, which are then transmitted to the decoder. In the decoder, the input of each stage is a fusion of the upsampled output of the previous stage and the features output by the corresponding encoder stage. Furthermore, the output features of the corresponding encoder stage pass through a CBAM convolutional attention module before fusion. A salient object supervision branch and an edge supervision branch are deployed simultaneously in the side path of each decoder block to optimize salient object prediction and edge extraction. Finally, the salient object predictions of all decoder blocks are fused with the edge information generated by the first decoder block to generate the final salient object prediction map. This enables accurate defect detection in complex defect scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a framework diagram of the method of the present invention;

[0045] Figure 2 Schematic diagram of the cascade feature fusion module in the method of the present invention;

[0046] Figure 3 This is a result diagram of the method of the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further described below with reference to the accompanying drawings.

[0048] Step (1). Dataset preprocessing, the specific method is as follows:

[0049] The public strip steel surface defect dataset SD-saliency-900 was selected as a dataset to verify the effectiveness of the proposed method. SD-saliency-900 consists of 900 images, including three types of strip steel surface defects: impurities, plaques, and scratches, with 300 images for each defect at a resolution of 200×200. A training set of 1,620 images was first generated. Specifically, 180 images were selected from each defect type, resulting in an initial training set of 540 images. Then, 90 images of each defect type were further selected from the initial training set and salt and pepper noise (ρ = 20%) was added to generate a noise-corrupted training set of 270 images. The initial training set and the noise-corrupted training set were combined to obtain a final training set containing a total of 810 images. After this, horizontal flipping was performed to enhance the training set, resulting in 1,620 images. Finally, during the training phase, each image I was resized to 256×256 and then normalized. In addition, because edge supervision is introduced, the Sobel operator is used to obtain 1620 edge maps of the GT image.

[0050] Step (2). Feature extraction, the specific method is as follows:

[0051] First, we build a deep feature extraction network and use ResNet-34 to build three high / medium / low resolution convolution branches. First, for the high-resolution convolution branch, that is, Figure 1 In the blue branch, the convolution layer with a 7×7 kernel (channel=64, stride=2) in the “conv1” of ResNet-34 is replaced by a convolution layer with a 3×3 kernel (channel=64, stride=1). At the same time, the last maximum pooling layer of “conv1” is discarded. Then, combined with “conv2_x”, a new convolution block Conv-E1 is constructed. After that, Conv-E2, Conv-E3, and Conv-E4 respectively adopt the three residual learning blocks (“conv3_x”, “conv4_x”, and “conv5_x”) of ResNet-34. In addition, in order to expand the receptive field of the network, a maximum pooling layer (kernel size=2×2, stride=2, padding=0) and three basic Basic Blocks (channel=512) are deployed after Conv-E4. These layers constitute Conv-E5 in the encoder convolution block. For the medium-resolution convolution branch, that is, Figure 1 For the orange branch of , we first add a maximum pooling layer (stride = 2) before the first convolution block Conv-E1. For the low-resolution convolution branch, that is, Figure 1In the green branch, a maximum pooling layer (stride = 4) is added before the first convolution block Conv-E1, and the rest of the structure is the same as the high-resolution convolution branch.

[0052] According to this architecture, high-level deep semantic features can be obtained and Afterwards, in order to aggregate these three deep semantic features, a cascaded feature fusion module is specially designed, such as Figure 2 shown.

[0053] Step (3). Fusion of deep features. The specific method is as follows:

[0054] First, the deep features Passed to a convolution block Conv (i.e. three convolution layers) to obtain enhanced depth features Right now:

[0055]

[0056] Among them, each convolutional layer in Conv (kernel size = 3×3, stride = 1) is followed by a batch normalization (BN) layer and a ReLU layer.

[0057] Afterwards, the enhanced depth features are transformed into Adjust to the same level by upsampling The same size. Then, these two deep features are combined and passed to a convolution block Conv. The process can be defined as:

[0058]

[0059] Among them up ×2 (.) indicates a 2x upsampling operation through bilinear interpolation, and "+" is an element-wise summation operation. Finally, the deep semantic features are And two enhanced depth features and Also passed to a convolution block Conv to obtain enhanced depth features The process can be defined as:

[0060]

[0061] Among them up ×4 (.) refers to a 4× upsampling operation using bilinear interpolation.

[0062] In this way, we can aggregate three high-level semantic deep features and Generating deep features Finally, in order to expand the receptive field of the encoder, we further deploy a convolutional block ConvB (denoted by hB), which contains three expanded convolutional layers (kernel size = 3×3, stride = 1, dilation = 2, channel = 512), three batch normalization layers (BN) and three ReLU layers. In this way, we can finally obtain the fused rich high-level deep semantic features F D Passed to the decoder. The process can be defined as:

[0063]

[0064] Step (4). Convolutional attention module, the specific method is as follows:

[0065] The output features of the corresponding stage of the high-resolution convolution branch of the encoder are passed through a CBAM (Convolutional Block Attention Module) module before being fused with the decoder information, i.e., concat. The input features are filtered by the channel attention module and the spatial attention module in sequence, and finally the recalibrated features are obtained. The purpose is to emphasize important features and compress unimportant features. This process can be defined as:

[0066]

[0067] Where CBAM represents the convolutional attention module, FC i Represents the output of the i-th CBAM module.

[0068] The CBAM module multiplies the output features of the channel attention module and the spatial attention module element-wise to produce the final attention-enhanced features. The convolutional attention module first uses a convolutional layer to extract features from the input feature map. A learnable convolution kernel then assigns different weights to different regions of the feature map. Finally, the weighted feature maps are fused together through a weighted summation to produce the final output feature map. The spatial attention module utilizes a self-attention mechanism to calculate the relationship between each position in the input feature map and all other positions. The features of each position are then updated based on these relationships. The feature updates for each position are weighted based on its similarity or correlation with other positions.

[0069] Step (5): Edge guidance and salient object fusion. The specific method is as follows:

[0070] For the decoder network, there are 5 convolutional blocks Each convolutional block consists of three convolutional layers, followed by a batch normalization layer and a ReLU layer. The input of each stage is the concatenation (C) of the upsampled output of the previous stage and the output of the corresponding stage in the encoder. In this way, we can get the input of the i-th decoder block Conv-Di. The process can be defined as:

[0071]

[0072] in represents the i-th decoder block Conv-Di, is the output of Conv-Di.

[0073] Then, the edge extraction branch and the salient object prediction branch are deployed in the side path of each decoder block. Specifically, these two operations are to extract the deep features. Processing, namely:

[0074]

[0075] Among them, f s is the function of the salient target prediction branch, A i is the output of the ith salient target prediction branch, i.e., the attention map, f e is the function of the edge extraction branch, E i is the output of the i-th edge extraction branch, i.e., edge information. Here, both the salient object prediction branch and the edge extraction branch are a 3×3 convolutional layer. In addition, in order to provide in-depth supervision for the salient object prediction branch and the edge extraction branch, we deploy upsampling operations after these branches. Figure 1 The orange and blue double-headed arrow shown in .

[0076] Afterwards, we attempt to merge the side outputs of all salient object prediction branches, namely the salient object map At the same time, the first edge information E1 is selected to participate in the salient target fusion process. In addition, it should be noted that in the salient target fusion, both the attention map and the edge information are not processed by the Sigmoid activation function. In addition, in order to concatenate the attention map and the edge information, the attention maps A2, A3, A4, and A5 are first adjusted to the same size through an upsampling operation. Finally, under the guidance of the edge information E1, the salient target fusion can be defined as

[0077] S=f[E1,A1,up ×2 (A2), up ×4 (A3), up ×8 (A4), up ×16 (A5)]

[0078] Where S is the final salient object map, [,] represents the concat operation, and f represents the convolution operation and the sigmoid activation function. This produces a high-quality salient object map with good boundary details, effectively highlighting defective areas on the strip surface.

[0079] The present invention deploys supervision to the salient object prediction branch and the edge extraction branch, i.e. and In addition, the cascade feature fusion module CFFM module and the final output of the entire network are supervised and defined as and ls6. Therefore, the total loss L of the entire network can be defined as

[0080]

[0081] The three loss functions are defined using mixed loss. For example, the rest are similar, it can be defined as

[0082]

[0083] in and They represent BCE loss, IoU loss and SSIM loss respectively.

[0084] Binary classification tasks usually adopt BCE (binary cross entropy) loss, which can be written as:

[0085]

[0086] Among them, l B , GT and S represent the BCE loss, ground truth and predicted saliency map, respectively.

[0087] IoU loss is often used to evaluate the similarity between GT and S, which can be written as:

[0088]

[0089] where l I is the IoU loss.

[0090] For SSIM loss, l S It can be defined as:

[0091]

[0092] in Represents P S and P GT The mean and standard deviation of represents the covariance of the two patches, C u and Cσ Usually set to 0.01 2 and 0.0 2 .

[0093] Step (6). Result error analysis, the specific method is as follows:

[0094] like Figure 3 As shown in the figure, it can be seen that when facing scenes with obvious impurities, spots, and scratches, this method can well detect significant targets. Whether it is a small target or the integrity of the object, the edge contour can be well detected. Figure 3 For the impurity defects in the first column and the scratch defects in the sixth column, when facing small salient target objects, the salient target map predicted by this method can better separate the object and the background. Figure 3 In the fourth column, for the scene with large speckle defects, the salient target map predicted by this method also has good results, and the object can be completely segmented, and the prediction result is relatively clear. Figure 3 The second column shows impurity defects, the third column shows spot defects, and the fifth column shows scratch defects. In terms of edge contours, this method can accurately detect all three defects, and the results are very close to the true value images.

[0095] To quantitatively evaluate the results, this method uses five evaluation metrics: mean absolute error (MAE), weighted F-measure (WF), overlap ratio (OR), structure metric (SM), and Pratt metric (PFOM). The above metrics for this method are 0.0119, 0.9262, 0.8448, 0.9418, and 0.9195, respectively, demonstrating the effectiveness of this method.

Claims

1. A steel surface defect detection method based on deep learning, characterized in that: The following steps are involved: Step 1. Obtain a steel strip surface defect dataset and preprocess it by adding noise, flipping, and normalizing it. Step 2. Build a deep feature extraction network, extract features from the data set, and obtain deep features; The deep feature extraction network uses ResNet-34 to construct three high / medium / low resolution convolution branches; For the high-resolution convolution branch, the convolution layer with a 7×7 kernel in conv1 of ResNet-34 is replaced by a convolution layer with a 3×3 kernel; at the same time, the last maximum pooling layer of conv1 is discarded; then, combined with conv2_x, a new convolution block Conv-E1 is constructed; after that, Conv-E2, Conv-E3, and Conv-E4 respectively adopt the three residual learning blocks of ResNet-34: conv3_x, conv4_x, and conv5_x; in addition, a maximum pooling layer and three residual blocks are deployed after Conv-E4 to constitute Conv-E5 in the encoder convolution block; For the medium-resolution convolution branch, a maximum pooling layer is added before the first convolution block Conv-E1 of ResNet-34; For the low-resolution convolution branch, a maximum pooling layer is added before the first convolution block Conv-E1 of ResNet-34, and the rest of the structure is the same as the high-resolution convolution branch; According to the three convolution branches, three deep semantic features are obtained respectively. and Step 3. Construct a cascade feature fusion module to fuse deep features; The specific implementation process of the cascade feature fusion module is as follows: The deep features Pass it to a convolution block Conv to get enhanced depth features Among them, each convolution layer in Conv is followed by a batch normalization layer and a ReLU layer; Afterwards, the enhanced depth features are transformed into Adjust to the same level by upsampling The two deep features are combined and passed into a convolutional block: Among them up ×2 (.) indicates a 2x upsampling operation through bilinear interpolation, and + is an element-wise summation operation; Deep semantic features And two enhanced depth features and Also passed to a convolution block to obtain enhanced depth features Among them, up ×4 (.) refers to a 4x upsampling operation using bilinear interpolation; Finally, a convolution block ConvB is deployed, which contains three extended convolution layers, three batch normalization layers and three ReLU layers to obtain the fused semantic features F D : Step 4. Based on the output features of the corresponding stage of the deep feature extraction network, the convolutional attention CBAM module is used to enhance the attention of the features and obtain the recalibrated features; Step 5. Based on the recalibrated features and the fused deep features, edge guidance and salient target fusion are performed to output the detection results.

2. The steel surface defect detection method based on deep learning according to claim 1, characterized in that: The specific implementation process of step 4 is as follows: The output features of the corresponding stage of the high-resolution convolution branch pass through a convolutional attention CBAM module, and the input features are filtered by the channel attention module and the spatial attention module in turn, and finally the recalibrated features are obtained. This process is defined as: Where CBAM represents the convolutional attention module, FC i Represents the output of the i-th CBAM module.

3. The steel surface defect detection method based on deep learning according to claim 2, characterized in that: The specific process of edge guidance and salient object fusion is as follows: For the decoder network, there are 5 convolutional blocks Each convolutional block consists of three convolutional layers, followed by a batch normalization layer and a ReLU layer; the input of each stage is the concatenation of the upsampled output of the previous stage and the output of the corresponding stage in the encoder, which is the input of the i-th decoder block Conv-Di. The process is defined as: where f i D represents the i-th decoder block Conv-Di, is the output of Conv-Di; In the side path of each decoder block, an edge extraction branch and a salient object prediction branch are deployed. These two operations are based on deep features. Processing, namely: Among them, f s is the function of the salient target prediction branch, A i is the output of the ith salient target prediction branch, i.e., the attention map, f e is the function of the edge extraction branch, E i is the output of the i-th edge extraction branch, i.e., edge information; both the salient object prediction branch and the edge extraction branch are a 3×3 convolutional layer; upsampling operations are deployed after these branches; Then the side outputs of all salient object prediction branches are combined to form the salient object map At the same time, the first edge information E1 is selected to participate in the salient target fusion process; in addition, the attention maps A2, A3, A4, and A5 are adjusted to the same size through upsampling operations; finally, under the guidance of the edge information E1, the salient target fusion is defined as: S=f[E1,A1,up ×2 (A2),up ×4 (A3),up ×8 (A4),up ×16 (A5)] Among them, S is the final salient target map, [,] represents the Concat operation, and f represents the convolution operation and Sigmoid activation function.

Images