Deep steel defect detection method based on multi-receptive field downsampling and channel attention cooperation

Abstract

The application discloses a deep steel defect detection method with multi-receptive field downsampling and channel attention cooperation, and the method comprises the following steps: acquiring steel surface defect image data in advance, labeling defect positions and defect types, and dividing the labeled data into a training set and a test set; constructing a deep steel defect detection network, taking YOLO as a main body architecture, and replacing all the downsampling modules in a backbone network with double-branch downsampling modules DPDM; adding an attention module SE to an output part of a multi-scale feature fusion PaFPN; determining a loss function of the deep steel defect detection network; training the constructed deep steel defect detection network by using the training set; and performing model performance evaluation by using the test set to obtain an optimal model and realize steel defect detection. The application solves the problems of multi-scale defects and insufficient feature saliency in the steel surface defect detection task, and significantly improves the steel defect detection precision.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a method for detecting deep steel surface defects using multi-receptor field downsampling and channel attention coordination. Background Technology

[0002] In the steel plate production process, due to factors such as production technology and environment, various defects or flaws such as cracks, scratches, and blemishes often appear on the surface of steel plates. To ensure product quality, traditional steel surface defect detection usually relies on manual monitoring. However, this method not only consumes a lot of manpower but also suffers from a significant decline in efficiency and accuracy under fatigue conditions. Therefore, designing an intelligent algorithm to achieve automated and accurate detection of steel plate surface defects is of great significance.

[0003] With the development of computer vision technology, traditional image processing techniques have made some progress in defect recognition and detection tasks. However, traditional methods that rely on rigorous mathematical formulas often lack broad adaptability to real-world task scenarios. For example, traditional methods that rely on handcrafted feature descriptors and subspace feature extraction algorithms are generally not adaptable to the diverse variations in defect types. In addition, traditional methods are also limited by a single background and struggle to cope with interference from the surrounding background of the defect, resulting in a large number of false detections.

[0004] In recent years, deep learning technology has made groundbreaking progress, demonstrating comprehensive performance advantages in various visual tasks. Currently, deep object detection technology, represented by the single-stage YOLO detection model, has been widely adopted and applied to various scenarios. For example, pedestrian and vehicle detection in security monitoring and autonomous driving has reached industrial-scale standards. However, in industrial scenarios, technology for detecting product defects still faces performance limitations. This is because the appearance of surface flaws or defects on products often exhibits characteristics such as small size or indistinct features, making it difficult for deep models to fully utilize their semantic integration capabilities in feature extraction.

[0005] Unlike target detection tasks in real-life scenarios, the simple implementation of existing depth detection algorithms for detecting surface defects in steel often fails to achieve industrial-scale performance levels. This is mainly due to the following two problems.

[0006] 1) Multi-scale problem of defect features: In terms of appearance, steel plate defects often exhibit a variety of types, especially with a trend towards numerous micro-defects. This conflicts with the feature extraction mode of deep detection models. This is mainly because the semantic integration advantage of deep neural networks relies heavily on feature dimension compression performed by cascaded convolutions. This dimension compression mechanism often leads to the loss or weakening of feature information from micro-defects.

[0007] 2) Insufficient saliency of defect features: In terms of appearance, steel plate defects often exhibit diversity in color, texture, and shape. This phenomenon makes it difficult for the model to capture defect information that is not visually significant. For example, when the difference between the defect's color and texture and the surrounding background is small, it is difficult to accurately obtain its defect information. Furthermore, the intra-class differentiation caused by the diverse appearances of similar defects can also lead to misclassification by the model.

[0008] Therefore, it is necessary to focus on steel surface defects and design a deep steel plate surface defect detection method that can solve the above technical problems. Summary of the Invention

[0009] Purpose of the invention: This invention proposes a method for detecting deep steel surface defects using multi-receptive field downsampling and channel attention coordination, which solves the problems of multi-scale defects and insufficient feature saliency in steel surface defect detection tasks.

[0010] Technical solution: The present invention provides a method for detecting deep steel surface defects using multi-receptive field downsampling and channel attention coordination, comprising the following steps:

[0011] (1) Acquire image data of steel surface defects in advance, label the defect locations and types, and divide the labeled data into training and testing sets;

[0012] (2) Construct a deep steel defect detection network with YOLO as the main architecture, replace all downsampling modules in the backbone network with dual-branch downsampling modules DPDM; add attention module SE to the output part of multi-scale feature fusion PaFPN;

[0013] (3) Determine the loss function of the deep steel defect detection network;

[0014] (4) The deep steel defect detection network is trained using the training set; the model performance is evaluated using the test set to obtain the optimal model for steel defect detection.

[0015] Furthermore, the dual-branch downsampling module DPDM described in step (2) includes an initial downsampling branch IDS and a large receptive field downsampling branch STD; the STD uses a spatial-to-depth operation to increase the receptive field scale of the downsampling operation.

[0016] Furthermore, the workflow of the dual-branch downsampling module DPDM is as follows:

[0017]

[0018] in, For containing C inThe input features of each channel are sampled by two branches to obtain the output features. and The features of the two branches are combined at the channel level; after channel compression through pointwise convolution, the terminal output is obtained.

[0019] Furthermore, the STD workflow of the large receptive field downsampling branch is as follows:

[0020] By increasing the number of channels, the feature representation of high-dimensional features in low-dimensional space is carried out; interval feature value information is obtained with a fixed step size, and small-scale features in low-dimensional space are assembled; the small-scale features are arranged in order by channel.

[0021] Furthermore, the STD downsampling branch and the IDS downsampling branch maintain the same number of channels.

[0022] Furthermore, the feature maps output by the STD downsampling branch and the IDS downsampling branch are spliced ​​together at a 1:1 ratio.

[0023] Furthermore, the attention module (SE) workflow is as follows:

[0024] SE first executes feature F through global pooling. in Feature statistics at each channel level generate a global information vector I containing information about each channel. G :

[0025]

[0026] Among them, F in The input features are w and h, which correspond to the width and height of the feature map, respectively. After global pooling, a fully connected layer is used to convert them into channel importance weights through the parameter W.

[0027]

[0028] Wherein, vector I EX It is the activation module EX(I) in the SE attention mechanism G The weighting coefficients generated by W);

[0029] Perform the original input features F in Weighting coefficient I of the corresponding channel EX Dot product operation to obtain terminal features F out :

[0030] F out =F in iI EX

[0031] The dot product (i) operation represents combining the weight vector I2 with the original feature map F. in Perform channel-by-channel weighted multiplication.

[0032] Furthermore, the loss function in step (3) includes the bounding box regression loss L. reg Category loss L cls And confidence loss L obj The overall loss function is:

[0033] L total =L reg +L cls +L obj

[0034] Among them, L cls and L obj Both methods employ binary cross-entropy loss, which is constructed as follows:

[0035]

[0036] Where y is the true value in the training set. is the predicted value of the model, and n is the number of categories.

[0037] Furthermore, the excitation module includes two serial fully connected layers W1 and W2, and the weight generation operation is a nested function in the form of Sigmoid[W2, ReLU(W1, I G [); where Sigmoid and ReLU are the activation functions of layers W1 and W2, respectively.

[0038] Furthermore, the regression loss L reg The accuracy of the predicted bounding box is verified by calculating the intersection-union ratio (IoU) between the ground truth bounding box and the predicted bounding box, as described below:

[0039]

[0040] In this context, A and B represent the ground truth bounding box and the predicted bounding box, respectively.

[0041] Beneficial Effects: Compared with the prior art, the beneficial effects of this invention are as follows: For the backbone feature extraction network, this invention designs a dual-branch downsampling method, forming a plug-and-play DPDM downsampling module. Compared with the ordinary downsampling method in traditional networks, it can provide richer and more diverse features for subsequent detection modules. The detailed information it covers is an important support for ensuring the accuracy of defect detection. To address the feature redundancy problem, this invention injects an SE channel attention module into the subsequent FPN part of the model, focusing on the feature information that is more critical to defect detection, thereby improving the accuracy of steel plate defect detection. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the deep steel defect detection network structure proposed in this invention;

[0043] Figure 2 This is a schematic diagram of the workflow of the dual-branch downsampling module;

[0044] Figure 3 This is a schematic diagram illustrating the principle of spatial depth transformation.

[0045] Figure 4 This is a schematic diagram illustrating the working principle of the SE module;

[0046] Figure 5 This is a schematic diagram of the SE module structure;

[0047] Figure 6 This is a diagram showing the defect detection results. Detailed Implementation

[0048] The present invention will now be described in further detail with reference to the accompanying drawings.

[0049] This invention proposes a method for detecting deep surface defects in steel using multi-receptive field downsampling and channel attention coordination, specifically including the following steps:

[0050] Step 1: Data collection and preprocessing of steel surface defects.

[0051] Image data of steel surface defects were collected, and the location and type of defects were manually labeled. The labeled data was divided into training and testing sets for model training and performance testing, respectively. This embodiment uses the publicly available Northeastern University steel surface defect dataset NEU-DET, which contains 1800 images in 6 categories. The training and testing sets were divided at equal intervals in a 6:4 ratio.

[0052] Step 2: Construct as follows Figure 1 The deep steel defect detection network shown is based on the mainstream YOLO detection model. Improvements were made to the network structure of the backbone feature extraction network and the subsequent Feature Pyramid Network (FPN) detection module. A Dual Path Down-sampling Method (DPDM) module was designed to promote the diversity of intermediate features in the model, and a Squeeze-and-Excitation (SE) channel attention module was injected to enhance the model's ability to acquire features of insignificant defects. Specifically, all downsampling modules in the backbone network were replaced with DPDM modules, and an SE module was added to the output of PaFPN.

[0053] (2.1) Dual-branch downsampling module (DPDM).

[0054] The feature representation capability of the backbone feature extraction network is the main support for the performance advantages of deep detection models. The downsampling operation is a key step in performing semantic integration. However, the small receptive field mechanism commonly used in existing downsampling methods often leads to insufficient detailed information in the sampled features. Therefore, addressing the multi-scale problem of steel plate surface defect features and the insufficient saliency of defect appearance features, this part adopts a downsampling approach that considers both large and small receptive fields, designing a dual-branch downsampling module (DPDM), such as... Figure 2 As shown.

[0055] The main goal of DPDM downsampling is to increase the diversity or richness of sampled features by utilizing different receptive field mechanisms. According to... Figure 3 The working principle illustrated is that DPDM includes an Initial Down-Sampling (IDS) branch and a Space To Depth (STD) large receptive field downsampling branch. The dual-branch mode, composed of receptive fields of different scales, can achieve better results than any single branch. Its working principle is briefly described below:

[0056] 1) IDS downsampling branch: Adhering to the small receptive field sampling setting, this branch directly uses the downsampling methods in the original model, such as the commonly used mean pooling, max pooling and 3×3 stride sampling method of convolution kernel.

[0057] 2) STD downsampling branch: Adhering to the large receptive field sampling mechanism, this branch adopts a space-to-depth operation in its structural design to increase the receptive field scale of the downsampling operation.

[0058] Overall branch workflow: This branch first performs a spatial-to-channel feature transformation (STD) operation, followed by a channel reduction operation using a 3x3 convolutional kernel. The channel compression step performs downsampling. Thus, this branch achieves a large receptive field sampling effect with a small 3x3 convolutional kernel, and its low parameter settings offer efficiency advantages.

[0059] The STD module works by increasing the number of channels to represent high-dimensional features in a low-dimensional space. For example... Figure 3 As shown, STD can be broken down into stride sampling and channel placement steps. ① Stride sampling: Using a fixed stride, interval feature values ​​are obtained to assemble small-scale features in low-dimensional space. ② Channel placement: The small-scale features are arranged sequentially by channel for subsequent convolution operations.

[0060] Dual-branch downsampling module workflow: The DPDM method processing flow can be formalized as follows in parallel and serial descriptions:

[0061]

[0062] in, For containing C in The input features of each channel are downsampled by two branches (IDS, STD) to obtain the output features. and This represents the joint of the two branch features at the channel level. Finally, after channel compression through pointwise convolution, the terminal output is obtained. The terminal output features and the downsampled features of the original model are consistent in the number of channels, so as to form a plug-and-play multi-receptive field downsampling module.

[0063] In this implementation, the downsampling branch of the onboard IDS uses the same downsampling method as YOLO, namely, a 3×3 convolution of length 2. Furthermore, the STD downsampling branch maintains the same number of channels as the IDS downsampling branch. Finally, the feature maps output from the two downsampling branches are concatenated at a 1:1 ratio.

[0064] (2.2) SE Attention Module.

[0065] The DPDM downsampling method in the backbone feature extraction network provides sufficient detailed information for the subsequent FPN detection module by promoting the diversity of sampled features. However, its multi-receptor field joint mechanism may produce a large amount of redundant information. Therefore, to enhance the detection capability of the detection head from the perspective of feature representation, this invention injects an SE attention module into the subsequent FPN detection module, providing the detection head with features that have better channel-level representation to perform feature information optimization. Figure 1 As shown, the SE attention module is attached to the front of the terminal detection head of the subsequent FPN module.

[0066] like Figure 5 As shown, this module first uses global average pooling to compress the input features into a vector, then uses a fully connected layer to activate the features, and finally multiplies it with the input features to enhance the input features. In the figure, w, h, and c represent the length, width, and height of the feature, respectively, where Global Average Pooling represents global average pooling, and FC represents a fully connected layer. SE attention mechanism is a channel-level feature refinement technique designed to optimize the feature representation capability of the network model. The core principle of SE attention mechanism lies in adjusting the contribution of each channel to subsequent predictions through a channel weighting strategy. Figure 4 This illustrates the working principle of SE, the core of which is... Figure 5The weight generation operation for the upper-middle auxiliary branch works as follows: First, a global reddening operation is performed on the features of each channel to obtain one-dimensional feature information for each channel. Second, a fully connected network is used to generate weighting coefficients. Finally, a dot-multiplication weighting operation is performed based on the generated weighting coefficients to achieve the function of highlighting or suppressing the importance of each channel. Its principle can be formalized as follows.

[0067] Given feature F in SE first executes feature F through global pooling. in Feature statistics at each channel level generate a global information vector I containing information about each channel. G This operation can be formalized as follows:

[0068]

[0069] Among them, F in The input features are w and h, which correspond to the width and height of the feature map, respectively. After global pooling, this branch then uses a fully connected layer and converts the input features into channel importance weights using the parameter W. This process can be formalized as follows:

[0070]

[0071] Wherein, vector I EX It is composed of the EX module EX(I) G The weights generated by the weights are, in practice, the EX module typically contains two sequential fully connected layers W1 and W2. Therefore, the weight generation operation can be further expressed as the following nested function form: Sigmoid[W2, ReLU(W1, I...]). G [ ] ]. Here, Sigmoid and ReLU are the activation functions of layers W1 and W2, respectively.

[0072] Finally, the original input features F are processed. in Weighting coefficient I of the corresponding channel EX Dot product operation to obtain terminal features F out This process can be formalized as follows:

[0073] F out =F in ·I EX

[0074] The dot product (·) operation represents combining the weight vector I2 with the original feature map F. in Perform channel-by-channel weighted multiplication.

[0075] Step 3: Determine the loss function of the deep steel defect detection network.

[0076] The loss function is composed of the bounding box regression loss L. reg Category loss L cls And confidence loss L obj Composed of three parts, the overall loss function can be expressed in the following form:

[0077] L total =L reg +L cls +L obj

[0078] L cls and L obj Both methods employ binary cross-entropy loss (BCELoss), which is constructed as follows:

[0079]

[0080] Among them, y i For the true values ​​in the training set, Here, represents the model's predicted values, and n represents the number of classes. The regression loss L... reg Intersection over Union (IOU) loss was used:

[0081]

[0082] In this context, A and B represent the ground truth bounding box and the predicted bounding box, respectively.

[0083] Step 4: Train the constructed deep steel defect detection network using the training set; evaluate the model performance using the test set to obtain the optimal model for steel defect detection.

[0084] The training dataset from step 1 underwent data preprocessing. The input image size for the model was set to 640×640, and Mosaic was used for data augmentation. The model was trained for 350 epochs, with data augmentation disabled in the last 100 epochs. A dynamic learning rate was used, initially set to 0.01, and adjusted using cosine annealing. Finally, the weight decay coefficient was set to 0.0005. The optimizer used a standard SGD algorithm. Furthermore, in this embodiment, the random seed for all experiments was uniformly fixed at 0. The training process involved repeatedly executing the following two steps according to the training settings:

[0085] 1) Perform forward computation: Input image data into the model, perform forward computation, and calculate the loss L based on the predicted values. total 2) Perform backpropagation: based on the loss L total The optimizer calculates the gradient values ​​of the model parameters and performs weight parameter updates.

[0086] After the model training is completed, use AP 50:95The performance indicators are used as the benchmarks, and the model performance is evaluated using steel plate defect test data. After meeting the defect detection performance indicators, the model is deployed and defect detection applications are implemented.

[0087] For the model trained in step 4, the detection of surface defects on steel is implemented. This part is divided into model testing and inference. The effectiveness of the improvement scheme is determined through experiments on a specified dataset. The model is deployed on a server, and the detection effect is observed on a given input image.

[0088] Unlike the SE module, DPDM is a module designed in this invention, and therefore its effectiveness needs to be verified separately. In this embodiment, improvement strategies are implemented for different object detection algorithms, and ablation experiments are conducted on the VOC2007 dataset. To ensure experimental fairness, the following three principles are adopted in the experimental design:

[0089] ① To eliminate the influence of random numbers on the detection model in deep learning, the random seed was consistently fixed to 0 in the experiments. ② To eliminate the influence of the model evaluation algorithm implementation on the results, the model accuracy evaluation part was consistently implemented using the standard pycocotools library. ③ To eliminate the influence of hyperparameter tuning operations on the results, the hyperparameter settings consistently used the options recommended by the corresponding algorithm authors.

[0090] Table 1 presents the results of the ablation experiments. Each model includes three methods: Baseline, STD, and DPDM. These represent the baseline model, the improved version using spatial-to-depth (STD) operation instead of downsampling, and the improved version using the DPDM module of this invention instead of downsampling, respectively.

[0091] Table 1. Ablation experimental results of different algorithms and methods on the VOC2007 dataset.

[0092]

[0093] As shown in Table 1, STD achieves a certain improvement in overall performance, but is slightly inferior to DPDM. This indicates that the spatial-to-depth operation used in STD can effectively alleviate the multi-scale detection problem. On the other hand, the DPDM module designed in this invention is not a simple superposition of the two downsampling methods; the dual-branch mode of STD and the original downsampling can further improve the model accuracy.

[0094] The training results on the welding defect dataset are shown in Table 2, which presents the ablation experiment results for each version of the above model, based on the training settings described in step 4.

[0095] Table 2 shows the ablation test results in the steel defect dataset.

[0096]

[0097] Table 2 lists the names of the baseline model and each improved model. "Baseline" refers to the YOLOXs model without any modifications. "DPDM" refers to the model where the downsampling module in the YOLOXs backbone network is replaced with a DPDM module, based on the Baseline model. "DPDM+SE" refers to the model where the SE attention module is added to the output of the PaFPN, based on the DPDM model. The results in Table 4 show that, on the AP50:95 metric, using DPDM improves accuracy by 1.3%. Adding the SE attention module further improves accuracy by 0.5%.

[0098] After validating the model, it can be deployed on a server for inference. In this embodiment, the model is deployed on a server using the Ubuntu operating system, and an NVIDIA 3090 on the server is used to implement the model's image detection. Figure 6 The results of steel image detection using the present invention are shown.

Claims

1. A method for deep steel defect detection using multi-receptor field downsampling and channel attention coordination, characterized in that, Includes the following steps: (1) Acquire image data of steel surface defects in advance, label the defect location and defect type, and divide the labeled data into training set and test set; (2) Construct a deep steel defect detection network, with YOLO as the main architecture, and replace all downsampling modules in the backbone network with dual-branch downsampling modules DPDM; An attention module (SE) is added to the output of the multi-scale feature fusion PaFPN. (3) Determine the loss function of the deep steel defect detection network; (4) The deep steel defect detection network is trained using the training set; the model performance is evaluated using the test set to obtain the optimal model for steel defect detection; The dual-branch downsampling module DPDM mentioned in step (2) includes an initial downsampling branch IDS and a large receptive field downsampling branch STD; the STD uses a spatial-to-depth operation to increase the receptive field scale of the downsampling operation; The workflow of the dual-branch downsampling module DPDM is as follows: in, For inclusion The input features of each channel are sampled by two branches to obtain the output features. and ; The features of the two branches are combined at the channel level; after channel compression through pointwise convolution, the terminal output is obtained. ; The STD workflow of the large receptive field downsampling branch is as follows: By increasing the number of channels, the feature representation of high-dimensional features in low-dimensional space is carried out; interval feature value information is obtained with a fixed step size, and small-scale features in low-dimensional space are assembled; the small-scale features are arranged in order by channels. The STD downsampling branch and the IDS downsampling branch maintain the same number of channels; The feature maps output by the STD downsampling branch and the IDS downsampling branch are spliced ​​together at a 1:1 ratio.

2. The method for deep steel defect detection based on multi-receptive field downsampling and channel attention coordination according to claim 1, characterized in that, The attention module (SE) workflow is as follows: SE first executes features through global pooling. Feature statistics at each channel level generate a global information vector containing information from each channel. : in, These are input features. and The width and height of the feature map are respectively assigned; after global pooling, a fully connected layer is then used to convert the values ​​into channel importance weights using parameter W: Among them, the weight vector It is the incentive module in the SE attention mechanism The generated weighting coefficients; Perform original input features Weight vector of the corresponding channel Dot product operation to obtain terminal features : Among them, dot product The operation represents the weight vector Compared with the original feature map Perform channel-by-channel weighted multiplication.

3. The method for deep steel defect detection based on multi-receptive field downsampling and channel attention coordination according to claim 1, characterized in that, The loss function in step (3) includes bounding box regression loss. Category loss and confidence loss The overall loss function is: in, and Both methods employ binary cross-entropy loss, which is constructed as follows: in, For the true values ​​in the training set, is the predicted value of the model, and n is the number of categories.

4. The method for deep steel defect detection using multi-receptive field downsampling and channel attention coordination according to claim 2, characterized in that, The excitation module comprises two serial fully connected layers. and The weight generation operation is in the form of nested functions. ;in, and They are and The activation function of the layer.

5. The method for deep steel defect detection based on multi-receptive field downsampling and channel attention coordination according to claim 3, characterized in that, The regression loss The accuracy of the predicted bounding box is verified by calculating the intersection-union ratio (IoU) between the ground truth bounding box and the predicted bounding box, as described below: In this context, A and B represent the ground truth bounding box and the predicted bounding box, respectively.

Images