Strip steel surface defect detection method based on multi-branch parallel attention model

Abstract

The invention discloses a strip steel surface defect detection method based on a multi-branch parallel attention model, and belongs to the technical field of industrial defect target detection. On the basis of a YOLOv8 network, space depth conversion convolution is used for replacing all down-sampling convolution in a backbone network CSPDarkNet, a multi-branch parallel attention module is used for replacing a C2f module, and an internal feature interaction module is added behind the backbone network, so that the strip steel surface defect detection precision of the model is improved. The strip steel surface defect detection precision is effectively improved while model parameters are slightly increased, and compared with an existing strip steel surface defect model, the problem that the image detection effect is poor due to the fact that background interference is strong, various defects are large in difference, the number of small defects is large, and defect targets are likely to be confused in an industrial scene can be better solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of industrial defect target detection, and in particular relates to a strip surface defect detection method based on a multi-branch parallel attention model. Background Art

[0002] Strip surface defect detection is one of the key technologies in the fields of industrial manufacturing, quality control and safety inspection. With the rapid development of automation and intelligent manufacturing, convolutional neural networks based on deep learning have shown significant progress in strip surface defect recognition. With its powerful feature extraction ability, convolutional neural networks can effectively capture complex patterns and details, and thus have gradually become the mainstream method in strip defect detection. However, although the existing deep learning methods have achieved certain results in many cases, the current technology still faces some challenges. In particular, the existing models often show shortcomings in accurately capturing the detailed features of low contrast and tiny defects. These tiny defects may be difficult to identify due to the interference of surface roughness, lighting conditions and background noise, resulting in missed detection or false detection. In addition, environmental changes, such as lighting changes and background interference, further reduce the robustness and adaptability of the detection system. These factors make it difficult for the detection accuracy to meet the high requirements of the industry for quality control in practical applications.

[0003] Therefore, improving the accuracy and stability of strip surface defect detection has become a key issue that needs to be solved urgently. Summary of the invention

[0004] The present invention provides a strip surface defect detection method based on a multi-branch parallel attention model, which is used to solve the problems in the prior art that tiny defects are easily disturbed by background information, missed detection and have low defect detection accuracy.

[0005] The present invention is based on the YOLOv8 network and uses spatial depth conversion convolution to replace all downsampling convolutions in its backbone network CSPDarkNet, reducing the loss of small-scale feature information and the model's missed detection rate of small-scale defects. The multi-branch parallel attention module is used to replace the C2f module to capture small-scale defect information. An internal feature interaction module is added after the backbone network to perform internal information interaction of multi-scale features, thereby improving the model's detection accuracy for strip surface defects.

[0006] The method specifically comprises the following steps:

[0007] S100 is based on the built strip surface defect detection platform. It uses industrial cameras to collect images of strips with surface defects and annotates the collected defect images with rectangular frames.

[0008] S200 builds a dataset of surface defect images of strip steel, and uses random cropping, translation, brightness change, and noise addition to enhance the data of the annotated defect images. The images are divided into training and test sets in a ratio of 8:2.

[0009] S300 replaces all standard convolutions used for downsampling in the backbone network CSPDarkNet with spatial depth conversion convolutions;

[0010] S400 uses a multi-branch parallel attention module to replace all C2f modules in the backbone network CSPDarkNet and the neck feature fusion network;

[0011] S500 adds an intra-scale feature interaction module to the backend of the backbone network;

[0012] S600 constructs a strip steel surface defect detection model based on a multi-branch parallel attention model according to the spatial depth conversion convolution, the multi-branch parallel attention module, and the internal feature interaction module, trains the detection model using a training set in the constructed strip steel surface defect image data set, and evaluates the model performance using a test set;

[0013] S700 detects and identifies the strip surface defect image using the trained strip surface defect detection model, and obtains the category, confidence level and location information of the strip surface defect output by the model.

[0014] Furthermore, the defect image acquisition first sets the light source, and adjusts the brightness, color and angle of the light source according to the detection environment and the reflective characteristics of the strip surface to ensure that the defects are clearly visible in the image; secondly, the industrial camera is installed on the platform bracket to ensure that the camera's viewing angle covers the strip surface that needs to be inspected, and the industrial camera is used to photograph the strips with different defects.

[0015] Furthermore, the annotation is to mark the surface defect image of the strip with rectangular boxes according to six categories, namely cracks, impurities, plaques, pitting, pressed iron oxide scale, and scratches. The image is annotated by software Labelimg. The annotation file format is TXT, which contains information on the category, the XY position coordinates of the upper left corner point of the rectangular box, and the XY position coordinates of the upper left corner point of the rectangular box.

[0016] Furthermore, the spatial depth conversion convolution includes a spatial depth conversion layer and a non-strided convolution, which is used to improve the detection accuracy of small-scale targets in low-resolution images.

[0017] Furthermore, the multi-branch parallel attention module includes a multi-branch feature extraction module and an attention mechanism module, which can effectively capture the multi-scale features of defects, especially for the extraction of small-scale defect information.

[0018] Furthermore, the internal feature interaction module performs internal information interaction on the extracted multi-scale features, including multi-head self-attention and feedforward neural networks, to enhance the feature expression capability of the backbone network.

[0019] The beneficial effects of the present invention are as follows:

[0020] The present invention is based on the YOLOv8 algorithm. By replacing all standard convolutions used for downsampling in the backbone network CSPDarkNet with spatial depth conversion convolutions, the feature extraction capability of small-scale defect targets in low-resolution images is enhanced; a multi-branch parallel attention module is used to replace all C2f modules in the backbone network CSPDarkNet and the neck feature fusion network to effectively capture the multi-scale features of defects, especially for the extraction of small-scale defect information; an intra-scale feature interaction module is added to the back end of the backbone network to perform internal information interaction on the extracted multi-scale features to enhance the feature expression capability of the backbone network. The present invention effectively improves the detection accuracy of strip surface defects while slightly increasing the model parameters. Compared with the current strip surface defect model, it can better solve the problem of poor image detection effect caused by strong background interference, large differences in various defects, a large number of small defects, and easily confused defect targets in industrial scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flow chart of a strip steel surface defect detection method based on a multi-branch parallel attention model;

[0022] Figure 2 It is a schematic diagram of a multi-branch feature extraction module;

[0023] Figure 3 Schematic diagram of the attention mechanism module;

[0024] Figure 4 Schematic diagram of the intra-scale feature interaction module. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is described in detail below in combination with specific embodiments (but not limited to the embodiments) and the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] like Figure 1 As shown, a strip surface defect detection method based on a multi-branch parallel attention model comprises the following steps:

[0027] S100 is based on the built strip surface defect detection platform and uses industrial cameras to capture images of strips with surface defects and annotate the captured defect images with rectangular frames.

[0028] Furthermore, defect image acquisition first sets the light source, and adjusts the brightness, color and angle of the light source according to the detection environment and the reflective characteristics of the strip surface. Ensure that the defects are clearly visible in the image. Secondly, install the industrial camera on the platform bracket, ensure that the camera's viewing angle can cover the strip surface to be inspected, and use the industrial camera to shoot strips with different defects.

[0029] Furthermore, the rectangular box is annotated with six categories of surface defect images of the strip, namely cracks, impurities, plaques, pitting, pressed iron oxide scale and scratches, and the image is annotated by the software Lableimg. The annotation file format is TXT, which contains information on the category, the XY position coordinates of the upper left corner point of the rectangular box and the XY position coordinates of the upper left corner point of the rectangular box.

[0030] S200 constructs a dataset of surface defect images of strip steel, and performs data enhancement on the annotated defect images by random cropping, translation, brightness change, and noise addition, and divides them into training and test sets in a ratio of 8:2.

[0031] Furthermore, the strip surface defect image dataset is NEU-DET, which contains 1800 images of six types of defects, and is randomly divided into a training set of 1440 images and a test set of 360 images in a ratio of 8:2.

[0032] S300 replaces all standard convolutions used for downsampling in the backbone network CSPDarkNet with spatial depth conversion convolutions.

[0033] Furthermore, the spatial depth conversion convolution includes a spatial depth conversion layer and a non-strided convolution layer, which are used to improve the detection accuracy of small-scale targets in low-resolution images. The spatial depth conversion layer converts defect features from space to depth, and the process can be expressed as When the low-resolution defect feature sub-image X passes through the spatial depth conversion layer, the width and height dimensions are sliced ​​into four sub-features f 0,0 =X[0:S:scale,0:S:scale], f 0,1 =X[0:S:scale,1:S:scale],f 1,0 =X[1:S:scale,0:S:scale],f 1,1 =X[1:S:scale,1:S:scale], where S represents the step size and scale represents the number of complete slices of the image. Then the four sub-features are concatenated in the channel dimension to obtain XC , X C Contains more fine-grained information of defect targets without losing the original information in the downsampling process. The non-strided convolution layer uses a standard convolution with a step size of 1 to input X C Pixel-by-pixel processing can reduce oversampling of the spatial depth conversion layer while retaining more small-scale feature information.

[0034] S400 uses a multi-branch parallel attention module to replace all C2f modules in the backbone network CSPDarkNet and the neck feature fusion network.

[0035] Furthermore, the multi-branch parallel attention module utilizes hierarchical fusion and multi-branch parallel multi-branch attention mechanism to extract and enhance the feature representation of small targets, performs defect feature extraction and fusion at different global and local scales and levels, and fuses the local fine-grained features of defects with the global background information, thereby further improving the detection accuracy of small target defects.

[0036] Furthermore, the multi-branch parallel attention module is mainly composed of a multi-branch feature extraction module and an attention mechanism module, such as Figure 2 As shown in Figure 1, the multi-branch feature extraction module includes three branches: global attention, local attention, and serial convolution. First, the input feature F is adjusted by point-by-point convolution to obtain F 1 , the global attention branch uses a global attention with a patch size of 4 on F 1 Processing to obtain F global , the local attention branch uses a local attention with a patch size of 2 on F 1 Processing to obtain F local The serial convolution branch uses three consecutive standard convolutions with a kernel size of 3 to process the input feature F to obtain three different features F conv1 、F conv2 and F conv3 , F 1 、F global 、F local 、F conv1 、F conv2 and F conv3 The final output feature F of the multi-branch feature extraction module is obtained by adding D .like Figure 3 As shown in the figure, the attention mechanism module includes serial channel attention and spatial attention. First, F D Input channel attention to get F channel , then F channel Input spatial attention to get F spatial Finally, we use the dropout layer with a random dropout rate of 0.1, the BN layer, and the SiLU activation function tospatial Processing to get output F output .

[0037] Furthermore, if Figure 2 As shown in the figure, the global attention and local attention establish the distinction between local and global branches by controlling the patch size parameter, which is achieved by aggregating and shifting non-overlapping patches in the spatial dimension, and calculating the attention matrix between non-overlapping patches, with the purpose of extracting and interacting local and global features. Specifically, the unfold and reshape operations are used to change the dimension of F1, dividing it into a series of spatially continuous blocks, and then averaging the features on each channel, and then using a feedforward neural network for linear calculation to increase the ability of the attention module to learn abstract features to obtain F. FFN The probability distribution of linear computation features in the spatial dimension is obtained through the Softmax activation function and compared with F FFN Multiplication is used to weight the local output features. The feature selection strategy uses the learnable prompt to give the ability to adjust attention according to high-level goals or tasks. Specifically, the cosine similarity is limited to between 0 and 1, and a mask is generated to filter feature information. The cosine similarity between the local output feature and the prompt is calculated and normalized, and a mask is generated to filter features. The purpose is to enhance the focus on important features and suppress the focus on unimportant features, which can more effectively capture the key information in the input image.

[0038] Furthermore, if Figure 3 As shown, the channel attention first inputs the original feature image F D , all channels of all strip surface defect feature sub-images are globally averaged and pooled, and then the channel weights are generated using one-dimensional convolution, and the corresponding probabilities of different channels are calculated using the Sigmoid activation function, and then compared with the input feature F D Multiply it as the input F of the next layer channel .

[0039] Furthermore, if Figure 3 As shown, the spatial attention first focuses on the input feature map F channel Perform global maximum pooling and global average pooling in the channel dimension to obtain two feature maps F with a channel number of 1 maxpool and F avgpool ; The purpose of pooling in the channel dimension is to compress the channel size to facilitate the subsequent learning of spatial features, and then F maxpool and F avgpool According to channel splicing, we get a feature map F with 2 channels. pool , and finally to F poolPerform a standard 7x7 convolution operation to obtain a feature map F with a channel number of 1 conv7 , then F conv7 The Sigmoid activation function is used to increase the nonlinear characteristics of the feature map to obtain the spatial attention weight matrix F sigmoid Finally, F sigmoid With F channel Multiply to get the output F spatial .

[0040] S500 adds an intra-scale feature interaction module to the backend of the backbone network.

[0041] Furthermore, the intra-scale feature interaction module improves the efficiency and effectiveness of feature extraction by introducing intra-scale feature interactions based on the attention mechanism. The core of this module is to use the attention mechanism between features of the same scale to enhance the network's focusing ability and promote richer feature fusion. Compared with traditional feature fusion involving features of different scales, the focus is on feature fusion within the same scale, which helps capture more fine-grained information.

[0042] Furthermore, if Figure 4 As shown in the figure, the internal feature interaction module first converts the deep features F output by the multi-branch parallel attention module into output Convert to a high-dimensional vector and embed F with randomly generated two-dimensional sine and cosine positions pos_sincos After adding, input the multi-head self-attention enhanced feature representation to get S ma , then F output With S ma After residual connection and layer normalization, S is obtained a . Then S a The input is a feed-forward neural network consisting of two fully connected layers with a hidden dimension of 2048 and a ReLU activation function to learn the nonlinear relationship between input and output and adapt to complex patterns to obtain S mlp Finally, S a With S mlp After residual connection and layer normalization, S is obtained FFN .

[0043] S600 constructs a strip surface defect detection model based on a multi-branch parallel attention model according to the spatial depth conversion convolution, the multi-branch parallel attention module and the internal feature interaction module, trains the detection model using a training set in the constructed strip surface defect image data set, and evaluates the model performance using a test set.

[0044] Furthermore, compared with the standard convolution used for downsampling, the spatial depth conversion convolution can retain more spatial detail information. The spatial features are converted to the channel dimension through slicing and splicing operations. The fusion of multi-scale information is helpful for small-scale target detection. The multi-branch parallel attention module uses the global attention with a patch of 4 in the multi-branch extraction module to capture large-scale target features, the local attention with a patch of 2 to capture small-scale target features, and the continuous 3×3 standard convolution to capture medium-scale target features, which can extract and interact local and global features. At the same time, the feature selection strategy used in global attention and local attention uses learnable prompts to give the ability to adjust attention according to high-level goals or tasks. The task-related tags are measured by cosine similarity and given higher weights. Enhancing the focus on important features can more effectively capture the key information in the input image and thus improve the detection accuracy. The attention mechanism module further learns and fuses important features in the channel and spatial dimensions. The internal feature interaction module adopts multi-head self-attention to enhance feature representation, which strengthens the model's ability to focus on defect features. It uses a feedforward neural network to enrich the complex feature representation extracted by the model, strengthen the model's nonlinear feature extraction capabilities, and better adapt to the complex feature learning tasks of strip surface defects.

[0045] S700 detects and identifies the strip surface defect image using the trained strip surface defect detection model, and obtains the category, confidence level and location information of the strip surface defect output by the model.

[0046] Furthermore, the strip surface defect image is first loaded, and preprocessing operations such as adjusting the image size and normalization are performed on it. Secondly, the preprocessed image is input into the trained strip surface defect detection model, and the defect feature extraction of different levels and scales is completed through spatial depth conversion convolution and multi-branch parallel attention module. After the features pass through the model prediction layer, the category prediction, confidence score and position prediction including the center coordinates, height and width of the prediction box are generated for each defect target detection box. Finally, in order to eliminate redundancy and improve detection accuracy, NMS non-maximum suppression is performed after the output result, the prediction box with the highest confidence is retained, and redundant detection boxes are removed when multiple prediction boxes overlap and the confidence is lower than a certain threshold. The output contains the category, confidence and precise position coordinates of the strip surface defect.

[0047] Furthermore, in order to verify the effectiveness of the strip surface defect detection method based on the multi-branch parallel attention model proposed in the present invention, a test was carried out on the public strip surface defect image dataset NEU-DET.

[0048] Table 1 Comparison of indicators between the multi-branch parallel attention model and the original model

[0049]

[0050]

[0051] The results in Table 1 show that the multi-branch parallel attention model has a slight decrease in accuracy compared to the original model, but the three accuracy indicators of recall rate, mAP@50 and mAP@50:95 have all been significantly improved. This shows that the method of the present invention has certain advantages and is more accurate in identifying surface defects of strip steel, especially in terms of recall rate and mAP@50.

[0052] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A strip surface defect detection method based on a multi-branch parallel attention model, characterized in that: Based on the YOLOv8 network, we use spatial depth conversion convolution to replace all downsampling convolutions in its backbone network CSPDarkNet, use multi-branch parallel attention modules to replace the C2f module, and add an internal feature interaction module after the backbone network. The specific steps include: S100 is based on the built strip surface defect detection platform. It uses industrial cameras to collect images of strips with surface defects and annotates the collected defect images with rectangular frames. S200 builds a dataset of surface defect images of strip steel, and uses random cropping, translation, brightness change, and noise addition to enhance the data of the annotated defect images. The images are divided into training and test sets in a ratio of 8:

2. S300 replaces all standard convolutions used for downsampling in the backbone network CSPDarkNet with spatial depth conversion convolutions; S400 uses a multi-branch parallel attention module to replace all C2f modules in the backbone network CSPDarkNet and the neck feature fusion network; S500 adds an intra-scale feature interaction module to the backend of the backbone network; S600 constructs a strip steel surface defect detection model based on a multi-branch parallel attention model according to the spatial depth conversion convolution, the multi-branch parallel attention module, and the internal feature interaction module, trains the detection model using a training set in the constructed strip steel surface defect image data set, and evaluates the model performance using a test set; S700 detects and identifies the strip surface defect image using the trained strip surface defect detection model, and obtains the category, confidence level and location information of the strip surface defect output by the model.

2. The method for detecting surface defects of a strip steel based on a multi-branch parallel attention model according to claim 1, characterized in that: The defect image acquisition in S100 first sets the light source, and adjusts the brightness, color and angle of the light source according to the detection environment and the reflective characteristics of the strip surface to ensure that the defects are clearly visible in the image; secondly, the industrial camera is installed on the platform bracket to ensure that the camera's viewing angle covers the strip surface to be inspected, and the industrial camera is used to shoot strips with different defects.

3. The method for detecting surface defects of a strip steel based on a multi-branch parallel attention model according to claim 1, characterized in that: The annotation in S200 is to mark the surface defect image of the strip with rectangular frames according to six categories: cracks, impurities, plaques, pitting, pressed iron oxide scale, and scratches. The image is annotated by the software Labelimg. The annotation file format is TXT, which contains information about the category, the XY position coordinates of the upper left corner of the rectangular frame, and the XY position coordinates of the upper left corner of the rectangular frame.

4. The method for detecting surface defects of a strip steel based on a multi-branch parallel attention model according to claim 1, characterized in that: The spatial depth conversion convolution of S300 includes a spatial depth conversion layer and a non-strided convolution.

5. The method for detecting surface defects of a strip steel based on a multi-branch parallel attention model according to claim 1, characterized in that: The multi-branch parallel attention module of S400 includes a multi-branch feature extraction module and an attention mechanism module, which is used to capture multi-scale features of defects.

6. The method for detecting surface defects of a strip steel based on a multi-branch parallel attention model according to claim 1, characterized in that: The internal feature interaction module of S500 performs internal information interaction on the extracted multi-scale features, including multi-head self-attention and feedforward neural network, to enhance the feature expression capability of the backbone network.

Images