A steel surface defect detection method, device and computer storage medium

Abstract

The present invention discloses a steel surface defect detection method, device and computer storage medium, and relates to the technical field of steel defect detection. The method comprises: obtaining a steel surface defect image data set, preprocessing the steel surface defect images in the data set, and dividing the data set into a training set and a test set; constructing a Mask RCNN based on a multi-head attention mechanism and Swin Transformer, and training the Mask RCNN using the training set; using the trained Mask RCNN to perform steel surface defect detection on the steel surface defect images in the test set, and obtaining the steel surface defect detection result. The multi-head self-attention mechanism and Swin Transformer disclosed in the present invention can retain small pixel information and reduce the amount of calculation; thus, the detection accuracy can be improved, especially the accuracy when detecting small-sized defects can be greatly improved; at the same time, the calculation speed can be improved to achieve real-time detection.

Description

Technical Field

[0001] The present invention relates to the technical field of steel defect detection, and in particular to a steel surface defect detection method, device and computer storage medium based on a Mask RCNN network with a multi-head self-attention mechanism and SwinTransformer. Background Art

[0002] The application of steel involves almost every field of human society, such as construction, automobiles, aerospace, precision instruments, etc. In the foreseeable future, there is no material that can completely replace steel, and steel will still be an indispensable material in human society. During the production, transportation, and storage of steel, due to the long transportation time, complex and changeable natural conditions, it is not uncommon for steel to have quality problems. When the purchaser accepts the steel, it is necessary to determine the type and degree of defects in the steel, which requires manual processing. However, the manual processing workload is huge and the response speed is slow, which has a great impact on subsequent production and processing.

[0003] At present, the main methods for steel surface defect recognition are traditional artificial feature classification detection methods and machine learning classification detection methods. Traditional defect recognition algorithms are divided into three categories:

[0004] 1) Statistical recognition algorithm

[0005] This type of algorithm calculates the probability distribution of defects by statistical inference through the regularity and periodicity of statistical pixel distribution, and judges metal surface defects by calculating probability and expectation. This type of method mainly includes: critical value calculation, clustering, boundary detection, two-dimensional fractal, grayscale statistics, conjugate matrix calculation, local binary pattern, morphological feature discrimination and other calculation methods. The advantages of this type of method are small amount of calculation and low hardware requirements; the disadvantages are that the metal surface defects need to have regularity and periodicity, require more accurate initial values, are sensitive to noise data, have average prediction accuracy, and cannot detect small defects such as small scratches and pits.

[0006] 2) Spectroscopy

[0007] This method uses a light source to illuminate the metal surface, analyzes the spectrum of the reflected light, identifies defects and background based on various calculation methods, and then determines whether the metal surface has defects. This type of method mainly includes: Fourier transform, Gabor filter, improved FIR filter, wavelet transform, multi-scale geometric analysis, Hough transform and other calculation methods. The advantages of this type of method are simple structure, mature algorithm, easy implementation and operation; the disadvantage is that it requires a high-stability light source and multiple high-precision spectral receivers, and the equipment investment is large. In the high temperature, high humidity and high corrosion environment of the factory, the equipment damage rate is high and it is not easy to maintain. In addition, only filters are used for data analysis, and the prediction accuracy is poor.

[0008] 3) Model method

[0009] The model method predicts whether the metal surface has defects by converting the original image data into a low-dimensional distribution. This type of method mainly includes: Markov random field model, Weibull distribution model, dynamic contour model and other methods. This type of method is mainly used for the detection and classification of surface materials. Due to the low prediction accuracy, it is generally not used for surface defect detection.

[0010] Among machine learning classification detection methods, common deep learning image recognition algorithms are Faster RCNN, YOLOV5 and other network architecture algorithms. Although common deep learning algorithms have made certain breakthroughs in the field of image recognition, they still cannot meet the needs of steel surface defect recognition, and have low recognition accuracy and slow recognition speed for small-scale defects. Summary of the invention

[0011] In view of this, the purpose of the present invention is to propose a Mask RCNN network steel surface defect detection method, device and computer storage medium based on multi-head self-attention mechanism and Swin Transformer, and to design a deep learning method for the characteristics of steel surface defects to meet the requirements of high-precision and rapid identification of large, medium and small-sized steel surface defects of various categories.

[0012] To achieve the above object, the present invention provides the following technical solutions:

[0013] In one aspect, the present invention provides a method for detecting surface defects of steel, the method comprising:

[0014] Obtain a data set of steel surface defect images, preprocess the steel surface defect images in the data set, and divide them into a training set and a test set; the steel surface defect images are images containing cracks, inclusions, plaques, pitting surfaces, rolling scales or scratches;

[0015] A mask region convolutional neural network Mask RCNN based on a multi-head attention mechanism and a rotational deformation Swin Transformer is constructed, and the Mask RCNN is trained using the training set; the Mask RCNN includes a fixed window multi-head self-attention module, an offset window multi-head self-attention module, a fully connected module, a layer normalization module, a sub-image fusion module, a feature pyramid network FPN, a region proposal network RPN, a region of interest fusion RoI align and a fully connected network; the Mask RCNN uses a multi-head self-attention mechanism and a Swin Transformer as a backbone network, uses FPN to extract a fused feature map, uses RPN to extract a region of interest, obtains a candidate suggestion box through RoI align, uses a fully connected layer and a linear layer to judge the category and extract the suggestion box, and then obtains the defect instance segmentation through the mask branch;

[0016] The trained Mask RCNN is used to perform steel surface defect detection on the steel surface defect images in the test set to obtain steel surface defect detection results.

[0017] Furthermore, the Mask RCNN is trained using the training set, including:

[0018] Input the steel surface defect images in the training set into the backbone network with self-attention mechanism and Swin Transformer to extract feature maps at four different stages;

[0019] Using the FPN method, the feature maps of adjacent stages in the feature maps of the four different stages are fused to obtain four feature maps with different resolutions;

[0020] Input the four feature maps with different resolutions into RPN, and obtain a predetermined number of candidate suggestion boxes according to the original image size;

[0021] The candidate proposal boxes are classified into binary values ​​and pooled using RoI align. They are then classified and regressed through a fully connected network to obtain the detection results and corresponding losses. At the same time, the mask branch is used to segment and annotate defect instances.

[0022] Use the stochastic gradient descent method SGD to adjust the weight of the backbone network based on the multi-head self-attention mechanism and SwinTransformer by calculating the loss. When the number of iterations reaches the set value, the training ends and the weight file is obtained;

[0023] The weight file is loaded into the Swin Transformer architecture based on the multi-head self-attention mechanism to obtain the trained Mask RCNN based on the multi-head attention mechanism and Swin Transformer.

[0024] Furthermore, the feature maps of adjacent stages in the feature maps of the four different stages are fused using the FPN method to obtain four feature maps with different resolutions, including:

[0025] Perform a 1x1 convolution operation on the feature map S5 output from stage 4 to obtain a feature map F5; perform a 3x3 convolution operation on the feature map F5 to obtain a feature map P5;

[0026] Perform a 1x1 convolution operation on the feature map S4 output from stage 3 to obtain the feature map F4, upsample the feature map F5 by 2 times and fuse it with the feature map F4, and then perform a 3x3 convolution operation to obtain the feature map P4;

[0027] Perform a 1x1 convolution operation on the feature map S3 output from stage 2 to obtain the feature map F3, upsample the feature map F4 by 2 times and fuse it with the feature map F3, and then perform a 3x3 convolution operation to obtain the feature map P3;

[0028] A 1x1 convolution operation is performed on the feature map S2 output from stage 1 to obtain the feature map F2. The feature map F3 is upsampled by 2 times and fused with the feature map F2. Then a 3x3 convolution operation is performed to obtain the feature map P2.

[0029] Furthermore, the four feature maps of different resolutions are input into the RPN, and a predetermined number of candidate suggestion boxes are obtained according to the size of the original image, including:

[0030] The original image and feature map are input into the candidate box generator to generate a predetermined number of candidate boxes of 1:1, 1:2, and 2:1 sizes. Then, regression prediction is used to obtain candidate suggestion boxes, and non-maximum suppression is used to obtain filtered candidate suggestion boxes.

[0031] Furthermore, the steel surface defect images in the data set are preprocessed, including:

[0032] The data enhancement operation of cropping and filling is used to obtain the preprocessed steel surface defect image and mark the location and category of the defects.

[0033] Furthermore, the steel surface defect images in the training set are input into the backbone network with self-attention mechanism and SwinTransformer to extract feature maps at four different stages, including:

[0034] Build the swin_transformer_block basic module;

[0035] Build the patch_partition basic module to create subgraphs;

[0036] Construct a stage 1 module, the input of which is the sub-image obtained by the patch_partition basic module; perform mapping and layer normalization operations on the input sub-image; use a convolution kernel of the same size as the sub-image to perform a convolution operation on the input image to obtain a first feature map; use the first feature map as the input of swin_transformer_block, and perform swin_transformer_block operation on the output feature map again, and the obtained second feature map is the input of stage 2;

[0037] Construct a patch_merging basic module, the input of which is a 4x4 feature map. Each 2x2 adjacent pixel is used as a partition sub-map, and the pixels at the same position in each partition sub-map are spliced ​​to obtain four 2x2 feature maps; these four feature maps are spliced ​​in the depth channel direction, and the depth channel of the feature map is doubled through layer normalization operation and a fully connected layer is used to perform linear transformation in the depth direction of the feature map, and a 2x downsampled feature map is output;

[0038] Construct a stage 2 module, perform a patch_merging operation on the input second feature map to obtain a third feature map; use the third feature map as the input of swin_transformer_block, and perform a swin_transformer_block operation on the output feature map again, and the fourth feature map obtained is the input of stage 3;

[0039] Construct a stage 3 module, wherein the stage 3 module performs a patch_merging operation on the input fourth feature map to obtain a fifth feature map; the fifth feature map is used as the input of swin_transformer_block, the output feature map is again subjected to swin_transformer_block operation, and the swin_transformer_block is input twice in a loop, and the obtained sixth feature map is used as the input of stage 4;

[0040] Construct a stage 4 module, wherein the stage 4 module performs a patch_merging operation on the input sixth feature map to obtain a seventh feature map; the seventh feature map is used as an input of a swin_transformer_block, and the output feature map is again subjected to a swin_transformer_block operation to obtain an eighth feature map.

[0041] Furthermore, the swin_transformer_block basic module is constructed, including:

[0042] The input feature map is normalized by LayerNorm operation, and then the feature map obtained by the operation is fused with the original feature map using a fixed window self-attention mechanism, which is recorded as the first fused feature map; the fixed window attention mechanism is to divide the feature map into windows of size 7x7 and perform a multi-head self-attention operation in each window;

[0043] Using a layer normalization LayerNorm operation and a multi-layer perceptron MLP operation on the first fused feature map, fusing the operated features with the fused feature map for a second time to obtain a secondary fused first feature map;

[0044] The first feature map of the secondary fusion is subjected to a layer normalization LayerNorm operation, and then the feature map obtained by the operation is fused with the original feature map using an offset window self-attention mechanism, which is recorded as a second fused feature map; the offset window attention mechanism is to divide the feature map into windows of size 7x7, and offset the windows to realize information interaction between different windows;

[0045] The second fused feature map is subjected to a layer normalization LayerNorm operation and a multi-layer perceptron MLP operation, and the operated features are fused with the fused feature map for a second time to obtain a secondary fused second feature map.

[0046] In another aspect, the present invention further provides a steel surface defect detection device, the device comprising:

[0047] A data set unit is used to obtain a data set of steel surface defect images, preprocess the steel surface defect images in the data set, and divide them into a training set and a test set; the steel surface defect images are images containing cracks, inclusions, plaques, pitting surfaces, rolling oxide scales or scratches;

[0048] A training unit, used to construct a mask region convolutional neural network Mask RCNN based on a multi-head attention mechanism and a rotational deformation Swin Transformer, and train the Mask RCNN using the training set obtained by the data set unit; the Mask RCNN includes a fixed window multi-head self-attention module, an offset window multi-head self-attention module, a fully connected module, a layer normalization module, a sub-image fusion module, a feature pyramid network FPN, a region proposal network RPN, a region of interest fusion RoI align and a fully connected network; the Mask RCNN uses a multi-head self-attention mechanism and a Swin Transformer as a backbone network, uses FPN to extract a fused feature map, uses RPN to extract a region of interest, obtains a candidate suggestion box through RoI align, uses a fully connected layer and a linear layer to judge the category and extract the suggestion box, and then obtains the defect instance segmentation through the mask branch;

[0049] The detection unit is used to perform steel surface defect detection on the steel surface defect images in the test set obtained by the data set unit using the Mask RCNN trained by the training unit to obtain a steel surface defect detection result.

[0050] Furthermore, the training unit comprises:

[0051] The first feature map extraction subunit is used to input the steel surface defect images in the training set into the backbone network with self-attention mechanism and Swin Transformer to extract feature maps of four different stages;

[0052] The second feature map extraction subunit is used to fuse the feature maps of adjacent stages in the feature maps of the four different stages obtained by the first feature map extraction unit by using the FPN method to obtain four feature maps with different resolutions;

[0053] a candidate suggestion box acquisition subunit, configured to input the four feature maps with different resolutions obtained by the second feature map extraction unit into the RPN, and acquire a predetermined number of candidate suggestion boxes according to the original image size;

[0054] An instance segmentation subunit is used to perform binary classification on the candidate suggestion boxes obtained by the candidate suggestion box acquisition unit, and use RoI align for pooling synthesis, and then perform classification and regression through a fully connected network to obtain a detection result and a corresponding loss, and use a mask branch to perform defect instance segmentation and annotation;

[0055] The weight file acquisition subunit is used to adjust the weight of the backbone network based on the multi-head self-attention mechanism and Swin Transformer by calculating the loss using the stochastic gradient descent method SGD. When the number of iterations reaches the set value, the training is completed and the weight file is obtained; the weight file is loaded into the Swin Transformer architecture based on the multi-head self-attention mechanism to obtain the trained Mask RCNN based on the multi-head attention mechanism and Swin Transformer.

[0056] On the other hand, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer instruction set, and when the computer instruction set is executed by a processor, the above-mentioned steel surface defect detection method is implemented.

[0057] Advantages and positive effects of the present invention:

[0058] The present invention discloses a multi-head self-attention mechanism and Swin Transformer that can retain small pixel information and reduce the amount of calculation; thereby, the detection accuracy can be improved, especially the accuracy in detecting small-size defects can be greatly improved; at the same time, the calculation speed can be improved to achieve real-time detection; in addition, the Mask RCNN network can not only detect steel surface defects, but also realize defect instance segmentation, helping front-line workers to more accurately determine the defect location and shape. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0060] Figure 1 Schematic diagram of the Mask RCNN network structure based on the multi-head self-attention mechanism and Swin Transformer in an embodiment of the present invention;

[0061] Figure 2 Schematic diagram of the process of the steel surface defect detection method in an embodiment of the present invention;

[0062] Figure 3 This is a picture detection result diagram of steel surface defects in an embodiment of the present invention;

[0063] In the figure: 1-crack, 2-scratch, 3-pitting surface, 4-plaque, 5-rolling scale, 6-inclusion, and the black box is the predicted defect box. DETAILED DESCRIPTION

[0064] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0065] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0066] Mask RCNN (MaskRegin-based Convolutional Neural Networks) based on multi-head self-attention mechanism and Swin Transformer (rotation deformation) is a deep learning method, such as Figure 1 As shown in the figure, the Mask RCNN based on the multi-head self-attention mechanism and Swin Transformer consists of a fixed window multi-head self-attention module, an offset window multi-head self-attention module, a fully connected module, a layer normalization module, a sub-image fusion module, a Feature Pyramid Network (FPN), a Region Proposal Network (RPN), RoI (Region of Interest) align and a fully connected network. When using the Mask RCNN based on the multi-head self-attention mechanism and Swin Transformer for steel defect detection, the multi-head self-attention mechanism and Swin Transformer are used as the backbone network, the FPN is used to extract the fused feature map, the RPN is used to extract the RoI region of interest, the candidate suggestion box is obtained through RoI align, the fully connected layer and the linear layer are used to judge the category and extract the suggestion box, and then the defect instance segmentation is obtained through the mask branch. Various defects on the surface of steel can be accurately and quickly detected, and the defects can be marked and segmented.

[0067] like Figure 2 As shown, it shows a flowchart of a Mask RCNN steel surface defect detection method based on a multi-head self-attention mechanism and SwinTransformer in an embodiment of the present invention. The method includes the following specific steps:

[0068] S101, obtaining steel surface defect images, performing data preprocessing on the image data, and dividing the image data into a training set and a test set;

[0069] Among them, the steel surface defect images are images containing cracks, inclusions, plaques, pitting surfaces, rolling scale or scratch defects, and the obtained steel surface defect images are divided into a training set and a test set at a ratio of 7:3. In the embodiment of the present invention, a hot-rolled strip surface defect database is used, including 6 types of defects: inclusions, scratches, rolling scale, cracks, pitting surfaces and plaques, 300 images of each type of defect, a total of 1,800 steel surface defect images, and the image size is 200px*200px.

[0070] In the specific implementation, it is necessary to first perform preprocessing operations on the steel surface defect images in the training set. Specifically, the preprocessing operations include: data enhancement operations of cropping and filling to obtain new steel surface defect images with a size of 224×224×3 (H×W×C), and mark the location and category of the defects.

[0071] S102, input the training set images into the backbone network with self-attention mechanism and Swin Transformer, and extract feature maps S2, S3, S4, and S5 at four different stages;

[0072] In step S102, the original image is input into the SwinTransformer architecture with a multi-head self-attention mechanism with an attention module to extract feature maps S2, S3, S4, and S5 at four different stages.

[0073] S103, using the FPN feature pyramid network method, the feature maps of adjacent stages are fused to obtain four feature maps P2, P3, P4, and P5 with different resolutions;

[0074] In step S103, a 1x1 convolution operation is performed on the feature map S5 output from stage 4 to obtain a feature map F5; a 3x3 convolution operation is performed on the feature map F5 to obtain a feature map P5. A 1x1 convolution operation is performed on the feature map S4 output from stage 3 to obtain a feature map F4, the feature map F5 is upsampled by 2 times and fused with the feature map F4, and then a 3x3 convolution operation is performed to obtain a feature map P4. A 1x1 convolution operation is performed on the feature map S3 output from stage 2 to obtain a feature map F3, the feature map F4 is upsampled by 2 times and fused with the feature map F3, and then a 3x3 convolution operation is performed to obtain a feature map P3. A 1x1 convolution operation is performed on the feature map S2 output from stage 1 to obtain a feature map F2, the feature map F3 is upsampled by 2 times and fused with the feature map F2, and then a 3x3 convolution operation is performed to obtain a feature map P2.

[0075] S104, input feature maps P2, P3, P4, and P5 into the RPN region proposal network, obtain a predetermined number of proposal boxes according to the original image size, and generate multiple candidate proposal boxes for an object;

[0076] In step S104, the original image and the feature map are input into the candidate box generator to generate a certain number of candidate boxes of 1:1, 1:2, and 2:1 sizes, and then regression prediction is used to obtain the suggestion boxes, and non-maximum suppression is used to obtain the filtered suggestion boxes.

[0077] S105, input the candidate suggestion box into the regressor for binary classification, then use RoI align for pooling synthesis, and then use the fully connected network for classification and regression to obtain the detection result and the corresponding loss, and use mask_branch to segment and annotate the defect instance;

[0078] In step S105, the candidate suggestion boxes are input into RoI align for pooling synthesis, and then input into the fully connected network for classification and candidate box regression, and 6-channel and 24-channel outputs are obtained, representing the classification and regression results, and the steel surface defect detection loss is calculated. Mask_branck is used for pixel alignment and upsampling operations, and each feature map is restored to the original size, with the number of channels being 6, to obtain defect instance segmentation annotations.

[0079] S106, using the stochastic gradient descent method SGD to adjust the weight of the backbone network based on the multi-head self-attention mechanism and SwinTransformer by calculating the loss. When the number of iterations reaches the set value, the training ends, the weight file is obtained, and then the test set is predicted;

[0080] In step S106, the stochastic gradient descent method SGD is used, the number of images input in each batch during training is batch_size 2, the learning rate is set to 0.00125*batch_size*gpu_nums, gpu_nums is the number of GPU graphics cards, the momentum is set to 0.9, the weight decay rate is set to 0.00005-0.0001, and the weight of the entire Swin Transformer architecture based on the multi-head self-attention mechanism is adjusted through the loss function. The number of iterations is set to 12-24. When the number of iterations is reached, the training ends and the weight file is obtained; the weight file is loaded into the Swin Transformer architecture based on the multi-head self-attention mechanism, and then the images of the test set are predicted to obtain the location, category and confidence of the surface defects of the steel.

[0081] For ease of understanding, step S102 is described in detail below. Step S102 specifically includes the following steps:

[0082] S201. Define a swin_transformer_block basic module.

[0083] In the first step, the input feature map is normalized by LayerNorm operation, and then the fixed window self-attention mechanism is used to fuse the feature map obtained by the operation with the original feature map, which is recorded as the fused feature map. Then the fused feature map is subjected to the normalized LayerNorm operation and the multi-layer perceptron MLP operation, and the operated features are fused with the fused feature map for the second time to obtain the secondary fused feature map as the input for the next step.

[0084] In the second step, the secondary fusion feature map obtained in the first step is subjected to the normalized LayerNorm operation, and then the offset window self-attention mechanism is used to fuse the feature map obtained by the operation with the original feature map, which is recorded as the fused feature map. Then, the fused feature map is subjected to the normalized LayerNorm operation and the multi-layer perceptron MLP operation, and the operated features are fused with the fused feature map for the second time to obtain the secondary fused feature map.

[0085] The fixed window attention mechanism divides the feature map into 7x7 windows and performs multi-head self-attention operations in each window. The offset window attention mechanism divides the feature map into 7x7 windows and offsets the windows to achieve information interaction between different windows, that is, offset rows 0, 1, and 2 to the rightmost side of the feature map, and offset columns 0, 1, and 2 to the bottom of the feature map.

[0086] Then, a multi-head self-attention operation is performed on each window after segmentation. The multi-head self-attention operation treats each window as a feature map, the input feature map sequence length is 4i, and the input node is x i , map the input to vector a through Input Embedding i , vector a i Through three transformation matrices, namely the query matrix W q , keyword matrix W k , information matrix W v Get the corresponding query vector q i , keyword vector k i 、Information vector v i , according to the number of heads h used, the query vector q is further obtained i , keyword vector k i 、Information vector v i Divide the query vector q into h equal parts and divide it into i Then, the query vector q of each head is i,hj With keyword vector k i,hjTo match, v i Represents the input vector a i The information obtained in q i With each k i Perform a dot product operation and divide by d 1 / 2 Get the corresponding attention weight α, where d represents the keyword vector k i The length of the matched vector is written uniformly in the form of matrix multiplication, and then softmax processing is performed on each row to obtain the weight of the information vector v. Each weight is then weighted and uniformly written in the form of matrix multiplication. Then the results obtained from each head are concat-joined, and the concatenated results are passed through the weight parameter matrix W to be trained. o Fusion is performed to obtain the final new feature map b i .

[0087] S202. Define a patch_partition basic module, which implements the creation of subgraph patches.

[0088] The input image is saved in a file with a size of 224x224x3 (H i xW i xC i ) of the tensor x i In , the original image is divided into 56×56×3 sub-image patches using the sub-image size (patch_size is 4x4), and the number of heads is set to 3. Each sub-image patch is expanded one-dimensionally to obtain the number of channels C patch It is a 16x3 pixel sequence, and then the pixel sequence is reassembled according to the sub-image patch position, that is, the original image is downsampled by patch_size.

[0089] S203. Define a Stage 1 module, which takes as input a 56×56×48 sub-image patch and sets the number of heads to 3.

[0090] First, the sub-image is mapped and the layer normalization LayerNorm operation is performed to transform it into a feature map with a channel number C of 96. This process can also be understood as using the sub-image patch to split the original image and input channel C input is 3, the number of output channels is C output is 96. Use the convolution kernel kernel_size of the same size as the sub-image patch_size and a step size of 4 to perform a convolution operation on the input image to obtain a 56x56x96 feature map. This feature map is used as the input of swin_transformer_block, and the output feature map is again subjected to swin_tansformer_block operation, and the obtained feature map is the input of Stage 2.

[0091] S204. Define a patch_merging basic module, which inputs a feature map and outputs a 2x downsampled feature map.

[0092] First, a 4x4 feature map is selected, and then each 2x2 adjacent pixel is used as a partition patch, and then the pixels at the same position in each partition patch are spliced ​​to obtain four 2x2 feature maps. Then these four feature maps are spliced ​​in the depth channel direction, and then a normalization LayerNorm operation and a fully connected layer are used to perform a linear transformation in the depth direction of the feature map to double the depth channel of the feature map.

[0093] S205. Define a stage 2 module whose input is a 56x56x96 feature map and sets the number of heads to 6.

[0094] First, the patch_merging operation is performed on the feature map to obtain a 28x28x192 feature map. This feature map is used as the input of swin_transformer_block, and the output feature map is again subjected to swin_transformer_block operation, and the obtained feature map is the input of Stage 3.

[0095] S206. Define a stage 3 module whose input is a 28x28x192 feature map and set the number of heads to 12.

[0096] First, the feature map is patch_merging to obtain a 14x14x384 feature map. This feature map is used as the input of swin_transformer_block, and the output feature map is again swin_transformer_block operated, and then swin_transformer_block is input twice, and the obtained feature map is the input of Stage 4.

[0097] S207. Define a stage 4 module whose input is a 14x14x384 feature map and set the number of heads to 24.

[0098] First, perform patch_merging operation on the feature map to obtain a 28x28x192 feature map. This feature map is used as the input of swin_transformer_block, and the output feature map is again subjected to swin_transformer_block operation.

[0099] In order to evaluate the detection effect of the steel surface defect detection method in the present invention on steel surface defect images, the mAP (mean Average Precision) evaluation index is used, that is, the average of the average precision AP (Average Precision) of different categories, where the average precision AP is the area under the precision-recall curve. The present invention uses the mean average precision mAP to measure the model performance of defect detection at a threshold of 0.5 for the intersection over union (IoU), where the intersection over union (IoU) refers to the ratio of the intersection area of ​​the predicted box and the true box to the combined area. The detection results are as follows: Figure 3 shown.

[0100] The present invention provides a new method for steel surface defect detection, which uses a multi-head self-attention mechanism and a SwinTransformer backbone network and an FPN feature pyramid network as an effective means of feature extraction, can effectively solve the problem of information loss in small-size pixel areas, improve detection accuracy, and effectively promote the development of steel surface defect detection technology, which has practical promotion value. Moreover, the present invention adopts the Swin Transformer architecture algorithm. Compared with other common deep learning algorithms, Swin Transformer has the advantages of simple structure, higher prediction accuracy, fast prediction speed, ability to detect and classify a variety of surface defects, and accurate identification of even tiny defects.

[0101] Corresponding to a steel surface defect detection method in the present invention, the present invention also provides a steel surface defect detection device, the device comprising:

[0102] A data set unit is used to obtain a data set of steel surface defect images, preprocess the steel surface defect images in the data set, and divide them into a training set and a test set; the steel surface defect images are images containing cracks, inclusions, plaques, pitting surfaces, rolling oxide scales or scratches;

[0103] A training unit, used to construct a mask region convolutional neural network Mask RCNN based on a multi-head attention mechanism and a rotational deformation Swin Transformer, and train the Mask RCNN using the training set obtained by the data set unit; the Mask RCNN includes a fixed window multi-head self-attention module, an offset window multi-head self-attention module, a fully connected module, a layer normalization module, a sub-image fusion module, a feature pyramid network FPN, a region proposal network RPN, a region of interest fusion RoI align and a fully connected network; the Mask RCNN uses a multi-head self-attention mechanism and a Swin Transformer as a backbone network, uses FPN to extract a fused feature map, uses RPN to extract a region of interest, obtains a candidate suggestion box through RoI align, uses a fully connected layer and a linear layer to judge the category and extract the suggestion box, and then obtains the defect instance segmentation through the mask branch;

[0104] The detection unit is used to perform steel surface defect detection on the steel surface defect images in the test set obtained by the data set unit using the Mask RCNN trained by the training unit to obtain a steel surface defect detection result.

[0105] Wherein, the training unit specifically includes:

[0106] The first feature map extraction subunit is used to input the steel surface defect images in the training set into the backbone network with self-attention mechanism and Swin Transformer to extract feature maps of four different stages;

[0107] The second feature map extraction subunit is used to fuse the feature maps of adjacent stages in the feature maps of the four different stages obtained by the first feature map extraction unit by using the FPN method to obtain four feature maps with different resolutions;

[0108] a candidate suggestion box acquisition subunit, configured to input the four feature maps with different resolutions obtained by the second feature map extraction unit into the RPN, and acquire a predetermined number of candidate suggestion boxes according to the original image size;

[0109] An instance segmentation subunit is used to perform binary classification on the candidate suggestion boxes obtained by the candidate suggestion box acquisition unit, and use RoI align for pooling synthesis, and then perform classification and regression through a fully connected network to obtain a detection result and a corresponding loss, and use a mask branch to perform defect instance segmentation and annotation;

[0110] The weight file acquisition subunit is used to adjust the weight of the backbone network based on the multi-head self-attention mechanism and Swin Transformer by calculating the loss using the stochastic gradient descent method SGD. When the number of iterations reaches the set value, the training is completed and the weight file is obtained; the weight file is loaded into the Swin Transformer architecture based on the multi-head self-attention mechanism to obtain the trained Mask RCNN based on the multi-head attention mechanism and Swin Transformer.

[0111] For the steel surface defect detection device in the embodiment of the present invention, since it corresponds to the steel surface defect detection method in the above embodiment, the description is relatively simple. For relevant similarities, please refer to the description of the steel surface defect detection method in the above embodiment, which will not be described in detail here.

[0112] An embodiment of the present invention further discloses a computer-readable storage medium, in which a computer instruction set is stored. When the computer instruction set is executed by a processor, the steel surface defect detection method provided in any of the above embodiments is implemented.

[0113] In the several embodiments provided by the present invention, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0114] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0115] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0116] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting surface defects of steel, characterized in that: The method comprises: Obtain a dataset of steel surface defect images, preprocess the steel surface defect images in the dataset, and divide them into a training set and a test set; the steel surface defect images are images containing cracks, inclusions, plaques, pitting surfaces, rolling scales or scratches; A mask region convolutional neural network Mask RCNN based on a multi-head attention mechanism and a rotational deformation Swin Transformer is constructed, and the Mask RCNN is trained using the training set; the Mask RCNN includes a fixed window multi-head self-attention module, an offset window multi-head self-attention module, a fully connected module, a layer normalization module, a sub-image fusion module, a feature pyramid network FPN, a region proposal network RPN, a region of interest fusion RoI align and a fully connected network; the Mask RCNN uses a multi-head self-attention mechanism and a Swin Transformer as a backbone network, uses FPN to extract a fused feature map, uses RPN to extract a region of interest, obtains a candidate suggestion box through RoI align, uses a fully connected layer and a linear layer to judge the category and extract the suggestion box, and then obtains the defect instance segmentation through the mask branch; Using the trained Mask RCNN to perform steel surface defect detection on the steel surface defect images in the test set to obtain a steel surface defect detection result; using the training set to train the Mask RCNN, including: Input the steel surface defect images in the training set into the backbone network with self-attention mechanism and Swin Transformer to extract feature maps at four different stages; Using the FPN method, the feature maps of adjacent stages in the feature maps of the four different stages are fused to obtain four feature maps with different resolutions; Input the four feature maps with different resolutions into RPN, and obtain a predetermined number of candidate suggestion boxes according to the original image size; The candidate proposal boxes are classified into binary values ​​and pooled using RoI align. They are then classified and regressed through a fully connected network to obtain the detection results and corresponding losses. At the same time, the mask branch is used to segment and annotate defect instances. Use the stochastic gradient descent method SGD to adjust the weight of the backbone network based on the multi-head self-attention mechanism and Swin Transformer by calculating the loss. When the number of iterations reaches the set value, the training ends and the weight file is obtained; The weight file is loaded into the Swin Transformer architecture based on the multi-head self-attention mechanism to obtain the trained Mask RCNN based on the multi-head attention mechanism and Swin Transformer.

2. A method for detecting surface defects of steel according to claim 1, characterized in that: Using the FPN method, the feature maps of adjacent stages in the feature maps of the four different stages are fused to obtain four feature maps with different resolutions, including: Perform a 1x1 convolution operation on the feature map S5 output from stage 4 to obtain a feature map F5; perform a 3x3 convolution operation on the feature map F5 to obtain a feature map P5; Perform a 1x1 convolution operation on the feature map S4 output from stage 3 to obtain the feature map F4, upsample the feature map F5 by 2 times and fuse it with the feature map F4, and then perform a 3x3 convolution operation to obtain the feature map P4; Perform a 1x1 convolution operation on the feature map S3 output from stage 2 to obtain the feature map F3, upsample the feature map F4 by 2 times and fuse it with the feature map F3, and then perform a 3x3 convolution operation to obtain the feature map P3; A 1x1 convolution operation is performed on the feature map S2 output from stage 1 to obtain the feature map F2. The feature map F3 is upsampled by 2 times and fused with the feature map F2. Then a 3x3 convolution operation is performed to obtain the feature map P2.

3. A method for detecting surface defects of steel according to claim 1, characterized in that: The four feature maps with different resolutions are input into the RPN, and a predetermined number of candidate suggestion boxes are obtained according to the size of the original image, including: The original image and feature map are input into the candidate box generator to generate a predetermined number of candidate boxes of 1:1, 1:2, and 2:1 sizes. Then, regression prediction is used to obtain candidate suggestion boxes, and non-maximum suppression is used to obtain filtered candidate suggestion boxes.

4. A method for detecting surface defects of steel according to claim 1, characterized in that: The steel surface defect images in the data set are preprocessed, including: The data enhancement operation of cropping and filling is used to obtain the preprocessed steel surface defect image and mark the location and category of the defects.

5. A method for detecting surface defects of steel according to claim 1, characterized in that: The steel surface defect images in the training set are input into the backbone network with self-attention mechanism and Swin Transformer to extract feature maps at four different stages, including: Build the swin_transformer_block basic module; Build the patch_partition basic module to create subgraphs; Construct a stage 1 module, the input of which is the sub-image obtained by the patch_partition basic module; perform mapping and layer normalization operations on the input sub-image; use a convolution kernel of the same size as the sub-image to perform a convolution operation on the input image to obtain a first feature map; use the first feature map as the input of swin_transformer_block, and perform swin_transformer_block operation on the output feature map again, and the obtained second feature map is the input of stage 2; Construct a patch_merging basic module, the input of which is a 4x4 feature map. Each 2x2 adjacent pixel is used as a partition sub-map, and the pixels at the same position in each partition sub-map are spliced ​​to obtain four 2x2 feature maps; these four feature maps are spliced ​​in the depth channel direction, and the depth channel of the feature map is doubled through layer normalization operation and a fully connected layer is used to perform linear transformation in the depth direction of the feature map, and a 2x downsampled feature map is output; Construct a stage 2 module, perform a patch_merging operation on the input second feature map to obtain a third feature map; use the third feature map as the input of swin_transformer_block, and perform a swin_transformer_block operation on the output feature map again, and the fourth feature map obtained is the input of stage 3; Construct a stage 3 module, wherein the stage 3 module performs a patch_merging operation on the input fourth feature map to obtain a fifth feature map; the fifth feature map is used as the input of swin_transformer_block, the output feature map is again subjected to swin_transformer_block operation, and the swin_transformer_block is input twice in a loop, and the obtained sixth feature map is used as the input of stage 4; Construct a stage 4 module, wherein the stage 4 module performs a patch_merging operation on the input sixth feature map to obtain a seventh feature map; the seventh feature map is used as an input of a swin_transformer_block, and the output feature map is again subjected to a swin_transformer_block operation to obtain an eighth feature map.

6. A method for detecting surface defects of steel according to claim 5, characterized in that: Build the swin_transformer_block basic module, including: The input feature map is normalized by LayerNorm operation, and then the feature map obtained by the operation is fused with the original feature map using a fixed window multi-head self-attention mechanism, which is recorded as the first fused feature map; the fixed window multi-head attention mechanism is to divide the feature map into windows of size 7x7 and perform a multi-head self-attention operation in each window; Using a layer normalization LayerNorm operation and a multi-layer perceptron MLP operation on the first fused feature map, fusing the operated features with the fused feature map for a second time to obtain a secondary fused first feature map; The first feature map of the secondary fusion is subjected to a layer normalization LayerNorm operation, and then the feature map obtained by the operation is fused with the original feature map using an offset window self-attention mechanism, which is recorded as a second fused feature map; the offset window attention mechanism is to divide the feature map into windows of size 7x7, and offset the windows to realize information interaction between different windows; The second fused feature map is subjected to a layer normalization LayerNorm operation and a multi-layer perceptron MLP operation, and the operated features are fused with the fused feature map for a second time to obtain a secondary fused second feature map.

7. A steel surface defect detection device, characterized in that: The device comprises: A data set unit is used to obtain a data set of steel surface defect images, preprocess the steel surface defect images in the data set, and divide them into a training set and a test set; the steel surface defect images are images containing cracks, inclusions, plaques, pitting surfaces, rolling oxide scales or scratches; A training unit, used to construct a mask region convolutional neural network Mask RCNN based on a multi-head attention mechanism and a rotational deformation Swin Transformer, and train the Mask RCNN using the training set obtained by the data set unit; the Mask RCNN includes a fixed window multi-head self-attention module, an offset window multi-head self-attention module, a fully connected module, a layer normalization module, a sub-image fusion module, a feature pyramid network FPN, a region proposal network RPN, a region of interest fusion RoI align and a fully connected network; the Mask RCNN uses a multi-head self-attention mechanism and a Swin Transformer as a backbone network, uses FPN to extract a fused feature map, uses RPN to extract a region of interest, obtains a candidate suggestion box through RoI align, uses a fully connected layer and a linear layer to judge the category and extract the suggestion box, and then obtains the defect instance segmentation through the mask branch; A detection unit is used to perform steel surface defect detection on the steel surface defect images in the test set obtained by the data set unit using the Mask RCNN trained by the training unit to obtain a steel surface defect detection result; the training unit includes: The first feature map extraction subunit is used to input the steel surface defect images in the training set into the backbone network with self-attention mechanism and Swin Transformer to extract feature maps of four different stages; The second feature map extraction subunit is used to fuse the feature maps of adjacent stages in the feature maps of the four different stages obtained by the first feature map extraction unit by using the FPN method to obtain four feature maps with different resolutions; a candidate suggestion box acquisition subunit, configured to input the four feature maps with different resolutions obtained by the second feature map extraction unit into the RPN, and acquire a predetermined number of candidate suggestion boxes according to the original image size; An instance segmentation subunit is used to perform binary classification on the candidate suggestion boxes obtained by the candidate suggestion box acquisition unit, and use RoI align for pooling synthesis, and then perform classification and regression through a fully connected network to obtain a detection result and a corresponding loss, and use a mask branch to perform defect instance segmentation and annotation; The weight file acquisition subunit is used to adjust the weight of the backbone network based on the multi-head self-attention mechanism and Swin Transformer by calculating the loss using the stochastic gradient descent method SGD. When the number of iterations reaches the set value, the training is completed and the weight file is obtained; the weight file is loaded into the Swin Transformer architecture based on the multi-head self-attention mechanism to obtain the trained Mask RCNN based on the multi-head attention mechanism and Swin Transformer.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer instruction set, and when the computer instruction set is executed by the processor, the steel surface defect detection method provided by any one of claims 1 to 6 is implemented.

Images