A metal surface defect detection method based on STM R-CNN

Abstract

The application provides a metal surface defect detection method based on an STM R-CNN, uses a Swin Transformer as a backbone feature extraction network, uses a Mix-FPN mixed feature pyramid as a feature extraction layer, designs a metal surface image detection algorithm of a cascaded region convolutional neural network framework, applies a transformer structure to a metal surface flaw detection field, and compared with a deep learning target detection algorithm based on a convolution network alone, can achieve more accurate detection effects. First, the Swin Transformer is used as the backbone feature extraction network to replace a conventional residual network structure, and the feature network is enhanced to extract deep semantic information hidden in an image. Then, a Mix-FPN mixed feature pyramid network is designed, different feature layer information is fused through the feature pyramid, a Multi-stage R-CNN cascaded structure is further designed, each stage is focused on detecting a specific range of Region Proposal through different IoU thresholds. Finally, a soft non-maximum suppression (Soft-NMS) and FP16 mixed precision training are used to optimize and improve the model performance.

Description

TECHNICAL FIELD

[0001] The application relates to application of a deep learning algorithm to traditional industrial metal defect detection, in particular to a metal surface defect detection method based on an STM R-CNN. BACKGROUND

[0002] Industrial metal profiles are widely used in various industrial construction engineering construction, and are also main raw materials of large steel structures, and play an important role in various application scenarios. As a basic material for industry, the production quality of metal profiles affects the safety, quality and service life of engineering construction at all times, and in some high-precision industrial application scenarios, the quality and appearance of materials have extremely strict standards. However, due to the limitations of steel production process and production environment, various problems are inevitable in steel. In order to meet the needs of different standards of steel in various industrial application scenarios, corresponding steel quality detection is also emerging. The present application mainly solves the problem of hot-rolled strip surface defect detection. Industrial hot-rolled strip refers to strip steel with a thickness of 0.1-2 mm and a width of 60-200 cm, which is widely used in the production and manufacturing of automobile, shipbuilding, electrical equipment, engineering construction and other industrial equipment. The completeness and flawlessness of the surface of the strip steel directly affect the service life and value of the subsequent products, so the strip surface defect detection is an important link in the production quality detection of hot-rolled strip.

[0003] Traditional steel surface detection methods include manual detection, magnetic powder detection, penetration detection, eddy current detection, X-ray detection and ultrasonic detection technology, and machine vision detection method. Manual detection is low in efficiency, ultrasonic waves are not suitable for materials with complex surfaces, and the previous machine vision detection method has the problems of difficulty in implementation, high requirement for equipment precision, high design threshold, unfriendliness to experiment designers, high maintenance cost and great environmental impact.

[0004] In recent years, with the rapid development of deep learning, the feature extraction characteristics of convolution operators have greatly developed the technology of image processing. The rapid development of integrated circuits and system architecture according to Moore's law has solved the device limitation of digital image processing technology, brought rapid capacity improvement, and made more powerful GPUs available for large-scale parallel computing, solving the computing power problem that has always limited deep learning. The traditional digital image processing has been revolutionarily improved, and the target detection method based on image processing has been rapidly applied and developed.

[0005] The target detection algorithm based on deep learning can be divided into two-stage algorithm and one-stage algorithm from structure, and the representatives are Faster-RCNN, YOLO series and SSD. The above methods have achieved good results in metal surface defect detection, and the detection effect basically meets the demand of industrial application, but with the update iteration of technology, more advanced methods are needed to further improve the quality and accuracy of defect detection. SUMMARY

[0006] In view of the problems in the prior art: the defect types on the surface of the metal profile are various and similar in characteristics, the defect shapes differ greatly, the area sizes differ greatly, and it is difficult to realize the rapid convergence of the algorithm and accurately detect under the condition of a small amount of data set. The present application proposes a metal surface defect detection algorithm based on STM R-CNN to solve the above problems.

[0007] The technical scheme adopted by the present application to achieve the above-mentioned purpose is:

[0008] A metal surface defect detection method based on STM R-CNN, comprising the following steps:

[0009] S1, collecting metal surface image data and data enhancement, label classification, and establishing a paired data set with classification labels;

[0010] S2, establishing a metal surface defect detection network of STM R-CNN, including the following four network modules: a backbone feature extraction network module, using a transformer operation unit to extract features of different dimensions from the input data; a Mix-FPN mixed feature extraction network module, further mixing the feature maps of different dimensions to obtain enhanced features; an RPN network module, used for iterative training of the enhanced features, outputting a region of interest and a defect boundary prediction box; a Multi-stage R-CNN multi-stage detection network module, used for iterative training on the region of interest output by the RPN network module, combining flexible non-maximum suppression to further output a defect boundary prediction box and a predicted classification label step by step;

[0011] S3, real-time collection of metal surface image data, input into the metal surface defect detection network of STM R-CNN, automatic positioning of the defect boundary prediction box and output of the predicted classification label.

[0012] The label classification is artificial classification of defects.

[0013] The backbone feature extraction network module extracts features of different dimensions from the input data, including:

[0014] 1) Cut the original image of HxWx3 into 4x4 image blocks patch, then expand it into linear dimension, and add the pixel position of each image block patch in the image;

[0015] 2) Input the sequentially connected transformer operation unit, respectively obtain four different dimension feature maps stage1-stage4.

[0016] The Mix-FPN mixed feature extraction network module adopts a cross-layer cross-data fusion method to output five enhanced feature maps p1-p5 for the four feature maps stage1-stage4 input.

[0017] The cross-layer cross-data fusion method comprises:

[0018] a. Adopt T4+T2, T3+T1, T4+T2+T3, T3+T1+T4 feature fusion, and then perform convolution operation to output p1-p4 feature information;

[0019] b. Perform 3x3 and stride=2 convolution operation on stage4 to obtain feature information p5;

[0020] Among them, T1-T4 are respectively four different dimension feature maps stage1-stage4 output by the backbone feature extraction network module, which are obtained by 1x1 convolution channel transformation.

[0021] The T4+T2, T3+T1, T4+T2+T3, T3+T1+T4 feature fusion comprises:

[0022] 1) Perform 4 times nearest neighbor up-sampling on T4, then perform add operation with T2 to obtain new

[0023] 2) Perform 4 times nearest neighbor up-sampling on new , then perform add operation with T1 to obtain new

[0024] 3) Perform 8 times random down-sampling on new , then perform add operation with T4 to obtain new

[0025] 4) Perform 2 times random down-sampling on new , then perform add operation with T3 to obtain new

[0026] 5) Perform 3x3 convolution on new all, to obtain four scale final outputs p1-p4.

[0027] The loss function of the RPN network module and the Multi-stage R-CNN multi-cascade detection network module is a classification cross-entropy loss and a bounding box regression loss.

[0028] The present application has the following advantages and benefits:

[0029] 1. The most advanced transformer-based feature extraction backbone architecture is used as the basic feature extraction network to realize the feature extraction capability of enhancing long semantic information.

[0030] 2. A Mix-FPN mixed feature pyramid network (Mixed dense feature pyramid networks, Mix-FPN) framework is designed to enhance the characteristics of the algorithm adapting to the large-scale changes of the detection target by mixing high and low layer feature semantic information.

[0031] 3. A Multi-stage R-CNN multi-cascade structure is designed to realize a multi-threshold step-up strategy through cascading R-CNN detection stages, so as to improve the detection accuracy.

[0032] 4. Soft non-maximum suppression and FP16 mixed precision training are used to realize fast convergence, improve the accuracy of detection, and reduce the training time. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 The STM R-CNN metal surface defect detection algorithm flowchart of the present application;

[0034] Figure 2 The Swin Transformer data processing process schematic diagram of the present application;

[0035] Figure 3 The Swin Transformer backbone algorithm model structure diagram used in the present application;

[0036] Figure 4 The Mix-FPN mixed feature pyramid extraction algorithm model structure diagram of the present application;

[0037] Figure 5 The region proposal network structure diagram of the present application;

[0038] Figure 6 The Multi-stage R-CNN multi-cascade detection network R-CNN algorithm model diagram of the present application. DETAILED DESCRIPTION

[0039] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the specific implementation method of the present application is described in detail below in combination with the drawings. In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the spirit of the present application, so the present application is not limited to the specific implementation disclosed below.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0041] As Figure 1 shown, the metal surface defect detection algorithm based on STM R-CNN proposed herein mainly consists of four parts, and the specific functions are as follows:

[0042] (1) Swin Transformer backbone network: transformer is developed from self-attention operation unit, which is different from the classical convolution operation unit structure in convolutional neural network. It has outstanding long semantic information acquisition ability in natural language processing technology, which will replace the latter, and unify the trend of computer vision and natural language processing two fields;

[0043] (2) Mix-FPN mixed feature extraction network: the feature maps output by the backbone network are fused according to the mixed mode to obtain five fused feature maps. The feature fusion is according to the high-level semantic information and the bottom layer fusion, the cross-layer fusion rule, which maximizes the cross mixing of feature information and enhances the representation ability of feature image;

[0044] (3) Multi-stage R-CNN layer: the single stage detection threshold used in the traditional way is subject to single condition, which cannot realize the unification of detection precision and threshold value, and seriously limits the further improvement of detection precision. We use multi-stage method, which can effectively avoid the linear constraint relationship between threshold value and detection precision through the setting of multiple threshold values, realize stronger relationship fitting, and further improve the detection precision;

[0045] (4) Soft-NMS: the flexible non-maximum suppression (NMS) algorithm will force the score of adjacent detection frame to be zero, which will cause the overlapping real object to be forced to zero, resulting in detection failure and decrease of average detection precision (average precision, AP).

[0046] Traditional NMS reset function:

[0047]

[0048] s i For detection box score, iou(M, b i ) is the intersection over union function of the real box and the detection box, N t is the set overlap threshold; using Soft-NMS algorithm, by reducing the score of overlapping detection box instead of forced zero operation, the overlapping detection box is retained.

[0049] Soft-NMS reset function:

[0050]

[0051] When the detection box exceeds the overlap threshold, the detection box score is reset linearly decays, the detection box close to M decays more, and the detection box far away from M is less affected.

[0052] As Figure 2 , 3 shown, Swin Transformer backbone network data processing algorithm, directly divide the image into fixed size batch patchs, get batch embedding patch through linear transformation, similar to linear embedding in natural language processing, sequence input transformer after picture classification and other operations. Swin Transformer backbone network first divides the HxWx3pix picture into 4x4 picture blocks (patch), flattens the patch into linear dimension, and then converts it into tokens embedding. Add position embedding based on token embedding. Input into the custom number of Transformer Encoder modules. (Each (H / 4)x(W / 4)x3pix patch represents a token.)

[0053] Backbone has a total of 4 stages, and each stage outputs a feature map. The size of the output feature map is (C=96):

[0054] ①:

[0055] ②:

[0056] ③:

[0057] ④:

[0058] As shown in Figure 4 The Backbone outputs 4 feature maps stage1, stage2, stage3, stage4, which are transformed into T1~T4 through 1x1 convolution channel, and the channel is unified to 256.

[0059] From T4, first perform 4 times nearest neighbor upsampling, and then add operation with T2 to obtain new

[0060] The new perform 4 times nearest neighbor upsampling, and then add operation with T1 to obtain new

[0061] The new perform 8 times random downsampling, and then add operation with T4 to obtain new

[0062] The new perform 2 times random downsampling, and then add operation with T3 to obtain new

[0063] The new fused all perform 3x3 convolution respectively to obtain 4 scale final outputs p4~p1;

[0064] In order to provide a feature map with a very large receptive field and detect large scale features, stage4 is subjected to 3x3 convolution operation with stride=2 to obtain p5. The Mix-FPN module realizes stage1~stage4, 4 feature map inputs, p1~p5, and 5 feature map output sizes:

[0065] p1:

[0066] p2:

[0067] p3:

[0068] p4:

[0069] p5:

[0070] As shown in Figure 5 , 6As shown, the feature map from the feature extraction network enters the RPN network, first passes through a 3x3 convolution, and then passes through a 1x1 convolution, respectively, to generate class division and boundary box prediction, respectively. The classification prediction is mainly used for binary classification prediction of foreground and background, and the boundary box is used as the input of the Multi-stage R-CNN cascade network. The results of simultaneous prediction will be combined with the anchor frame generator to generate a region pre-frame and a label for loss calculation.

[0071] The loss function of RPN and Multi-stage R-CNN network is:

[0072]

[0073] In formula (3), i represents the anchor index, p i represents the prediction probability of the i-th prediction frame anchor as the real label, represents 1 for the corresponding positive sample and 0 for the negative sample, which ensures that when the anchor is a negative sample, there is no boundary box regression loss. i represents the boundary box regression value of the i-th predicted anchor, represents the corresponding i-th real frame value, which calculates the offset of the anchor and the real frame. N cls Minimum batch size, N reg is the number of prediction frames Anchor Location. L cls is the cross-entropy loss, L reg is the SmoothL1Loss.

[0074] Formula (3), the loss function of the RPN network is the classification cross-entropy loss and the boundary box regression loss.

[0075] ① Classification cross-entropy loss formula (Cross EntropyLoss): The classifier in the RPN network divides the candidate frame into foreground and background, which is a binary classification problem. The prediction result has only two p and 1-p, and the formula is:

[0076]

[0077] where p i represents the probability of the i-th anchor prediction as the real label, 1 when it is a positive sample, and 0 when it is a negative sample;

[0078] ② The multi-class function of the classification cross-entropy loss will be used in the R-CNN module:

[0079]

[0080] where M is the number of classes, y jc is the indicator function (0 or 1), taking 1 if the true class of sample j equals c, and 0 otherwise; p jc is the predicted probability that observation j belongs to class c.

[0081] ③Bounding box regression loss:

[0082]

[0083]

[0084]

[0085]

[0086]

[0087] where x, y, w, h represent the coordinate positions of the real (label) box respectively, and x a ,y a ,w a ,h a represent the coordinates of the anchor prediction box respectively. The regression loss function is calculated by using formula (7) to calculate the loss function of the anchor prediction box and the real box.

[0088] As Figure 6 shown, the data set of the application adopts a hot-rolled strip steel public data set, which has been labeled and classified, and the classification includes six types of rolled-in scale (RS), plaque (Pa), crack (Cr), pitting surface (PS), inclusion (In) and scratch (Sc), each type has 300 pictures, and a total of 1800 pictures.

[0089] The detection results of traditional R-CNN network have high antagonism regardless of the threshold setting. If the threshold is set high, the predicted bounding box (x) and the real bounding box (y) will contain a lot of background, making it difficult for the network to obtain positive sample data. If the threshold is low, the network can obtain more positive samples, but there will be more non-real samples. Therefore, it is difficult to set the threshold through a single network model. In order to improve the detection ability of the model, a cascade architecture is constructed, and the threshold of the detector module is continuously improved, which is (0.55, 0.65, 0.75) respectively. By using the regression output of the previous stage for resampling, some extreme values are removed by increasing the IoU threshold, optimizing the deep detector, and improving the overall performance. The intersection and union ratio of the predicted bounding box (bounding box predict) and the real bounding box (real bounding box) is calculated:

[0090]

[0091] The predicted value of the bounding box is obtained by the IoU of the RPN network, which is sent to the first stage of the R-CNN network. If the IoU value exceeds the set threshold, it is sent to the second stage. The IoU threshold of the second stage is higher, and it is further filtered and sent to the third stage. Therefore, in the RPN network and the Multi-stage R-CNN network, the Soft-NMS flexible non-maximum suppression algorithm is adopted in the metal surface defect detection, and the detection accuracy is improved by realizing the nearest neighbor decay strategy. The formula of Soft-NMS is:

[0092]

[0093] Where g y is the detection frame score, IoU(x, y) is the intersection and union ratio of the detection frame and the real frame, and u is the threshold of non-maximum suppression.

[0094] The feature map of Banckbone is input into the RPN and Multi-stage R-CNN network parts at the same time, receives box_pred_0 from the RPN network, and adjusts box_pred and cls_logist through cascading multiple R-CNN modules, each module sets different thresholds. The final test result is the average value of n R-CNN modules, and the prediction of bbox is the output of the last R-CNN.

[0095] The above is the preferred embodiment of the present application, it should be pointed out that, for those skilled in the art, without departing from the principles described in the present application, can also make several improvements and refinements, these improvements and refinements should be considered as the scope of protection of the present application.

Claims

1. A method for metal surface defect detection based on STM R-CNN, characterized in that, The method comprises the following steps: S1, collecting metal surface image data and data enhancement, label classification, and establishing a paired data set with classification labels; S2, establishing an STM R-CNN metal surface defect detection network, comprising the following four network modules: a backbone feature extraction network module, using a transformer operation unit to extract features of different dimensions from the input data; a Mix-FPN mixed feature extraction network module, further mixing the feature maps of different dimensions to obtain enhanced features; an RPN network module, used for iterative training of the enhanced features, outputting a region of interest and a defect boundary prediction box; a Multi-stage R-CNN multi-cascade detection network module, used for iterative training on the region of interest output by the RPN network module, combining flexible non-maximum suppression to further output a defect boundary prediction box and a predicted classification label at each level; S3, real-time collection of metal surface image data, input into the STM R-CNN metal surface defect detection network, automatic positioning of the defect boundary prediction box and output of the predicted classification label.

2. The metal surface defect detection method based on STM R-CNN according to claim 1, wherein, The label classification is performed by manual classification of defects.

3. The metal surface defect detection method based on STM R-CNN according to claim 1, wherein, The backbone feature extraction network module extracts features of different dimensions from the input data, comprising: 1) cutting an original image of HxWx3 into 4x4 image blocks patch, then expanding it into a linear dimension, and adding the pixel position of each image block patch in the image; 2) inputting into sequentially connected transformer operation units to obtain four feature maps stage1-stage4 of different dimensions, respectively.

4. The metal surface defect detection method based on STM R-CNN according to claim 1, characterized in that, The Mix-FPN mixed feature extraction network module uses a cross-layer cross-data fusion method to input four feature maps stage1-stage4 to output five enhanced feature maps p1-p5.

5. The metal surface defect detection method based on STM R-CNN according to claim 4, characterized in that, The cross-layer cross-data fusion method comprises: a. using T4+T2, T3+T1, T4+T2+T3, T3+T1+T4 feature fusion, then performing convolution operation to output p1-p4 feature information; b. performing 3x3 and stride=2 convolution operation on stage4 to obtain feature information p5; Wherein, T1-T4 are respectively stage1-stage4, four different dimension feature maps output by the backbone feature extraction network module, obtained through 1x1 convolution channel transformation.

6. The metal surface defect detection method based on STM R-CNN according to claim 1, wherein, The T4+T2, T3+T1, T4+T2+T3, T3+T1+T4 feature fusion comprises: 1) 4x nearest neighbor upsample T4, then add with T2 to get new 2) new = new + T1 4x nearest neighbor up-sampling is performed, followed by an add operation with T1 to get the new 3) new = new + T4 8x random down-sampling followed by an add operation with T4 to get new 4) new 2x random down-sampling followed by an add operation with T3 to get a new 5) on the new fused Each of them is followed by a respective 3x3 convolution to obtain the final outputs p1~p4 at 4 scales.

7. The metal surface defect detection method based on STM R-CNN according to claim 1, wherein, The loss function of the RPN network module and the Multi-stage R-CNN multi-cascade detection network module is a classification cross-entropy loss and a boundary box regression loss.

Images