A steel surface defect detection method based on an infrared thermal imager

Abstract

The application discloses a steel surface defect detection method based on an infrared thermal imager, relates to the technical field of steel surface defect detection, and compared with the prior art, the performance of a detection model is improved by improving a YOLOv5 network model algorithm; compared with a conventional image, an infrared image can better highlight surface defect features and improve defect contrast; and the steel to be detected is divided into multiple images for collection by collecting local images at a time and moving a workbench to collect the images in batches, so that the resolution of infrared image collection is ensured, and the steel production of various specifications is adapted to.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of steel surface defect detection, and particularly relates to a steel surface defect detection method based on an infrared thermal imager. BACKGROUND

[0002] Steel is used in a variety of projects, including the construction of houses, electrical appliances, roads, and other infrastructure such as railways. In today's manufacturing environment, especially in the context of the upcoming Industry 4.0, steel production is of paramount importance. Steel defects can reduce the corrosion resistance, wear resistance, and load-bearing capacity of the product, resulting in significant financial losses. Due to improper operation during the production process, surface defects such as coarse grains and other phenomena can occur. When steel is left in the environment for a long time, it forms an oxide layer on the surface and rusts. When the steel is rolled, it is in contact with the air at high temperatures, forming rolling scales. Due to the contact of the steel's oxide layer or other foreign matter with the high-temperature rolled piece and scratches, various defects can occur on the surface of the steel. Some common surface defects are cracks, inclusions, patches, pitting, rolling scales, and scratches. In actual production processes, different types of steel surface defects often occur. With a variety of defect types and varying scales and shapes, it is a challenging task to detect all types of defects using the same method.

[0003] Historically, expert evaluators have conducted manual steel surface inspections, which are both time-consuming and labor-intensive. Researchers have proposed various automatic detection methods to assist or replace manual inspections. With the development of technology, production efficiency in industrial production is continuously improving, and many automated, non-contact, non-destructive testing techniques have emerged to meet the production testing needs of metal industrial products such as steel plates, which cannot be met by manual detection. Therefore, many enterprises have begun to adopt some technically sophisticated detection methods, such as eddy current and magnetic flux detection in the field of electromechanical technology; there are also some detection methods based on optical technology, such as infrared and laser. However, these methods have many detection limitations, such as limited defect types and sizes that can be detected, insufficient defect resolution, slow detection speed, and poor maintainability and scalability, making it difficult to ensure high quality requirements. The rapid development of image processing, machine learning, and deep learning has brought new detection methods to steel defect detection, which have broad application prospects in steel defect detection tasks due to their non-contact detection, high detection efficiency, and stable detection effect. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a steel surface defect detection method based on an infrared thermal imager, which can not only improve detection accuracy, reduce detection cost and misjudgment rate, but also improve detection stability and efficiency, thereby improving the precision of steel defect detection.

[0005] The technical problems to be solved by the present application are solved by the following technical solutions:

[0006] A steel surface defect detection method based on an infrared thermal imager, comprising the following steps:

[0007] (1) Obtain infrared image data of the steel surface, label and linearly transform and enhance the data to obtain a surface defect image dataset, and divide the dataset images into a training set and a test set according to a proportion.

[0008] (2) Construct an improved YOLOv5 network model by adding an attention mechanism, improving an activation function and adding a small target detection layer, and train the network model using the training set images processed in step (1) to obtain a trained improved YOLOv5 network model.

[0009] (3) Build an image acquisition and detection system, place the measured steel on a displacement workbench, set an infrared thermal imager at a position perpendicular to the plane of the workbench, connect the infrared thermal imager to an industrial computer, pre-load the improved YOLOv5 network model trained in step (2) in the industrial computer, and place an infrared light source on one side to obliquely irradiate the surface of the steel. If the defect is irradiated, it will be diffusely reflected and collected by the thermal imager. The infrared thermal imager and the infrared light source are fixed, the sample position is changed by the displacement workbench, the infrared thermal imager collects images of the corresponding orientation, and uploads them to the industrial computer for detection.

[0010] (4) According to the fixed height of the infrared thermal imager, calculate the range of local collection, divide the steel and its surrounding area into n x m detection parts according to the size of the range, collect local image data of each part of the steel surface by the infrared thermal imager, obtain local infrared images of the steel and send them to the industrial computer, and the industrial computer uses the improved Yolov5 network model trained in step (2) to detect defects.

[0011] (5) Control the displacement workbench to move the steel, move the next detection part of the steel to the collection range of the infrared thermal imager, repeat step (4), and output the final detection result after splicing.

[0012] In the specific technical solution, step (1) is to collect the infrared image data of the steel surface from industrial production, label the defects in the image using LabelImg, and perform rotation, tangent, random cropping, and color space transformation processing on the image to realize data enhancement, and obtain an infrared defect image database of the steel surface.

[0013] In the specific technical solution, step (2) comprises the following steps:

[0014] (2.1) Select YOLOv5 convolutional neural network as the basic framework, build a network model for defect detection, the model is divided into four parts, which are Input, Backbone, Neck and Head.

[0015] (2.2) The original convolution module is improved, and the activation function is changed from LeakyReLU to SiLU, so the original convolution module is improved to CBS module. SiLU activation function is an activation function similar to LeakyReLU, and the advantages of SiLU are as follows: first, SiLU activation function is smooth in the whole input range, without the discontinuity of LeakyReLU, which makes SiLU function more stable in calculating gradient, which helps to improve the convergence speed of the model; second, SiLU function has stronger nonlinearity, which can better fit the complex data distribution; third, the derivative form of SiLU function is simple, which can better propagate the gradient.

[0016] (2.3) Add new detection layer and detection head to the original network structure, which imports the 160x160 detection feature map in the network, used to detect 4x4 and above targets, to enhance the effect of small size target detection.

[0017] (2.4) Add SE attention mechanism in Backbone part, first perform global average pooling on input feature map, compress the feature map from [W, H, C] to [1, 1, C], then use two fully connected layers for excitation operation, build the correlation between channels, generate a weight value for each feature channel, and finally multiply the weight value with the original feature map to weight the features in each channel.

[0018] (2.5) Before training the training set images using the improved Yolov5 network model, the K-means++ clustering algorithm is also used to cluster the surface defect image dataset to obtain the prior box suitable for steel defect detection.

[0019] (2.6) When training, the network training initial parameters need to be set, including picture input size, momentum parameter, weight decay regularization term parameter, picture angle change parameter, saturation and exposure change parameter, hue change parameter, initial learning rate and training iteration number, adjust the momentum parameter, weight decay regularization term parameter, initial learning rate and training iteration number, after training, use mAP to evaluate the network performance, get the trained improved YOLOv5 network model.

[0020] The expression of the activation function SiLU is:

[0021] f LeakyReLU (x)=max(ax,x) (1)

[0022]

[0023] where a is a constant less than 1.

[0024] Compared with the activation function LeakyReLU, the SiLU function has better smoothness and non-linear data capturing ability, and has non-zero gradient in the whole input range, so that the weight is better updated. The improvement of the activation function can alleviate the problems of gradient disappearance and gradient explosion, improve the efficiency, training stability and convergence of the model.

[0025] The SE attention mechanism performs the following operation:

[0026]

[0027] S = sigma(W2(max(0, W1(Z)))) (4)

[0028] F1 = SxF (5)

[0029] where Z(C) represents the scalar value of the Cth channel in the compressed feature map; W represents the width of the feature map; H represents the height of the feature map; C represents the number of channels; F(j, i, C) represents the pixel value of the ith row and jth column of the cth channel in the feature map F; W1 and W2 represent the weight matrices of the two fully connected layers respectively; sigma represents the normalization function Sigmoid, and the size of S is [1, 1, C]; F1 is the weighted feature map, and the size is the same as the original feature map F.

[0030] The steel surface defect types include cracks, inclusions, patches, pitting, rolling scales and scratches.

[0031] The beneficial effects of the present application are: compared with the prior art, the present application improves the performance of the detection model by improving the YOLOv5 network model algorithm; compared with the conventional image based on the infrared image, the infrared image can better highlight the surface defect features and improve the defect contrast; and the present application divides the measured steel into multiple images for acquisition by means of single acquisition of local image and multi-time acquisition by moving workbench, so as to ensure the resolution of infrared image acquisition. Therefore, the steel surface defect detection method provided by the present application can effectively improve the resolution of the acquired infrared image under the premise of ensuring automatic high-speed acquisition, and improve the defect detection precision by improving the performance of the detection model, so as to process large-size steel, thereby being suitable for steel production of various specifications. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is a flowchart of the detection method of the present application;

[0033] Figure 2It is a common steel surface defect data graph;

[0034] Figure 3 It is a network structure diagram of the original YOLOv5 network model;

[0035] Figure 4 It is a network structure diagram of the improved YOLOv5 network model of the application;

[0036] Figure 5 It is a schematic diagram of the image acquisition and detection system of the detection method of the application;

[0037] Figure 6 It is a schematic diagram of the image acquisition scheme of the detection method of the application;

[0038] Figure 7 It is a schematic diagram of the system flow of the detection method of the application in actual application;

[0039] Figure 8 It is an accuracy comparison diagram of the improved YOLOv5 network model and the original YOLOv5 network model in multiple detection experiments;

[0040] Figure 9 It is a detection result comparison diagram of the improved YOLOv5 network model and the original YOLOv5 network model in one detection experiment;

[0041] Figure 10 It is a detection accuracy comparison diagram of the improved YOLOv5 network model and the original YOLOv5 network model for different types of steel defects. DETAILED DESCRIPTION

[0042] In order to make the technical means, creative features, purposes and effects realized by the application easy to understand, the application will be further described below in combination with specific examples and drawings.

[0043] As shown in Figure 1 The application provides a steel surface defect detection method based on an infrared thermal imager, which comprises the following steps:

[0044] (1) Obtain infrared image data of the steel surface, label and perform linear transformation enhancement processing on the data, obtain surface defect image data set, and divide the data set image into a training set and a test set according to a proportion.

[0045] In specific embodiments, some infrared steel images with defects can be collected from the network or taken from factories, production lines and the like, and then the obtained steel images are labeled according to defect types to form a steel defect data set. The defect types of the steel images include cracks, inclusions, patches, pitting, rolling scales and scratches, etc. Figure 2The figures are shown. Among them, figure (a) is a crack, figure (b) is an inclusion, figure (c) is a plaque, figure (d) is a pimple, figure (e) is a rolling scale, and figure (f) is a scratch.

[0046] (2) An improved YOLOv5 network model is constructed by adding an attention mechanism, improving the activation function, and adding a small target detection layer, and the network model is trained using the training set image processed in step (1) to obtain a trained improved YOLOv5 network model.

[0047] (2.1) Select the YOLOv5 convolutional neural network basic framework to build a network model for defect detection. The model is divided into four parts, namely Input, Backbone, Neck and Head. The backbone feature extraction network uses a CSP network, which is composed of 4 C3 residual blocks. Each residual block is composed of convolution, batch normalization and activation function. The feature pyramid network is used to fuse the shallow feature map and the deep feature map. The input picture size is 640x640. The final network will pass through three YOLOv5 prediction heads to produce feature maps of 20x20, 40x40 and 80x80 to predict the results. The network structure is as shown in Figure 3 .

[0048] (2.2) The original convolution module is improved by changing the activation function from LeakyReLU to SiLU. Thus, the original convolution module is improved to a CBS module, and the expression is:

[0049] f LeakyReLU (x)=max(ax,x) (1)

[0050]

[0051] Where a is a constant less than 1.

[0052] (2.3) A new detection layer and detection head (New Detect) is added to the original network structure to enhance the detection effect of small size targets. The detection head imports the 160x160 detection feature map in the network for detecting targets of 4x4 and above.

[0053] (2.4) To further improve the accuracy of steel defect detection, a SE attention module (Squeeze-and-Excitation) is embedded before the output of the backbone of YOLOv5. The module automatically obtains the importance of each feature channel through learning, and uses the importance to enhance the features and suppress the features that are not important to the current task. Further, let the input feature map be F, with a size of [W, H, C], where W and H represent the width and height of the feature map, and C represents the number of channels. First, the input feature map is globally averaged pooled:

[0054]

[0055] wherein Z(C) represents the scalar value of the Cth channel in the compressed feature map, F(j, i, C) represents the pixel value of the ith row and jth column of the Cth channel in the feature map F, and the size of the final output Z is [1, 1, C].

[0056] Then, the weight of each channel is learned through two fully connected layers:

[0057] S = σ(W2(max(0, W1(Z)))) (4)

[0058] wherein W1 and W2 represent the weight matrices of the two fully connected layers respectively, σ represents the normalization function Sigmoid, and the size of S is [1, 1, C].

[0059] Finally, the obtained weight S is multiplied by the original feature map F to weight each channel:

[0060] F1 = SxF (5)

[0061] wherein F1 is the weighted feature map, and the size is the same as the original feature map F.

[0062] According to the above steps, the original YOLOv5 network model is improved to obtain an improved YOLOv5 network model, as shown in Figure 4 .

[0063] (2.5) Further, before training the improved YOLOv5 network model by the steel infrared defect data set, further comprising: using a K-means++ clustering algorithm to cluster the steel defect data set to obtain a prior box suitable for steel defect detection. The specific steps are as follows: randomly select a point in the label information set of the steel defect data set as the cluster center point; calculate the shortest distance D(x) of each point in the label information set and the existing cluster center point, calculate the probability of each point being selected as the next cluster center point, then use the roulette method to select the next cluster center, and repeat this step to select 8 cluster center points; 9 cluster center points can be obtained from the above process, calculate the distance of each point in the label information set to the 9 cluster center points, and according to the size of the distance, the 9 clusters can be obtained by assigning each cluster center; the average value of each cluster is taken as the center of each cluster, and then the distance is repeatedly calculated until the cluster center no longer changes or changes very little.

[0064] (3) Build an image acquisition and detection system, place the measured steel on the displacement workbench, set the infrared thermal imager at a position perpendicular to the plane of the workbench, connect the infrared thermal imager to the industrial computer, pre-load the improved YOLOv5 network model trained in step (2) on the industrial computer, and place an infrared light source on one side to irradiate the surface of the steel at an angle. If the defect is irradiated, it will be diffusely reflected and collected by the thermal imager. The infrared thermal imager and the infrared light source are fixed, so the sample position needs to be changed by the displacement table to make the infrared thermal imager collect the image of the corresponding orientation and upload it to the industrial computer for detection.

[0065] The built image acquisition and detection system is shown in Figure 5 The bottom white part is the displacement workbench, the dark gray part is the sample to be measured, and the vertical dashed box in the figure is the range of single local data collection of the infrared thermal imager. The infrared light is input from the side at an angle, and the infrared thermal imager is fixed at a position perpendicular to the plane of the measured sample. In this way, the dark field imaging method is used to increase the contrast of the defect image of the target surface to be measured, and the collected infrared image data is sent to the industrial computer. The industrial computer loads the trained weight file in advance, and performs real-time defect detection and saves the results whenever it receives image data.

[0066] When the industrial computer performs defect detection, firstly, the input image will undergo a series of preprocessing steps, including image scaling, normalization, padding and the like, to ensure that the input image meets the requirements of the model; after preprocessing, the image will be input into the improved YOLOv5 network model for forward propagation, in which the image will pass through the backbone network, the feature pyramid network and the neck network to extract the features of the image; then in the head network, the feature map is further processed through the calculation of convolution layers, fully connected layers and activation functions to generate a series of bounding boxes, each bounding box containing the class, position and confidence of a target to predict the class and position of the target; since there may be overlapping bounding boxes, a non-maximum suppression algorithm is used to remove redundant detection results, the bounding boxes are sorted according to the confidence, and the bounding boxes with high overlap are gradually removed, and only the bounding box with the highest confidence is retained as the final detection result; finally, the model outputs the detected target bounding box for target positioning and identification.

[0067] (4) According to the fixed height of the infrared thermal imager, the range of local collection is calculated, and the steel and its periphery are divided into n x m detection parts according to the range size. Each part of the steel surface is collected by the infrared thermal imager to obtain the local infrared image of the steel, and the image is sent to the industrial computer. The industrial computer uses the improved YOLOv5 network model in step (2) to perform defect detection.

[0068] Wherein, the distance from the infrared thermal imager to the sample plane is D, the length range that can be collected is H, and the width range is W:

[0069]

[0070]

[0071] Wherein, α is the horizontal field of view angle of the infrared thermal imager, and β is the vertical field of view angle of the infrared thermal imager.

[0072] The calculated W and H are the sizes of a single detection part, and are also the X and Y direction displacement steps of the displacement workbench.

[0073] (5) Control the displacement workbench to move, move the next detection part of the steel to the collection range of the infrared thermal imager, repeat step (4), until all the detection part images are collected, and the final detection result is output after splicing.

[0074] As Figure 6As shown, taking the example of being divided into 2x6 parts by steel materials, the parts are numbered 1 to 12, the initial position of collection is the leftmost No. 1 area, the collection process needs to displace the displacement table along the X direction 6 times, after each time of displacement along the X direction and collection, it also needs to displace along the Y direction and collect again, and the No. 1 area is sequentially collected in the numerical order until the No. 12 area, thus, after 12 times of collection and detection, the industrial computer saves 12 steel material infrared image detection results, which are spliced and output.

[0075] Whether the detection effect of the modified YOLOv5 network model is improved compared with the original YOLOv5 network model needs to be verified, and mAP (mean Average Precision) is used as an index for measuring the performance of the model, which is an index for evaluating the accuracy of the algorithm at different confidence thresholds by calculating the overlap (IoU) between the predicted frame and the real labeled frame, and sorting the predicted frame according to the confidence, and mAP@0.5 represents the average precision when the IoU threshold is 0.5, which is one of the commonly used evaluation indexes in target detection, and it can comprehensively consider the performance of the detection algorithm on different categories, and has a certain reference value for comparing the accuracy of different algorithms or different models.

[0076] Table 1

[0077]

[0078] In the ablation experiment, four commonly used attention mechanisms, namely CBAM (Convolutional Block Attention Module), ECA (Efficient Channel Attention Module), CA (Coordinate Attention) and SE (Squeeze-and-Excitation) network, are tested. Theoretically, the performance of CBAM is the best, but the effect is not good when applied to steel plate surface defect detection. After comparison, the SE network has the best detection effect and the strongest universality, and can adapt to the situation of low image data contrast and lack of features in the steel detection process, and improve the model precision.

[0079] Figure 8 The mAP@0.5 comparison chart of the improved YOLOv5s network model of the application and the original YOLOv5s network model for the detection experiment of the steel data set. From Figure 8 It can be seen that after 200 times of training, the recognition result of the improved YOLOv5s network model of the application for the steel surface defect is obviously higher than that of the original YOLOv5 network model.

[0080] Further, Figure 2The six types of defects shown represent pictures, and the test pictures are detected by the improved YOLOv5s network model and the original YOLOv5 network model respectively, and the detection results are as shown in Figure 9 , wherein (a) is the detection result of the original YOLOv5 network model, and (b) is the detection result of the improved YOLOv5s network model of the application. Figure 9 It can be seen that the model performance of the improved YOLOv5s network model of the application has certain improvement compared with the original YOLOv5s network model, and the detection of each defect is more accurate.

[0081] Further, the detection accuracy of the two models is counted according to the types of steel surface defects, and taking the six types of defects of cracks, inclusions, patches, pits, rolling scales and scratches as examples, the detection accuracy of each type of defect of the improved YOLOv5s network model and the original YOLOv5s network model is as shown in Figure 10 , wherein (a) is the detection accuracy of the original YOLOv5 network model, and (b) is the detection accuracy of the improved YOLOv5s network model of the application. Figure 10 It can be seen that the detection accuracy of the improved YOLOv5s network model of the application for cracks, inclusions, patches and scratches is greatly improved, and the detection accuracy for pits and rolling scales is the same as that of the original YOLOv5s network model.

[0082] The above shows and describes the basic principles and main features of the application and the advantages of the application. It should be understood by those skilled in the art that the application is not limited by the above examples, and the above examples and descriptions in the specification are only to illustrate the principles of the application, and various changes and improvements can be made without departing from the spirit and scope of the application, and these changes and improvements all fall within the scope of the claimed application. The scope of protection of the application is defined by the appended claims and their equivalents.

Claims

1. A method for detecting surface defects of steel material based on an infrared thermal imager, characterized in that, Comprise the following steps: (1) Obtain the infrared image data of the surface of the steel, label and linearly transform and enhance the data, obtain the surface defect image dataset, and divide the dataset images into training set and test set according to the proportion; (2) Construct an improved YOLOv5 network model by adding attention mechanism, improving activation function and adding small target detection layer, and train the network model using the training set images processed in step (1) to obtain the trained improved YOLOv5 network model; 2.1 Select YOLOv5 convolutional neural network as the basic framework to build a network model for defect detection, which is divided into four parts, namely Input, Backbone, Neck and Head; 2.2 Improve the original convolution module, change the activation function from LeakyReLU to SiLU, and thus improve the original convolution module to CBS module; 2.3 Add a new detection layer and detection head to the original network structure, which imports the 160x160 detection feature map in the network for detecting targets of 4x4 and above; 2.4 Add SE attention mechanism to the Backbone part, first perform global average pooling on the input feature map, compress the feature map from [W, H, C] to [1, 1, C], then use two fully connected layers for excitation operation, build the correlation between channels, and generate a weight value for each feature channel, and finally multiply the weight value with the original feature map to weight the features in each channel; (3) Build an image acquisition and detection system, place the measured steel on the displacement workbench, set the infrared thermal imager perpendicular to the plane of the workbench, connect the infrared thermal imager to the industrial computer, pre-load the improved YOLOv5 network model trained in step (2) into the industrial computer, and place an infrared light source on one side to irradiate the surface of the steel at an angle. If the defect is irradiated, it will be diffusely reflected and collected by the thermal imager. The infrared thermal imager and the infrared light source are fixed, the sample position is changed by the displacement workbench, the infrared thermal imager collects the image of the corresponding orientation, and uploads it to the industrial computer for detection; (4) According to the fixed height of the infrared thermal imager, calculate the range of local acquisition, divide the steel and its surrounding area into n*m detection parts according to the size of the range, collect local image data of each part of the steel surface by the infrared thermal imager, obtain the local infrared image of the steel and send it to the industrial computer. The industrial computer uses the improved Yolov5 network model trained in step (2) to detect defects; (5) Control the displacement workbench to move the steel, move the next detection part of the steel to the infrared thermal imager acquisition range, repeat step (4), until all the detection part images are collected, and output the final detection result after splicing.

2. The steel surface defect detection method according to claim 1, characterized by, Step (1) is to collect the infrared image data of the surface of the steel from industrial production, label the defects in the image using LabelImg, and perform rotation, tangent, random cropping, and color space transformation to realize data enhancement, and obtain the infrared defect image database of the surface of the steel.

3. The steel surface defect detection method according to claim 1, characterized by, Step (2) comprises the following steps: 2.5 Before training the training set images using the improved Yolov5 network model, the surface defect image dataset is also clustered by using the K-means++ clustering algorithm to obtain the prior box suitable for steel defect detection; 2.6 When training, the network training initial parameters need to be set, including picture input size, momentum parameter, weight decay regularization term parameter, picture angle change parameter, saturation and exposure change parameter, hue change parameter, initial learning rate and training iteration number, adjust the momentum parameter, weight decay regularization term parameter, initial learning rate and training iteration number, after training, use mAP to evaluate the network performance, get the trained improved YOLOv5 network model.

4. The steel surface defect detection method according to claim 3, characterized by, The expression of the activation function SiLU is: f LeakyReLU (x) = max(ax, x) (1) Wherein, a is a constant less than 1.

5. The steel surface defect detection method according to claim 3, characterized by, The SE attention mechanism performs the following operation: S=σ(W2(max(0,W1(Z)))) (4) F1=S×F (5) Wherein, Z(C) represents the scalar value of the Cth channel in the compressed feature map; W represents the width of the feature map; H represents the height of the feature map; C represents the number of channels; F(j, i, C) represents the pixel value of the ith row and jth column of the cth channel of the feature map F; W1 and W2 represent the weight matrices of the two fully connected layers respectively; sigma represents the normalization function Sigmoid, and the size of S is [1, 1, C]; F1 is the weighted feature map, and the size is the same as the original feature map F.

6. The method of inspecting a surface of a steel material according to any one of claims 1 to 5, characterized in that, The steel surface defect types include cracks, inclusions, patches, pitting, rolling scales and scratches.

Images