A method for detecting surface defects of hot-rolled strip steel based on deep learning

Abstract

The present invention provides a hot-rolled strip surface defect detection method based on deep learning, comprising: S1 to S2: constructing a hot-rolled strip surface defect image data set and dividing it into a training set and a test set; S3: constructing a YOLOv5s network model; S4: modifying a backbone network convolution module to a RepVGG Block module; S5: changing an upsampling mode to a CARAFE upsampling mode; S6: changing a downsampling mode to a Pyconv downsampling mode; S7: model pruning; S8: optimizing a loss function; S9: training and testing a hot-rolled strip surface defect detection model based on a training set, a test set and an optimized loss function; S10: inputting a hot-rolled strip surface defect image to be detected into a qualified hot-rolled strip surface defect detection model to obtain a hot-rolled strip surface defect detection result; the method is used to improve the accuracy and speed of defect detection and reduce the requirements on the performance of equipment running the detection method.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a hot-rolled strip surface defect detection method based on deep learning. Background Art

[0002] At present, surface defect detection is a very important research topic in the field of machine vision. It is also called AOI (Automated optical inspection) or ASI (Automated surface inspection). It uses machine vision equipment to obtain images to determine whether there are defects in the captured images. Traditional surface defect detection methods based on machine vision often use conventional image processing algorithms or manually designed features plus classifiers. Generally speaking, the different properties of the inspected surface or defects are usually used to design imaging schemes. Reasonable imaging schemes help to obtain images with uniform illumination and clearly reflect the surface defects of objects. In recent years, with the successful application of deep learning models represented by convolutional neural networks (CNN) in many computer vision (CV) fields, many defect detection methods based on deep learning are also widely used in various industrial scenarios. At present, target detection algorithms used in the field of defect detection are mainly divided into two categories. One type is the detection algorithm that relies on anchor boxes, including two-stage networks represented by Faster R-CNN (Fast Region-based Convolutional Network) and one-stage networks represented by SSD (Single Shot multibox Detector) and another type of YOLO (You Only LookOnce) series.

[0003] However, since the detection targets of hot-rolled strip surface defects are small and the details are blurred, the detection accuracy is low. When the existing defect detection algorithms are applied to hot-rolled strip surface defects, the calculation speed is not fast enough, the equipment performance requirements are high, and the detection accuracy is not high enough.

[0004] Therefore, the present invention proposes a hot-rolled strip surface defect detection method based on deep learning. Summary of the invention

[0005] The present invention provides a hot-rolled strip surface defect detection method based on deep learning, which modifies the backbone network of the YOLOv5s network model into a RepVGG module to improve the program running speed; uses an optimized loss function to accelerate convergence and improve regression accuracy; adopts a CARAFE upsampling method to improve the receptive field; adds a Pyconv module to improve detection performance; then adds a small target detection layer to improve the accuracy; and optimizes the structure through model pruning to reduce the requirements for installation equipment; thereby reducing the complexity of the algorithm and the requirements for the performance of equipment running the detection method, and improving the accuracy and detection speed of defect detection.

[0006] The present invention provides a method for detecting surface defects of hot-rolled strip steel based on deep learning, comprising:

[0007] S1: Obtain massive amounts of hot-rolled strip surface defect image data;

[0008] S2: Based on the massive hot-rolled strip surface defect image data, a hot-rolled strip surface defect image dataset is constructed, and the hot-rolled strip surface defect image dataset is divided into a training set and a test set;

[0009] S3: Build a YOLOv5s network model;

[0010] S4: Modify the backbone network convolution module of the YOLOv5s network model to the RepVGG Block module;

[0011] S5: Change the upsampling method of the YOLOv5s network model to the CARAFE upsampling method;

[0012] S6: Change the downsampling method of the YOLOv5s network model to the Pyconv downsampling method;

[0013] S7: Pruning the YOLOv5s network model to obtain a hot-rolled strip surface defect detection model based on the improved YOLOv5s network model;

[0014] S8: Based on the optimized loss function, an optimized loss function is obtained;

[0015] S9: Based on the training set, the test set and the optimized loss function, the hot-rolled strip surface defect detection model is trained and tested in sequence to obtain a qualified hot-rolled strip surface defect detection model;

[0016] S10: Inputting the surface defect image of the hot-rolled steel strip to be detected into the qualified hot-rolled steel strip surface defect detection model to obtain the hot-rolled steel strip surface defect detection result.

[0017] Preferably, the hot-rolled strip surface defect detection method based on deep learning, S2: based on massive hot-rolled strip surface defect image data, construct a hot-rolled strip surface defect image dataset, and divide the hot-rolled strip surface defect image dataset into a training set and a test set, including:

[0018] Based on the manual marking instruction, the manual marking defect area in each hot-rolled strip surface defect image data is marked;

[0019] All the marked hot-rolled strip surface defect image data are aggregated to obtain a hot-rolled strip surface defect image dataset;

[0020] The hot-rolled strip surface defect image dataset is divided into a training set and a test set based on a preset ratio.

[0021] Preferably, in the hot-rolled strip surface defect detection method based on deep learning, the YOLOv5s network model constructed in step S3 includes:

[0022] Input, Back bone, Neck network, Head Prediction;

[0023] Among them, Input includes Mosaic data enhancement, adaptive anchor box calculation, and adaptive image scaling;

[0024] Backbone includes Focus module, SPP module and C3 module;

[0025] The neck network Neck includes FPN module, PAN module and C3 module;

[0026] The header includes the Bounding box loss function and NMS.

[0027] Preferably, the hot-rolled strip surface defect detection method based on deep learning, S4: modifying the backbone network convolution module of the YOLOv5s network model to a RepVGG Block module, comprising:

[0028] S41: Training multi-branch model: During training, add parallel 1×1 convolution branches and identity mapping branches to each 3×3 convolution layer;

[0029] S42: Convert the multi-branch model to a single-path model: Consider the 1×1 convolution as a 3×3 convolution with many zeros in the convolution kernel, and the identity mapping is a special 1×1 convolution; according to the additivity principle of convolution, the three branches of each RepVGG Block module can be merged into a 3×3 convolution;

[0030] S43: Structural parameter reconstruction: Transfer the weights of the multi-branch network to a simple network through the actual data flow.

[0031] Preferably, the hot-rolled strip surface defect detection method based on deep learning, S5: changing the upsampling mode of the YOLOv5s network model to the CARAFE upsampling mode, comprises:

[0032] S51: Feature map channel compression: Use 1×1 convolution to compress the number of channels of the input feature map of shape C×H×W to C m , where C, H, and W represent the number of channels, height, and width, respectively;

[0033] S52: Content encoding and upsampling kernel prediction: using k encoder ×k encoder The convolution layer predicts the upsampling kernel, and expands the channel dimension of the input feature map compressed in step S51 in the spatial dimension to obtain a shape of The upsampling kernel is, where the upsampling ratio is σ and the upsampling convolution kernel is k up ×k up , the number of input channels is C m , the number of output channels is

[0034] S53: Upsampling kernel normalization: Use softmax to normalize the upsampling kernel k obtained in step S52 up ×k up Normalize each channel of so that the sum of the weights of all channels is 1, and obtain the output feature map;

[0035] S54: Map each position in the output feature map back to the input feature map to obtain the mapped feature map, and then map the k positions at the center of the mapped feature map. up ×k up The dot product is performed between the area of ​​​​the predicted upsampling kernel of the point to obtain the output value, where different channels at the same position share the same upsampling kernel.

[0036] Preferably, the optimized loss function of the hot-rolled strip surface defect detection method based on deep learning comprises:

[0037]

[0038] Where, L IoU is the loss value output by the optimized loss function, IoU is the intersection-over-union ratio between two overlapping rectangular boxes, is the penalty term for the distance between the center point of the target box and the bounding box, b is the coordinate representation of the center point of the target box, and b gt is the coordinate representation of the center point of the bounding box, ρ 2(b,b gt ) is the Euclidean distance between the center point of the target frame and the bounding box, c is the diagonal length of the bounding box, is the penalty term for the wide distance between the target box and the bounding box, w is the coordinate representation of the width of the target box, and w gt is the coordinate representation of the width of the bounding box, ρ 2 (w,w gt ) is the Euclidean distance between the target box and the wide line segment of the bounding box, c w is the width of the bounding box, is the penalty term for the height distance between the target box and the bounding box, h is the coordinate representation of the height of the target box, and h gt is the coordinate representation of the height of the external frame, ρ 2 (h,h gt ) is the Euclidean distance between the target box and the height line segment of the external box, c h The height of the bounding box.

[0039] Preferably, the hot-rolled strip surface defect detection method based on deep learning, S9: based on the training set and the test set and the optimized loss function, the hot-rolled strip surface defect detection model is trained and tested in sequence to obtain a qualified hot-rolled strip surface defect detection model, including:

[0040] The hot-rolled strip surface defect detection model is trained based on the training set and the optimized loss function to obtain a trained hot-rolled strip surface defect detection model;

[0041] Input the test set into the trained hot-rolled strip surface defect detection model to obtain the model-marked defect area of ​​each hot-rolled strip surface defect image data in the test set;

[0042] The model test accuracy is calculated based on the manually marked defect areas and model marked defect areas of each hot-rolled strip surface defect image data in the test set;

[0043] Determine whether the model test accuracy is not less than the accuracy threshold, and if so, regard the trained hot-rolled steel strip surface defect detection model as a qualified hot-rolled steel strip surface defect detection model;

[0044] Otherwise, the trained hot-rolled strip surface defect detection model is retrained.

[0045] Preferably, the hot-rolled strip surface defect detection method based on deep learning calculates the model test accuracy based on the manually marked defect area and the model marked defect area of ​​each hot-rolled strip surface defect image data in the test set, including:

[0046] Generate a first angle sequence based on the tangent angle of each pixel on the contour of the manually marked defect area determined in sequence, take the pixel corresponding to the sudden change point in the first angle sequence as the first sudden change pixel, and mark a first dividing boundary between the first sudden change pixel and the adjacent pixel corresponding to the contour of the manually marked defect area in the opposite direction of the direction of determining the tangent angle;

[0047] A second angle sequence is generated based on the tangent angle of each pixel on the contour of the model mark defect area determined in sequence, a pixel corresponding to the sudden change point in the second angle sequence is regarded as a second sudden change pixel, and a second dividing boundary is marked between the second sudden change pixel and an adjacent pixel corresponding to the contour of the model mark defect area in the opposite direction of the direction of determining the tangent angle;

[0048] Dividing the contour of the artificially marked defect area based on all the determined first dividing boundaries to obtain a plurality of first curve segments, and at the same time, dividing the contour of the model marked defect area based on all the determined second dividing boundaries to obtain a plurality of second curve segments;

[0049] The model test accuracy is calculated based on all the first curve segments in the manually marked defect area and all the second curve segments in the model marked defect area of ​​each hot-rolled strip surface defect image data in the test set.

[0050] Preferably, the hot-rolled strip surface defect detection method based on deep learning calculates the model test accuracy based on all first curve segments in the manually marked defect area and all second curve segments in the model marked defect area of ​​each hot-rolled strip surface defect image data in the test set, including:

[0051] Determine a first area of ​​a region enclosed by a line connecting two end points of the first curve segment and a center point of a corresponding manually marked defect region and the corresponding first curve segment, and simultaneously determine a second area of ​​a region enclosed by a line connecting two end points of the second curve segment and a center point of a corresponding model marked defect region and the corresponding second curve segment;

[0052] Determine a first angle difference between tangent lines of two end points of the first curve segment of the contour of the artificially marked defect area, and use the ratio of the first angle difference to the length of the first curve segment as a first approximate curvature of the first curve segment;

[0053] Determine a second angle difference between tangent lines of two end points of the second curve segment of the contour of the model marked defect area, and take the ratio of the second angle difference to the length of the second curve segment as a second approximate curvature of the second curve segment;

[0054] The model test accuracy is calculated based on the first area and first approximate curvature of all first curve segments in the manually marked defect area of ​​each hot-rolled strip surface defect image data in the test set and the second area and second approximate curvature of all second curve segments in the model marked defect area.

[0055] Preferably, the hot-rolled strip surface defect detection method based on deep learning calculates the model test accuracy based on the first area and the first approximate curvature of all first curve segments in the manually marked defect area of ​​each hot-rolled strip surface defect image data in the test set and the second area and the second approximate curvature of all second curve segments in the model marked defect area, including:

[0056] Based on the relative positions between all the first curve segments in the manually marked defect area and the relative positions between all the second curve segments in the corresponding model marked defect area, all the first curve segments in the manually marked defect area are matched one by one with all the second curve segments in the corresponding model marked defect area to obtain a curve segment matching result;

[0057] Calculate the model recognition accuracy of the hot-rolled steel strip surface defect image data based on the curve segment correspondence results, the first areas and first approximate curvatures of all first curve segments, and the second areas and second approximate curvatures of all second curve segments;

[0058] The average of the model recognition accuracy of all hot-rolled strip surface defect image data in the test set is taken as the model test accuracy.

[0059] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0060] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0062] Figure 1 is a detailed flow chart of a method for detecting surface defects of hot-rolled strip steel based on deep learning in an embodiment of the present invention;

[0063] Figure 2 is a general flow chart of a method for detecting surface defects of hot-rolled strip steel based on deep learning in an embodiment of the present invention;

[0064] Figure 3 Schematic diagram of the overall YOLOv5s network structure in an embodiment of the present invention;

[0065] Figure 4 This is a module structure diagram of the backbone network RepVGG Block in an embodiment of the present invention;

[0066] Figure 5 Schematic diagram of the PyConv structure in an embodiment of the present invention;

[0067] Figure 6 It is a detection effect diagram in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0069] Embodiment 1:

[0070] The present invention provides a method for detecting surface defects of hot-rolled strip steel based on deep learning. Figures 1 to 5 ,include:

[0071] S1: Obtain massive amounts of hot-rolled strip surface defect image data;

[0072] S2: Based on the massive hot-rolled strip surface defect image data, a hot-rolled strip surface defect image dataset is constructed, and the hot-rolled strip surface defect image dataset is divided into a training set and a test set;

[0073] S3: Build a YOLOv5s network model;

[0074] S4: Modify the backbone network convolution module of the YOLOv5s network model to the RepVGG Block module;

[0075] S5: Change the upsampling method of the YOLOv5s network model to the CARAFE upsampling method;

[0076] S6: Change the downsampling method of the YOLOv5s network model to the Pyconv downsampling method;

[0077] S7: Pruning the YOLOv5s network model to obtain a hot-rolled strip surface defect detection model based on the improved YOLOv5s network model;

[0078] S8: Based on the optimized loss function, an optimized loss function is obtained;

[0079] S9: Based on the training set, the test set and the optimized loss function, the hot-rolled strip surface defect detection model is trained and tested in sequence to obtain a qualified hot-rolled strip surface defect detection model;

[0080] S10: Inputting the surface defect image of the hot-rolled steel strip to be detected into the qualified hot-rolled steel strip surface defect detection model to obtain the hot-rolled steel strip surface defect detection result.

[0081] In this embodiment, the hot-rolled steel strip surface defect image data is a surface image of the hot-rolled steel strip containing defects.

[0082] In this embodiment, the hot-rolled steel strip surface defect image data set is a set of images consisting of a large amount of manually labeled hot-rolled steel strip surface defect image data.

[0083] In this embodiment, the training set and the test set are obtained by dividing the hot-rolled strip surface defect image dataset according to a preset ratio, for example, according to a ratio of 7 to 3, with the training set accounting for 7 parts and the test set accounting for 3 parts.

[0084] In this embodiment, the YOLOv5s network model is a target detection algorithm that uses an Anchor-based detection method and is a single-stage target detection method. Compared with Yolov4, Yolov5 has faster speed and higher accuracy and is currently one of the industry's leading target detection algorithms. The SE algorithm was proposed in 2019. It adaptively recalibrates channel feature responses by explicitly modeling the interdependencies between channels; it shows that these blocks can be stacked together to form a SENet architecture, which generalizes very effectively on different data sets.

[0085] In this embodiment, the RepVGG Block module is a module that implements feature extraction of the input image based on the Repvgg algorithm. The Repvgg algorithm is a ResNet-style multi-branch model proposed in 2021.

[0086] In this embodiment, the backbone network convolution module (Backbone) is the module in the YOLOv5s network model that is mainly responsible for extracting features from the input image.

[0087] In this embodiment, the upsampling method is an operation of enlarging the original image so that it can be displayed on a display device with higher resolution. When performing image segmentation, the image needs to be classified at the pixel level. Therefore, after the convolution extracts the abstract features, the feature map needs to be restored to the original image size through upsampling. Common upsampling methods include bilinear interpolation, transposed convolution, upsampling (unsampling) and unpooling (unpooling).

[0088] In this embodiment, the CARAFE upsampling method is a lightweight upsampling method with a large receptive field. It guides the recombination process according to the input features. At the same time, the entire operator is relatively lightweight and only brings very small additional parameters and calculations.

[0089] In this embodiment, the down sampling method is an operation of reducing the image.

[0090] In this embodiment, the Pyconv downsampling method is a downsampling method implemented based on pyramid convolution (Pyramidal Convolution, PyConv).

[0091] In this embodiment, model pruning is an operation to reduce delays in real-time tasks and transplant large networks to terminal devices with weaker computing power through heuristics (manually defining the importance of each pruned unit to delete unimportant units in the neural network) or optimization (considering model pruning as an optimization problem to automatically find pruning locations).

[0092] In this embodiment, the hot-rolled strip surface defect detection model is a model for identifying defect areas of a hot-rolled strip surface defect image to be detected, which is obtained after executing the above steps S4 to S7 on the YOLOv5s network model.

[0093] In this embodiment, the qualified hot-rolled strip surface defect detection model is a qualified hot-rolled strip surface defect detection model obtained by training and testing the hot-rolled strip surface defect detection model in sequence based on the training set, the test set and the optimized loss function.

[0094] In this embodiment, the hot-rolled steel strip surface defect image to be detected is the hot-rolled steel strip surface defect image whose defect areas need to be identified by the qualified hot-rolled steel strip surface defect detection model.

[0095] In this embodiment, the hot-rolled steel strip surface defect detection result is the result of the defect area in the hot-rolled steel strip surface defect image to be detected by the qualified hot-rolled steel strip surface defect detection model.

[0096] In this embodiment, various index parameters for experimental comparison between the embodiment of the present invention and the existing target detection model are as follows:

[0097]

[0098] The beneficial effects of the above technologies are as follows: the backbone network of the YOLOv5s network model is modified to the RepVGG module to improve the program running speed; the optimized loss function is used to accelerate convergence and improve regression accuracy; the CARAFE upsampling method is used to improve the receptive field; the Pyconv module is added to improve the detection performance; a small target detection layer is added to improve the accuracy; and the structure is optimized through model pruning to reduce the requirements for installation equipment; thereby reducing the complexity of the algorithm and the requirements for the performance of the equipment running the detection method, and improving the accuracy and speed of defect detection.

[0099] Embodiment 2:

[0100] On the basis of Example 1, the hot-rolled strip surface defect detection method based on deep learning, S2: based on massive hot-rolled strip surface defect image data, construct a hot-rolled strip surface defect image dataset, and divide the hot-rolled strip surface defect image dataset into a training set and a test set, including:

[0101] Based on the manual marking instruction, the manual marking defect area in each hot-rolled strip surface defect image data is marked;

[0102] All the marked hot-rolled strip surface defect image data are aggregated to obtain a hot-rolled strip surface defect image dataset;

[0103] The hot-rolled strip surface defect image dataset is divided into a training set and a test set based on a preset ratio (for example, 7 to 3, with 7 sets for training and 3 sets for testing).

[0104] In this embodiment, the manual marking instruction is an instruction manually input for marking the manually identified defect area in each hot-rolled strip surface defect image data.

[0105] In this embodiment, the manually marked defect area is the defect area marked in the hot-rolled strip surface defect image data based on the manual marking instruction.

[0106] The beneficial effects of the above technology are: completing the manual marking of defect areas of the massive hot-rolled strip surface defect image data obtained, summarizing the hot-rolled strip surface defect image data set, and dividing it into training sets and test sets for training and testing the hot-rolled strip surface defect detection model.

[0107] Embodiment 3:

[0108] On the basis of Example 1, the hot-rolled strip surface defect detection method based on deep learning, the YOLOv5s network model constructed in step S3 includes:

[0109] Input, Back bone, Neck network, Head Prediction;

[0110] Among them, Input includes Mosaic data enhancement, adaptive anchor box calculation, and adaptive image scaling;

[0111] Backbone includes Focus module, SPP module and C3 module;

[0112] The neck network Neck includes FPN module, PAN module and C3 module;

[0113] The header includes the Bounding box loss function and NMS.

[0114] The beneficial effect of the above technology is: constructing a YOLOv5s network model including Input, Back bone, neck network Neck, and head Prediction.

[0115] Embodiment 4:

[0116] On the basis of Example 1, the hot-rolled strip surface defect detection method based on deep learning, S4: modifying the backbone network convolution module of the YOLOv5s network model to a RepVGG Block module, comprising:

[0117] S41: Training multi-branch model: During training, add parallel 1×1 convolution branches and identity mapping branches to each 3×3 convolution layer;

[0118] S42: Convert the multi-branch model to a single-path model: Consider the 1×1 convolution as a 3×3 convolution with many zeros in the convolution kernel, and the identity mapping is a special 1×1 convolution; according to the additivity principle of convolution, the three branches of each RepVGG Block module can be merged into a 3×3 convolution;

[0119] S43: Structural parameter reconstruction: Transfer the weights of the multi-branch network to a simple network through the actual data flow.

[0120] The beneficial effects of the above technology are: the backbone network convolution module of the YOLOv5s network model is modified to a RepVGGBlock module, which improves the detection speed.

[0121] Embodiment 5:

[0122] On the basis of Example 1, the hot-rolled strip surface defect detection method based on deep learning, S5: changing the upsampling mode of the YOLOv5s network model to the CARAFE upsampling mode, comprising:

[0123] S51: Feature map channel compression: Use 1×1 convolution to compress the number of channels of the input feature map of shape C×H×W to C m , where C, H, and W represent the number of channels, height, and width, respectively;

[0124] S52: Content encoding and upsampling kernel prediction: using k encoder ×k encoder The convolution layer predicts the upsampling kernel, and expands the channel dimension of the input feature map compressed in step S51 in the spatial dimension to obtain a shape of The upsampling kernel is, where the upsampling ratio is σ and the upsampling convolution kernel is k up ×k up , the number of input channels is C m , the number of output channels is

[0125] S53: Upsampling kernel normalization: Use softmax to normalize the upsampling kernel k obtained in step S52 up ×k up Normalize each channel of so that the sum of the weights of all channels is 1, and obtain the output feature map;

[0126] S54: Map each position in the output feature map back to the input feature map to obtain the mapped feature map, and then map the k positions at the center of the mapped feature map. up ×k up The dot product is performed between the area of ​​​​the predicted upsampling kernel of the point to obtain the output value, where different channels at the same position share the same upsampling kernel.

[0127] The beneficial effect of the above technology is: changing the upsampling method of the YOLOv5s network model to the CARAFE upsampling method improves the receptive field.

[0128] Embodiment 6:

[0129] On the basis of Example 1, the hot-rolled strip surface defect detection method based on deep learning, the optimized loss function includes:

[0130]

[0131] Where, L IoU is the loss value output by the optimized loss function, IoU is the intersection-over-union ratio between two overlapping rectangular boxes, is the penalty term for the distance between the center point of the target box and the bounding box, b is the coordinate representation of the center point of the target box, and b gt is the coordinate representation of the center point of the bounding box, ρ 2 (b,b gt) is the Euclidean distance between the center point of the target frame and the bounding box, c is the diagonal length of the bounding box, is the penalty term for the wide distance between the target box and the bounding box, w is the coordinate representation of the width of the target box, and w gt is the coordinate representation of the width of the bounding box, ρ 2 (w,w gt ) is the Euclidean distance between the target box and the wide line segment of the bounding box, c w is the width of the bounding box, is the penalty term for the height distance between the target box and the bounding box, h is the coordinate representation of the height of the target box, and h gt is the coordinate representation of the height of the external frame, ρ 2 (h,h gt ) is the Euclidean distance between the target box and the height line segment of the external box, c h The height of the bounding box.

[0132] The beneficial effects of the above technology are: using the optimized loss function to accelerate convergence and improve regression accuracy.

[0133] Embodiment 7:

[0134] On the basis of Example 1, the hot-rolled strip surface defect detection method based on deep learning, S9: based on the training set and the test set and the optimized loss function, the hot-rolled strip surface defect detection model is trained and tested in sequence to obtain a qualified hot-rolled strip surface defect detection model, including:

[0135] The hot-rolled strip surface defect detection model is trained based on the training set and the optimized loss function to obtain a trained hot-rolled strip surface defect detection model;

[0136] Input the test set into the trained hot-rolled strip surface defect detection model to obtain the model-marked defect area of ​​each hot-rolled strip surface defect image data in the test set;

[0137] The model test accuracy is calculated based on the manually marked defect areas and model marked defect areas of each hot-rolled strip surface defect image data in the test set;

[0138] Determine whether the model test accuracy is not less than the accuracy threshold, and if so, regard the trained hot-rolled steel strip surface defect detection model as a qualified hot-rolled steel strip surface defect detection model;

[0139] Otherwise, the trained hot-rolled strip surface defect detection model is retrained.

[0140] In this embodiment, the model-marked defect area is the defect area marked by the trained hot-rolled steel strip surface defect detection model, which is obtained after the test set is input into the trained hot-rolled steel strip surface defect detection model.

[0141] In this embodiment, the model test accuracy is the defect detection accuracy of the trained hot-rolled strip surface defect detection model calculated based on the manually marked defect area and the model marked defect area of ​​each hot-rolled strip surface defect image data in the test set.

[0142] In this embodiment, the accuracy threshold is the judgment threshold of the model test accuracy based on which whether the trained hot-rolled strip surface defect detection model is qualified is judged.

[0143] In this embodiment, retraining the trained hot-rolled steel strip surface defect detection model means: retraining the trained hot-rolled steel strip surface defect detection model using newly acquired massive hot-rolled steel strip surface defect image data.

[0144] The beneficial effects of the above technology are: based on the training set, the test set and the optimized loss function, the hot-rolled strip surface defect detection model is trained and tested in sequence, and the model test accuracy is determined based on the manually marked defect area and the model marked defect area of ​​each hot-rolled strip surface defect image data in the test set during the test process, and the model test accuracy is judged to obtain a qualified hot-rolled strip surface defect detection model.

[0145] Embodiment 8:

[0146] On the basis of Example 7, the hot-rolled strip surface defect detection method based on deep learning calculates the model test accuracy based on the manually marked defect area and the model marked defect area of ​​each hot-rolled strip surface defect image data in the test set, including:

[0147] A first angle sequence is generated based on the tangent angle of each pixel point on the contour of the artificially marked defect area that is determined in sequence (for example, starting from the pixel point when the line connecting the contour of the artificially marked defect area and the origin of the preset coordinate system is 0 degrees in the preset coordinate system and determined in a counterclockwise direction), a pixel point corresponding to the sudden change point in the first angle sequence is regarded as a first sudden change pixel point, and a first dividing boundary is marked between the first sudden change pixel point and an adjacent pixel point corresponding to the contour of the artificially marked defect area in the opposite direction of the direction of determining the tangent angle;

[0148] A second angle sequence is generated based on the tangent angle of each pixel on the contour of the model mark defect area determined in sequence, a pixel corresponding to the sudden change point in the second angle sequence is regarded as a second sudden change pixel, and a second dividing boundary is marked between the second sudden change pixel and an adjacent pixel corresponding to the contour of the model mark defect area in the opposite direction of the direction of determining the tangent angle;

[0149] Dividing the contour of the artificially marked defect area based on all the determined first dividing boundaries to obtain a plurality of first curve segments, and at the same time, dividing the contour of the model marked defect area based on all the determined second dividing boundaries to obtain a plurality of second curve segments;

[0150] The model test accuracy is calculated based on all the first curve segments in the manually marked defect area and all the second curve segments in the model marked defect area of ​​each hot-rolled strip surface defect image data in the test set.

[0151] In this embodiment, the first angle sequence is a sequence generated by the tangent angle of each pixel point determined sequentially on the contour of the artificially marked defect area.

[0152] In this embodiment, the sudden change point is a point where a sudden change occurs in the first angle sequence.

[0153] In this embodiment, the first sudden change pixel point is a pixel point corresponding to the sudden change point in the first angle sequence.

[0154] In this embodiment, the first dividing boundary is the boundary between the first sudden change pixel point and the corresponding adjacent pixel point in the contour of the artificially marked defect area in the opposite direction of the direction of determining the tangent angle.

[0155] In this embodiment, the second angle sequence is a sequence generated by the tangent angle of each pixel point determined sequentially on the contour of the model mark defect area.

[0156] In this embodiment, the second sudden change pixel point is the pixel point corresponding to the sudden change point in the second angle sequence.

[0157] In this embodiment, the second dividing boundary is the boundary between the second sudden change pixel point and an adjacent pixel point corresponding to the contour of the model mark defect area in the opposite direction of the direction of determining the tangent angle.

[0158] In this embodiment, the first curve segment is a curve segment obtained by dividing the outline of the artificially marked defect area based on all the determined first division boundaries.

[0159] In this embodiment, the second curve segment is a curve segment obtained by dividing the outline of the model marked defect area based on all the determined second dividing boundaries.

[0160] The beneficial effects of the above technology are as follows: a dividing boundary is determined by analyzing and identifying the position where the angle of the tangent at each pixel point on the contour of the artificially marked defect area and the model marked defect area suddenly changes, and the contours of the artificially marked defect area and the model marked defect area are divided based on the dividing boundary, so that the tangent angle changes of the obtained first curve segment and the second curve segment are more regular, thereby making the approximate curvature calculated subsequently using the curvature and the line segment length more accurate.

[0161] Embodiment 9:

[0162] On the basis of Example 8, the hot-rolled strip surface defect detection method based on deep learning calculates the model test accuracy based on all first curve segments in the manually marked defect area and all second curve segments in the model marked defect area of ​​each hot-rolled strip surface defect image data in the test set, including:

[0163] Determine a first area of ​​a region enclosed by a line connecting two end points of the first curve segment and a center point of a corresponding manually marked defect region and the corresponding first curve segment, and simultaneously determine a second area of ​​a region enclosed by a line connecting two end points of the second curve segment and a center point of a corresponding model marked defect region and the corresponding second curve segment;

[0164] Determine a first angle difference between tangent lines of two end points of the first curve segment of the contour of the artificially marked defect area, and use the ratio of the first angle difference to the length of the first curve segment as a first approximate curvature of the first curve segment;

[0165] Determine a second angle difference between tangent lines of two end points of the second curve segment of the contour of the model marked defect area, and take the ratio of the second angle difference to the length of the second curve segment as a second approximate curvature of the second curve segment;

[0166] The model test accuracy is calculated based on the first area and first approximate curvature of all first curve segments in the manually marked defect area of ​​each hot-rolled strip surface defect image data in the test set and the second area and second approximate curvature of all second curve segments in the model marked defect area.

[0167] In this embodiment, the first area is the area of ​​a region enclosed by a line connecting two end points of the first curve segment and a center point of the corresponding artificially marked defect region and the corresponding first curve segment.

[0168] In this embodiment, the second area is the area of ​​the region enclosed by the line connecting the two end points of the second curve segment and the center point of the corresponding model marked defect region and the corresponding second curve segment.

[0169] In this embodiment, the first angle difference is the angle difference between the tangent lines of the contour of the artificially marked defect area at two end points of the first curve segment.

[0170] In this embodiment, the first approximate curvature is the ratio of the first angle difference to the length of the first curve segment.

[0171] In this embodiment, the second angle difference is the angle difference between the tangent lines of the contour of the model mark defect area at two end points of the second curve segment.

[0172] In this embodiment, the second approximate curvature is the ratio of the second angle difference to the length of the second curve segment.

[0173] The beneficial effects of the above technology are: by determining the ratio of the angle difference between the two end points of the first curve segment and the second curve segment whose tangent angle changes more regularly after division and the length of the corresponding first curve segment or the second curve segment, the curvature of the first curve segment and the second curve segment can be approximated, and then based on the first area and the first approximate curvature of all first curve segments in the manually marked defect area of ​​each hot-rolled strip surface defect image data in the test set and the second area and the second approximate curvature of all second curve segments in the model marked defect area, the model test accuracy can be accurately calculated from the two perspectives of shape and size.

[0174] Embodiment 10:

[0175] On the basis of Example 9, the hot-rolled strip surface defect detection method based on deep learning calculates the model test accuracy based on the first area and the first approximate curvature of all first curve segments in the manually marked defect area of ​​each hot-rolled strip surface defect image data in the test set and the second area and the second approximate curvature of all second curve segments in the model marked defect area, including:

[0176] Based on the relative positions between all the first curve segments in the manually marked defect area and the relative positions between all the second curve segments in the corresponding model marked defect area, all the first curve segments in the manually marked defect area are matched one by one with all the second curve segments in the corresponding model marked defect area to obtain a curve segment matching result;

[0177] Calculate the model recognition accuracy of the hot-rolled steel strip surface defect image data based on the curve segment correspondence results, the first areas and first approximate curvatures of all first curve segments, and the second areas and second approximate curvatures of all second curve segments;

[0178] The average of the model recognition accuracy of all hot-rolled strip surface defect image data in the test set is taken as the model test accuracy.

[0179] In this embodiment, based on the relative positions between all the first curve segments in the manually marked defect area and the relative positions between all the second curve segments in the corresponding model marked defect area, all the first curve segments in the manually marked defect area are matched one-to-one with all the second curve segments in the corresponding model marked defect area to obtain the curve segment matching result, including:

[0180] Starting from the first curve segment and the second curve segment that intersect the horizontal coordinate of the preset coordinate system, determine the order of all first curve segments in the manually marked defect area and the order of all second curve segments in the model marked defect area in a counterclockwise direction, match the first curve segments and the second curve segments with the same ordinal number, and obtain the curve segment correspondence result.

[0181] In this embodiment, the curve segment correspondence result is a result obtained by one-to-one correspondence between all first curve segments in the manually marked defect area and all second curve segments in the corresponding model marked defect area.

[0182] In this embodiment, based on the curve segment correspondence results of the hot-rolled steel strip surface defect image data, the first areas and first approximate curvatures of all first curve segments, and the second areas and second approximate curvatures of all second curve segments, the model recognition accuracy of the corresponding hot-rolled steel strip surface defect image data is calculated, including:

[0183] The ratio of the area difference between the first area of ​​each first curve segment and the second area of ​​the corresponding second curve segment in the curve segment correspondence result and the first area is regarded as the area deviation rate of the corresponding first curve segment;

[0184] The ratio of the difference between the first approximate curvature of each first curve segment and the second approximate curvature of the corresponding second curve segment in the curve segment correspondence result to the first approximate curvature is regarded as the curvature deviation rate of the corresponding first curve segment;

[0185] The average of the area deviation rate and the curvature deviation rate of each first curve segment and the corresponding second curve segment in the curve segment corresponding result is taken as the curve segment deviation rate;

[0186] The average value of the curve segment deviation rates of all first curve segments and corresponding second curve segments in the curve segment corresponding results of all hot-rolled strip surface defect image data is taken as the model recognition error rate;

[0187] The difference between 1 and the model recognition error rate is taken as the model recognition accuracy.

[0188] In this embodiment, the model recognition accuracy rate refers to the accuracy rate of the trained hot-rolled strip surface defect detection model in identifying defect areas.

[0189] The beneficial effects of the above technology are: based on the first area and first approximate curvature of all first curve segments in the manually marked defect area of ​​each hot-rolled strip surface defect image data in the test set and the second area and second approximate curvature of all second curve segments in the model marked defect area, the model test accuracy can be accurately calculated from the two perspectives of shape and size.

[0190] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for detecting surface defects of hot-rolled strip steel based on deep learning, characterized in that: include: S1: Obtain massive hot-rolled strip surface defect image data; S2: Based on the massive hot-rolled strip surface defect image data, a hot-rolled strip surface defect image dataset is constructed, and the hot-rolled strip surface defect image dataset is divided into a training set and a test set; S3: Build a YOLOv5s network model; S4: Modify the backbone network convolution module of the YOLOv5s network model to the RepVGG Block module; S5: Change the upsampling method of the YOLOv5s network model to the CARAFE upsampling method; S6: Change the downsampling method of the YOLOv5s network model to the Pyconv downsampling method; S7: Pruning the YOLOv5s network model to obtain a hot-rolled strip surface defect detection model based on the improved YOLOv5s network model; S8: optimize the loss function to obtain an optimized loss function; S9: Based on the training set, the test set and the optimized loss function, the hot-rolled strip surface defect detection model is trained and tested in sequence to obtain a qualified hot-rolled strip surface defect detection model; S10: inputting the surface defect image of the hot-rolled steel strip to be detected into the qualified hot-rolled steel strip surface defect detection model to obtain the hot-rolled steel strip surface defect detection result; Among them, step S9: based on the training set and the test set and the optimized loss function, the hot-rolled strip surface defect detection model is trained and tested in sequence to obtain a qualified hot-rolled strip surface defect detection model, including: The hot-rolled strip surface defect detection model is trained based on the training set and the optimized loss function to obtain a trained hot-rolled strip surface defect detection model; Input the test set into the trained hot-rolled strip surface defect detection model to obtain the model-marked defect area of ​​each hot-rolled strip surface defect image data in the test set; The model test accuracy is calculated based on the manually marked defect areas and model marked defect areas of each hot-rolled strip surface defect image data in the test set; Determine whether the model test accuracy is not less than the accuracy threshold, and if so, regard the trained hot-rolled steel strip surface defect detection model as a qualified hot-rolled steel strip surface defect detection model; Otherwise, the trained hot-rolled strip surface defect detection model is retrained; The model test accuracy is calculated based on all first curve segments in the manually marked defect area and all second curve segments in the model marked defect area of ​​each hot-rolled strip surface defect image data in the test set, including: Determine a first area of ​​a region enclosed by a line connecting two end points of the first curve segment and a center point of a corresponding manually marked defect region and the corresponding first curve segment, and simultaneously determine a second area of ​​a region enclosed by a line connecting two end points of the second curve segment and a center point of a corresponding model marked defect region and the corresponding second curve segment; Determine a first angle difference between tangent lines of two end points of the first curve segment of the contour of the artificially marked defect area, and use the ratio of the first angle difference to the length of the first curve segment as a first approximate curvature of the first curve segment; Determine a second angle difference between tangent lines of two end points of the second curve segment of the contour of the model marked defect area, and take the ratio of the second angle difference to the length of the second curve segment as a second approximate curvature of the second curve segment; Calculate the model test accuracy based on the first area and the first approximate curvature of all first curve segments in the manually marked defect area of ​​each hot-rolled strip surface defect image data in the test set and the second area and the second approximate curvature of all second curve segments in the model marked defect area; Among them, based on the manually marked defect area and model marked defect area of ​​each hot-rolled strip surface defect image data in the test set, the model test accuracy is calculated, including: Generate a first angle sequence based on the tangent angle of each pixel on the contour of the manually marked defect area determined in sequence, take the pixel corresponding to the sudden change point in the first angle sequence as the first sudden change pixel, and mark a first dividing boundary between the first sudden change pixel and the adjacent pixel corresponding to the contour of the manually marked defect area in the opposite direction of the direction of determining the tangent angle; A second angle sequence is generated based on the tangent angle of each pixel on the contour of the model mark defect area determined in sequence, a pixel corresponding to the sudden change point in the second angle sequence is regarded as a second sudden change pixel, and a second dividing boundary is marked between the second sudden change pixel and an adjacent pixel corresponding to the contour of the model mark defect area in the opposite direction of the direction of determining the tangent angle; Dividing the contour of the artificially marked defect area based on all the determined first dividing boundaries to obtain a plurality of first curve segments, and at the same time, dividing the contour of the model marked defect area based on all the determined second dividing boundaries to obtain a plurality of second curve segments; The model test accuracy is calculated based on all the first curve segments in the manually marked defect area and all the second curve segments in the model marked defect area of ​​each hot-rolled strip surface defect image data in the test set.

2. The method for detecting surface defects of hot-rolled strip steel based on deep learning according to claim 1, characterized in that: S2: Based on the massive hot-rolled strip surface defect image data, a hot-rolled strip surface defect image dataset is constructed, and the hot-rolled strip surface defect image dataset is divided into a training set and a test set, including: Based on the manual marking instruction, the manual marking defect area in each hot-rolled strip surface defect image data is marked; All the marked hot-rolled strip surface defect image data are aggregated to obtain a hot-rolled strip surface defect image dataset; The hot-rolled strip surface defect image dataset is divided into a training set and a test set based on a preset ratio.

3. The method for detecting surface defects of hot-rolled strip steel based on deep learning according to claim 1, characterized in that: The YOLOv5s network model constructed in step S3 includes: Input, Back bone, Neck network, Head Prediction; Among them, Input includes Mosaic data enhancement, adaptive anchor box calculation, and adaptive image scaling; Backbone includes Focus module, SPP module and C3 module; The neck network Neck includes FPN module, PAN module and C3 module; The header includes the Bounding box loss function and NMS.

4. The method for detecting surface defects of hot-rolled strip steel based on deep learning according to claim 1, characterized in that: S4: Modify the backbone network convolution module of the YOLOv5s network model to the RepVGG Block module, including: S41: Training multi-branch model: During training, add parallel 1×1 convolution branches and identity mapping branches to each 3×3 convolution layer; S42: Convert the multi-branch model to a single-path model: Consider the 1×1 convolution as a 3×3 convolution with many zeros in the convolution kernel, and the identity mapping is a special 1×1 convolution; according to the additivity principle of convolution, the three branches of each RepVGG Block module can be merged into a 3×3 convolution; S43: Structural parameter reconstruction: Transfer the weights of the multi-branch network to a simple network through the actual data flow.

5. The method for detecting surface defects of hot-rolled strip steel based on deep learning according to claim 1, characterized in that: S5: Change the upsampling method of the YOLOv5s network model to the CARAFE upsampling method, including: S51: Feature map channel compression: Use 1×1 convolution to compress the number of channels of the input feature map of shape C×H×W to C m , where C, H, and W represent the number of channels, height, and width, respectively; S52: Content encoding and upsampling kernel prediction: using k encoder ×k encoder The convolution layer predicts the upsampling kernel, and expands the channel dimension of the input feature map compressed in step S51 in the spatial dimension to obtain a shape of The upsampling kernel is, where the upsampling ratio is σ and the upsampling convolution kernel is k up ×k up , the number of input channels is C m , the number of output channels is S53: Upsampling kernel normalization: Use softmax to normalize the upsampling kernel k obtained in step S52 up ×k up Normalize each channel of so that the sum of the weights of all channels is 1, and obtain the output feature map; S54: Map each position in the output feature map back to the input feature map to obtain the mapped feature map, and then map the k positions at the center of the mapped feature map. up ×k up The dot product is performed between the area of ​​​​the predicted upsampling kernel of the point to obtain the output value, where different channels at the same position share the same upsampling kernel.

6. The method for detecting surface defects of hot-rolled strip steel based on deep learning according to claim 1, characterized in that: The optimized loss function includes: Where, L IoU is the loss value output by the optimized loss function, IoU is the intersection-over-union ratio between two overlapping rectangular boxes, is the penalty term for the distance between the center point of the target box and the bounding box, b is the coordinate representation of the center point of the target box, and b gt is the coordinate representation of the center point of the bounding box, ρ 2 (b,b gt ) is the Euclidean distance between the center point of the target frame and the bounding box, c is the diagonal length of the bounding box, is the penalty term for the wide distance between the target box and the bounding box, w is the coordinate representation of the width of the target box, and w gt is the coordinate representation of the width of the bounding box, ρ 2 (w,w gt ) is the Euclidean distance between the target box and the wide line segment of the bounding box, c w is the width of the bounding box, is the penalty term for the height distance between the target box and the bounding box, h is the coordinate representation of the height of the target box, and h gt is the coordinate representation of the height of the external frame, ρ 2 (h,h gt ) is the Euclidean distance between the target box and the height line segment of the external box, c h The height of the bounding box.

7. The method for detecting surface defects of hot-rolled strip steel based on deep learning according to claim 1, characterized in that: Based on the first area and the first approximate curvature of all the first curve segments in the manually marked defect area of ​​each hot-rolled strip surface defect image data in the test set and the second area and the second approximate curvature of all the second curve segments in the model marked defect area, the model test accuracy is calculated, including: Based on the relative positions between all the first curve segments in the manually marked defect area and the relative positions between all the second curve segments in the corresponding model marked defect area, all the first curve segments in the manually marked defect area are matched one by one with all the second curve segments in the corresponding model marked defect area to obtain a curve segment matching result; Calculate the model recognition accuracy of the hot-rolled steel strip surface defect image data based on the curve segment correspondence results, the first areas and first approximate curvatures of all first curve segments, and the second areas and second approximate curvatures of all second curve segments; The average of the model recognition accuracy of all hot-rolled strip surface defect image data in the test set is taken as the model test accuracy.

Images