A multi-level network fusion round steel surface defect detection method and image acquisition device

Abstract

The present invention discloses a multi-level network-integrated round steel surface defect detection method and image acquisition device. The detection method comprises: acquiring real-time round steel surface images, establishing a round steel surface defect dataset from images of round steel surfaces with defects, and establishing round steel surface non-defect image samples from images of round steel surfaces without defects; using the round steel surface defect dataset to create a round steel surface defect detection network model that integrates an Efficient Net classification priority network and a YOLOv5 network, enabling the model to classify and identify the types of round steel surface defects and locate the locations of round steel surface defects; training the round steel surface defect detection network model using transfer learning; and adding the non-defect image samples to the round steel surface defect detection network model and training the model. The image acquisition device includes a bracket, a line scan camera, and a ring-shaped green LED light source. The detection method has strong generalization capability and good robustness, effectively improving the training accuracy and detection accuracy of the detection network model.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision, deep learning, target detection, and metal surface defect detection equipment and methods, and in particular to a multi-level network fusion round steel surface defect detection method and device. Background Art

[0002] As a specialty steel grade and industrial raw material, round steel is widely used in the manufacture of various industrial parts for automobiles, aerospace, and other industries. Its quality directly determines the quality of the finished industrial products it serves as raw material. As the manufacturing industry continues to transition toward automated, information-based, and data-driven intelligent production, round steel production continues to accelerate, placing higher demands on quality assurance testing technologies. Therefore, ensuring round steel production quality is crucial to improving a company's competitiveness. Mastering real-time, efficient, stable, and high-precision round steel testing technology is a crucial step for steelmakers to seize the commanding heights of the new industrial revolution.

[0003] At present, with the continuous upgrading of round steel production lines, the pace of steel production is getting faster and faster. How to ensure production quality in a fast production pace is the most concerned issue in the current round steel production. The existing manual visual inspection method is no longer suitable for defect detection in the current round steel production line due to its timeliness and lack of life safety guarantees. The existing non-destructive testing methods, due to the limitations of these detection technologies themselves, have poor algorithm anti-chemical capabilities and are mostly used for offline detection. The real-time performance is poor and it is difficult to adapt to the needs of today's high-speed online production of round steel surface defect detection. The existing machine vision method designs feature extractors (threshold segmentation, edge detection, contrast detection and other methods) to obtain the characteristics of defects, but due to the complexity of the algorithm design process and the susceptibility to complex and changing environmental influences, the algorithm has insufficient generalization ability, poor robustness and low detection accuracy. Summary of the Invention

[0004] The present invention provides a multi-level network fusion round steel surface defect detection method and image acquisition device. The existing round steel surface defect detection has the following problems: the algorithm design process is complicated and is easily affected by complex and changeable environments, resulting in insufficient algorithm generalization ability, poor robustness and low detection accuracy.

[0005] To solve the above technical problems, the embodiments of the present invention provide the following solutions:

[0006] On the one hand, an embodiment of the present invention provides a multi-level network fusion round steel surface defect detection method, the detection method comprising:

[0007] Acquire real-time round steel surface images, establish round steel surface images with defects as round steel surface defect datasets, and establish round steel surface images without defects as round steel surface non-defect image samples;

[0008] Using the round steel surface defect dataset, a round steel surface defect detection network model was created by integrating the Efficient Net classification priority network with the YOLOv5 network. The model was able to classify and identify the types of round steel surface defects and locate the locations of round steel surface defects.

[0009] Using transfer learning to train the round steel surface defect detection network model;

[0010] Non-defective image samples of round steel surfaces are added to the round steel surface defect detection network model and the model is trained.

[0011] Preferably, establishing a round steel surface defect dataset includes:

[0012] Use LabelImg to label the surface defect image of round steel and mark the defect category and defect location information.

[0013] Preferably, establishing the round steel surface defect dataset further includes:

[0014] Create surface defect image samples of round steel including pits, scratches, roller marks, warping, ears, iron oxide belts and cracks;

[0015] The round steel surface images with more iron oxide belts and ear defects that are easily misdetected are established as round steel surface non-defect image samples.

[0016] Preferably, the creation of a round steel surface defect detection network model includes:

[0017] A YOLOv5 target detection network model is designed to obtain the classification and border positioning information of round steel surface defects, and an EfficientNet network is designed to prioritize the classification of round steel surface defects.

[0018] Preferably, in the process of detecting defects using the round steel surface defect detection network model, if the EfficientNet classification priority network has a classification defect with a confidence level greater than 0.9, it is determined that the sample image must have such a defect; otherwise, a confidence score of not less than 0.3 is taken as the possible defect category, and the final detection result is obtained by further combining the YOLOv5 network.

[0019] Preferably, in training the round steel surface defect detection network model using transfer learning, the following steps are included:

[0020] The weights obtained from training on the ImageNet dataset are used as the initial weights for backbone network training. Pre-training is performed on the hot-rolled strip surface defect dataset to obtain pre-trained weights, which are used as the initial weights for backbone network training.

[0021] The round steel surface character dataset is divided into training set, validation set and test set with a ratio of 7:1:2.

[0022] Preferably, the round steel surface character dataset is divided into a training set, a validation set, and a test set, including:

[0023] Freeze the shallow pre-trained weights of the backbone convolutional neural network and use the Adam optimizer with a learning rate of 0.001 to pre-train the network model;

[0024] Unfreeze the shallow pre-trained weights of the backbone convolutional neural network and reduce the learning rate to 0.0001 for training again.

[0025] On the other hand, an embodiment of the present invention provides an image acquisition device, which is used to acquire a round steel surface image in the multi-level network fusion round steel surface defect detection method, and the device includes:

[0026] The device includes a bracket, on which four line scan cameras are evenly distributed around the circumference, with each line scan camera having an angle of 45° with the horizontal line and an angle of 45° with the vertical line;

[0027] An annular green LED light source is installed on the bracket.

[0028] Preferably, the line scan camera has a GIGE digital interface, and the collected round steel surface image is transmitted to the server over a long distance via a network.

[0029] The above solution of the present invention includes at least the following beneficial effects:

[0030] In the above scheme, the detection method integrates the Efficient Net classification priority network and the YOLOv5 network into a two-level network to obtain a round steel surface defect network model based on multi-level network fusion, which effectively solves the problems of false alarms and missed detections in surface defect detection, meets the needs of round steel surface defect detection, has sufficient generalization ability and good robustness, and improves the detection accuracy. When training the round steel surface defect detection network model, transfer learning is used to train the entire round steel surface defect detection network model, which solves the problem that insufficient number of samples easily leads to an imbalance in the number of defect samples of different categories and easily causes overfitting during the training process. A type of round steel surface non-defect image sample is added to the training of the round steel surface defect detection network model, making the sample division more reasonable and effectively improving the training accuracy and detection accuracy of the round steel surface defect detection network model.

[0031] The image acquisition device collects real-time round steel surface images and transmits them to the server for round steel surface defect detection. The images collected by the image acquisition device have no blind spots and good image uniformity, and are suitable for round steel surface defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a flow chart of Example 1 of the multi-level network fusion round steel surface defect detection method of the present invention;

[0033] Figure 2 The process of the multi-level network fusion round steel surface defect detection method of the present invention is as follows Figure 1 ;

[0034] Figure 3 The process of the multi-level network fusion round steel surface defect detection method of the present invention is as follows Figure 2 ;

[0035] Figure 4 The process of the multi-level network fusion round steel surface defect detection method of the present invention is as follows Figure 3 ;

[0036] Figure 5 The process of the multi-level network fusion round steel surface defect detection method of the present invention is as follows Figure 4 ;

[0037] Figure 6a This is the round steel surface pit defect image sample of the present invention;

[0038] Figure 6b This is a sample of the round steel surface scratch defect image of the present invention;

[0039] Figure 6c This is a sample of the roller mark defect image on the round steel surface of the present invention;

[0040] Figure 6d This is an image sample of a round steel surface warping defect of the present invention;

[0041] Figure 6e This is an image sample of the round steel surface ear defect of the present invention;

[0042] Figure 6f This is an image sample of the iron oxide belt defect on the round steel surface of the present invention;

[0043] Figure 6g This is a round steel surface crack defect image sample of the present invention;

[0044] Figure 6h The non-defective image sample of the round steel surface of the present invention;

[0045] Figure 7a This is a rendering of the round steel surface pit defect image detection effect of the present invention;

[0046] Figure 7b This is an image detection effect diagram of scratch defects on the surface of round steel according to the present invention;

[0047] Figure 7c This is a rendering of the round steel surface roller mark defect image detection effect of the present invention;

[0048] Figure 7d This is an image detection effect diagram of round steel surface peeling defects of the present invention;

[0049] Figure 7e This is a rendering of the round steel surface ear defect image detection effect of the present invention;

[0050] Figure 7f This is an image detection effect diagram of iron oxide belt defects on the surface of round steel of the present invention;

[0051] Figure 7g This is a rendering of the round steel surface crack defect image detection effect of the present invention;

[0052] Figure 7h This is a rendering of the non-defect image detection effect of the round steel surface of the present invention;

[0053] Figure 8 This is an algorithm framework diagram of the multi-level network fusion round steel surface defect detection method of the present invention;

[0054] Figure 9 This is a flow chart of Example 2 of the multi-level network fusion round steel surface defect detection method of the present invention;

[0055] Figure 10 is a schematic diagram of an image acquisition device of the present invention;

[0056] Figure 11 It is a schematic diagram of the annular LED green light source of the image acquisition device of the present invention.

[0057] Reference numerals:

[0058] 1. Bracket; 2. Line scan camera; 3. Green LED light source. DETAILED DESCRIPTION

[0059] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0060] Example 1

[0061] like Figure 1 and Figure 8 As shown, this embodiment provides a multi-level network fusion round steel surface defect detection method, the algorithm framework diagram of the multi-level network fusion round steel surface defect detection method is as shown in FIG. Figure 8As shown, the detection method includes:

[0062] S100, acquiring real-time round steel surface images, establishing round steel surface images with defects as a round steel surface defect dataset, and establishing round steel surface images without defects as round steel surface non-defect image samples;

[0063] S200, using a round steel surface defect dataset, creating a round steel surface defect detection network model that integrates an Efficient Net classification priority network and a YOLOv5 network, wherein the model is capable of classifying and identifying types of round steel surface defects and locating positions of round steel surface defects;

[0064] S300, using transfer learning to train the round steel surface defect detection network model;

[0065] S400: adding the non-defective image samples of the round steel surface to the round steel surface defect detection network model and training the model.

[0066] The detection effect of round steel surface defect image and round steel surface non-defect image is shown in the figure Figure 7a to Figure 7h As shown, the detection method of this embodiment establishes a data set of round steel surface defects and round steel surface non-defect image samples. In order to solve the problem that the feature extraction capability of a single network module is limited and it is difficult to fully extract the features of round steel surface defects, the detection method of this embodiment fuses the Efficient Net classification priority network with the YOLOv5 network in a two-level network to obtain a round steel surface defect network model based on multi-level network fusion, which effectively solves the problems of false alarms and missed detections in surface defect detection and meets the needs of round steel surface defect detection. It has sufficient generalization ability and good robustness, and improves the detection accuracy. When training the round steel surface defect detection network model, transfer learning is used to train the entire round steel surface defect detection network model, which solves the problem that insufficient number of samples easily leads to an imbalance in the number of defect samples of different categories and easily causes overfitting during the training process. A round steel surface non-defect image sample is added to the training of the round steel surface defect detection network model, so that the sample division is more reasonable, which effectively improves the training accuracy and detection accuracy of the round steel surface defect detection network model.

[0067] like Figure 2 The step S100 of establishing a round steel surface defect dataset includes:

[0068] S110. Label the round steel surface defect images using LabelImg to identify the defect categories and locations, and establish a dataset. Specifically, manually select round steel surface defect images and round steel surface non-defect images suitable for network training and label them using Labelimg.

[0069] S120, respectively establish round steel surface defect image samples of pits, scratches, roller marks, peeling, ears, iron oxide belts and cracks, a total of 7 types of round steel surface defect image samples that affect the quality and appearance of the finished product. Round steel surface pit defect image samples are as follows: Figure 6a As shown, Figure 6b Round steel surface scratch defect image sample, round steel surface roller mark defect image sample Figure 6c As shown, the round steel surface peeling defect image sample is as follows Figure 6d As shown, the sample image of the ear defect on the round steel surface is as follows Figure 6e As shown, the image sample of iron oxide belt defect on the round steel surface is as follows Figure 6f As shown, the round steel surface crack defect image sample is as follows Figure 6g shown.

[0070] S130, the round steel surface images with a large number of iron oxide belts and ear defects that are easily misdetected are established as round steel surface non-defective image samples. The round steel surface non-defective image samples do not affect the quality of the round steel finished product. The round steel surface non-defective image samples are as follows: Figure 6h shown.

[0071] like Figure 3 In step S200, the round steel surface defect detection network model is created, which includes:

[0072] S210. Design a YOLOv5 target detection network model to obtain the classification and border positioning information of round steel surface defects, and design an EfficientNet network to prioritize the classification of round steel surface defects.

[0073] S220. During the defect detection process using the round steel surface defect detection network model, if the EfficientNet classification priority network classifies a defect with a confidence score greater than 0.9, the sample image is determined to definitely contain that type of defect. Otherwise, the confidence score is taken as no less than 0.3 as the possible defect category, and the YOLOv5 network is further combined to obtain the final detection result. Defect detection is performed on round steel surface image samples collected on-site. Generally, no more than three types of defects appear in a single image sample.

[0074] YOLOv5 of this embodiment has the advantages of fast detection speed and small model parameters. The specific advantages are as follows: First, data enhancement and other optimizations are performed at the input end. The specific methods include: using data enhancement methods such as random scaling, random cropping, and random arrangement for splicing to enhance the feature extraction ability of the network, which is conducive to improving the detection effect of small targets; by initially setting the length and width of the anchor frame, the weight parameters are continuously updated during network training to find the anchor frame value that is optimal for the distribution of the data set; since the image size needs to be adjusted during the training process (filling the border black pixels to meet the size requirements), the adaptive method is used to add the least black border pixels, which can reduce the extraction of redundant feature information, improve the inference speed, reduce the amount of calculation, and improve the target detection speed. Second, design a lighter backbone network (Backbone) with stronger feature extraction capabilities: design a Focus network structure so that feature maps can be sliced, and the convolution kernel parameters are shared to improve feature extraction efficiency; design a CSP1 network structure, which is a residual structure, which can speed up parameter updates and ensure accuracy while reducing the amount of calculation; third, improve the network feature fusion capability in subsequent network structures: adopt the FPN+PAN structure, and design a CSP2 structure to replace the conventional convolution structure to improve feature fusion capabilities.

[0075] This implementation uses a round steel surface defect detection network model that integrates the Efficient Net classification priority network and the YOLOv5 network. Using a single YOLOv5 target detection network model for defect classification often leads to missed detections and false detections due to incorrect defect classification. Based on the YOLOv5 network's acquisition of target classification and location, this implementation uses the EfficientNet classification priority network, which has strong feature extraction capabilities and small model parameters, as the classification priority network. The classification priority network only performs a single defect classification task, and its image feature extraction is more consistent with defect classification requirements.

[0076] It effectively solves the problems of false alarms and missed detections in surface defect detection and meets the needs of round steel surface defect detection.

[0077] like Figure 4 and Figure 5 In step S300, the round steel surface defect detection network model is trained by transfer learning, including:

[0078] S310. Use the weights obtained from training on the ImageNet dataset as the initialization weights for backbone network training, perform pre-training on a hot-rolled strip surface defect dataset to obtain pre-training weights, and use the pre-training weights as the initialization weights for backbone network training; specifically, the hot-rolled strip surface defect dataset is the publicly available Northeastern University hot-rolled strip surface defect dataset.

[0079] S320, dividing the round steel surface character dataset into a training set, a validation set, and a test set in a ratio of 7:1:2;

[0080] S321. Freeze the shallow pre-trained weights of the backbone convolutional neural network and use the Adam optimizer with a learning rate of 0.001 to pre-train the network model. Specifically, the batch size is set to 45 and the number of iterations is set to 200 epochs.

[0081] S322. Unfreeze the shallow pre-trained weights of the backbone convolutional neural network, reduce the learning rate to 0.0001, and train again. Specifically, adaptively adjust the batch size to 20, and train again for 300 epochs.

[0082] In step S300, the training hardware for training the round steel surface defect detection algorithm model is: the CPU uses the Intel W-2235 model, the GPU uses the RTX 3080 model, the memory is 32G, and the training environment: the deep learning environment of pytorch 1.8.0 is used.

[0083] In step S400, to enhance the network model's ability to extract defect features, a non-defective round steel surface sample was added to the training process, given the similarities in features across multiple defect classes. This resulted in a more rational sample partitioning process, allowing the network model to focus more on extracting defect features during training and effectively improving the detection accuracy of the trained network model. Furthermore, the use of transfer learning for model training addressed the issues of an imbalance in the number of defect samples across different classes, which can easily lead to overfitting during training due to insufficient sample size.

[0084] In the actual production process, according to the characteristics of the round steel production line, this embodiment designs an automatic image acquisition device suitable for collecting round steel defect samples, which is used for automatic acquisition of round steel surface images;

[0085] Example 2

[0086] like Figure 9 As shown, this embodiment provides a multi-level network fusion round steel surface defect detection method, the algorithm framework diagram of the multi-level network fusion round steel surface defect detection method is as shown in FIG. Figure 8 As shown, the detection method includes:

[0087] S100. Design an image acquisition device based on the specifications and diameters of round steel according to the requirements of actual application scenarios. Use the acquisition device to acquire real-time round steel surface images. Build round steel surface images with defects into a round steel surface defect dataset. Build round steel surface images without defects into round steel surface non-defect image samples.

[0088] S200, using a round steel surface defect dataset, creating a round steel surface defect detection network model that integrates an Efficient Net classification priority network and a YOLOv5 network. The model is capable of classifying and identifying the types of round steel surface defects and locating the locations of round steel surface defects.

[0089] S300, using transfer learning to train the round steel surface defect detection network model;

[0090] S400: adding non-defective image samples of round steel surfaces to the round steel surface defect detection network model and training the model.

[0091] Example 3

[0092] As shown in FIG10 , this embodiment provides an image acquisition device suitable for round steel with a specification and diameter of 50-130 mm, the device being used to acquire the round steel surface image in the multi-level network fusion round steel surface defect detection method, the device comprising:

[0093] The device includes a bracket on which four line scan cameras are evenly distributed around the circumference. Each line scan camera is positioned at a 45° angle to the horizontal and a 45° angle to the vertical. The line scan cameras have a GIGE digital interface, and the collected images of the round steel surface are transmitted remotely via a network to a server, which stores the images. Specifically, the four line scan cameras are spaced 90° apart in the radial direction of the round steel. The arrangement of the four line scan cameras ensures that the entire circumference of the round steel is captured without any blind spots. The line scan cameras use a 2048-resolution line scan camera with a maximum acquisition line frequency of 26K.

[0094] like Figure 11 As shown, a ring-shaped green LED light source is installed on the bracket. The green LED light source can effectively distinguish the wavelength band of the red-hot steel coil's own light, so that the image is not affected by the heating state of the steel coil itself, effectively preventing interference from the on-site lighting environment. The green LED light source has a large illumination area, high brightness, simple image adjustment, and good image uniformity. Compared with laser light sources, it is more suitable for hot rolling sites. The green LED ring light source used not only meets the requirements of illuminating the entire circumference of the round steel but also effectively reduces the unevenness within the illumination range (unevenness ≤ 5%).

[0095] Based on the image acquisition device of this embodiment, the number and arrangement of line scan cameras are adjusted to be applicable to the production of round steels of other specifications and diameters.

[0096] The image acquisition device of this embodiment collects real-time round steel surface images and transmits them to the server for round steel surface defect detection. The images collected by the image acquisition device have no blind spots and good image uniformity, and are suitable for round steel surface defect detection.

[0097] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A multi-level network fusion round steel surface defect detection method, characterized in that: The detection method comprises: Acquire real-time round steel surface images, establish round steel surface images with defects as round steel surface defect datasets, and establish round steel surface images without defects as round steel surface non-defect image samples; A round steel surface defect detection network model was constructed by integrating the Efficient Net classification priority network with the YOLOv5 network. The model was able to classify and identify the types of round steel surface defects and locate the locations of round steel surface defects. Using transfer learning to train the round steel surface defect detection network model; Adding non-defective image samples of round steel surfaces to the round steel surface defect detection network model and training the model; Among them, the creation of the round steel surface defect detection network model includes: Design a YOLOv5 target detection network model to obtain the classification and bounding box positioning information of round steel surface defects, and design an EfficientNet network to prioritize the classification of round steel surface defects; Among them, in the process of round steel surface defect detection network model detecting defects, if the EfficientNet classification priority network has a classification defect with a confidence level greater than 0.9, it is determined that the sample image must have this type of defect; otherwise, the confidence score is taken as not less than 0.3 as the possible defect category, and further combined with the YOLOv5 network to obtain the final detection result.

2. The multi-level network fusion round steel surface defect detection method according to claim 1 is characterized in that: The round steel surface defect dataset is established, including: Use LabelImg to label the surface defect image of round steel and mark the defect category and defect location information.

3. The multi-level network fusion round steel surface defect detection method according to claim 2 is characterized in that: The establishment of the round steel surface defect dataset also includes: Create surface defect image samples of round steel including pits, scratches, roller marks, warping, ears, iron oxide belts and cracks; The round steel surface images with more iron oxide belts and ear defects that are easily misdetected are established as round steel surface non-defect image samples.

4. The multi-level network fusion round steel surface defect detection method according to claim 1 is characterized in that: In training the round steel surface defect detection network model using transfer learning, the following steps are included: The weights obtained from training on the ImageNet dataset are used as the initial weights for backbone network training. Pre-training is performed on the hot-rolled strip surface defect dataset to obtain pre-trained weights, which are used as the initial weights for backbone network training. The round steel surface character dataset is divided into training set, validation set and test set with a ratio of 7:1:

2.

5. The multi-level network fusion round steel surface defect detection method according to claim 1 is characterized in that: The round steel surface character dataset is divided into training set, validation set and test set, including: Freeze the shallow pre-trained weights of the backbone convolutional neural network and use the Adam optimizer with a learning rate of 0.001 to pre-train the network model; Unfreeze the shallow pre-trained weights of the backbone convolutional neural network and reduce the learning rate to 0.0001 for training again.

Images