A steel pipe small target defect identification and positioning method based on deep learning

Abstract

The present application belongs to the field of deep learning industrial machine vision, specifically a steel pipe small target defect recognition positioning method based on deep learning, comprising the following steps: extracting the feature information of the target image through the backbone layer of Yolo5, introducing the Transformer network in the extraction process, further extracting the position and feature information of multiple small target defects in the target image, and inputting the extracted feature information of the position of multiple small target defects in the target image into the network layer; optimizing the network structure in the network layer, fusing the features of the feature information of the target image extracted by the backbone layer, obtaining the feature information extracted by the network layer; performing feature screening on the feature information extracted by the network layer to obtain the target size of the steel pipe defect, and finally outputting the steel pipe defect recognition result in the target image. The present application makes the accuracy and precision of the recognition of small target defects of the steel pipe higher.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of deep learning industrial machine vision, in particular to a steel pipe small target defect recognition and positioning method based on deep learning. BACKGROUND

[0002] With the development of the Internet in recent years, computer vision in the field of deep learning has developed rapidly, and more and more enterprises combine computer vision with industrial production and maintenance. In the early days when computer vision did not exist, surface defects were mainly detected by manual and photoelectric methods. In recent years, computer vision has been increasingly widely used in the industrial field. Compared with other methods, computer vision technology has the advantages of low cost, fast positioning, and high recognition accuracy. Therefore, computer vision target detection technology has great development prospects in steel pipe defect detection.

[0003] Steel pipes are used in many industrial fields as raw materials. Higher requirements are put forward for the quality of steel pipes. If the partial defects of the steel pipes are not timely processed, it may pose a great threat to the life and property safety of enterprises and employees. In the existing target recognition model, due to the high cost of industrial data set collection, the training and verification data set is mainly composed of medium and large target data sets, so the recognition effect of the steel pipe small target defect is not good, and there are problems of low accuracy and poor recognition effect of the steel pipe small target defect. SUMMARY

[0004] The purpose of the present application is to provide a more accurate steel pipe small target defect recognition method based on deep learning, which can accurately recognize the steel pipe small target defect and solve the problems of low accuracy and poor recognition effect. A new steel pipe small target defect recognition scheme is proposed to overcome the defects of the existing target recognition model.

[0005] The technical scheme adopted by the present application to achieve the above purpose is: a steel pipe small target defect recognition and positioning method based on deep learning, comprising the following steps:

[0006] 1) The feature information of the target image is extracted by the backbone layer of Yolo5, and the Transformer network is introduced in the extraction process to further extract the position and feature information of multiple small target defects in the target image, and the extracted feature information of the position of multiple small target defects in the target image is input into the network layer;

[0007] 2) The network structure in the network layer of Yolov5 is optimized to improve the ability of the network model to fuse the feature information of each dimension, and the feature information of the target image extracted by the backbone layer is fused to obtain the feature information extracted by the network layer;

[0008] 3) performing feature screening on the feature information extracted by the network layer to obtain a target size of the steel pipe defect, and finally outputting a steel pipe defect recognition result in a target image.

[0009] In step 1), the feature information of the target image is extracted by the backbone layer of Yolo5, including the following steps:

[0010] 1-1) performing slice processing on the target image to realize down-sampling without information loss, obtaining a plurality of images, and splicing the plurality of images;

[0011] 1-2) obtaining local spatial information in the target image by a convolution layer in the backbone layer;

[0012] 1-3) further increasing the network depth by the Bottlneck module to facilitate the extraction of the features of the target image;

[0013] 1-4) introducing a transformer module to further extract the position and feature information of the small target defects in the image by optimizing the encoding module in the backbone layer;

[0014] 1-5) performing a maximum pooling operation on the input target image by the SPP module to form a group of feature maps with different sizes.

[0015] The Transformer network is introduced in the extraction process to further extract the position and feature information of the multiple small target defects in the target image, specifically:

[0016] The PE encoding module in the Transformer network is optimized, and the expression of the optimized PE encoding module is:

[0017]

[0018]

[0019] wherein, pos is the position of the word, i is the dimension of the word, d modle is a feature vector of the position encoding, LR is a LeakyRelu function, and is a nonlinear function to prevent gradient disappearance;

[0020] According to the PE module, the speed of linear transformation is slowed down as i increases, the instability of the Transformer in the training process is alleviated, and the ability of the module to extract detailed features is improved.

[0021] The step 2) is specifically:

[0022] The network structure of Yolov5 is optimized by the MHSA function, that is:

[0023] In Yolov5, the network layer cross-fuses features of different scales through FPN_PANet, and replaces the convolution at the lowermost layer of the FPN_PANet feature fusion layer with a multi-head self-attention module MHSA.

[0024] The network layer uses upsampling to continuously increase the size of the feature map, and uses downsampling to continuously decrease the size of the feature map, and fuses features and semantic features of different levels to obtain more comprehensive feature information.

[0025] The step 3) is specifically:

[0026] 2-1) In the network layer, the prediction frame is self-defined, and prediction frames for large targets, medium targets and small targets are respectively generated;

[0027] 2-2) The medium target prediction frame and the small target prediction frame are retained, and the large target prediction frame is removed, and the minimum anchor frame generated by the Kmean clustering method is used to replace the large target prediction frame;

[0028] 2-3) The prediction frame is screened by using the maximum suppression, and the recognition result is finally output.

[0029] The minimum anchor frame generated by the Kmean clustering method is used to replace the large target prediction frame, which is specifically:

[0030] (1) Randomly select points from the target defect data set as the center points of the initial clustering, and assume that the randomly selected points are 1…K, then the center points N={n1, n w ,…,n k};

[0031] (2) Calculate the distance d of the horizontal coordinates of different sample labels in the target data set to each center point N;

[0032] (3) Put the sample with the smallest distance d into the class of the divided clustering center;

[0033] (4) Recalculate the clustering center of each class for different classes of samples in the class of the divided clustering center;

[0034] (5) Repeat steps (3) to (4), and calculate by iteration until the position of the clustering center no longer changes;

[0035] (6) Add the three generated prediction frames and take the average to obtain the finally generated minimum anchor frame.

[0036] The divided cluster center, in particular is:

[0037] 3-1) first randomly select k samples from the target defect data set as cluster centers;

[0038] 3-2) and calculate the distance between all samples and the k cluster centers;

[0039] 3-3) for each sample, divide it into the cluster where the cluster center with the closest distance is located;

[0040] 3-4) update the center of each cluster, that is, calculate according to the average value, and then take it as the new clustering center.

[0041] The present application has the following beneficial effects and advantages:

[0042] 1. The present application proposes a steel pipe small target defect recognition method based on deep learning, which not only solves the problems of poor recognition effect and low accuracy of steel pipe small target defect recognition, but also improves the overall stability of the model. Compared with other schemes and methods, the present model has higher accuracy, stronger stability and faster convergence speed.

[0043] 2. The present application can make the recognition accuracy and precision of steel pipe small target defects higher.

[0044] 3. In the present application, the neck network FPN_PANet also introduces MHSA (multi-head self-attention) for optimization to further enhance the ability of the model to fuse small target defect position information.

[0045] 4. The present application uses the optimized Transformer module to further extract the overall position information of the defect through PE encoding. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 The flow chart of the steel pipe small target defect recognition method of the present application;

[0047] Figure 2 The structure diagram of the steel pipe defect feature extraction of the present application;

[0048] Figure 3 The structure diagram of the steel pipe defect feature fusion of the present application;

[0049] Figure 4a The example diagram of the target prediction frame in the defect recognition of the present application;

[0050] Figure 4b The example diagram of the small target prediction frame in the defect recognition of the present application;

[0051] Figure 4cThis is an example diagram of the prediction box for identifying large, defective targets according to the present invention. Detailed Implementation

[0052] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0053] The overall process of this invention combines a YOLOv5 network model with a Transformer network to effectively solve the problem of identifying small target defects in steel pipes. First, image enhancement methods are used to expand and enhance the data collected from the steel pipes to address the issue of limited dataset size. In the identification of small targets in the steel pipe defect data, firstly, during the feature extraction stage of the network model, the backbone layer of YOLOv5 is used to extract feature information, which is then introduced into a Transformer network, and its encoding module is optimized to further extract the location and feature information of multiple small target defects in the image. Secondly, during the feature fusion stage of the network model, the neck network FPN_PANet of YOLOv5 is also introduced into MHSA (multi-head self-attention) for optimization to further enhance the model's ability to fuse the location information of small target defects. Finally, the large target prediction box in the YOLOv5 head network is optimized using the k-means clustering method to generate a prediction box that more accurately reflects the size of the steel pipe defect target.

[0054] This invention provides a method for identifying small target defects in steel pipes based on deep learning. The specific details will be described below to help people better understand this invention.

[0055] like Figure 1 The diagram shown is a flowchart of the method for identifying small target defects in steel pipes according to the present invention, illustrating the identification process of small target defects in steel pipes. It mainly includes three parts: extraction and fusion of defect feature information, and output of results.

[0056] Step 1) In the feature extraction stage of the network model, the backbone layer of YOLO5 is used to extract feature information, and then the Transformer network is introduced and its encoding module is optimized to further extract the location and feature information of multiple small target defects in the image.

[0057] Step 2) Use MHSA (multi-head self-attention) to optimize the Yolov5 network structure and further improve the network model's ability to fuse feature information from various dimensions.

[0058] MHSA is a multi-head self-attention module used to extract long-range structural information in images. Therefore, this paper adds MHSA to the bottom after the encoder of UNet. The main function of MHSA is to establish the relationship between each element in the abstract feature map, and the corresponding receptive field is the whole image.

[0059] The specific implementation is: replace the convolutions between C5 and C6 in NHSA.

[0060] Step 3) Optimize the large target prediction boxes of the Yolov5 head network using the k-means clustering method to better match the target size of the steel pipe defect, and use maximum suppression (NMS) for filtering to finally output the recognition results.

[0061] The recognition result is shown in Figure 4, which is a specific example of the flowchart of the defect recognition algorithm of the present invention, and is a specific recognition result diagram of the flowchart of the defect recognition algorithm of the present invention.

[0062] like Figure 2 The diagram shown illustrates the structural diagram for steel pipe defect feature extraction in this invention. It is based on the YOLOv5 backbone layer for feature extraction, and then utilizes an optimized Transformer to further extract the location and feature information of multiple small target defects in the image. The YOLOv5 backbone layer consists of one Focus module, four 3x3 convolutional layers with a stride of 1, 24 Bottleneck CSP modules, and one SPP pooling module. The Focus module slices the image before convolution, allowing the backbone layer to extract more feature information. The 3x3 convolutional layers extract and transform local spatial information from the steel pipe image. The Bottleneck modules with 128, 256, 512, and 1024 channels increase the network depth, which is beneficial for feature extraction. Finally, the SPP module performs max pooling on the input steel pipe image, forming a set of feature maps of different sizes. Before the SPP max pooling operation, the optimized Transformer module is used to further extract the overall location information of the defect through PE encoding. The detailed optimization process is as follows:

[0063] By introducing a Transformer network and optimizing its encoding module, the location and feature information of multiple small target defects in the image are further extracted. Specifically:

[0064] When Tranformer extracts image feature information, it goes through an encoding module PositionEmbedding. We optimize it based on the characteristics of steel pipe defects to extract the position and feature information of multiple small target defects in the image. Finally, it goes through the Spatial Pyramid Pooling network structure to input the extracted feature information into the network layer for further extraction.

[0065] The original encoding module formula was:

[0066]

[0067]

[0068] sin(∝+β)=sinαcosβ+cosαsinβ and cos(∝+β)=cosαcosβ-sinαsinβ, we can see that the feature vector at position n changes linearly. Although the PE encoding module is very helpful in extracting relative position information, its stability is poor during training. It is necessary to reduce the speed of linear change. The optimized PE formula is:

[0069]

[0070]

[0071] In the formula above, pos represents the position of the word, i represents the dimension of the word, and d represents the position of the word. modle This represents a feature vector for positional encoding. LR stands for the LeakyReLU function, which is a non-linear function that can effectively prevent gradient vanishing. As i increases, the PE module slows down the linear transformation to alleviate the instability of the Transformer during training and improve the module's ability to extract detailed features.

[0072] Figure 3 The diagram shown illustrates the structural fusion of steel pipe defect features according to the present invention. The neck network, composed of an FPN_PANet network structure, primarily fuses feature information from different levels through convolution stacking and channel number variation. Specifically, the neck network uses upsampling to continuously increase the feature map size from the shallower feature information obtained from the backbone layer, and downsampling to continuously decrease the feature map size, thus fusing graphic and semantic features from different levels to obtain more comprehensive feature information.

[0073] In the head network (layer network), the original predicted bounding boxes are custom-defined, categorized as large, medium, and small. Specifically, as follows... Figures 4a to 4c, respectively, are defect recognition example graphs of middle, small and large prediction boxes;

[0074] In the steel pipe image, there is a labeled label, and the generated anchor box is closer to the label. Because most of the steel pipe defects are small targets, the self-defined large target anchor box may not be suitable for the size of the steel pipe defect, so the middle and small prediction boxes are retained, and the large target prediction box is removed. The minimum anchor box generated by the Kmean clustering method is used instead. The maximum suppression (NMS) is used to screen the prediction box, and the final output recognition result is output.

[0075] The process of the specific Kmean clustering method for generating the minimum anchor box is as follows:

[0076] Randomly selected points from the steel pipe defect data set are used as the initial clustering center points, which are assumed to be 1…K, and the center points N={n1, n2,…, n k}; Then the horizontal coordinates of different sample labels in the steel pipe data set are calculated, and the distance d from each N is calculated. The sample with the smallest d is placed in the cluster center class. Then the cluster centers of different classes in the class are recalculated. Finally, only the calculation is performed by iteration until the position of the cluster center no longer changes. Then the three prediction boxes generated are added and averaged to obtain the final generated minimum anchor box.

[0077] Among them, the divided cluster center is specifically:

[0078] 3-1) First, randomly select k samples from the target defect data set as cluster centers;

[0079] 3-2) Calculate the distance between all samples and the k cluster centers;

[0080] 3-3) For each sample, divide it into the cluster center of the cluster where the nearest cluster center is located;

[0081] 3-4) Update the center of each cluster, i.e. calculate it according to the average value, and then use it as the new cluster center.

[0082] Among them, n anchor boxes are obtained by using K-means clustering. ② Use genetic algorithm to randomly mutate the wh of the anchor box. If the effect becomes better after mutation, keep this anchor box. Finally, generally 3 suitable anchor boxes are generated, but in this paper, the small and medium anchor boxes manually set are retained as the anchor boxes of the medium and large targets, and then the 3 suitable anchor boxes generated by keras above are added and divided by 3, and then the minimum target anchor box is obtained.

[0083] The above merely illustrates the embodiments of the present application, and any modification made by those skilled in the art within the spirit and principle of the present application shall fall within the protection scope of the present application.

Claims

1. A method for identifying and locating small-target defects in steel pipes based on deep learning, characterized in that, Includes the following steps: 1) The feature information of the target image is extracted through the backbone layer of YOLO5. During the extraction process, the Transformer network is introduced to further extract the location and feature information of multiple small target defects in the target image. The extracted feature information of the location of multiple small target defects in the target image is input into the network layer. The process of introducing a Transformer network during extraction further extracts the location and feature information of multiple small target defects in the target image. Specifically: By introducing a Transformer network to optimize the PE encoding module in the pole layer, the optimized PE encoding module expression is as follows: ； ； Where pos is the position of the word, and i is the dimension of the word. This is a feature vector encoded by position. LR is the LeakyReLU function, which is a non-linear function to prevent gradient vanishing. According to the PE module, as i increases, the speed of linear transformation is slowed down, the instability of Transformer operation during training is alleviated, and the module's ability to extract detailed features is improved. 2) Optimize the network structure in the Yolov5 network layer to improve the network model's ability to fuse feature information from various dimensions. Fuse the feature information of the target image extracted from the backbone layer to obtain the feature information extracted by the network layer. Step 2), specifically: The Yolov5 network structure is optimized using the MHSA function, namely: In YOLOv5, the network layers cross-fuse features at different scales through FPN_PANet, and the convolution at the bottom layer of the FPN_PANet feature fusion layer is replaced with a multi-head self-attention module MHSA. The network layers use upsampling to continuously increase the size of the feature map from the backbone layer and downsampling to continuously decrease the size of the feature map. They also fuse graphic and semantic features from different levels to obtain more comprehensive feature information. 3) The feature information extracted from the network layer is filtered to obtain the target size of the steel pipe defect, and finally the steel pipe defect recognition result in the target image is output.

2. The method for identifying and locating small target defects in steel pipes based on deep learning according to claim 1, characterized in that, Step 1), the extraction of feature information from the target image through the YOLO5 backbone layer, includes the following steps: 1-1) Slice the target image to achieve downsampling without information loss, obtain multiple images, and stitch the multiple images together; 1-2) The stitched image is processed through convolutional layers in the backbone layer to obtain local spatial information of the target image; 1-3) Then pass the target image through the Bottelneck module to increase the network depth, which is beneficial for the extraction of target image features; 1-4) Introducing the transformer module, by optimizing the encoding module in the backbone layer, further extracting the location and feature information of small target defects in the image; 1-5) The extracted feature information is used to perform max pooling on the input target image through the SPP module to form a set of feature maps of different sizes.

3. The method for identifying and locating small target defects in steel pipes based on deep learning according to claim 1, characterized in that, Step 3) specifically involves: 2-1) In the network layer, the prediction boxes are custom-defined, namely prediction boxes for large targets, medium targets, and small targets; 2-2) Retain the predicted boxes for medium and small targets, remove the predicted boxes for large targets, and replace the predicted boxes for large targets with the smallest anchor boxes generated by the K-means clustering method. 2-3) Use maximum suppression to filter the predicted bounding boxes and finally output the recognition results.

4. The method for identifying and locating small target defects in steel pipes based on deep learning according to claim 3, characterized in that, The method of replacing the large target prediction box with the minimum anchor box generated by K-means clustering is as follows: (1) Randomly selected points from the target defect dataset will be used as the center points for the initial clustering. Assuming the randomly selected points are 1…K, then the center points will be… ; (2) Calculate the distance d from the x-coordinate of the different sample labels in the target dataset to each center point N; (3) Place the sample with the smallest distance d into the class of the divided cluster centers; (4) For samples of different categories in the class of the divided cluster centers, recalculate the cluster centers of each category; (5) Repeat steps (3) to (4) and perform calculations iteratively until the position of the cluster center no longer changes; (6) Add the three generated prediction boxes together and then take the average to obtain the final generated minimum anchor box.

5. The method for identifying and locating small target defects in steel pipes based on deep learning according to claim 4, characterized in that, The cluster centers defined in the division are specifically as follows: 3-1) First, randomly select k samples from the target defect dataset as cluster centers; 3-2) And calculate the distance between all samples and the centers of these k clusters; 3-3) For each sample, assign it to the cluster containing the nearest cluster center; 3-4) Update the center of each cluster, i.e. calculate it according to the average value, and then use it as the new cluster center.

Images