A deep learning-based X-ray image weld defect detection method

Abstract

The application provides a kind of X-ray image weld defect detection method based on deep learning, utilizes FasterRCNN network to carry out weld defect detection to X-ray image, compared with traditional FasterRCNN, the ResNet50 module of backbone network is improved in the application, gi information gain module is added behind the last convolution layer in bottleneck, the feature extraction capability is improved, at the same time, the preset anchor frame generation ratio in RPN network is modified, the target length-width ratio to be detected is clustered using K-Means++ algorithm, the defect feature length-width ratio distribution more in line with current detection data set is obtained, and the defect detection effect is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning technology and pipeline weld defect detection technology, and particularly relates to an X-ray image weld defect detection method based on deep learning. BACKGROUND

[0002] Welding technology plays a very important role in manufacturing industry. However, due to the limitation of welding technology, welding defects such as round defects, strip defects, incomplete fusion, undercut, internal concave, cracks and the like still exist in the welding process. Therefore, the detection of welding defects is very important to ensure the reliability and safety of the welding products. At present, the most common method for defect detection is still manual detection, which has low detection efficiency and the accuracy is limited by the professionalism of the detection personnel. Therefore, relevant researchers begin to combine machine learning algorithms to conduct research on automatic detection of welding defects on X-ray images of welds.

[0003] Early defect detection algorithms identify each candidate region generated by the image processing stage through machine learning algorithms to determine the defect location and category. For example, Duan Feng et al. (DUAN Feng, YIN Shifan, SONG Peipei, et al. Automatic Welding Defect Detection of X-Ray Images by Using Cascade AdaBoost with Penalty Term[J]. IEEE Access, 2019, 7: 125929-125938.) obtained potential defect regions in the image through image preprocessing, and used cascade AdaBoost to judge the real defects in the potential defect regions. However, this method is not accurate enough in position prediction. With the wide application of deep learning algorithms, FasterRCNN and YOLOv3 are used for weld defect detection tasks. For example, Guo Wenming et al. (Guo Wenming, Liu Kai, Qu Huifan. Welding Defect Detection of X-ray Images Based on FasterR-CNN Model[J]. Journal of Beijing University of Posts and Telecommunications, 2019, 42(06): 20-28.) designed a welding defect detection system based on Faster RCNN, which made a simple attempt at defect detection. Zhong Jiajun et al. (Zhong Jiajun, He Deqiang, Miao Jian, Chen Yanjun, Yao Xiaoyang. Subway Vehicle Welding Defect Detection Based on Improved FasterR-CNN[J]. Journal of Railway Science and Engineering, 2020, 17(04): 996-1003.) introduced the Unet model and Resnet model to improve the original FasterR-CNN framework, enhancing the feature extraction capability of the model and improving the recognition accuracy. Wei Ruoyu et al. (Wei Ruoyu, Li Shuting, Wu Songrong, et al. Track Fastener Defect Detection Based on Improved YOLOV3 Algorithm[J]. Railway Standard Design, 2020, 64(12): 30-36.) used two prediction branches in YOLOv3 to detect defects. However, the accuracy of the model is still low, and the existing research direction mainly focuses on multi-scale defect target detection and small target defect detection. For the problem of multi-scale defects, Chen Haiyong et al. (Chen Haiyong, Zhao Peng, Yan Haowei. Crack Detection of Multi-scale FasterRCNN with Attention Fusion[J]. Optoelectronic Engineering, 2021, 48(1): 64-74.) added a self-downward path FPN in the backbone network of FasterRCNN, combined with the FocalLoss loss function, to improve the defect detection accuracy. Zhu Ankang et al. (Zhu Ankang, Wang Luofen, Zhang Zhenyu. Design of Aluminum Surface Defect Detection System Based on Deep Learning[J]. Sensors and Microsystems, 2022, 41(08): 96-99+103.) added deformable convolution to the FasterRCNN network model to improve detection accuracy due to the uneven size of defects.For the problem of small target defect detection, the most common method is to use feature pyramid network (FPN) to predict in multiple layers of feature maps. Wu Ren et al. (Wu Ren, Sun Yuan. Welding defect detection based on combination of cascade structure and Faster R-CNN [J]. Combination machine tool and automatic processing technology, 2022 (02): 59-62 + 67.) proposed an improved Faster R-CNN welding defect detection method, which improved the RPN network, used K-means clustering method to generate anchor box closer to the target area, combined with deep residual network (ResNet), obtained smaller welding defect detail features, and used a three-layer cascade structure based on IOU value to obtain accurate defect position. Chen Yongbin et al. (CHEN Yongbin, WANG Jingran, WANG Guitang. Intelligent welding defect detection model based on improved R-CNN [J]. IETE Journal of Research, 2022: 1-10.) improved the feature extraction layer of Faster RCNN using Res2Net residual network, and combined with FPN module with weighted features to improve the detection ability of small targets. The above research has done a lot of work around defect detection, but the existence of multi-size defects and small defects makes the detection model still need to continue to improve the accuracy, Arthur D et al. (Arthur D, Vassilvitskii S. K-Means++: The advantages of careful seeding [C] / / Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2007, New Orleans, Louisiana, USA, January 7-9, 2007. ACM, 2007.) proposed an improved K-Means clustering algorithm, which improves the clustering effect by carefully selecting the initial seed point. Among various existing detection methods, the K-Means++ algorithm is the mainstream scheme for generating detection anchor frames through clustering. SUMMARY

[0004] The application provides a kind of X ray image welding defect detection method based on deep learning, can accurately detect the welding defect in X ray image.

[0005] To achieve the above technical purpose, the application adopts the following technical scheme:

[0006] 1. An X-ray image welding defect detection method based on deep learning, characterized by comprising the following steps:

[0007] Step one: make the data set X for training and testing, the specific steps are as follows:

[0008] (1) The welding seam image data is cropped, the original welding seam image size is w*h, wherein w<h, the value range of w and h is positive integer, the sliding window is used to crop each welding seam image, the sliding window step is S, S∈(0, h), and the original welding defect label information is mapped to obtain the defect label information of the cropped image, finally a group of welding seam image data set D with size of h*h is obtained;

[0009] (2) The data in the data set D is divided, the data set D is randomly divided into training set D1, verification set D2 and test set D3 according to a:b:c ratio, wherein a, b and c are positive integers, and a+b+c=10, D1, D2 and D3 together constitute the data set X;

[0010] Step two: build the improved FasterRCNN network model, the specific steps are as follows:

[0011] (1) X is input into the FasterRCNN network model as the input of the network;

[0012] (2) The data is input into the improved residual network ResNet50 model for training, the improved ResNet50 is composed of different number of Bottleneck modules, Bottleneck is the basic module unit in the network, the structure is inplanes is the input channel number, planes is the output channel number, the improved ResNet50 is composed of 3 Bottleneck in the first part, 4 Bottleneck in the second part, 6 Bottleneck in the third part, and 3 Bottleneck in the fourth part, and the multi-scale features output by the four parts are respectively denoted as C2, C3, C4 and C5, wherein the specific implementation method of the improved ResNet50 network is as follows:

[0013] The improved ResNet50 network is composed of five convolutional layers: the first convolutional layer contains a [7x7, 64] convolution with stride=2 and a [3x3] maximum pooling with stride=2; the second convolutional layer is composed of three Bottleneck Network structure, wherein gi is the abbreviation of globalinfo block, which is a global information gain module; the third convolutional layer is composed of four Bottleneck Network structure; the fourth convolutional layer is composed of six Bottleneck Network structure; the fifth convolutional layer is internally composed of three Bottleneck Network structure

[0014] The calculation formula of the gi module, denoted as G, is as follows:

[0015]

[0016] where x represents the input of different Bottleneck modules, gp x represents the feature map after channel adjustment grouping of the input x, and the calculation method is gp x =R(x), where b, c, h, and w respectively represent the batch number, the channel number, the height, and the width, g is the grouping size, R(·) represents adjusting the input shape to [b×g, c / / g, h, w], R1 represents adjusting the output shape to [b×g, h×w, 1], R2 represents adjusting the output shape to [b×g, c / / g, h×w], R3 represents adjusting the output shape to [b×g, 1, c / / g], R4 represents adjusting the output shape to [b×g, c / / g, h×w], R5 represents adjusting the output shape to [b×g, 1, h, w], R6 represents adjusting the output shape to [b, c, h, w], represents a 1×1 convolution with the input channel number c / / g and the output channel number 1, represents a 1×1 convolution with the input and output channel numbers both being c / / g, represents a 1×1 convolution with the input channel number 3×c / / g and the output channel number c / / g, represents a 3×3 convolution with the input and output channels both being c / / g, represents a 3×3 convolution with the input and output channels both being c / / g, the dilation being 2, and the padding being 2, represents a 3×3 convolution with the input and output channels both being c / / g, the dilation being 3, and the padding being 3, and softmax and sigoid represent two kinds of activation functions, respectively, where the symbol / / represents the integer division operation;

[0017] (3) Taking the multi-scale features C2, C3, C4, and C5 obtained in (2) as inputs, inputting them into the feature pyramid network FPN for multi-scale feature fusion, fully utilizing different scale feature information, and outputting a group of fused multi-scale features, denoted as P2, P3, P4, P5, and P6, wherein P2, P3, P4, P5, and P6 are respectively generated by feature splicing of the feature maps generated by C2, C3, C4, and C5 and the upsampling operation, and P6 is generated by one down-sampling based on P5, and the formula is as follows:

[0018]

[0019] wherein i represents a feature map index, i is an integer, i∈[2, 6], C i represents the multi-scale feature output of the i-th layer in the input FPN, C i+1 represents the output corresponding to the i+1-th layer in the FPN, C i-1 represents the input corresponding to the i-1-th layer in the FPN, Upsample and Downsample represent 2 times up-sampling and 2 times down-sampling respectively, Conv 1×1 and Conv 3×3 respectively represent two kinds of convolution with 1×1 and 3×3 convolution kernels, and Add represents an element-wise addition operation;

[0020] (4) taking the obtained multi-scale feature map as input, constructing a region recommendation network layer to generate region candidate proposals, judging the proposal frame to be a positive sample or a negative sample through Softmax, and using bounding box regression to obtain an accurate detection frame, specifically, the region recommendation network layer includes a front background classification branch and a bounding box regression branch, in the classification branch, it is judged whether a defect exists at each position of the feature map, and a negative example is marked if no defect is contained, and a positive example is marked if a defect is contained, in the regression branch, a new anchor box ratio is used to replace the original anchor box ratio generated, the new anchor box ratio is generated using the K-Means++ algorithm, and the anchor box of the position where the defect exists is corrected according to the offset of the center point coordinates, width and height of the basic anchor box, and the region proposal is obtained by comprehensively considering the results of the two branches, wherein the specific steps of the K-Means++ algorithm for anchor box ratio clustering are as follows:

[0021] 1. Group all target labeled anchor boxes according to the IOU distance, that is, similar anchor boxes are placed in the same group;

[0022] The IOU distance calculation formula is:

[0023]

[0024] wherein X and Y respectively represent two target frames in the target labeling information, Area represents the area operation, ∩ represents the intersection operation, ∪ represents the union operation, and IOU(X, Y) represents the calculation of the IOU distance of X and Y;

[0025] 2. Randomly select an anchor box from each group as the center of the group;

[0026] 3. For each anchor box, calculate the distance to all centers of mass, select the group where the nearest center of mass is located, and add the anchor box to the group;

[0027] 4. Repeat ② and ③ until all anchor boxes are divided into a group;

[0028] 5Calculate the proportion of all anchor boxes in each group as the proportion of the target labeled anchor box;

[0029] (5) The fused multi-scale feature map and the detection frame are input into the ROIPooling layer, and according to the positive example, the negative example and the position offset information obtained in step three, the corresponding feature map is extracted, the feature is extracted, the corresponding feature map is obtained, and the output is adjusted to a uniform size, wherein the formula for selecting the feature map during feature extraction is:

[0030]

[0031] Wherein m is the number of feature maps, the value range is a positive integer, m0 is the initial feature map number, the value range is a positive integer, h and w are the high and wide after offset respectively, the value range is a positive integer, T is a constant, the value range is a positive number, Indicates the floor operation;

[0032] (6) The feature map obtained after the ROIPooling layer is taken as input and sent to the classification and regression layer composed of two fully connected layers, wherein one fully connected layer classifies the defect categories, and the other fully connected layer regresses the defect position, so as to obtain the defect category and position, and finally produce a detection frame with defect category and confidence;

[0033] Step three: use error back propagation to optimize and update the parameters in the improved FasterRCNN network model, and save all the network parameters P with the highest mAP value in the FasterRCNN model detection result in each training round, the specific steps are as follows:

[0034] (1) Adopt error back propagation, use cross entropy loss function to measure the loss of classification model, use L1 loss to regress the frame position, and the joint loss function formula of the two loss functions is:

[0035] L=-∑(y*log(p)+(1-y)*log(1-p))+λ∑(|p b -g t |)

[0036] Wherein L represents the joint loss function value, y is the real category label, p is the predicted probability of the category, p b Indicates the predicted frame coordinates, g t Indicates the real frame coordinates, |·| represents the absolute value, λ is used to balance the weight of the two loss terms, -∑(y*log(p)+(1-y)*log(1-p)) represents the cross entropy loss function, and λ∑(|p b -g t |) represents the L1 loss function;

[0037] (2) Through each training of the training sample X, the loss value L is made to be back propagated in the improved network model by using the stochastic gradient descent function SGD, so that the model parameter P of the i th round of training of the FasterRCNN i The random change in the negative gradient direction is used to optimize the network, and the calculation formula of the model parameter updated by the SGD is as follows:

[0038]

[0039] Wherein, P i represents the model parameter of the i th round of training of the FasterRCNN, P i-1 represents the model parameter of the i-1 th round of training of the FasterRCNN, and η represents the optimization step of the SGD, represents the derivative of the training loss L with respect to the model parameter P i-1 of the i-1 th round of training of the FasterRCNN,

[0040] (3) The mean average precision mAP of the model is calculated, and the calculation formula is as follows:

[0041]

[0042] Wherein, mAP represents the mean average precision, N represents the number of categories, and AP represents the mean average precision, and the calculation formula is as follows:

[0043]

[0044] Wherein, P(R) represents the curve of the precision relative to the recall rate, and in practice, the AP value is the average value of the maximum precision in a certain number of equal-interval recall rate intervals, and the formula of the precision is as follows:

[0045]

[0046] Wherein, P is the precision, TP is the number of samples that are predicted to be positive examples and are actually positive examples, and FP is the number of samples that are predicted to be positive examples and are actually negative examples.

[0047] (4) The weight model with the highest mAP value in the model is saved as the final trained weight model P;

[0048] Step six: the test stage needs to post-process the prediction bounding box output by the last layer of the model, remove the bounding box generated in the detection that does not meet the requirements, and the processing method is: remove the bounding box whose coordinates exceed the image boundary, subdivide the remaining bounding box according to the category, sort the bounding box of the same category according to the score obtained by classification, remove the bounding box whose score is low and whose intersection over union with the high-score bounding box exceeds k, the value range of k is (0, 1), and the remaining bounding box is used as the final prediction result to predict the defect position and defect category.

[0049] The X-ray image weld defect detection method based on deep learning provided by the application uses an improved FasterRCNN network to detect weld defects in X-ray images, compared with the traditional Faster RCNN, the gi module is used to improve the Resnet50 feature extraction network part, the feature extraction capability is increased, at the same time, the K-Means++ algorithm is used to cluster the length-width ratio of the target to be detected, and the anchor ratio of the anchor frame distribution most suitable for the data to be detected is obtained, so that the accuracy of defect prediction is enhanced.

[0050] Beneficial effects:

[0051] Compared with the prior art, the application has the following beneficial effects:

[0052] The improved ResNet50 feature extraction backbone network can enhance the feature extraction capability; the K-Means++ algorithm can fully utilize the defect length-width ratio distribution information of the to-be-detected data set, so that the position and shape of the generated candidate frame can be more fully close to the real labeled frame, thereby obtaining better detection precision. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 It is a flowchart of a kind of deep learning based radiographic image weld defect detection method, wherein the C2~C5 part of feature extraction network in FasterRCNN network is modified;

[0054] Figure 2 It is the structure diagram of ResNet50 improved BottleNeck base module, gi module is added on the basis of original information aggregation;

[0055] Figure 3 It is gi module structure diagram, which is composed of 1x1 convolution module, 3x3 convolution module, softmax and sigmoid module;

[0056] Figure 4For the example of the weld defect detection, the rectangular frame in the shown figure represents the detected defect position, and the number thereon represents the defect category, wherein "1" represents a circular defect, "2" represents a strip defect, "3" represents an incomplete fusion defect, "4" represents an undercut defect, "5" represents an overlap defect, and "6" represents a crack defect. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0058] Embodiment:

[0059] 1. An X-ray image weld defect detection method based on deep learning, the specific calculation method comprising the following steps:

[0060] Step one: making a data set X for training and testing, the specific steps being as follows:

[0061] (1) performing a clipping process on the weld image data, the original weld image size being 2560*224, and using a sliding window to clip each weld image, the sliding window step being S, S∈(0, 224), and at the same time, mapping the original weld defect annotation information to obtain the defect annotation information of the clipped image, finally obtaining a group of weld image data sets D with a size of 224*224;

[0062] (2) dividing the data in the data set D, and randomly dividing the data set D into a training set D1, a verification set D2 and a test set D3 according to a ratio of 8:1:1, D1, D2 and D3 together constituting the data set X;

[0063] Step two: constructing an improved FasterRCNN network model, the specific steps being as follows:

[0064] (1) taking X as the input of the network and inputting it into the FasterRCNN network model;

[0065] (2) inputting the data into an improved residual network ResNet50 model for training, the improved ResNet50 mainly adding a gi module at the end of the Bottleneck, the gi module including three branches, the first two branches using a 1x1 convolution module and a softmax function to encode spatial information and embed the information into the channel dimension to realize long-range interaction, and the other branch using 3x3 dilated convolution with dilated rates of 1, 2 and 3 to extract features of different receptive fields for fusion to capture local cross-channel information interaction and make up for the deficiency of the local receptive field in the 1x1 convolution, wherein the calculation formula of the gi module, denoted as G, is:

[0066]

[0067] where x represents the input of different Bottleneck modules, gP x represents the feature map after channel adjustment grouping of input x, and the calculation method is gp x = R(x), and 1, 64, 56, 56 are batch number, channel number, height and width, respectively, 2 is the grouping size, R(·) represents adjusting the input shape to [1x2, 64 / / 2, 56, 56], R1 represents adjusting the output shape to [1x2, 56x56, 1], R2 represents adjusting the output shape to [1x2, 64 / / 2, 56x56], R3 represents adjusting the output shape to [1x2, 1, 64 / / 2], R4 represents adjusting the output shape to [1x2, 65 / / 2, 56x56], R5 represents adjusting the output shape to [1x2, 1, 56, 56], R6 represents adjusting the output shape to [1, 64, 56, 56], represents a 1x1 convolution with an input channel number of 64 / / 2 and an output channel number of 1, represents a 1x1 convolution with an input and output channel number of 64 / / 2, represents a 1x1 convolution with an input channel number of 3x64 / / 2 and an output channel number of 64 / / 2, represents a 3x3 convolution with an input and output channel number of 64 / / 2, represents a 3x3 convolution with an input and output channel number of 64 / / 2, a dilation of 2 and a padding of 2, represents a 3x3 convolution with an input and output channel number of 64 / / 2, a dilation of 3 and a padding of 3, and softmax and sigmoid represent two kinds of activation functions, respectively;

[0068] (3) The multi-scale features C2, C3, C4, C5 obtained in (2) are input into the feature pyramid network FPN for multi-scale feature fusion, so as to fully utilize different scale feature information, and output a group of fused multi-scale features, denoted as P2, P3, P4, P5, P6, wherein P2, P3, P4, P5, P6 are generated by feature splicing of the feature maps generated by C2, C3, C4, C5 and the up-sampling operation, and P6 is generated by one down-sampling based on P5, and the formula is as follows:

[0069]

[0070] where i represents the feature map index, i is an integer, i∈[2, 6], C idenotes the multi-scale feature output of the i-th layer in the input FPN, C' i+1 denotes the output corresponding to the i+1-th layer in the FPN, C i-1 denotes the input corresponding to the i-1-th layer in the FPN, Upsample and Downsample denote 2x upsampling and 2x downsampling respectively, Conv 1×1 and Conv 3×3 denote two kinds of convolution with 1x1 and 3x3 convolution kernels respectively, and Add denotes an element-wise addition operation;

[0071] (4) The obtained multi-scale feature map is taken as input to construct a region recommendation network layer to generate region candidate proposals, and Softmax is used to judge whether the proposal frame input is a positive sample or a negative sample, and a bounding box regression is used to obtain an accurate detection frame. Specifically, the region recommendation network layer includes a front and a back background classification branch and a bounding box regression branch. In the classification branch, it is judged whether a defect exists at each position of the feature map. If no defect exists, it is marked as a negative example, and if a defect exists, it is marked as a positive example. In the regression branch, a new set of anchor box ratios is used to replace the original anchor box generation ratios. The new anchor box ratios are generated using the K-Means++ algorithm. The anchor box for the position where the defect exists is corrected according to the offset of the center point coordinates, width and height of the basic anchor box for the position where the defect exists. The results of the two branches are combined to obtain the region proposal. The specific steps of the K-Means++ algorithm for anchor box ratio clustering are as follows:

[0072] 1. Group all target labeled anchor boxes according to the IOU distance, that is, place similar anchor boxes in the same group;

[0073] 2. Randomly select an anchor box from each group as the centroid of the group;

[0074] 3. For each anchor box, calculate the distance to all centroids, select the group where the nearest centroid is located, and add the anchor box to the group;

[0075] 4. Repeat ② and ③ until all anchor boxes are grouped into a group;

[0076] 5. Calculate the ratio of all anchor boxes in each group as the anchor box ratio of the target label;

[0077] (5) The fused multi-scale feature map and the detection frame are input into the ROIPooling layer, and the corresponding feature map is extracted according to the positive example, negative example and position offset information obtained in step three to extract the feature, obtain the corresponding feature map and adjust the output to a uniform size. When extracting the feature, the formula for selecting the feature map is:

[0078]

[0079] wherein m is the number of the feature map, the value range is positive integer, m0 is the initial feature map number, the value range is positive integer, h and w are the offset height and width respectively, the value range of each is positive integer,

[0080] T takes 224, represents the floor operation;

[0081] (6) the feature map obtained after the ROIPooling layer is input into the classification and regression layer composed of two fully connected layers, one of which classifies the defect categories, and the other of which regresses the defect positions, so as to obtain the defect categories and positions, and finally generate a detection frame with defect categories and confidence;

[0082] Step three: use error back propagation to optimize and update the parameters in the improved FasterRCNN network model, and save the network parameters P with the highest mAP value in the FasterRCNN model detection result in all training rounds;

[0083] Step four: the prediction frame output by the last layer of the model needs to be post-processed in the test stage to remove the frames generated in the detection that do not meet the requirements, and the processing method is: remove the frames whose coordinates exceed the image boundary, subdivide the remaining frames according to the categories, sort the frames of the same category according to the scores obtained by classification, remove the frames whose scores are low and whose intersection over union with the frames with high scores exceeds 0.5, and the remaining frames are used as the final prediction result to predict the defect position and defect category, thereby completing the X-ray image weld defect detection.

[0084] The above description is not intended to limit the present application in any form, although the present application has been disclosed by the above examples, however, it is not intended to limit the present application, any person skilled in the art, without departing from the technical solution of the present application, can make changes or modifications to the equivalent examples of equivalent changes by using the above disclosed technical content, but any simple modification, equivalent change and modification of the above examples according to the technical essence of the present application, all still belong to the scope of the technical solution of the present application.

Claims

1. A method for detecting weld defects in X-ray images based on deep learning, characterized in that It includes the following steps: Step 1: Make a dataset X for training and testing. The specific steps are as follows: (1) Crop the weld image data. The size of the original weld image is w×h, where w < h, and the values of w and h are both positive integers. Use a sliding window to crop each weld image. The step size of the sliding window is S, S ∈ (0, h). At the same time, map the original weld defect annotation information to obtain the defect annotation information of the cropped image. Finally, obtain a set of weld image datasets D with a size of h*h; (2) Divide the data in dataset D. Randomly divide dataset D into a training set D1, a validation set D2, and a test set D3 according to the ratio of a:b:c, where a, b, and c are all positive integers, and a + b + c = 10. D1, D2, and D3 together form dataset X; Step 2: Build an improved Faster RCNN network model. The specific steps are as follows: (1) Take X as the input of the network and input it into the Faster RCNN network model; (2) The data is input into the improved residual network ResNet50 model for feature extraction. The improved ResNet50 is composed of different numbers of Bottleneck modules. Bottleneck is the basic module unit in the network and its structure is Inplanes is the number of input channels, planes is the number of output channels, the first part of the improved ResNet50 is 3 Bottlenecks, the second part is 4 Bottlenecks, the third part is 6 Bottlenecks, and the fourth part is 3 Bottlenecks. The multi-scale features output by the four parts are recorded as C2, C3, C4, and C5 respectively. The specific implementation of the improved ResNet50 network is as follows: The improved ResNet50 network consists of five convolutional layers: The first convolutional layer contains a [7x7, 64], Convolution with stride=2 and maximum pooling of [3x3] and stride=2; the second convolution layer has three Bottleneck layers inside. Network structure, where gi is the abbreviation of global info block, which is a global information gain module; the third convolutional layer has four Bottleneck Network structure; the fourth convolutional layer has six Bottleneck layers. Network structure; the fifth convolutional layer has three Bottleneck layers inside. Network structure; Among them, the calculation formula of the gi module, denoted as G, is: Among them, x represents the input of different Bottleneck modules, gp x Represents the feature map after channel adjustment and grouping of input x, calculated by gp x =R(x), where b, c, h, and w are the number of batches, the number of channels, the height, and the width, respectively. g is the group size, and g is a positive integer. R(·) represents adjusting the input shape to [b×g,c / / g,h,w], R1 represents adjusting the output shape to [b×g,h×w,1], R2 represents adjusting the output shape to [b×g,c / / g,h×w], R3 represents adjusting the output shape to [b×g,1,c / / g], R4 represents adjusting the output shape to [b×g,c / / g,h×w], R5 represents adjusting the output shape to [b×g,1,h,w], and R6 represents adjusting the output shape to [b,c,h,w]. It represents a 1×1 convolution with c / / g as the input channel and 1 as the output channel. represents a 1×1 convolution with both input and output channels c / / g. It represents a 1×1 convolution with 3×c / / g input channels and c / / g output channels. It means that both the input channel and the output channel are 3×3 convolution of c / / g. It represents a 3×3 convolution with both input and output channels c / / g, dilation and padding of 2. It represents a 3×3 convolution with both input and output channels c / / g, dilation and padding of 3, softmax and sigmoid represent two activation functions respectively, and the above symbol / / represents an integer division operation; (3) Take the multi-scale features C2, C3, C4, C5 obtained in (2) as the input and input them into the Feature Pyramid Network FPN for multi-scale feature fusion. Make full use of the feature information of different scales and output a set of fused multi-scale features, denoted as P2, P3, P4, P5, P6. Among them, P2, P3, P4, P5, P6 are respectively generated by the feature maps generated by C2, C3, C4, C5 and the upsampling operation for feature splicing. P6 is generated by performing a downsampling operation on the basis of P5. The formula is as follows: Among them, i represents the feature map index, i is an integer, i∈[2,6], C i Represents the multi-scale feature output of the i-th layer in the input FPN, C′ i+1 represents the output corresponding to the i+1 layer in FPN, C i-1 Indicates the input corresponding to the i-1 layer in FPN, Upsample and Downsample represent 2x upsampling and 2x downsampling respectively, Conv 1×1 and Conv 3×3 They represent two types of convolution with convolution kernels of 1×1 and 3×3 respectively, and Add represents the element-by-element addition operation; (4) Take the obtained multi-scale feature map as the input, build a region proposal network layer to generate region candidate proposals, judge whether the proposed box is a positive sample or a negative sample through Softmax, and use bounding box regression to obtain accurate detection boxes. Specifically, the region proposal network layer includes foreground and background classification branches and a bounding box regression branch. In the classification branch, judge whether there is a defect at each position of the feature map. If there is no defect, it is marked as a negative example. If there is a defect, it is marked as a positive example. In the regression branch, use a new set of anchor box ratios to replace the original ratio for generating anchor boxes. The new anchor box ratios are generated using the K-Means++ algorithm. Calculate the offsets of the center point coordinates, width, and height of the defect position relative to the base anchor box to correct the anchor boxes at the defect positions. Combine the results in the two branches to obtain region proposals. Among them, the specific steps of the K-Means++ algorithm for anchor box ratio clustering are: ① Group all the target-annotated anchor boxes according to the IOU distance, that is, put similar anchor boxes in the same group; The calculation formula of the IOU distance is: Among them, X and Y respectively represent two target boxes in the target annotation information, Area represents the area-taking operation, ∩ represents the intersection-taking operation, ∪ represents the union-taking operation, and IOU(X,Y) represents the calculation of the IOU distance between X and Y; ② Randomly select an anchor box from each group as the centroid of the group; ③For each anchor box, calculate its distance to all centroids, select the group with the closest centroid, and add the anchor box to the group; ④ Repeat ② and ③ until all anchor boxes are grouped into one group; ⑤Calculate the ratio of all anchor boxes in each group and use it as the anchor box ratio of the target annotation; (5) The fused multi-scale feature map and detection box are input into the ROI Pooling layer. According to the positive examples, negative examples and position offset information obtained in step 3, the corresponding feature map is extracted to obtain the corresponding feature map and the output is adjusted to a uniform size. The formula for selecting the feature map during feature extraction is: Among them, m is the number of the feature map, the value range is a positive integer, m0 is the initial feature map number, the value range is a positive integer, h and w are the height and width after offset, the value range is a positive integer, T is a constant, the value range is a positive number, Indicates a round-down operation; (6) The feature map obtained after the ROI Pooling layer is used as input and sent to the classification and regression layer composed of two fully connected layers. One fully connected layer classifies the defect category, and the other fully connected layer regresses the defect location, thereby obtaining the defect category and location, and finally generating a detection box with the defect type and confidence level. Step 3: Use error backpropagation to optimize and update the parameters in the improved Faster RCNN network model, and save the network parameter P with the highest mAP value in the Faster RCNN model detection results in all training rounds. The specific steps are as follows: (1) Error back propagation is adopted, cross entropy loss function is used to measure the classification model loss, and L1 loss is used to regress the border position. The joint loss function formula of the two loss functions is: L=-∑(y*log(p)+(1-y)*log(1-p))+λ∑(|p b -g t |) Among them, L represents the joint loss function value, y is the true category label, p is the probability of the predicted category, and p b Represents the predicted bounding box coordinates, g t represents the true bounding box coordinates, |·| represents the absolute value, λ is used to balance the weights of the two loss terms, λ∑(|p b -g t |) represents the L1 loss function; (2) Through each training of the training sample X, the stochastic gradient descent function SGD is used to make the loss value L back propagate in the improved network model, so that the model parameter P of the Faster RCNN i-th round training i Random changes are made in the direction of negative gradient to optimize the network. The calculation formula for updating the model parameters using SGD is as follows: Among them, P i Represents the model parameters of Faster RCNN in the i-th round of training, P i-1 represents the model parameters of the Faster RCNN i-1 round of training, η represents the optimization step size of SGD, η∈(0,1), Represents the training loss L for the model parameter P of Faster RCNN i-1 round training i-1 Perform derivation; (3) Calculate the mean average precision (mAP) of the model. The calculation formula is: Among them, mAP represents mean average precision, N represents the number of categories, and AP represents mean average precision. The calculation formula is: Among them, P(R) represents the curve of precision rate relative to recall rate. In practice, the AP value is the average of the maximum precision rate under a certain number of equally spaced recall rate intervals. The formula for precision rate is: Among them, P is the precision rate, TP is the number of samples predicted to be positive and actually positive, and FP is the number of samples predicted to be positive and actually negative. (4) Save the weight model with the highest mAP value in the model as the final trained weight model P; Step 4: Use P saved in step 3 as the network parameter of the Faster RCNN model, and input the model parameter P after training as the training for testing. In the test phase, the predicted bounding box output by the last layer of the model needs to be post-processed to remove the unqualified bounding boxes generated during the detection. The processing method is as follows: remove the bounding boxes whose coordinates exceed the image boundary, subdivide the remaining bounding boxes by category, sort the bounding boxes of the same category according to the scores obtained by classification, remove the bounding boxes with low scores and whose intersection-over-union ratio with the bounding boxes with high scores exceeds k, and the value range of k is (0,1). The remaining bounding boxes are used as the final prediction results to predict both the defect location and defect category, and finally obtain the defect detection results with defect location and defect type.

Images