A method for weld defect recognition in radiographic images based on multi-branch convolutional neural network

Abstract

The present invention provides a method for identifying weld defects in radiographic images based on a multi-branch convolutional neural network, which consists of a basic convolutional network layer and multiple classification branches. Compared with traditional convolutional neural networks, the present invention uses multiple classification branches for defect identification, and the classification branches are adaptively generated by a classification branch generator. First, the basic convolutional network layer preliminarily classifies the defects and obtains the error rate for each defect type. Then, the defect types are merged in order from high to low error rates. Then, classification branches are generated according to the number of merged defect types. Finally, the classification branches are used to further classify the merged defects. The model uses the basic convolutional network layer and the classification branches to classify the defect types in the image twice, which can effectively improve the accuracy of the model.

Description

Technical Field

[0001] The present invention relates to the fields of pipeline weld defect recognition technology and deep learning technology, and in particular to a method for weld defect recognition in radiographic images based on a multi-branch convolutional neural network. Background Art

[0002] Defect identification in welds is very important for evaluating welding quality. At present, the common way to identify defects is to use radiographic technology for non-destructive testing and manually evaluate whether there are defects in the radiographic film. However, with the development of industry, the efficiency of manual evaluation cannot meet industrial needs. Therefore, it becomes meaningful to use computer vision to perform intelligent evaluation of weld radiographic images to assist manual evaluation tasks. Intelligent evaluation models include traditional models and deep neural network models. Traditional models select feature description operators based on defect characteristics to describe features, and then use traditional machine learning algorithms to classify defects. For example, Zhang Zhifen et al. (Zhang Zhifen, Ren Wenjing, Yang Zhe, et al. Real-time seam defect identification for Al alloys in robotic arcwelding using optical spectroscopy and integrating learning [J]. Measurement, 2020, 156: 107546.) used a random forest classification algorithm for defect identification. However, this type of method relies on feature description methods, and the description of defect features is more based on general feature parameters, such as geometric features such as area, defect symmetry, aspect ratio, and texture statistical parameters such as mean, standard deviation, and entropy. It is difficult to design special features with defect characteristics, which restricts the accuracy of traditional models in defect identification. Therefore, deep neural networks can use convolutional network layers to automatically extract features, bringing a solution. Kumaresan et al. (Kumaresan S, Aultrin KSJ, Kumar SS, et al. Transfer learning with CNN for classification of weld defect [J]. IEEE Access, 2021, 9: 1.) extracted feature vectors through a pre-trained ResNet50 network and used support vector machines for defect identification.Hou Wenhui et al. (Hou Wenhui, Rao Lulu, Zhu Andong, et al. Feature fusion for weld defect classification with small dataset [J]. Journal of Sensors, 2022, 2022: Article ID 8088202. (2022-02-23) [2022-10-01]. https: / / doi.org / 10.1155 / 2022 / 8088202.) used an autoencoder to obtain feature vectors, which were fused with feature vectors based on histograms and co-occurrence matrices to serve as the basis for defect classification using a support vector machine. Li Zhen et al. (Li Zhen, Ma Li, Gao Jingli. X-ray image classification method for weld defects based on deep learning [J]. Information Technology, 2019, (8): 49-52.) fine-tuned the structure of the AlexNet network to automatically identify defects in weld images. However, deep neural networks also face some problems when used for defect recognition, which leads to reduced model accuracy, such as small defects, small data sample size, and image noise interference. Chang Yasheng et al. (Chang Yasheng, Wang Weiku. A deep learning-based weld defect classification method using radiographic images with a cylindrical projection [J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-11.) Based on the SegNet network, they replaced some ordinary convolutions with dilated convolutions and reduced the number of pooling layers in the network to reduce the low accuracy of small-size defects due to detail loss. Shang Jiaze et al. (Shang Jiaze, An Weipeng, Liu Yu, et al. Oil pipeline weld defect identification system based on convolutional neural network. [J]. KSII Transactions on Internet and Information Systems, 2020, 14 (3): 1086-1103.) adjusted the number of network layers based on small sample data to obtain a unique convolutional neural network, and trained the network from scratch, so that the network has good performance on small sample data.HuAnde et al. (Hu Ande, Wu Lijian, Huang Jiankang. Recognition of weld defects from x-ray images based on improved convolutional neural network [J]. Multimedia Tools and Applications, 2022, 81 (11): 15085-15102.) use an adaptive correction factor to optimize the mean pooling operation in the convolutional network to reduce the impact of image noise on model recognition accuracy. However, different types of defects with similar characteristics are not easy to identify, which affects the accuracy of model recognition. In order to improve the recognition accuracy of defects in weld images and distinguish defects with similar characteristics, the present invention uses a multi-branch convolutional neural network to intelligently identify defects. Summary of the Invention

[0003] The present invention mainly overcomes the deficiencies in the prior art and provides a method for identifying weld defects in radiographic images based on a multi-branch convolutional neural network.

[0004] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0005] 1. A method for identifying weld defects in radiographic images based on a multi-branch convolutional neural network, comprising the following steps:

[0006] Step 1: Perform image preprocessing on the original image. The image preprocessing operations include grayscale stretching and median filtering to obtain the image after preprocessing. The grayscale stretching formula is:

[0007]

[0008] Among them, g is the pixel value after grayscale stretching, and its value range is [0, g max ], g max is the maximum pixel value of the stretched image, the value range is a positive integer, and g max Greater than f max , f is the pixel value of the original image, f max is the maximum pixel value of the original image, f min is the minimum pixel value of the original image, To round down, the median filter uses the median of all pixel values ​​within the m×n window size of the original pixel value to replace the original pixel value, and the value ranges of m and n are both positive integers;

[0009] Step 2: Use the preprocessed image in step 1 as the input image, and then scale the input image into a tensor of a×b×c, where the value ranges of a, b, and c are all positive integers, and then normalize it. The normalization formula is:

[0010]

[0011] Where u is the value after normalization, and its value range is the interval [-1, 1]; v is the value before normalization, and its value range is any real number; m is the mean of the pixel values ​​of all images to be normalized, and its value range is any real number; δ is the standard deviation of the pixel values ​​of all images to be normalized, and its value range is any real number;

[0012] Step 3: Use the normalized tensor as the input of the model and build a multi-branch convolutional neural network for defect recognition. This network structure adds N classification branches to the traditional convolutional neural network, and the N classification branches are adaptively generated by the branch generator. The value range of N is a positive integer. Building an adaptive multi-branch convolutional neural network includes the following steps:

[0013] (1) Build a ResNet50 network as the basic convolutional network layer and train it on the training set. The trained basic convolutional network layer classifies defects on the validation set and obtains the corresponding error rate. The image of the training set is I train The radiographic image of the pipeline weld is I verify Radiographic image of a pipe weld. train and I verify The value range of is a positive integer. The label of the training set and the label of the validation set are the types of defects in the image c, which are integers in the interval [0, C-1]. C is the number of defect types, which is a positive integer. The trained model selects the model with the smallest cross entropy loss during the training process. The formula for cross entropy loss is:

[0014]

[0015] Among them, H is the cross entropy loss, q i is the maximum value of the model output vector of the i-th image, and its value range is [0,1];

[0016] (2) Based on the error rate on the validation set in (1), the classification branch is generated using the branch generator. The error rate formula is:

[0017]

[0018] Among them, P i,j is the error rate of category i being misclassified as category j, and its value range is [0,1]. iis the number of images of category i in the validation set, and its value range is a positive integer. i,j is the number of images that the model classifies category i as category j, and its value range is a positive integer. The steps of the branch generator to generate branches are steps (3) to (5);

[0019] (3) Keep the value of i unchanged and compare the corresponding P i,j The value of the maximum P i,j As the maximum error rate of the model for category i in the validation set, denoted as p i,j ;

[0020] (4) Change the value of i to obtain the maximum error rate p of the model in all categories in the validation set i,j Then set the threshold t, the value range is [0,1], and use the threshold to classify all types of p i,j Screen and retain p values ​​greater than or equal to the threshold i,j , remove p values ​​less than the threshold i,j ;

[0021] (5) After filtering (4), p i,j Sort in descending order and put the largest p i,j The corresponding two categories are merged. If category i is incorrectly classified as category j, then category i is merged into category j, and neither category i nor category j can be merged into other categories. Then continue to use the remaining p i,j Repeat the above category merging process until all p i,j is used, and finally a classification branch is generated for all merged categories;

[0022] (6) All classification branches are trained. Each branch is trained separately. The training set is part of the images in the training set in (1). This part of the images contains defects in the merged category. The label of the training set is the category c of the defect in the image, and the value range is [0, C i -1], C i is the number of defect types, indicating the number of defect types contained in the i-th merged type. Its value range is a positive integer. During training, the first L convolutional layers in the basic convolutional network layer are shared. The value range of L is a positive integer. The parameters of this part of the convolutional layer are not updated during training. The trained model is the one with the smallest cross entropy loss during the training process. The formula of cross entropy loss is the same as in (1).

[0023] (7) After the basic convolutional network layer and the N classification branches are trained, the basic convolutional network layer and the N classification branches are combined. The basic convolutional network layer is first used to classify defects, and then the N classification branches are used to classify the merged types. The N classification branches share the first L convolutional layers with the basic convolutional network layer, and finally the types of defects contained in the image are obtained.

[0024] The present invention provides a method for identifying weld defects in radiographic images based on a multi-branch convolutional neural network, which consists of a basic convolutional network layer and multiple classification branches. Compared with traditional convolutional neural networks, the present invention uses multiple classification branches for defect identification, and the classification branches are adaptively generated by a classification branch generator. First, the basic convolutional network layer preliminarily classifies the defects and obtains the error rate for each defect type. Then, the defect types are merged in order from high to low error rates. Then, classification branches are generated according to the number of merged defect types. Finally, the classification branches are used to further classify the merged defects. The model uses the basic convolutional network layer and the classification branches to classify the defect types in the image twice, which can effectively improve the accuracy of the model.

[0025] Beneficial effects:

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The present invention provides a method for identifying weld defects in radiographic images based on a multi-branch convolutional neural network. The method can use classification branches to perform detailed classification of defect types that are difficult to distinguish, thereby improving the accuracy of the defect classification model. In this process, the error rate of the basic convolutional network layer is used to evaluate whether the defect types are easy to distinguish. The greater the error rate of one type of defect to another type of defect, the more difficult it is to distinguish the two types of defects. The defect types that are difficult to distinguish are merged, and the classification branches are used to perform detailed classification of the merged defect types, thereby improving the accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This figure shows a flowchart of a method for weld defect recognition in radiographic images based on a multi-branch convolutional neural network, including image preprocessing, scaling and normalization, and an adaptive multi-branch convolutional neural network. The adaptive multi-branch convolutional neural network includes a basic convolutional network layer and a multi-classification branch structure. The dotted lines in the figure indicate that they are not actual connections, but rather branches in the multi-classification branch structure generated by the classification branch generator.

[0029] Figure 2 It is a single classification branch network structure diagram, which shows the network structure of each classification branch. Multiple classification branches constitute Figure 1 Multi-classification branch structure in ;

[0030] Figure 3This is a diagram combining the basic convolutional network layer and multiple classification branches. It shows how, after the basic convolutional network layer classifies defects, the classification branches further classify the corresponding merged categories. The classification result is composed of these two parts. In the figure, "..." indicates omitted categories, merged categories, or classification branches. The connection method is the same as other parts.

[0031] Figure 4 This is an example diagram of defect types in weld defect identification, including circular defects, strip defects, lack of fusion defects, concave defects, undercut defects and crack defects. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0033] Example:

[0034] A method for identifying weld defects in radiographic images based on a multi-branch convolutional neural network is described in detail below:

[0035] Step 1: Perform image preprocessing on the original image. The image preprocessing operations include grayscale stretching and median filtering to obtain the image after preprocessing. The grayscale stretching formula is:

[0036]

[0037] Among them, g is the pixel value after grayscale stretching, g max is the maximum pixel value of the stretched image, which is 255, f is the pixel value of the original image, and f max is the maximum pixel value of the original image, f min is the minimum pixel value of the original image, To round down, a 3x3 window is selected for median filtering;

[0038] Step 2: Use the preprocessed image in step 1 as the input image, scale the input image into a 224x224x3 tensor, and then standardize it. The standardization formula is:

[0039]

[0040] Where u is the value after normalization, v is the value before normalization, m is the mean of the pixel values ​​of all images to be normalized. According to the size of the scaling, the input to be normalized has three channels, and the mean values ​​are 0.485, 0.456, and 0.406 respectively. δ is the standard deviation of the pixel values ​​of all images to be normalized. According to the size of the scaling, the input to be normalized has three channels, and the standard deviation values ​​are 0.229, 0.224, and 0.225 respectively.

[0041] Step 3: Use the normalized tensor as the input of the model and build a multi-branch convolutional neural network for defect recognition. This network structure adds N classification branches to the traditional convolutional neural network, and the N classification branches are adaptively generated by the branch generator. Building an adaptive multi-branch convolutional neural network includes the following steps:

[0042] (1) Build a ResNet50 network as the basic convolutional network layer and train it on the training set. The trained basic convolutional network layer classifies defects on the validation set and obtains the corresponding error rate. The image of the training set is I train The radiographic image of the pipeline weld is I verify Radiographic image of a pipe weld. train and I verify The values ​​of are 6000 and 300. The labels of the training set and the validation set are the types of defects c in the image. The number of defect types is C, and the value of C is 7, including non-defects, circular defects, strip defects, lack of fusion, undercut, concave, and cracks, which correspond to numbers from 0 to 6 respectively. Therefore, the value of c is an integer in [0, 6]. The trained model selects the model with the smallest cross entropy loss during the training process. The formula for cross entropy loss is:

[0043]

[0044] Among them, H is the cross entropy loss, q i is the maximum value in the model output vector of the i-th image;

[0045] (2) Based on the error rate on the validation set in (1), the classification branch is generated using the branch generator. The error rate formula is:

[0046]

[0047] Among them, P i,j is the error rate of category i being misclassified as category j, I i is the number of images of category i in the validation set, I i,j The number of images that the model classifies category i as category j, and the steps of the branch generator to generate branches are steps (3) to (5);

[0048] (3) Keep the value of i unchanged and compare the corresponding P i,j The value of the maximum P i,j As the maximum error rate of the model for category i in the validation set, denoted as p i,j ;

[0049] (4) Change the value of i to obtain the maximum error rate p of the model in all categories in the validation set i,j Then set the threshold t to 0.05, and use the threshold to calculate the p i,j Screen and retain p values ​​greater than or equal to the threshold i,j , remove p values ​​less than the threshold i,j ;

[0050] (5) After filtering (4), p i,j Sort in descending order and put the largest p i,j The corresponding two categories are merged. If category i is incorrectly classified as category j, then category i is merged into category j, and neither category i nor category j can be merged into other categories. Then continue to use the remaining p i,j Repeat the above category merging process until all p i,j is used, and finally a classification branch is generated for all merged categories;

[0051] (6) All classification branches are trained. Each branch is trained separately. The training set is part of the images in the training set in (1). This part of the images contains defects in the merged category. The label of the training set is the category c of the defect in the image, and the value is in the interval [0, C i -1], C i is the number of defect types, indicating the number of defect types contained in the i-th merged type. During training, the first L convolutional layers of the basic convolutional network layer are shared. The value of L is 40, that is, the parameters of the first 40 convolutional layers are not updated during training. The trained model is the one with the smallest cross entropy loss during the training process. The formula for cross entropy loss is the same as in (1);

[0052] (7) After the basic convolutional network layer and the N classification branches are trained, the basic convolutional network layer and the N classification branches are combined. The basic convolutional network layer is first used to classify defects, and then the N classification branches are used to classify the combined types. The N classification branches share the first 40 convolutional layers with the basic convolutional network layer, and finally the types of defects contained in the image are obtained;

[0053] Step 4: Evaluate the performance of different models on the same test set. The test set image is a radiographic image of a pipe weld. testThe value of is 700, the label of the test set is the type of defect in the image, the number of defect types is C, the value of C is 7, including non-defect, circular defect, strip defect, lack of fusion, undercut, concave, crack, each image corresponds to a type label, the label is manually marked, the evaluation index is the accuracy, the accuracy is the proportion of images in the test set that the model can correctly classify the defect type, among which, if the defect type classified by the model is the same as the defect type of the label, the defect type classified by the model is correct. The greater the accuracy of the model, the better the defect recognition effect of the model. The formula for the accuracy is:

[0054]

[0055] Among them, accuracy is the accuracy rate, I test is the number of images in the test set, I right In order to classify the number of correct images, the present invention is compared with ResNet50, a model based on AlexNet, RayNet, and a model based on DenseNet121 (wherein ResNet50 is a model from the literature (He Kaiming, Zhang Xiangyu, Ren Shaoqing, et al. Deep Residual Learning for Image Recognition [J]. 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.), the model based on AlexNet is a model from the literature (Li Zhen, Ma Li, Gao Jingli. Weld defect X-ray image classification method based on deep learning [J]. Information Technology, 2019, (8): 49-52.), and RayNet is a model from the literature (Shang Jiaze, An Weipeng, Liu Yu, et al. Oil pipeline weld defect identification system based on convolutional neural network. [J]. KSII Transactions on Internet and Information Systems, 2020, 14(3): 1086-1103.) The model based on DenseNet121 is the model of the literature (Gu Jing, Wang Qiwen, Zhang Min, et al. Weld defect detection and recognition based on DenseNet network [J]. Sensors and Microsystems, 2020, 39(9): 129-131.)). The comparison results are:

[0056]

[0057] The above description does not limit the present invention in any form. Although the present invention has been disclosed through the above embodiments, it is not intended to limit the present invention. Any technician familiar with the profession can use the technical content disclosed above to make changes or modifications to equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A method for identifying weld defects in radiographic images based on a multi-branch convolutional neural network, characterized in that The following steps are involved: Step 1: Perform image preprocessing on the original image. The image preprocessing operations include grayscale stretching and median filtering to obtain the image after preprocessing. The grayscale stretching formula is: Among them, g is the pixel value after grayscale stretching, and its value range is [0, g max ], g max is the maximum pixel value of the stretched image, the value range is a positive integer, and g max Greater than f max , f is the pixel value of the original image, f max is the maximum pixel value of the original image, f min is the minimum pixel value of the original image, To round down, the median filter uses the median of all pixel values ​​within the m×n window size of the original pixel value to replace the original pixel value, and the value ranges of m and n are both positive integers; Step 2: Use the preprocessed image in step 1 as the input image, and then scale the input image into a tensor of a×b×c, where the value ranges of a, b, and c are all positive integers, and then normalize it. The normalization formula is: Where u is the value after normalization, and its value range is the interval [-1, 1]; v is the value before normalization, and its value range is any real number; m is the mean of the pixel values ​​of all images to be normalized, and its value range is any real number; δ is the standard deviation of the pixel values ​​of all images to be normalized, and its value range is any real number; Step 3: Use the normalized tensor as the input of the model and build a multi-branch convolutional neural network for defect recognition. This network structure adds N classification branches to the traditional convolutional neural network, and the N classification branches are adaptively generated by the branch generator. The value range of N is a positive integer. Building an adaptive multi-branch convolutional neural network includes the following steps: (1) Build a ResNet50 network as the basic convolutional network layer and train it on the training set. The trained basic convolutional network layer classifies defects on the validation set and obtains the corresponding error rate. The image of the training set is I train The radiographic image of the pipeline weld is I verify Radiographic image of a pipe weld. train and I verify The value range of is a positive integer. The label of the training set and the label of the validation set are the types of defects in the image c, which are integers in the interval [0, C-1]. C is the number of defect types, which is a positive integer. The trained model selects the model with the smallest cross entropy loss during the training process. The formula for cross entropy loss is: Among them, H is the cross entropy loss, q i is the maximum value of the model output vector of the i-th image, and its value range is [0,1]; (2) Based on the error rate on the validation set in (1), the classification branch is generated using the branch generator. The error rate formula is: Among them, P i,j is the error rate of category i being misclassified as category j, and its value range is [0,1]. i is the number of images of category i in the validation set, and its value range is a positive integer. i,j is the number of images that the model classifies category i as category j, and its value range is a positive integer. The steps of the branch generator to generate branches are steps (3) to (5); (3) Keep the value of i unchanged and compare the corresponding P i,j The value of the maximum P i,j As the maximum error rate of the model for category i in the validation set, denoted as p i,j ; (4) Change the value of i to obtain the maximum error rate p of the model in all categories in the validation set i,j Then set the threshold t, the value range is [0,1], and use the threshold to classify all types of p i,j Screen and retain p values ​​greater than or equal to the threshold i,j , remove p values ​​less than the threshold i,j ; (5) After filtering (4), p i,j Sort in descending order and put the largest p i,j The corresponding two categories are merged. If category i is incorrectly classified as category j, then category i is merged into category j, and neither category i nor category j can be merged into other categories. Then continue to use the remaining p i,j Repeat the above category merging process until all p i,j is used, and finally a classification branch is generated for all merged categories; (6) All classification branches are trained. Each branch is trained separately. The training set is part of the images in the training set in (1). This part of the images contains defects in the merged category. The label of the training set is the category c of the defect in the image, and the value range is [0, C i -1], C i is the number of defect types, indicating the number of defect types contained in the i-th merged type. Its value range is a positive integer. During training, the first L convolutional layers in the basic convolutional network layer are shared. The value range of L is a positive integer. The parameters of this part of the convolutional layer are not updated during training. The trained model is the one with the smallest cross entropy loss during the training process. The formula of cross entropy loss is the same as in (1). (7) After the basic convolutional network layer and the N classification branches are trained, the basic convolutional network layer and the N classification branches are combined. The basic convolutional network layer is first used to classify defects, and then the N classification branches are used to classify the merged types. The N classification branches share the first L convolutional layers with the basic convolutional network layer, and finally the types of defects contained in the image are obtained.

Images