A deep learning-based radiographic image weld defect detection method

Abstract

The application provides a kind of based on deep learning's radiographic image weld defect detection method, utilizes Faster RCNN network to carry out target detection to radiographic image, on the basis of traditional Faster RCNN, the application increases background subtraction network layer, obtains the Faster RCNN network structure of five modules, the background subtraction network layer has with the parameter of updating, can constantly optimize and reduce the effect of background along with network training, highlights the feature of defect, simultaneously, utilizes three branch area recommendation network layer instead of traditional area recommendation network, it has two regression branches, so that the task of predicting defect position information is subdivided, one branch is responsible for predicting the horizontal coordinate and width of the center point of defect, one branch is responsible for predicting the vertical coordinate and height of the center point of defect, enhances the accuracy of defect position prediction.

Description

Technical Field

[0001] This invention relates to the fields of pipeline weld defect detection technology and deep learning technology, and in particular to a deep learning-based method for detecting weld defects using X-ray images. Background Technology

[0002] Welding technology has significant applications in pipe manufacturing. However, even with current welding techniques, welds still exhibit various defects such as circular defects, strip defects, cracks, lack of fusion, undercut, and concavity. Therefore, the detection of welding defects is crucial for ensuring the reliability and safety of welded products. Currently, manual inspection remains a common method for defect detection; however, this method is inefficient and its accuracy is limited by the expertise of the inspectors. Therefore, researchers have begun to explore the use of machine learning algorithms to automate weld defect detection on X-ray non-destructive testing images of welds.

[0003] Early defect detection algorithms utilized machine learning to identify candidate regions generated during image processing, determining defect locations and categories. For example, Duan Feng et al. (DUAN Feng, YIN Shifan, SONG Peipei, et al. Automatic Welding Defect Detection of X-Ray Images by Using Cascade AdaBoost with Penalty Term[J].IEEE Access,2019,7:125929-125938.) obtained potential defect regions in the image during image preprocessing and used cascaded AdaBoost to determine the true defects within these regions. However, this method's location prediction was not accurate enough. Subsequently, with the application of deep learning algorithms, Faster R-CNN and YOLOv3 were widely used for target detection tasks of weld defects. For example, Guo Wenming et al. (Guo Wenming, Liu Kai, Qu Huifan. Welding defect detection based on X-ray images using Faster R-CNN model [J]. Journal of Beijing University of Posts and Telecommunications, 2019, 42(06):20-28.) designed a welding defect detection system based on Faster R-CNN and made a simple attempt at defect detection; Wei Ruoyu et al. (Wei Ruoyu, Li Shuting, Wu Songrong, et al. Defect detection of track fasteners based on improved YOLO V3 algorithm [J]. Railway Standard Design, 2020, 64(12):30-36.) used two prediction branches in YOLOv3 to detect defects. However, the accuracy of the model still needs to be improved, and the research direction mainly focuses on the problems of small defect target detection, multi-scale defect target detection and detection efficiency. For the problem of small defect target detection, it is common to use the Feature Pyramid Network (FPN) to make predictions in multi-layer feature maps. Li Dongjie et al. (Li Dongjie, Li Ruohao. A method for detecting defects in mugs based on improved Faster RCNN[J]. Progress in Laser & Optoelectronics, 2020, 57(4):353-360.) improved the detection of small targets in mug defects by combining FPN with the Faster RCNN detection network. Chen Yongbin et al. (CHENYongbin, WANG Jingran, WANG Guitang. Intelligent Welding Defect Detection Model on Improved R-CNN[J]. IETE Journal of Research, 2022:1-10.) improved the feature extraction layer of Faster RCNN by using Res2Net residual network and combined it with FPN module with weighted features to improve the model's ability to detect small targets.Liu Weipeng et al. (LIU Weipeng, SHAN Shengqi, CHEN Haiyong, et al. X-ray WeldDefect Detection Based on AF-RCNN[J]. Welding in the World, 2022, 66(6): 1165-1177.) proposed the AF-RCNN network for detecting weld defects. This algorithm uses ResNet and FPN as the backbone network of Faster RCNN, and combines efficient convolutional attention modules and CIOU loss function to improve the detection capability of Faster RCNN for small defects. For the problem of multi-scale defects, Chen Haiyong et al. (Chen Haiyong, Zhao Peng, Yan Haowei. Crack detection of multi-scale Faster RCNN with attention fusion[J]. Optoelectronic Engineering, 2021, 48(1): 64-74.) added a bottom-up FPN to the backbone network of Faster RCNN, and combined it with Focal Loss loss function to improve the detection accuracy of defects. Zhu Ankang et al. (Zhu Ankang, Wang Raofen, Zhang Zhenyu. Design of a surface defect detection system for aluminum based on deep learning [J]. Sensors & Microsystems, 2022, 41(08): 96-99+103.) addressed the problem of low detection efficiency caused by uneven defect size by adding deformable convolution to the Faster RCNN network model to improve detection accuracy. In terms of detection efficiency, Tang Maojun et al. (Tang Maojun, Huang Haisong, Zhang Songsong, et al. Application of improved Faster-RCNN in weld defect detection [J]. Modular Machine Tools & Automated Manufacturing Technology, 2021, (12): 83-86.) added a decoupled classification refinement structure to improve detection speed on the basis of adding an FPN structure. Wu Weihao et al. (Wu Weihao, Li Qing. Defect detection of electrical connectors based on improved YOLOv3 [J]. Journal of Sensor Technology, 2020, 33(2):299-307.) reduced the number of DBL units in YOLOv3 and used single-scale feature maps for prediction in order to improve the defect detection speed. The above studies have done a lot of work on defect detection, but the existence of small defects and multi-size defects, as well as the requirements of the detection task on model efficiency, mean that the detection model still needs to continue to improve its accuracy and optimize its detection speed. Summary of the Invention

[0004] The present invention mainly overcomes the shortcomings of the prior art and provides a method for detecting weld defects based on X-ray images using deep learning.

[0005] To achieve the above technical objectives, the present invention adopts the following technical solution:

[0006] 1. A method for detecting weld defects using X-ray images based on deep learning, characterized by the following steps:

[0007] Step 1: Perform image preprocessing on the original image, including grayscale stretching, median filtering, and mean filtering, to obtain the preprocessed image. The grayscale stretching formula is:

[0008]

[0009] Where g is the pixel value after grayscale stretching, and its value ranges from [0, g] to [0, g]. max The integer within ], g max The maximum pixel value of the stretched image, taking values ​​in the range of positive integers, and g max Greater than f max f is the pixel value of the original image. max f is the maximum pixel value of the original image. min The minimum pixel value of the original image. To round down, median filtering uses the median of all pixel values ​​within an m×n window of the original pixel value to replace the original pixel value, while mean filtering uses the mean of all pixel values ​​within an m×n window of the original pixel value to replace the original pixel value. The values ​​of m and n are both positive integers.

[0010] Step 2: Use the preprocessed image as the input image, then scale the input image into a tensor of size a×b×c, where a, b, and c are all positive integers. Then, perform standardization using the formula:

[0011]

[0012] Where u is the standardized value, ranging from [-1, 1], v is the unstandardized value, ranging from any real number, m is the mean of the pixel values ​​of all images to be standardized, ranging from any real number, and δ is the standard deviation of the pixel values ​​of all images to be standardized, ranging from any real number.

[0013] Step 3: Using the standardized tensor as input, build a Faster R-CNN object detection network to detect defects in the image. Add a background subtraction network layer, changing the traditional four-part Faster R-CNN network to a five-part structure. Modify the regression branch in the region recommendation network layer to independently predict the location information of defects in the image. The defect location information includes the x-coordinate of the defect's center point, the y-coordinate of the defect's center point, the height of the defect, and the width of the defect. This Faster R-CNN network consists of five parts: a background subtraction network layer, a feature extraction network layer, a region recommendation network layer, an interest region pooling layer, and a classification / location regression layer. The construction steps are as follows:

[0014] (1) Construct a background subtraction network layer to reduce the influence of the background in the image and highlight the defect features. It consists of large-size mean pooling, convolution and skip connections of m×n. The values ​​of m and n are positive integers. The input of this layer is simulated after large-size mean pooling and convolution. Then, the input of this layer is subtracted from the simulated background through skip connections to obtain the output of this layer.

[0015] (2) Construct a feature extraction network layer to extract defect features. It consists of ResNet50, FPN, and SE attention mechanism. Among them, the feature pyramid network is a typical method for extracting feature maps of different sizes for subsequent tasks. In the defect detection task, it outputs N feature maps, where N is a positive integer, for the subsequent prediction process. Each feature map layer is responsible for anchor boxes of A sizes, where A is a positive integer. The size of the anchor box is represented by h. a ×w a The form is represented as h, where h a w represents the height of the anchor frame, and its value is a positive integer. a The width of the anchor box is represented by a positive integer. This layer uses the background minus the output of the network layer as input, and the output is N feature maps.

[0016] (3) Using the feature map as input, a region recommendation network layer is built to generate region suggestions. Based on the classification and regression dual branches of the traditional region recommendation network layer, the region recommendation network further subdivides the responsibility of the regression branch and independently predicts the location information of defects in the image. It includes three branches: front / background classification branch, width / center point horizontal coordinate regression branch, and height / center point vertical coordinate regression branch. These are referred to as the three-branch region recommendation network layer. In the classification branch, it judges whether each position in the feature map contains a defect. The position containing a defect is marked as a positive example, and the position not containing a defect is marked as a negative example. Furthermore, it makes a rough prediction of the position of the defect based on the anchor box. Here, two regression branches are used to independently predict the position of the defect. One branch is responsible for predicting the horizontal coordinate and width of the center point of the defect, and the other branch is responsible for predicting the vertical coordinate and height of the center point of the defect. The obtained position information is based on the anchor box position and is expressed as the offset of the anchor box position relative to the defect position in the horizontal coordinate, width, vertical coordinate, and height of the center point.

[0017] (4) Construct a region-of-interest pooling layer, adjust the number of convolutional layers K, where K is a positive integer. Based on the positive examples, negative examples, and offsets obtained in (3), extract features from the corresponding feature maps and output them in a uniform size. The formula for selecting feature maps when extracting features is:

[0018]

[0019] Where p is the selected feature map number, with a value in the range of positive integers; p0 is the initial feature map number, with a value in the range of positive integers; h is the height of the offset anchor box, with a value in the range of positive numbers; w is the width of the offset anchor box, with a value in the range of positive numbers; and T is a constant, with a value in the range of positive numbers. To round down;

[0020] (5) Construct a classification / location regression layer, which consists of two fully connected layers. One fully connected layer classifies the defect types, and the other fully connected layer regresses the defect locations to obtain the defect types and defect locations, forming a bounding box with defect types, called the predicted bounding box.

[0021] Step 4: Train the model on the training set, where the images in the training set are I. train X-ray image of the pipe weld, I train The values ​​of are positive integers. The labels of the training set are the type c of defects in the image and the location of the defects. The number of defect types is C, where C is a positive integer and c is a positive integer in the interval [0, C-1]. The location of the defect includes the x-coordinate of the center point of the defect. t The ordinate of the center point of the defect is y. t The height of the defect w t and the width h of the defect t Each image contains several defects, and each defect corresponds to a location. The labels are manually assigned. The training process is as follows:

[0022] (1) Calculate the offset of the defect location in the label relative to the anchor frame, including the offset of the horizontal axis, the offset of the vertical axis, the offset of the height, and the offset of the width. The formula for calculating the offset of the horizontal axis is:

[0023]

[0024] Among them, t x The offset of the horizontal axis, x t x is the x-coordinate of the center point of the defect in the label. a w is the x-coordinate of the center point of the anchor frame a x is the width of the anchor frame. t x a w a The value range is positive integers, and the formula for calculating the offset of the vertical axis is:

[0025]

[0026] Among them, t y The offset of the vertical axis, y t The y-coordinate of the center point of the defect in the label. a h is the x-coordinate of the center point of the anchor frame.a y is the height of the anchor frame. t y a h a The value range is positive integers, and the formula for calculating the width offset is:

[0027]

[0028] Among them, t w For the width offset, w t The width of the defect in the label is a positive integer, and the height offset is calculated using the following formula:

[0029]

[0030] Among them, t h For high offset, h t The value represents the height of the defect in the label, and its range is positive integer;

[0031] (2) The training model consists of three parts: the background subtraction network layer, the feature extraction network layer, and the three-branch region recommendation network layer. The parameters selected after training are those of the model with the smallest loss value during training. N is selected for each image. a The calculation is performed on each anchor frame, and the formula for calculating the loss value is as follows:

[0032]

[0033] Where L is the loss value, N a q represents the number of anchor boxes used for training per image, with values ​​ranging from positive integers. i t represents the maximum value of the model output vector for the i-th anchor box, with a value range of [0,1]. x,i Let t be the offset of the horizontal coordinate in the label corresponding to the i-th anchor box. y,i t represents the offset of the ordinate in the label corresponding to the i-th anchor box. w,i t is the offset of the width of the label corresponding to the i-th anchor box. h,i r is the height offset of the label corresponding to the i-th anchor box. x,i r is the offset of the x-coordinate of the model output corresponding to the i-th anchor box. y,i r is the offset of the ordinate of the model output corresponding to the i-th anchor box. w,i r is the offset of the width of the model output corresponding to the i-th anchor box. h,i Let S(...) be the offset of the height of the model output corresponding to the i-th anchor box. The values ​​of the above offsets are all real numbers. S(...) is the smoothing L1 loss, calculated by the following formula:

[0034]

[0035] Where S(x) is any input;

[0036] (3) Train the entire model, importing the parameters of the three-branch region recommendation network layer trained in (2). Select the model with the smallest loss value during training. The loss value calculation formula is the same as in (2), except that the offset of the smoothed L1 loss input is replaced with the offset based on the output of the three-branch region recommendation network layer. For each image, select N. r Calculate N candidate boxes. r Replace N a This ensures that the parameters of the three-branch region recommendation network layer are not updated during training.

[0037] (4) The training model consists of three parts: background subtraction network layer, feature extraction network layer and three-branch region recommendation network layer. The parameters of the background subtraction network layer, feature extraction network layer and three-branch region recommendation network layer trained in (3) are imported. The training process only updates the parameters of the three-branch region recommendation network layer. The rest of the training process is the same as (2).

[0038] (5) Train the entire model and import the parameters trained in (3), including the parameters of the background subtraction network layer, the parameters of the feature extraction network layer, the parameters of the interest region pooling layer and the parameters of the classification / location regression layer. Import the parameters of the three-branch region recommendation network layer trained in (4). During the training process, only update the parameters of the interest region pooling layer and the parameters of the classification / location regression layer. The rest of the training process is the same as (3).

[0039] Step 5: During model testing, the predicted bounding boxes output by the last layer of the model need to be post-processed to filter out the qualified predicted bounding boxes and remove the unqualified predicted bounding boxes output by the detection network. The processing is as follows: First, remove the predicted bounding boxes with the classification label "non-defect". Then, remove the predicted bounding boxes whose upper left or lower right corner coordinates exceed the image boundary. Finally, divide the remaining predicted bounding boxes according to the classification label. Predicted bounding boxes of the same category are sorted in descending order of classification score. Remove the predicted bounding boxes with low scores but whose intersection-union ratio with the predicted bounding boxes with higher scores exceeds W. The value of W is in the range of [0,1]. The remaining predicted bounding boxes are used as the final prediction results. The predicted bounding boxes in the prediction results include two aspects of information: defect type and defect location.

[0040] This invention provides a deep learning-based method for detecting weld defects in ray images. It utilizes a Faster R-CNN network for target detection in ray images. Building upon the traditional Faster R-CNN, this invention adds a background subtraction network layer, resulting in a five-module Faster R-CNN network structure. This background subtraction network layer has updated parameters, continuously optimizing its background reduction effect during network training to highlight defect characteristics. Simultaneously, a three-branch region recommendation network layer replaces the traditional region recommendation network. Its two regression branches further subdivide the task of predicting defect location information: one branch predicts the x-coordinate and width of the defect's center point, and the other predicts the y-coordinate and height of the defect's center point, enhancing the accuracy of defect location prediction.

[0041] Beneficial effects:

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] The background subtraction network layer can continuously improve the background removal effect with training, reduce the impact of the background on defect detection, and highlight the defect area; the three-branch region recommendation network layer contains two regression branches, which subdivide the defect location prediction task, making the obtained candidate boxes more accurate, and can reduce the number of candidate boxes output by the region recommendation network layer, thereby improving the detection speed. Attached Figure Description

[0044] Figure 1 This is a flowchart of a deep learning-based X-ray image weld defect detection method, including image preprocessing, scaling and normalization, Faster R-CNN network, and post-processing. The Faster R-CNN network is modified into a five-part structure, consisting of a background subtraction network layer, a feature extraction network layer, a three-branch region recommendation network layer, an interest region pooling layer, and a classification / location regression layer.

[0045] Figure 2 The background is subtracted from the network layer structure diagram, which consists of large-size mean pooling, convolution, and skip connections. The skip connections are the subtraction part in the diagram.

[0046] Figure 3 The recommended network layer structure diagram for the three-branch region consists of six convolutions and three branches.

[0047] Figure 4 This is an example diagram of weld defect detection. The results of the defect detection in the diagram include the type and location of the defect, forming a border with the defect type. The numbers above the border in the diagram represent the type of defect: "1" represents a circular defect, "2" represents a strip defect, "3" represents a lack of fusion defect, "4" represents an indentation defect, "5" represents an undercut defect, and "6" represents a crack defect. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0049] Example:

[0050] A deep learning-based method for detecting weld defects using X-ray images is described below:

[0051] Step 1: Perform image preprocessing on the original image to obtain the preprocessed image, including grayscale stretching, median filtering, and mean filtering. The grayscale stretching formula is:

[0052]

[0053] Where g is the pixel value after grayscale stretching, g max The maximum pixel value of the stretched image is 255, where f is the pixel value of the original image. max f is the maximum pixel value of the original image. min The minimum pixel value of the original image. To round down, a 3x3 window is selected for both median filtering and mean filtering.

[0054] Step 2: Use the preprocessed image as the input image, then scale the input image to a 224x224x3 tensor, and then standardize it. The standardization formula is:

[0055]

[0056] Where u is the standardized value, v is the unstandardized value, m is the mean of the pixel values ​​of all images to be standardized, and the input to be standardized has 3 channels depending on the scaling, with mean values ​​of 0.485, 0.456, and 0.406 respectively; δ is the standard deviation of the pixel values ​​of all images to be standardized, and the input to be standardized has 3 channels depending on the scaling, with standard deviation values ​​of 0.229, 0.224, and 0.225 respectively.

[0057] Step 3: Using the standardized tensor as input, build a Faster R-CNN object detection network to detect defects in the image. Add a background subtraction network layer, changing the traditional four-part Faster R-CNN network to a five-part structure. Modify the regression branch in the region recommendation network layer to independently predict the location information of defects in the image. The defect location information includes the x-coordinate of the defect's center point, the y-coordinate of the defect's center point, the height of the defect, and the width of the defect. This Faster R-CNN network consists of five parts: a background subtraction network layer, a feature extraction network layer, a region recommendation network layer, an interest region pooling layer, and a classification / location regression layer. The construction steps are as follows:

[0058] (1) Construct a background subtraction network layer to reduce the influence of the background in the image and highlight the defect features. It consists of a large-size 25x25 average pooling, convolution and skip connections. The input of this layer is simulated after large-size average pooling and convolution. Then, the input of this layer is subtracted from the simulated background through skip connections to obtain the output of this layer.

[0059] (2) Construct a feature extraction network layer to extract defect features. It consists of ResNet50, Feature Pyramid Network (FPN), and SE attention mechanism. Here, the Feature Pyramid Network outputs 5 feature maps for subsequent prediction processes. The 5 feature maps are denoted as P2, P3, P4, P5, and P6. Each feature map layer is responsible for 3 sizes of anchor boxes. The anchor box size of each feature map layer is obtained through clustering. Specifically, layer P2 is responsible for 8x49. The P3 layer is responsible for anchor boxes of 13x30 and 19x20, the P4 layer is responsible for anchor boxes of 16x98, 26x60 and 38x41, the P5 layer is responsible for anchor boxes of 26x159, 42x98 and 63x66, the P5 layer is responsible for anchor boxes of 38x236, 62x145 and 93x98, and the P6 layer is responsible for anchor boxes of 51x311, 83x191 and 122x129. This layer uses the background subtraction network layer output as input and outputs 5 feature maps.

[0060] (3) Using the feature map as input, a region recommendation network layer is built to generate region suggestions. Based on the classification and regression dual branches of the traditional region recommendation network layer, the region recommendation network further subdivides the responsibility of the regression branch and independently predicts the location information of defects in the image. It includes three branches: front / background classification branch, width / center point horizontal coordinate regression branch, and height / center point vertical coordinate regression branch. These are referred to as the three-branch region recommendation network layer. In the classification branch, it judges whether each position in the feature map contains a defect. The position containing a defect is marked as a positive example, and the position not containing a defect is marked as a negative example. Furthermore, it makes a rough prediction of the position of the defect based on the anchor box. Here, two regression branches are used to independently predict the position of the defect. One branch is responsible for predicting the horizontal coordinate and width of the center point of the defect, and the other branch is responsible for predicting the vertical coordinate and height of the center point of the defect. The obtained position information is based on the anchor box position and is expressed as the offset of the anchor box position relative to the defect position in the horizontal coordinate, width, vertical coordinate, and height of the center point.

[0061] (4) Construct a region of interest pooling layer and adjust the number of convolutional layers. Here, four convolutional layers are added to the region of interest pooling layer. Based on the positive examples, negative examples, and offsets obtained in (3), features are extracted from the corresponding feature maps, and the output is of a uniform size. The formula for selecting the feature map when extracting features is:

[0062]

[0063] Where p is the selected feature map number, p0 is the initial feature map number (here, a value of 5), h is the height of the offset anchor box, w is the width of the offset anchor box, and T is a constant (here, a value of 224). To round down; (6) Construct a classification / location regression layer, consisting of two fully connected layers. One fully connected layer classifies the defect types, and the other fully connected layer regresses the defect locations to obtain the defect types and defect locations, forming a bounding box with defect types, called the predicted bounding box;

[0064] Step 4: Train the model on the training set, which consists of 1200 ray images of pipe welds. The labels for the training set are the defect type *c* and the defect location. There are 7 defect types, and *c* takes the value of a positive integer in the interval [0, 6], corresponding to non-defect, circular defect, strip defect, lack of fusion defect, concave defect, undercut defect, and crack defect, respectively. The defect location includes the x-coordinate of the defect's center point. t The ordinate of the center point of the defect is y. t The height of the defect w t and the width h of the defect t Each image contains several defects, and each defect corresponds to a location. The labels are manually assigned. The training process is as follows:

[0065] (1) Calculate the offset of the defect location in the label relative to the anchor frame, including the offsets of the horizontal coordinate, vertical coordinate, height, and width. The formula for calculating the offset of the horizontal coordinate is:

[0066]

[0067] Among them, t x The offset of the horizontal axis, x t x is the x-coordinate of the center point of the defect in the label. a w is the x-coordinate of the center point of the anchor frame a The formula for calculating the offset of the vertical axis, where the width of the anchor frame is given, is:

[0068]

[0069] Among them, t y The offset of the vertical axis, y t The y-coordinate of the center point of the defect in the label. a h is the x-coordinate of the center point of the anchor frame. a The formulas for calculating the offset of the height and width of the anchor frame are as follows:

[0070]

[0071] Among them, t w For the width offset, w t The formula for calculating the width and height offset of the defect in the label is:

[0072]

[0073] Among them, t h For high offset, h t The high percentage of defects in the label;

[0074] (2) The training model consists of three parts: the background subtraction network layer, the feature extraction network layer, and the three-branch region recommendation network layer. The parameters selected after training are those of the model with the smallest loss value during training. 16 anchor boxes are selected for each image for calculation. The formula for calculating the loss value is as follows:

[0075]

[0076] Where L is the loss value, N a q represents the number of anchor boxes used for training per image, with a value of 16. i t represents the maximum value of the model output vector for the i-th anchor box. x,i Let t be the offset of the horizontal coordinate in the label corresponding to the i-th anchor box. y,i t represents the offset of the ordinate in the label corresponding to the i-th anchor box. w,it is the offset of the width of the label corresponding to the i-th anchor box. h,i r is the height offset of the label corresponding to the i-th anchor box. x,i r is the offset of the x-coordinate of the model output corresponding to the i-th anchor box. y,i r is the offset of the ordinate of the model output corresponding to the i-th anchor box. w,i r is the offset of the width of the model output corresponding to the i-th anchor box. h,i Let S(...) be the offset of the height of the model output corresponding to the i-th anchor box, and let S(...) be the smoothed L1 loss, calculated as follows:

[0077]

[0078] Where S(x) is any input;

[0079] (3) Train the entire model and import the parameters of the three-branch region recommendation network layer trained in (2). Select the model with the smallest loss value during the training process. The loss value calculation formula is the same as (2). The offset of the smooth L1 loss input is replaced with the offset based on the output of the three-branch region recommendation network layer. 16 candidate boxes are selected for each image for calculation to ensure that the parameters of the three-branch region recommendation network layer are not updated during the training process.

[0080] (4) The training model consists of three parts: background subtraction network layer, feature extraction network layer and three-branch region recommendation network layer. The parameters of the background subtraction network layer, feature extraction network layer and three-branch region recommendation network layer trained in (3) are imported. The training process only updates the parameters of the three-branch region recommendation network layer. The rest of the training process is the same as (2).

[0081] (5) Train the entire model, import the parameters of the background subtraction network layer, feature extraction network layer, interest region pooling layer and classification / location regression layer trained in (3), import the parameters of the three-branch region recommendation network layer trained in (4), and only update the parameters of the interest region pooling layer and classification / location regression layer during the training process. The rest of the training process is the same as (3).

[0082] Step 5: During model testing, the predicted bounding boxes output by the last layer of the model need to be post-processed to filter out the qualified predicted bounding boxes and remove the unqualified predicted bounding boxes output by the detection network. The processing is as follows: First, remove the predicted bounding boxes with the classification label of "non-defect". Then, remove the predicted bounding boxes whose upper left and lower right corner coordinates exceed the image boundary. Finally, divide the remaining predicted bounding boxes according to the classification label. Predicted bounding boxes of the same category are sorted in descending order of classification score. Remove the predicted bounding boxes with low scores but whose intersection-union ratio with higher-scoring predicted bounding boxes exceeds 0.5. The remaining predicted bounding boxes are used as the final prediction results. The predicted bounding boxes in the prediction results include two aspects of information: defect type and defect location.

[0083] Step Six: Evaluate the performance of different models on the same test set. The test set consists of 300 ray images of pipe welds. The labels for the test set are the type of defect 'c' and the location of the defect in the image. There are 7 defect types, and the value of 'c' is a positive integer in the interval [0,6], corresponding to non-defect, circular defect, strip defect, lack of fusion defect, concave defect, undercut defect, and crack defect, respectively. The location of the defect includes the x-coordinate of the center point of the defect. t The ordinate of the center point of the defect is y. t The height of the defect w t and the width h of the defect tEach image contains several defects, and each defect corresponds to a location. Labels are marked manually. Each image corresponds to a category label, and the labels are marked manually. The evaluation index is mean average precision. The higher the mean average precision of the model, the better the defect detection effect of the model. The mean average precision is used to evaluate the original Faster R-CNN, Faster R-CNN based on ResNet50 and FPN, the algorithm of this invention, and YOLOv3 (where the original Faster R-CNN is the model in the paper (REN Shaoqing, HE Kaiming, GIRSHICK Ross, et al. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137-1149.), and the Faster R-CNN based on ResNet50 and FPN is the model in the paper (LIN Tsungyi, DOLLAR Piotr, GIRSHICK Ross, et al. Feature Pyramid Networks for Object Detection[J]. 30th IEEE Conference on The model used in Computer Vision and Pattern Recognition, 2017:936-944., and the model used in YOLOv3 (REDMON Joseph, FARHADI Ali. YOLOv3: An Incremental Improvement[C]. IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA: IEEE, 2017:6517-6525.) were compared in terms of detection performance. The formula for the mean and average precision is:

[0084]

[0085] Where mAP represents the mean precision, N represents the number of classes, and AP represents the mean precision, calculated using the following formula:

[0086]

[0087] Where P(R) represents the precision-to-recall curve. In practice, since a continuous curve is not readily available, the AP value here is taken as the average of the maximum precision over a certain number of equally spaced recall intervals. The formula for precision is:

[0088]

[0089] Where P is precision, TP is the number of samples that were predicted as positive and were also positive, FP is the number of samples that were predicted as positive and were also negative, and the formula for the recall rate is:

[0090]

[0091] Where R is the recall rate, and FN is the number of samples that were predicted to be negative but were actually positive. The comparison results are as follows:

[0092]

[0093]

[0094] The above description is not intended to limit the present invention in any way. Although the present invention has been disclosed through the above embodiments, it is not intended to limit the present invention. Any person skilled in the art can make changes or modifications to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A deep learning-based radiographic image weld defect detection method, characterized by The method comprises the following steps: Step one: image preprocessing of the original image, including gray stretch, median filter and mean filter, to obtain the image after preprocessing, the gray stretch formula is: Where g is the pixel value after grayscale stretching, and its value ranges from [0, g] to [0, g]. max The integer within ], g max The maximum pixel value of the stretched image, taking values ​​in the range of positive integers, and g max Greater than f max f is the pixel value of the original image. max f is the maximum pixel value of the original image. min The minimum pixel value of the original image. To round down, median filtering uses the median of all pixel values ​​within an m×n window of the original pixel value to replace the original pixel value, while mean filtering uses the mean of all pixel values ​​within an m×n window of the original pixel value to replace the original pixel value. The values ​​of m and n are both positive integers. Step two: taking the preprocessed image as the input image, then scaling the input image into a tensor of a x b x c, the value range of a, b and c is positive integer, then standardizing, the formula of standardization is: Wherein, u is the value after standardization, the value range is interval [-1, 1], v is the value before standardization, the value range is any real number, m is the mean value of the pixel value of all images to be standardized, the value range is any real number, and delta is the standard deviation of the pixel value of all images to be standardized, the value range is any real number; Step three: taking the standardized tensor as the input, building a Faster RCNN target detection network to detect the defects in the image, adding a background subtraction network layer, changing the traditional Faster RCNN network from four parts to five parts, and modifying the regression branch in the region recommendation network layer to independently predict the position information of the defects in the image, including the horizontal coordinate of the center point of the defect, the vertical coordinate of the center point of the defect, the height of the defect and the width of the defect, the Faster RCNN network comprises a background subtraction network layer, a feature extraction network layer, a region recommendation network layer, an interest region pooling layer and a classification / position regression layer, and the building steps are as follows: (1) building a background subtraction network layer to reduce the influence of the background in the image and highlight the defect features, which is composed of m x n large size mean pooling, convolution and jump connection, wherein the value range of m and n is positive integer, the input of the layer is obtained after large size mean pooling and convolution, and then the simulated background is obtained, and then the input of the layer is subtracted from the simulated background through jump connection to obtain the output of the layer; (2) Building a feature extraction network layer to extract defect features, which is composed of ResNet50, FPN and SE attention mechanism, wherein the feature pyramid network is a typical method for extracting feature maps of different sizes for subsequent tasks, in the defect detection task, it takes N feature maps as output, N is a positive integer, which is used for the prediction process of the subsequent part, and each layer of feature map is responsible for A size of anchor box, A is a positive integer, and the size of the anchor box is represented in the form of h a ×w a , wherein h a represents the height of the anchor box, and the value range is a positive integer, w a represents the width of the anchor box, and the value range is a positive integer, and the layer uses the output of the background subtraction network layer as input, and outputs N feature maps; (3) taking the feature map as the input, building a region recommendation network layer to generate region suggestions, the region recommendation network is based on the classification and regression double branch of the traditional region recommendation network layer, and the responsibility of the regression branch is subdivided to independently predict the position information of the defects in the image, including three branches of front / background classification branch, width / center point horizontal coordinate regression branch and height / center point vertical coordinate regression branch, which are called three branch region recommendation network layer, in the classification branch, it is judged whether each position in the feature map contains defects, and the positions containing defects are marked as positive examples, and the positions not containing defects are marked as negative examples, and on the basis of the anchor frame, the position of the defect is roughly predicted, here, two regression branches are used to independently predict the position of the defect, one branch is responsible for predicting the center point horizontal coordinate and width of the defect, and the other branch is responsible for predicting the center point vertical coordinate and height of the defect, the position information obtained is based on the anchor frame position as the basis, which is expressed as the offset of the anchor frame position relative to the defect position in the center point horizontal coordinate, width, center point vertical coordinate and height; (4) build the interest region pooling layer, adjust the number of convolutional layers K, K is a positive integer, according to the positive examples, negative examples and offsets obtained in (3), extract the features from the corresponding feature maps, and output the unified size, wherein the formula for selecting the feature map when extracting the features is: wherein p is a selected feature map number, the value range of which is a positive integer, p0 is an initial feature map number, the value range of which is a positive integer, h is a height of the offset anchor frame, the value range of which is a positive number, w is a width of the offset anchor frame, the value range of which is a positive number, T is a constant, the value range of which is a positive number, is rounded down. (5) build the classification / position regression layer, which is composed of two fully connected layers, one fully connected layer classifies the defect types, and one fully connected layer regresses the defect positions to obtain the defect types and defect positions, which constitute a bounding box with defect types, called predicted bounding box; Step four: training the model on the training set, the images of the training set are I train Radiographic images of Zhangguandao welds, I train The value range of is a positive integer, the label of the training set is the type of defects c and the position of the defects in the image, the number of defect types is C, the value range of C is a positive integer, the value range of c is a positive integer in the interval [0, C-1], the position of the defects includes the horizontal coordinate x of the center point of the defects t , the vertical coordinate y of the center point of the defects t , the height w of the defects t , and the width h of the defects t , each image contains several defects, each defect corresponds to a position, and the label is marked manually, and the training process is: (1) calculate the offsets of the defect positions in the labels relative to the anchor boxes, including the offsets of the horizontal coordinates, the offsets of the vertical coordinates, the offsets of the heights and the offsets of the widths, the formula for calculating the offsets of the horizontal coordinates is: wherein t x is the offset of the horizontal coordinate, x t is the horizontal coordinate of the center point of the defect in the label, x a is the horizontal coordinate of the center point of the anchor frame, w a is the width of the anchor frame, x t , x a , w a are positive integers, and the offset of the vertical coordinate is calculated according to the following formula: wherein t y is the offset of the longitudinal coordinate, y t is the longitudinal coordinate of the center point of the defect in the label, y a is the longitudinal coordinate of the center point of the anchor frame, h a is the height of the anchor frame, y t , y a , h a The value range of the positive integer is: the offset calculation formula of the width is: where t w is a wide offset, w t is the width of the defect in the label, and is a positive integer, and the high offset calculation formula is: wherein t h is a high offset, h t is a height of a defect in the label, and is a positive integer; (2) The background subtraction network layer, the feature extraction network layer and the three-branch region recommendation network layer in the model are trained, and the parameters of the model with the minimum loss value in the training process are selected. N a anchor boxes are selected for each image to calculate the loss value, and the calculation formula of the loss value is: wherein L is a loss value, N a is the number of anchor boxes used for training for each image, and takes a positive integer value, q i is the maximum value of the model output vector of the i-th anchor box, taking the value in the interval [0, 1], t x,i is the horizontal coordinate offset in the label corresponding to the i-th anchor box, t y,i is the vertical coordinate offset in the label corresponding to the i-th anchor box, t w,i is the width offset in the label corresponding to the i-th anchor box, t h,i is the height offset in the label corresponding to the i-th anchor box, r x,i is the horizontal coordinate offset of the model output corresponding to the i-th anchor box, r y,i is the vertical coordinate offset of the model output corresponding to the i-th anchor box, r w,i is the width offset of the model output corresponding to the i-th anchor box, r h,i a high offset for the i-th anchor box corresponding model output, wherein the offset is a real number, S(...) is the smooth L1 loss, and the formula is: wherein S(x) is any input; (3) training the whole model, importing the parameters of the three branch region recommendation network layers trained in (2), selecting the model with the minimum loss value in the training process as the trained model, the loss value calculation formula being the same as that in (2), wherein the offset input of the smooth L1 loss is replaced by the offset based on the output of the three branch region recommendation network layer, N r candidate boxes are selected for each image to calculate, N r is replaced by N a , ensuring that the parameters of the three branch region recommendation network layers are not updated in the training process; (4) train the background subtraction network layer, the feature extraction network layer and the three-branch region recommendation network layer in the model, import the parameters of the background subtraction network layer, the feature extraction network layer and the three-branch region recommendation network layer trained in (3), and only update the parameters of the three-branch region recommendation network layer in the training process, and the remaining training process is the same as (2); (5) train the entire model, import the parameters trained in (3), including the parameters of the background subtraction network layer, the parameters of the feature extraction network layer, the parameters of the interest region pooling layer and the parameters of the classification / position regression layer, import the parameters of the three-branch region recommendation network layer trained in (4), and only update the parameters of the interest region pooling layer and the parameters of the classification / position regression layer in the training process, and the remaining training process is the same as (3); Step five: when the model is tested, the predicted bounding box output by the last layer of the model needs to be post-processed, and the predicted bounding box that meets the requirements is selected, and the unqualified predicted bounding box output by the detection network is removed, the processing process is: first, remove the predicted box with a classification label of non-defect, then remove the predicted box with a left upper corner coordinate or a right lower corner coordinate exceeding the image boundary, and finally, the remaining predicted boxes are divided according to the classification label, the predicted boxes of the same category are sorted in descending order according to the classification score, the predicted boxes with a low score but an intersection over union exceeding W with the predicted boxes with a higher score are removed, W is in the range of [0, 1], and the remaining predicted boxes are used as the final prediction result, the predicted boxes in the prediction result include defect types and defect positions.

Images