A method for defect annotation of digital images of weld seams based on deep learning

Abstract

To solve the problem that the defect information of the weld in the weld digital image generated by traditional DR ray detection is manually confirmed, resulting in low weld evaluation efficiency and high quality risk, an embodiment of the present invention provides a method and system for defect annotation of weld digital images based on deep learning, including: training a convolutional neural network with a data set to obtain a weld defect annotation model; wherein, the data set includes a training set; the training set includes a number of data units; each data unit includes a weld digital image and a defect annotation file generated by defect annotation of the weld digital image; using the weld defect annotation model to perform defect annotation on the weld digital image to be measured.

Description

Technical Field

[0001] The present invention relates to the field of image defect recognition, and particularly to a method for labeling weld digital image defects based on deep learning. Background Art

[0002] In manufacturing enterprises such as pressure vessels, boilers, and pipelines, DR ray detection technology is widely used to generate weld digital images. The defect information of the image welds is mainly completed by manual confirmation. Facing a large number of detection samples, usually more than 1000 weld images need to be evaluated in a single shift. The non-destructive testing personnel have a high work intensity, are prone to visual fatigue, and are also affected by the skills or subjective factors of the inspectors. There are problems such as high labor intensity, easy omission of inspection, and strong subjectivity, resulting in low efficiency of weld defect evaluation and high quality risk. Summary of the Invention

[0003] To solve the problem that the defect information of the welds in the weld digital images generated by traditional DR ray detection is manually confirmed, resulting in low efficiency of weld evaluation and high quality risk, an embodiment of the present invention provides a method for labeling weld digital image defects based on deep learning.

[0004] The embodiment of the present invention is implemented through the following technical solutions:

[0005] In a first aspect, an embodiment of the present invention provides a method for labeling weld digital image defects based on deep learning, including:

[0006] Training a convolutional neural network with a data set to obtain a weld defect labeling model;

[0007] Wherein, the data set includes a training set; the training set includes a number of data units; each data unit includes a weld digital image and a defect labeling file generated by labeling the defect of the weld digital image;

[0008] Using the weld defect labeling model to label the defect of the weld digital image to be measured.

[0009] Further, the convolutional neural network is a YOLO-v3 neural network; the weld defect labeling model adopts a shallow feature extraction network, and the number of convolutional layers of the weld defect labeling model is not more than 100 layers.

[0010] Further, the data set further includes a validation set and a test set; both the validation set and the test set include a number of the data units;

[0011] Training a convolutional neural network with a data set to obtain a weld defect labeling model; including:

[0012] S1. Perform the following steps on the weld digital images of each data unit in the training set to train the neural network:

[0013] a. Normalize the pixel values of the weld digital image and then input it into the input nodes of the neural network;

[0014] b. The weld digital image of the input nodes undergoes multi-layer linear processing and non-linear activation through the convolutional layer of the neural network to perform data dimensionality reduction and feature extraction;

[0015] c. Input the extracted features obtained through the feature extraction into the fully connected layer to obtain the one-hot encoding representing the defect judgment result;

[0016] d. Calculate the loss error between the one-hot encoding and the actual value of the weld defect;

[0017] e. Calculate the first-order derivative of the loss error with respect to the weights of the hidden layer in the network;

[0018] f. Iteratively update the weights of the hidden layer in the network based on the first-order derivative;

[0019] S2. Use the validation set and the test set to verify and test the trained neural network to obtain the weld defect annotation model.

[0020] Furthermore, the calculation formula for the learning process of the loss error is expressed by the following formula (1):

[0021] L = A + Z - K - D - E (1)

[0022] Where, L is the loss error; A is the loss for calculating the center point coordinates of the bounding box; Z is the loss for calculating the width and height of the bounding box; K is the loss for calculating the confidence of having an object in the bounding box; D is the loss for calculating the confidence of not having an object in the bounding box; E is the loss for calculating the object category of the bounding box.

[0023] Furthermore, the weld digital image of the input nodes undergoes multi-layer linear processing and non-linear activation through the convolutional layer of the neural network to perform data dimensionality reduction and feature extraction; including:

[0024] Divide the input image into S * S cells, and set B bounding boxes for each cell; where both S and B are positive integers greater than zero;

[0025] The calculation formula for A is expressed by formula (2) as:

[0026]

[0027] Where, λ coord is the weight of the bounding box with an object;

[0028] Indicates whether the j-th bounding box of the i-th cell in the input image partition is responsible for prediction. If so, it is taken as 1; otherwise, it takes the value of 0.

[0029] x ij Represents the predicted value of the x-coordinate of the center point of the j-th bounding box of the i-th cell.

[0030] Represents the true value of the x-coordinate of the center point of the j-th bounding box of the i-th cell.

[0031] y ij Represents the predicted value of the y-coordinate of the center point of the j-th bounding box of the i-th cell.

[0032] Represents the true value of the y-coordinate of the center point of the j-th bounding box of the i-th cell.

[0033] w ij Represents the predicted value of the width of the j-th bounding box of the i-th cell.

[0034] h ij Represents the predicted value of the height of the j-th bounding box of the i-th cell.

[0035] The calculation formula of Z is expressed as in formula (3):

[0036]

[0037] Among them, λ coord Is the weight of the object bounding box;

[0038] Indicates whether the j-th bounding box of the i-th cell in the input image partition is responsible for prediction. If so, it is taken as 1; otherwise, it takes the value of 0.

[0039] w ij Represents the predicted value of the width of the j-th bounding box of the i-th cell.

[0040] Represents the true value of the width of the j-th bounding box of the i-th cell.

[0041] h ij Represents the predicted value of the height of the j-th bounding box of the i-th cell.

[0042] Represents the true value of the height of the j-th bounding box of the i-th cell.

[0043] The calculation formula of K is expressed as in formula (4):

[0044]

[0045] Among them, indicates whether the j-th bounding box of the i-th cell in the input image partition is responsible for prediction. If so, it is taken as 1; otherwise, it takes the value of 0.

[0046] c ij represents the model prediction confidence. When there is an object in the cell, the prediction confidence is the intersection over union of the model-predicted bounding box and the actual bounding box; when there is no object in the cell, the prediction confidence is 0.

[0047] represents the true confidence, and its value is {0, 1}.

[0048] The calculation formula of D is expressed as in formula (5):

[0049]

[0050] Among them, λ noobj is the weight of the no-object bounding box.

[0051] indicates whether the j-th bounding box of the i-th cell in the input image partition is responsible for prediction. If so, it is taken as 0; otherwise, it takes the value of 1.

[0052] c ij represents the model prediction confidence. When there is an object in the cell, the prediction confidence is the intersection over union of the model-predicted bounding box and the actual bounding box; when there is no object in the cell, the prediction confidence is 0.

[0053] represents the true confidence, and its value is {0, 1}.

[0054] The calculation formula of E is expressed as in formula (6):

[0055]

[0056] Among them, indicates whether the j-th bounding box of the i-th cell in the input image partition is responsible for prediction. If so, it is taken as 1; otherwise, it takes the value of 0.

[0057] refers to the true probability value of class c;

[0058] p ij (c) is the predicted value belonging to class c.

[0059] Furthermore, the calculation formula of the network learning process of the convolutional layer is expressed as in the following formula (7):

[0060]

[0061] Among them, C represents the convolution layer, l represents the layer index of the convolution layer, and the value C of the current convolution layer l It is composed of the previous convolutional layer C l-1 Obtained through convolution calculation, W represents the weight of the convolution kernel, X represents the bias of the convolution kernel, i represents the convolution kernel index, and * represents the convolution operation.

[0062] Furthermore, the calculation formula of the network learning process of the fully connected layer F is expressed as follows:

[0063]

[0064] Among them, F represents the fully connected layer, l represents the layer index of the fully connected layer, and the value of the current fully connected layer F l is the fully connected layer F l-1 Obtained through dot product calculation, M represents the weight of the fully connected layer, y represents the bias of the fully connected layer, i represents the element index of the fully connected layer, and · represents the dot product operation.

[0065] Further, f. iteratively updating the hidden layer weights in the network based on the first-order derivative; comprising:

[0066] The hidden layer weight θ in the network is iteratively assigned and updated according to the following formula (9):

[0067]

[0068] Among them, lr is the learning rate, is the first-order derivative of the loss error with respect to the hidden layer weights in the network.

[0069] The learning rate lr adopts corresponding preset learning rate values ​​based on different training stages;

[0070] The learning rate lr adopts corresponding preset learning rate values ​​based on different training stages; including:

[0071] A first learning rate is used in the initial training stage of the neural network, and a second learning rate is used after the neural network learning enters a plateau period; wherein the second learning rate is less than the first learning rate; and the second learning rate decays to one tenth of the first learning rate.

[0072] Furthermore, before using the data set to train the convolutional neural network to obtain the weld defect annotation model, the method also includes: obtaining a digital image of the weld;

[0073] The obtaining of the weld digital image comprises:

[0074] Perform a random cropping and a random horizontal flipping on the original weld digital image respectively to obtain the original weld digital image, the cropped weld digital image, and the horizontally flipped weld digital image, so as to achieve data augmentation;

[0075] Obtain the original weld digital image, the cropped weld digital image, and the horizontally flipped weld digital image to get the weld digital image.

[0076] Compared with the prior art, the embodiment of the present invention has the following advantages and beneficial effects:

[0077] A method and system for defect annotation of weld digital images based on deep learning according to an embodiment of the present invention obtain a weld defect annotation model by training a convolutional neural network using a data set; use the weld defect annotation model to perform defect annotation on a to-be-tested weld digital image, and solve the defect that the defect information of the weld in the weld digital image generated by traditional DR ray detection is manually confirmed, resulting in low weld evaluation efficiency and high quality risk. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the drawings required to be used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0079] Figure 1 It is a flow schematic diagram of a method for defect annotation of weld digital images based on deep learning as an example.

[0080] Figure 2 It is a flow schematic diagram of another method for defect annotation of weld digital images based on deep learning as an example.

[0081] Figure 3 It is a schematic diagram of the structure of a weld defect annotation model. In the figure, I represents the input, C represents the convolutional layer, F represents the fully connected layer, and O represents the output.

[0082] Figure 4 It is a schematic diagram of the network modular configuration of the weld defect annotation model.

[0083] Figure 5 It is a schematic diagram of image data augmentation; where Figure (a) is the original image, Figure (b) is the random cropping, and Figure (c) is the random flipping.

[0084] Figure 6 It is a schematic diagram of the learning rate decay curve.

[0085] Figure 7It is a schematic structural diagram of a weld digital image defect annotation system based on deep learning. Specific embodiments

[0086] To make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below in conjunction with embodiments and drawings. The illustrative embodiments of the present invention and their descriptions are only used to explain the present invention and are not intended to limit the present invention.

[0087] In the following description, a large number of specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to those of ordinary skill in the art that: these specific details do not have to be employed to practice the present invention. In other embodiments, well-known structures, circuits, materials, or methods have not been specifically described in order to avoid obscuring the present invention.

[0088] Throughout the specification, references to "one embodiment", "an embodiment", "one example" or "an example" mean that a particular feature, structure, or characteristic described in connection with the embodiment or example is included in at least one embodiment of the present invention. Thus, the phrases "one embodiment", "an embodiment", "one example" or "an example" appearing throughout the specification do not necessarily all refer to the same embodiment or example. Additionally, the particular features, structures, or characteristics may be combined in any suitable combination and / or sub-combination in one or more embodiments or examples. Further, those of ordinary skill in the art should understand that the diagrams provided herein are for illustrative purposes only and are not necessarily drawn to scale. The term "and / or" used herein includes any and all combinations of one or more of the associated listed items.

[0089] In the description of the present invention, the orientation or positional relationship indicated by the terms "front", "rear", "left", "right", "upper", "lower", "vertical", "horizontal", "high", "low", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as limiting the protection scope of the present invention.

[0090] Embodiment

[0091] To solve the problems of low weld evaluation efficiency and high quality risk caused by manual confirmation of weld defect information in the weld digital images generated by traditional DR ray detection, on the one hand, the embodiments of the present invention provide a weld digital image defect annotation method based on deep learning, as shown in Figure 1 and includes:

[0092] T1. Training a convolutional neural network with a data set to obtain a weld defect annotation model;

[0093] Among them, the data set includes a training set; the training set includes a number of data units; each data unit includes a weld digital image and a defect annotation file generated by performing defect annotation on the weld digital image.

[0094] T2. Use the weld defect annotation model to perform defect annotation on the weld digital image to be measured.

[0095] Therefore, in the embodiment of the present invention, by training a convolutional neural network with a data set, a weld defect annotation model is obtained; the weld defect annotation model is used to perform defect annotation on the weld digital image to be measured, solving the problems of low weld evaluation efficiency and high quality risk caused by manual confirmation of the defect information of the weld in the weld digital image generated by traditional DR ray detection.

[0096] Further, the convolutional neural network is a YOLO-v3 neural network.

[0097] Exemplarily, a method for defect annotation of a weld digital image based on deep learning, as shown in Figure 2 includes the steps:

[0098] S10. Obtain a weld digital image;

[0099] Perform a random crop and a random horizontal flip on the original weld digital image respectively to obtain the original weld digital image, the cropped weld digital image, and the horizontally flipped weld digital image, so as to achieve data augmentation.

[0100] Obtain the original weld digital image, the cropped weld digital image, and the horizontally flipped weld digital image to obtain the weld digital image.

[0101] In actual welding production, since the actual welding process has undergone strict welding process evaluation, and the automation level is relatively high, the weld formation is beautiful, and there are few internal welding defects. The overall incidence rate of welding defects is only 3%. The small sample problem is one of the biggest obstacle problems of machine learning algorithms. This example proposes a method for enhancing the digital weld digital image. While maintaining the random enhancement probability of the data at 0.5 and on the basis of a single learning rate enhancement, the image data enhancement shown in Figure 5 is used. Perform a random crop and a random horizontal flip on the original weld digital image (a) respectively to obtain the original weld digital image, the cropped weld digital image (b), and the horizontally flipped weld digital image (c), effectively expanding the data sample capacity. Unexpectedly, it can also effectively remove the interference of the image background and the base metal image on learning, achieving a data enhancement effect of 1 + 1 > 2.

[0102] S20. Perform defect annotation on the weld digital image to generate an annotation file.

[0103] The defect annotation can be carried out in any existing way, and there is no restriction on this.

[0104] S30: Take the weld digital image and the corresponding annotation file as a data unit, and divide all data units into a training set, a validation set, and a test set according to a certain ratio;

[0105] S40: Use the training set, the validation set, and the test set to develop a training model for weld defect annotation. In this example, a convolutional neural network is used. For the schematic diagram of its basic structure, refer to Figure 3 and Figure 4 as shown. This convolutional neural network has the advantages of parameter sharing and feature translation invariance.

[0106] Specifically, S40 includes: S1. Perform the following steps on the weld digital image of each data unit in the training set to train the neural network: Normalize the pixel values of the weld digital image and input them into the input node I of the network. After multiple linear processes and non-linear activations of the convolutional layer network C, the effects of data dimensionality reduction and feature extraction are achieved. Input the features into the fully connected layer network F, and the output includes a defect box detection dictionary, including the two-dimensional coordinates (x, y) of the defect, the two-dimensional size (w, h), the confidence score of the defect box, and the defect category. The qualitative judgment of the defect is then converted according to the defect categories specified in the non-destructive testing standard. Use one-hot encoding O to represent the corresponding defect nature, and a total of 11 categories are distinguished, where 0 represents the weld, 1 represents cracks, 2 represents lack of fusion, 3 represents incomplete penetration, 4 represents internal concavity, 5 represents undercut, 6 represents porosity, 7 represents overlap, 8 represents poor profile, 9 represents burn-through, and 10 represents other defects such as foreign objects; In the case of random initialization of the network, calculate the loss error L between the output O value and the actual value of the welding defect, and calculate the first-order derivative of L with respect to the weights of the hidden layer in the network. Then, based on this first-order derivative, iterate and update the weights of the hidden layer in the network; The actual value of the weld defect can be obtained from the annotation file. Usually, the calculation is a simple linear function, so the first-order derivative of the loss error L is always differentiable, and the welding defect data can be trained. S2. Use the validation set and the test set to verify and test the trained neural network to obtain the weld defect annotation model.

[0107] S50: Automatically annotate the DR digital image with the trained weld defect annotation model.

[0108] Due to the limited computer memory, reasonable data batch processing hyperparameters must be set, and the welding defect data is sent into the network in batches, and then the model performs network learning according to the process.

[0109] The network learning process can be divided into two stages. The first stage is the forward calculation stage, and the main objective of this stage is to calculate the loss function through formulas (1)-(8); In the second stage, formula (9) is used to iteratively update the parameters in the network using the backpropagation algorithm. Through this iterative update, the loss function of the model can be continuously reduced. Corresponding to the weld data labels, the smaller the loss function, the smaller the difference between the predicted value of the model and the actual weld annotation value. Therefore, after iterative calculation, the model can achieve the prediction function for weld data.

[0110] Optionally, the calculation formula for the learning process of the loss error is expressed by the following formula (1):

[0111] L = A + Z - K - D - E (1)

[0112] Where L is the loss error; A is the loss for calculating the center point coordinates of the bounding box; Z is the loss for calculating the width and height of the bounding box; K is the loss for calculating the confidence of having an object in the bounding box; D is the loss for calculating the confidence of not having an object in the bounding box; E is the loss for calculating the object category of the bounding box.

[0113] Optionally, the input image is divided into S * S cells, and B bounding boxes are set for each cell; where both S and B are positive integers greater than zero;

[0114] The calculation formula for A is expressed as formula (2):

[0115]

[0116] Where λ coord is the weight of the bounding box with an object;

[0117] indicates whether the j-th bounding box in the i-th cell of the input image division is responsible for prediction. If so, it takes 1, otherwise it takes 0;

[0118] x ij represents the predicted value of the x coordinate of the center point of the j-th bounding box in the i-th cell;

[0119] represents the true value of the x coordinate of the center point of the j-th bounding box in the i-th cell;

[0120] y ij represents the predicted value of the y coordinate of the center point of the j-th bounding box in the i-th cell;

[0121] represents the true value of the y coordinate of the center point of the j-th bounding box in the i-th cell;

[0122] w ijThe predicted value of the width of the j-th bounding box of the i-th cell;

[0123] h ij The predicted value of the height of the j-th bounding box of the i-th cell;

[0124] The calculation formula of Z is expressed as in Formula (3):

[0125]

[0126] where λ coord is the weight of the object bounding box;

[0127] Indicates whether the j-th bounding box of the i-th cell in the divided input image is responsible for prediction. If so, it takes the value of 1; otherwise, it takes the value of 0;

[0128] w ij The predicted value of the width of the j-th bounding box of the i-th cell;

[0129] The true value of the width of the j-th bounding box of the i-th cell;

[0130] h ij The predicted value of the height of the j-th bounding box of the i-th cell;

[0131] The true value of the height of the j-th bounding box of the i-th cell;

[0132] The calculation formula of K is expressed as in Formula (4):

[0133]

[0134] where Indicates whether the j-th bounding box of the i-th cell in the divided input image is responsible for prediction. If so, it takes the value of 1; otherwise, it takes the value of 0;

[0135] c ij Indicates the model prediction confidence. When there is a target in the cell, the prediction confidence is the intersection over union of the model predicted bounding box and the actual bounding box; when there is no target in the cell, the prediction confidence is 0;

[0136] Indicates the true confidence, and its value is {0, 1};

[0137] The calculation formula of D is expressed as in Formula (5):

[0138]

[0139] where λ noobjis the weight of the objectless border;

[0140] Indicates whether the j-th bounding box of the i-th cell in the input image partition is responsible for prediction. If so, it is taken as 0; otherwise, it takes the value of 1.

[0141] c ij Indicates the model prediction confidence. When there is an object in the cell, the prediction confidence is the intersection over union of the model-predicted bounding box and the actual bounding box. When there is no object in the cell, the prediction confidence is 0.

[0142] Indicates the true confidence, and its value is {0, 1}.

[0143] The calculation formula of E is expressed as in formula (6):

[0144]

[0145] where, Indicates whether the j-th bounding box of the i-th cell in the input image partition is responsible for prediction. If so, it is taken as 1; otherwise, it takes the value of 0.

[0146] refers to the true probability value for class c;

[0147] p ij (c) is the predicted value belonging to class c.

[0148] Furthermore, the calculation formula of the network learning process of the convolutional layer is expressed as the following formula (7):

[0149]

[0150] where, C represents the convolutional layer, l represents the layer index of the convolutional layer, and the value C of the current convolutional layer l is obtained by convolution calculation from the previous convolutional layer C l-1 W represents the weight of the convolution kernel, X represents the bias of the convolution kernel, i represents the convolution kernel index, and * represents the convolution operation.

[0151] Furthermore, the calculation formula of the network learning process of the fully connected layer F is expressed as the following formula (8):

[0152]

[0153] where, F represents the fully connected layer, l represents the layer index of the fully connected layer, and the value F of the current fully connected layer l is from the previous fully connected layer F l-1Obtained through dot product calculation, where M represents the weights of the fully connected layer, y represents the biases of the fully connected layer, i represents the element indices of the fully connected layer, and · represents the dot product operation.

[0154] Furthermore, the f iteratively updates the weights of the hidden layers in the network based on the first-order derivative; it includes:

[0155] Iteratively assigns and updates the weights θ of the hidden layers in the network according to the following formula (9):

[0156]

[0157] where lr is the learning rate, is the first-order derivative of the loss error with respect to the weights of the hidden layers in the network.

[0158] Furthermore, the learning rate lr adopts corresponding preset learning rate values based on different training stages;

[0159] The learning rate lr adopts corresponding preset learning rate values based on different training stages; it includes:

[0160] Use the first learning rate in the initial training stage of the neural network, and use the second learning rate after the neural network learning enters the plateau period; where the second learning rate is less than the first learning rate.

[0161] In this example, when performing machine learning algorithm recognition on natural images, the widely used initial learning rate is 1e-3. This technology demonstrates through experiments that the 1e-3 learning rate is not conducive to the improvement of model performance, resulting in the model being unable to find the global optimal solution but oscillating at the local optimal solution. When the learning rate is small, the model training duration will be greatly increased. Based on the characteristics of the training of the above welding defect data images, this embodiment proposes a learning rate decay curve, and the learning rate lr adopts corresponding preset learning rate values based on different training stages; specifically, use the first learning rate in the initial training stage, and use the second learning rate after the model learning enters the plateau period; where the second learning rate is less than the first learning rate.

[0162] Furthermore, the second learning rate decays to one-tenth of the first learning rate. For example, if the first learning rate is 1e-4, the second learning rate decays to 1e-5.

[0163] That is to say, use a larger learning rate in the initial stage, and after the model learning enters the plateau period, decay the model learning rate to one-tenth of the original. Based on the variation relationship between the learning rate and the accuracy curve, the optimal learning rate decay curve is proposed. Refer to Figure 6 as shown, thereby improving the model accuracy while reducing the learning time.

[0164] Optionally, the weld defect annotation model of this example adopts a shallow feature extraction network, and the number of convolutional layers of the weld defect annotation model is no more than 100 layers. Further optionally, the number of convolutional layers of the weld defect annotation model is 10 layers.

[0165] Different from the common deep network feature extraction methods, the conventional YOLO-v3 neural network covers 107 convolutional layers. This example only uses 10 convolutional layers to extract effective feature information of the weld, and at the same time avoids the problem of feature disappearance in deep network feature extraction. In view of the characteristics of small-size weld defects, when using traditional deep networks to extract features, defect information will be lost deep in the network, which causes the subsequent feature recognition network to have no effective information to learn. Therefore, this example proposes a shallow feature extraction network, which maximally extracts the high-level semantic information of small-size welding defect features while avoiding the problem of feature disappearance.

[0166] In addition, the deep learning-based weld digital image defect annotation method of this example can combine the industry's judgment standard for the harm degree of welding defect types to achieve timely warning of harmful defects when in use; further, the deep learning-based weld digital image defect annotation method of the example can also predict and annotate the positions of the defect corner points, so as to realize the pre-judgment of the defect size, and through the combined use of the deep learning-based weld digital image defect annotation method of the example and the industry's judgment standard for the harm degree of welding defect sizes, achieve timely warning of overweight defects.

[0167] Thus, the deep learning-based weld digital image defect annotation method of this example solves the problem of small samples of welding defects by performing image data augmentation when acquiring weld digital images, reduces the learning time consumption and improves the accuracy of the weld defect annotation model by implementing the optimal learning rate decay curve strategy; by adopting a shallow feature extraction network to obtain weld defects, it realizes the extraction of small-size welding defect features and avoids the problem of feature disappearance. The deep learning-based weld digital image defect annotation method of this example can be combined with relevant industry judgment standards to achieve timely warning of harmful defects.

[0168] In a second aspect, an embodiment of the present invention provides a deep learning-based weld digital image defect annotation system, as shown in Figure 7 shown, including:

[0169] A training unit for training a convolutional neuron network using a data set to obtain a weld defect annotation model; wherein, the data set includes a training set; the training set includes a number of data units; each data unit includes a weld digital image and a defect annotation file generated by defect annotation of the one weld digital image;

[0170] An annotation unit for using the weld defect annotation model to perform defect annotation on a digital image of a weld to be measured.

[0171] The working principles of other possible implementation manners of the weld digital image defect annotation system based on deep learning in this embodiment are the same as those of the exemplary weld digital image defect annotation method based on deep learning, and will not be elaborated here.

[0172] Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the present invention can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. Optionally, they can be implemented by program codes executable by the computing device. Thus, they can be stored in a computer storage medium (ROM / RAM, magnetic disk, optical disc) and executed by the computing device. And in some cases, the steps shown or described can be executed in a sequence different from that here, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module to implement. Therefore, the present invention is not limited to any specific combination of hardware and software.

[0173] The above-mentioned specific implementation manners have further elaborated the purpose, technical solution and beneficial effects of the present invention. It should be understood that the above is only the specific implementation manners of the present invention and is not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for defect annotation of digital images of weld seams based on deep learning, characterized in that, Including: Training a convolutional neural network with a dataset to obtain a weld defect annotation model; Wherein, the dataset includes a training set; the training set includes a number of data units; each data unit includes a weld digital image and a defect annotation file generated by performing defect annotation on the one weld digital image; Using the weld defect annotation model to perform defect annotation on a weld digital image to be measured; The dataset further includes a validation set and a test set; both the validation set and the test set include a number of the data units; Training a convolutional neural network with a dataset to obtain a weld defect annotation model; including: S1. Performing the following steps on the weld digital image of each data unit in the training set to train the neural network: a. Normalizing the pixel values of the weld digital image and inputting it into the input node of the neural network; b. The weld digital image of the input node undergoes multi-layer linear processing and non-linear activation through the convolutional layer of the neural network to perform data dimensionality reduction and feature extraction; c. Inputting the extracted features obtained through the feature extraction into the fully connected layer to obtain a one-hot encoding representing the defect judgment result; d. Calculating the loss error between the one-hot encoding and the actual value of the weld defect; e. Calculating the first-order derivative of the loss error with respect to the weights of the hidden layer in the network; f. Iteratively updating the weights of the hidden layer in the network based on the first-order derivative; S2. Using the validation set and the test set to validate and test the trained neural network to obtain a weld defect annotation model; The calculation formula of the learning process of the loss error is expressed by the following formula (1): L = A + Z - K - D - E (1) Wherein, L is the loss error; A is the loss of calculating the center point coordinates of the bounding box; Z is the loss of calculating the width and height of the bounding box; K is the loss of calculating the confidence of having an object in the bounding box; D is the loss of calculating the confidence of having no object in the bounding box; E is the loss of calculating the object category of the bounding box; The weld digital image of the input node undergoes multi-layer linear processing and non-linear activation through the convolutional layer of the neural network to perform data dimensionality reduction and feature extraction; including: Dividing the input image into S * S cells and setting B bounding boxes for each cell; wherein, both S and B are positive integers greater than zero; The calculation formula of A is expressed as formula (2): (2) Among them, is the weight of the object border; Indicates whether the j-th bounding box of the i-th cell in the input image partition is responsible for prediction. If so, it is taken as 1; otherwise, it takes the value of 0. Represents the predicted value of the x coordinate of the center point of the j-th bounding box in the i-th cell; Denote the true value of the x coordinate of the center point of the j-th bounding box of the i-th cell; Represents the predicted value of the y coordinate of the center point of the j-th bounding box of the i-th cell; Denote the true value of the y coordinate of the center point of the j-th bounding box of the i-th cell; The predicted value of the width of the j-th bounding box of the i-th cell; Represents the predicted value of the height of the j-th bounding box of the i-th cell; The calculation formula of Z is expressed as formula (3): (3) Among them, is the weight of the object border; Indicates whether the j-th bounding box of the i-th cell in the division of the input image is responsible for prediction. If so, it is taken as 1; otherwise, it takes the value of 0. Denote the predicted value of the width of the j-th bounding box of the i-th cell; Denote the true value of the width of the j-th bounding box of the i-th cell; Denote the predicted value of the height of the j-th bounding box of the i-th cell; Represents the true value of the height of the j-th bounding box of the i-th cell; The calculation formula of K is expressed as formula (4): (4) Among them, indicates whether the j-th bounding box of the i-th cell in the input image partition is responsible for prediction. If so, it is taken as 1; otherwise, it takes the value of 0. Indicates the prediction confidence of the model. When the target exists in the cell, the prediction confidence is the intersection over union (IoU) between the model-predicted bounding box and the actual bounding box. When the target does not exist in the cell, the prediction confidence is 0. Represents the true confidence level, which takes values in {0, 1}; The calculation formula of D is expressed as formula (5): (5) Among them, is the weight of the objectless border; Indicates whether the j-th bounding box of the i-th cell in the input image partition is responsible for prediction. If so, it is taken as 0; otherwise, it takes the value of 1. Indicates the prediction confidence of the model. When the target exists in the cell, the prediction confidence is the intersection over union (IoU) between the model-predicted bounding box and the actual bounding box. When the target does not exist in the cell, the prediction confidence is 0. Represents the true confidence level, which takes values in {0, 1}; The calculation formula of E is expressed as formula (6): (6) Among them, indicates whether the j-th bounding box of the i-th cell in the divided input image is responsible for prediction. If so, it is taken as 1; otherwise, it takes the value of 0. It refers to that the C class is the true probability value; is the predicted value belonging to class C; The convolutional neural network is a YOLO-v3 neural network; the weld defect annotation model adopts a shallow feature extraction network, and the number of layers of the convolutional layer of the weld defect annotation model is not more than 100 layers.

2. The method for defect annotation of digital images of weld seams based on deep learning according to claim 1, characterized in that, The calculation formula of the network learning process of the convolutional layer is expressed by the following formula (7): （7） Among them, represents a convolutional layer, represents the layer index of the convolutional layer, and the value of the current convolutional layer is obtained by convolutional calculation from the previous convolutional layer through convolution calculation, represents the weight of the convolutional kernel, represents the bias of the convolutional kernel, represents the convolutional kernel index, represents the convolution operation.

3. The method for defect annotation of digital images of weld seams based on deep learning according to claim 1, characterized in that The calculation formula of the network learning process of the fully connected layer is expressed by the following formula (8): （8） Among them, represents a fully connected layer, represents the layer index of the fully connected layer, and the value of the current fully connected layer is obtained by the dot product calculation of the previous fully connected layer and is, represents the weight of the fully connected layer, represents the bias of the fully connected layer, represents the element index of the fully connected layer, represents the dot product operation.

4. The method for defect annotation of digital images of weld seams based on deep learning according to claim 1, characterized in that, The f. Iteratively updating the weights of the hidden layer in the network based on the first-order derivative; including: Iteratively assign and update the weights of the hidden layer in the network according to the following formula (9): ​ - （9） Among them, is the learning rate, is the first-order derivative of the loss error with respect to the weights of the hidden layer in the network.

5. The method for defect annotation of digital weld images based on deep learning according to claim 4, wherein Learning rate Adopt corresponding preset learning rate values based on different training stages; The learning rate Adopt corresponding preset learning rate values based on different training stages; including: Use the first learning rate during the initial training stage of the neural network, and use the second learning rate after the neural network learning enters the plateau period; wherein the second learning rate is less than the first learning rate; the second learning rate decays to one-tenth of the first learning rate.

6. The method for defect annotation of digital images of weld seams based on deep learning according to claim 1, wherein, Before training the convolutional neural network with the dataset to obtain the weld defect annotation model, it further includes: obtaining the weld digital image; The obtaining of the weld digital image includes: Perform a random cropping and a random horizontal flipping on the original weld digital image respectively to obtain the original weld digital image, the cropped weld digital image, and the horizontally flipped weld digital image, so as to achieve data augmentation; Obtain the original weld digital image, the cropped weld digital image, and the horizontally flipped weld digital image to obtain the weld digital image.

Images