Training method and device based on bi-lstm neural network

Abstract

The application provides a defect detection method and a training method based on a Bi-LSTM neural network. The defect detection method comprises the following steps: A', acquiring N frames of infrared thermal images of a test surface of a workpiece to be tested changing with time during a cooling process; C', acquiring a pulse thermal imaging data sequence SEQ' of each pixel point on the test surface of the workpiece to be tested changing with time from the N frames of infrared thermal images; and G', bringing the pulse thermal imaging data sequence SEQ' of each pixel point changing with time into a trained Bi-LSTM neural network respectively to acquire a defect condition of a corresponding position of each pixel point. The application does not need to know prior information such as material thermal properties, a reference area or a characteristic time, and can ensure the accuracy of each defect detection. Meanwhile, the application can realize full-automatic data analysis and is more convenient to apply.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of nondestructive testing of workpiece defects, and particularly relates to a training method and device based on a Bi-LSTM neural network. BACKGROUND

[0002] Pulse thermography nondestructive testing technology can be used for non-contact, large-area, qualitative and quantitative detection of internal defects of materials. With the increasing requirements for material quality, especially the safety and economy of in-service equipment, people pay more attention to the three-dimensional structure information of internal defects of materials. For quantitative measurement of the three-dimensional structure of internal defects of materials, researchers have proposed methods such as thermal contrast method (defect depth prediction), logarithmic second derivative peak value method (defect depth prediction), half-width method (defect size prediction), and deconvolution method (defect size prediction).

[0003] However, the above methods can realize three-dimensional reconstruction of defects, but require prior information such as thermal properties of materials, characteristic time or reference area, and such information is not known in many detection scenarios, or even if known, due to different material, shape and other parameters, the original prior information cannot guarantee the accuracy of detection. SUMMARY

[0004] (I) Technical problems to be solved

[0005] The present application aims to at least partially solve at least one of the above technical problems.

[0006] (II) Technical solutions

[0007] In order to achieve the above purpose, according to one aspect of the present application, a defect detection method based on a Bi-LSTM neural network is provided.

[0008] The defect detection method comprises:

[0009] Step A', obtaining N frames of infrared thermal images of the test surface of the workpiece to be tested changing with time during the cooling process;

[0010] Step C', obtaining a pulse thermography data sequence SEQ' of each pixel point on the test surface of the workpiece to be tested changing with time from the N frames of infrared thermal images;

[0011] Step G, respectively bringing the pulse thermography data sequence SEQ' of each pixel point changing with time into the trained Bi-LSTM neural network to obtain the defect condition of the corresponding position of each pixel point.

[0012] In some embodiments of the present application, 2≤W≤5, and the step C' further comprises: dividing the pulse thermal imaging data sequence SEQ' into W segments in time sequence, and obtaining W pulse thermal imaging data subsequences with a dimension of 1×N / W; and in the step G, the step of bringing the pixel point pulse thermal imaging data sequence SEQ' into the trained Bi-LSTM neural network comprises: bringing the W pulse thermal imaging data subsequences corresponding to the pulse thermal imaging data sequence SEQ' into the trained Bi-LSTM neural network.

[0013] In some embodiments of the present application, the step G comprises: when the defect depth of the pixel point corresponding position is greater than a depth threshold h th , determining that the pixel point corresponding position is a non-defect, wherein h th ≥0.9h, and h is the thickness of the corresponding position of the workpiece to be measured.

[0014] In some embodiments of the present application, in the step C', the pulse thermal imaging data sequence of all pixel points on the test surface of the workpiece to be measured is obtained; the step G comprises: obtaining the defect condition of the corresponding position of all pixel points on the test surface of the workpiece to be measured; and the step G further comprises: a step H of reconstructing the three-dimensional structure of the defects of the workpiece to be measured by using the mesh function in the Matlab software and the defect condition of the corresponding position of all pixel points.

[0015] In some embodiments of the present application, the step A' comprises: a sub-step A'1 of obtaining a background infrared thermal image of the test surface of the workpiece to be measured before heating by using an infrared thermal imager; a sub-step A'2 of heating the workpiece to be measured by using a high-energy flash lamp and obtaining N frames of original infrared thermal images of the test surface of the workpiece to be measured in the cooling process by using the infrared thermal imager; and a sub-step A'3 of obtaining N frames of infrared thermal images of the test surface of the workpiece to be measured in the cooling process by deducting the background infrared thermal image before heating from the N frames of original infrared thermal images in the cooling process.

[0016] In order to achieve the above-mentioned purpose, according to a second aspect of the present application, a training method based on a Bi-LSTM neural network is also provided.

[0017] The training method based on the Bi-LSTM neural network is used for training the Bi-LSTM neural network, and comprises:

[0018] A step A of obtaining N frames of infrared thermal images of the test surface of a training workpiece in a cooling process, wherein the training workpiece has the same thermal properties as the workpiece to be measured, the training workpiece has M defect regions with different depths, the depth of each defect region is known, and M≥2;

[0019] Step C, obtaining a pulse thermal imaging data sequence SEQ of each pixel point of the training workpiece test surface changing with time from the N frames of infrared thermal images;

[0020] Step D, constructing a defect training data set from the pulse thermal imaging data sequence of each pixel point of the defect area changing with time and the corresponding defect depth;

[0021] Step E, constructing a non-defect training data set from the pulse thermal imaging data sequence of each pixel point of the non-defect area changing with time and the corresponding defect depth;

[0022] Step F, training a Bi-LSTM neural network regression model by using the data in the defect training data set and the non-defect training data set, and obtaining a trained Bi-LSTM neural network.

[0023] In some embodiments of the present application, the defect training data set is: wherein S is the number of pixel points in the M defect areas, s = 1, 2, …, S, SEQ D,s is a pulse thermal imaging data sequence of the s-th pixel point in the defect area of the training workpiece test surface changing with time in N frames of infrared thermal images, is the normalized defect depth of the s-th pixel point at the corresponding position of the training workpiece:

[0024]

[0025] wherein D D,s is the physical depth of the s-th pixel point at the corresponding position of the training workpiece, D mean , D max , D min represents the average value, maximum value and minimum value of the depth corresponding to all thermal signals in the training set;

[0026] The non-defect data set is: T is the number of pixel points in the non-defect area, t = 1, 2, …, T, SEQ D,t is a pulse thermal imaging data sequence of the t-th pixel point in the non-defect area of the training workpiece test surface changing with time in N frames of infrared thermal images, is the normalized defect depth of the non-defect area of the training workpiece:

[0027]

[0028] wherein D U is the thickness of the non-defect area of the training workpiece.

[0029] In some embodiments of the present application, the non-defect region is an annular region outside the defect region, and the number of pixels T of the non-defect region satisfies: 0.5S≤T≤1.5S.

[0030] In some embodiments of the present application, the step C further comprises: dividing the pulse thermal imaging data sequence with a dimension of 1×N into W sub-sequences with a dimension of 1×N / W in time sequence, 2≤W≤5; in each training of the step F, the W sub-sequences with a dimension of 1×N / W are independently input into the Bi-LSTM neural network for training.

[0031] In some embodiments of the present application, the step C further comprises: a step B of acquiring a visible light digital photo of the test surface of the training workpiece; performing image segmentation on the visible light digital photo to acquire a binary digital photo of the defect region of the training workpiece; registering the binary digital photo of the defect region with the photo of the infrared thermal image to acquire the defect region digital mask map; and selecting an annular region as the non-defect region outside the defect region of the binary digital photo of the defect region after registration to acquire the non-defect region digital mask map.

[0032] In some embodiments of the present application, the step C comprises: multiplying each frame of the infrared thermal image with the defect region digital mask map to screen out each pixel of the defect region, wherein the pixel value of the defect region in the defect region digital mask map is 1, and the pixel value of other regions is 0; and multiplying each frame of the infrared thermal image with the non-defect region digital mask map to screen out each pixel of the non-defect region, wherein the pixel value of the non-defect region in the non-defect region digital mask map is 1, and the pixel value of other regions is 0.

[0033] In some embodiments of the present application, each training in the step F comprises: the forward LSTM network outputs a feature vector with a dimension of 1×H as O f , the backward LSTM network outputs a feature vector with the same dimension as O b , the two vectors are spliced to obtain a feature vector with a dimension of 1×2H, O is multiplied by a learning matrix with a dimension of 2H×1 to obtain a normalized depth regression value y gp , the normalized depth regression value y gp is transformed into a physical scale regression value y gm : y gm =y gp ×(D max -D min )+D mean , and a distance loss F g is calculated from the physical scale regression value and the corresponding true defect depth value y L1:F L1 =|y gp -y g |;The distance loss F L1 The data is fed into the Bi-LSTM neural network regression model and trained until the distance loss F is reached. L1 It is below the preset threshold.

[0034] To achieve the above objectives, according to a second aspect of the invention, an apparatus is also provided. The apparatus includes: a digital memory; and a processor electrically coupled to the digital memory, configured to execute the defect detection method described above, or the training method described above, based on instructions stored in the digital memory.

[0035] (III) Beneficial Effects

[0036] As can be seen from the above technical solution, the present invention has at least one of the following beneficial effects:

[0037] (1) The Bi-LSTM neural network is trained using the training dataset of the training workpiece with the same thermal properties as the workpiece to be tested. It does not require prior information such as material thermal properties, reference area or feature time. This facilitates detection while ensuring the accuracy of each defect detection.

[0038] Meanwhile, the Bi-LSTM neural network is trained using pulsed thermal imaging data sequences during the cooling process. These pulsed thermal imaging data sequences do not require any algorithmic processing (such as Fourier transform algorithms, principal component analysis algorithms, etc.) nor do they require manual extraction of feature data for network training. The Bi-LSTM neural network can achieve fully automated data analysis, making it more convenient to use.

[0039] (2) The pulsed thermal imaging data sequence is divided into different sub-sequences, and the length dependence between the signals in each interval is extracted using a bidirectional LSTM network. Experiments show that the length setting of the interval is sensitive, and this method of dividing the interval can help improve the accuracy of defect inversion.

[0040] (3) After obtaining the defect depth data of all pixels on the test surface of the workpiece to be tested, the three-dimensional structure of the internal defects of the workpiece to be tested can be reconstructed more intelligently by combining the mesh function of Matlab, which facilitates observation and use.

[0041] (4) In the process of predicting the depth of defects, non-defect data and defect data are used together as training data for the regression task to effectively predict the overall depth distribution of the workpiece to be tested.

[0042] (5) Considering the influence of transverse heat diffusion, the cooling sequence data of the corresponding pixels in the non-defect area adjacent to the defect is taken as the non-defect data set, and then the method of depth threshold value can effectively distinguish the defect area and the non-defect area. It has been proved in practice that compared with the non-defect area randomly selected, the model trained by the non-defect area adjacent to the defect is more accurate for the true defect range.

[0043] (6) The method combining the regression task and the depth threshold value effectively shortens the time of three-dimensional structure reconstruction of the defect. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 It is a working principle diagram of Bi-LSTM.

[0045] Figure 2 It is a network structure diagram of LSTM.

[0046] Figure 3A It is a flowchart of the training method based on Bi-LSTM neural network of the present application.

[0047] Figure 3B It is a flowchart of the defect detection method based on Bi-LSTM neural network of the present application.

[0048] Figure 4 It is a schematic diagram of the pulse infrared thermal imaging experiment of the training workpiece and the workpiece to be tested.

[0049] Figure 5 It is a flowchart of the normalization preprocessing process of the pulse thermal imaging data sequence in the cooling process in the Bi-LSTM network.

[0050] Figure 6 It is a framework diagram of the Bi-LSTM network predicting the depth of the defect.

[0051] Figure 7A It is a three-dimensional structure diagram of the carbon fiber flat-bottom hole standard training workpiece.

[0052] Figure 7B It is a three-dimensional structure diagram of the carbon fiber flat-bottom hole test workpiece.

[0053] Figure 8 It is a cooling data sequence diagram corresponding to the depth distribution of the present application in turn 1.1mm, 1.64mm, 2.19mm, 2.81mm, 10mm area.

[0054] Figure 9 It is a thermal map of the carbon fiber flat-bottom hole training workpiece 8s after flashing.

[0055] Figure 10 It is a thermal map of the carbon fiber flat-bottom hole test workpiece 8s after flashing.

[0056] Figure 11 A digital mask map A of defect regions of the workpiece for training.

[0057] Figure 12 A digital mask map B of non-defect regions of the workpiece for training.

[0058] Figure 13 A defect depth prediction map of the test workpiece.

[0059] Figure 14 A non-defect depth prediction map of the test workpiece.

[0060] Figure 15 A defect region mask map C of the test workpiece.

[0061] Figure 16 A flat-bottom hole defect three-dimensional structure map of the test workpiece. DETAILED DESCRIPTION

[0062] The present application is based on a Bi-LSTM neural network algorithm, a regression model of the Bi-LSTM neural network is trained by inputting a pulse thermal imaging data sequence, a defect prediction depth map and a non-defect prediction depth map are automatically generated by using the regression model and a depth threshold method obtained by training, and finally a three-dimensional structure of a defect is reconstructed.

[0063] First, before specifically introducing the embodiments of the present application, the Bi-LSTM neural network is briefly described.

[0064] The Bi-LSTM neural network is one of the recurrent neural networks, which has the advantage of being able to describe the forward and backward dependence between signals, and is very effective for processing sequence information. The Bi-LSTM neural network is to train two LSTMs at the same time, the parameters of the two LSTMs are independent, and the directions are opposite. Finally, the feature vectors extracted by the two LSTMs are spliced, so that the past and future features can be captured. The working principle of the Bi-LSTM is as shown in Figure 1 , O f and O b represent the feature vectors extracted by the forward LSTM and the reverse LSTM respectively. The LSTM sets up a gating mechanism, including a forgetting gate, an input gate and an output gate, and its structure is as shown in Figure 2 . The gating mechanism controls the signal transmission state, so as to selectively remember the important information features. In the figure, ⊙ represents the multiplication of the corresponding elements in the feature matrix, represents the addition of the corresponding elements in the feature matrix, and σ and tanh represent the sigmoid and tanh nonlinear activation functions respectively. The main calculations involved are as shown in formulas (1) to (6). The forgetting gate selectively forgets the output c f of the previous node through the function z t-1 , W fwith b f The weights and biases needed to learn for the forget gate. The input gate is controlled by the function z i to preserve the current moment c t The state information that needs to be stored, W i with b i The weights and biases needed to learn for the input gate. The output gate is controlled by the function z o to determine the current neural node c t The information needed to output to the hidden state h t , W o with b o The weights and biases needed to learn for the input gate, and also need to learn the network parameters W c and b c . In short, this unique gating mechanism is very suitable for processing sequence data, not only can learn the long-term dependence information between data, but also to a certain extent, solve the standard recurrent neural network in feature extraction will encounter the problem of gradient disappearance and gradient explosion.

[0065]

[0066] z f = σ(W f · x + b f ) (2)

[0067] z i = σ(W i · x + b i ) (3)

[0068] z o = σ(W o · x + b o ) (4)

[0069] c t = z f ⊙ c t-1 + z i ⊙ tanh(W c · x + b c ) (5)

[0070] h t = z o ⊙ tanh(c t ) (6)

[0071] Those skilled in the art can understand that the Bi-LSTM neural network is to train the forward LSTM network and the backward LSTM network simultaneously, the middle hidden layer units of the two LSTM networks can affect each other and promote each other. Then the feature vectors extracted by the forward LSTM network and the backward LSTM network are spliced, so that the global features of the sequence data can be captured. In addition, compared with a single LSTM network, the Bi-LSTM neural network can avoid the loss of previous signal features, infer the long-short term dependence relationship between the signal value at each time and the global signal value, and finally extract rich global data features, which can improve the accuracy of the regression task.

[0072] Secondly, the training part and the detection part involved in the application are described as follows.

[0073] The technical scheme of the application is divided into two parts, the first part is the training part, and the second part is the detection part. Those skilled in the art should understand that after the Bi-LSTM neural network is trained, it can be directly used for testing, therefore, the detection part can be implemented independently of the training part. Similarly, the training part can also be implemented independently of the detection part. In summary, the implementation of the application is either to implement the detection part independently, or to implement the training part independently, or to implement the detection part and the training part together, all of which are within the protection scope of the application. In the specific description part of the following embodiments, the similar technical contents of the training part and the detection part will be mainly described with respect to the differences, and the same contents will not be described in detail hereinafter, and those skilled in the art can refer to the foregoing contents.

[0074] In order to make the purpose, technical scheme and advantages of the application more clear and obvious, the application is further described in detail below in combination with specific embodiments and with reference to the drawings.

[0075] Figure 3A The flow chart of the training method based on the Bi-LSTM neural network of the application. Figure 3B The flow chart of the defect detection method based on the Bi-LSTM neural network of the application. The application is described below in combination with Figure 3A and Figure 3B .

[0076] I. Training part

[0077] Please refer to Figure 3A , the training method of the Bi-LSTM neural network of the application comprises:

[0078] Step A, obtaining N frames of infrared thermal images of the test surface of the training workpiece changing with time during the cooling process, wherein the training workpiece has the same thermal properties as the workpiece to be tested, the training workpiece has M defect regions with different depths, the depth of each defect region is known, and M≥2;

[0079] A training workpiece with the same thermal properties as the workpiece to be tested is prepared, and defects of a certain size and multiple depths are designed in the training workpiece. Moreover, the defect depth range of the training workpiece covers the defect depth range of the workpiece to be tested.

[0080] Figure 4 A schematic diagram of the pulse infrared thermal imaging experiment of the training workpiece and the workpiece to be tested is shown in FIG. 1. As shown in FIG. 1, the flash lamp and the infrared thermal imager are placed on the same side of the test piece. Before and after the surface of the test piece is heated by the flash, the infrared thermal imager monitors and records the thermal wave time sequence signal of the surface of the test piece changing with time in real time. Figure 4

[0081] Please refer to Figure 3A and Figure 4 , step A of the training part further comprises:

[0082] Sub-step A1, acquiring the background infrared thermal image of the test surface of the training workpiece before heating by using the infrared thermal imager;

[0083] Sub-step A2, heating the training workpiece by using the high-energy flash lamp, and acquiring N frames of original infrared thermal images of the test surface of the training workpiece changing with time during the cooling process by using the infrared thermal imager;

[0084] Sub-step A3, deducting the background infrared thermal image before heating from the N frames of original infrared thermal images during the cooling process to acquire N frames of infrared thermal images of the test surface of the training workpiece changing with time.

[0085] The specific operation process is as follows: (1) a training workpiece with the same thermal properties as the workpiece to be tested is prepared, which is used to train the regression model in the Bi-LSTM network. In addition, a workpiece to be tested is also prepared to evaluate the trained regression model and predict the depth distribution of the workpiece to be tested. The training workpiece and the workpiece to be tested are both designed with defects of a certain size and multiple depths, and the defect depth range of the training workpiece covers the defect depth range of the workpiece to be tested; (2) before pulse excitation, the infrared thermal imager is used to collect 10 frames of background infrared thermal image sequences of the training workpiece or the workpiece to be tested at a preset acquisition frequency, and store them in a general memory; (3) while the training workpiece and the workpiece to be tested are being pulse excited, the infrared thermal imager is used to acquire N frames of original infrared thermal images of the test piece surface at a preset acquisition frequency and acquisition time, and store them in a general memory. For deeper defects, the preset acquisition frequency can be appropriately reduced, and the acquisition time can be appropriately increased, and vice versa, but attention should be paid to the total amount of data collected not exceeding the memory of the thermal imager; (4) each frame of original infrared thermal image of the training workpiece and the workpiece to be tested acquired in step 3 is deducted from the average thermal image of the background infrared thermal image sequence without pulse excitation acquired in step 2 to acquire N frames of infrared thermal images of the training workpiece and the workpiece to be tested. ​

[0086] Those skilled in the art should understand that in step A' of the test step, the step of obtaining N frames of infrared thermal images is similar to this, and subsequent details will not be described.

[0087] Step B, obtaining a defect area digital mask image and a non-defect area digital mask image of the training workpiece;

[0088] The defect area digital mask image is a template for screening defect area pixels from the infrared thermal image. As the name implies, the corresponding area is the area with defects on the training workpiece.

[0089] Specifically, step B further comprises:

[0090] Sub-step B1, obtaining a visible light digital photo of the test surface of the training workpiece;

[0091] Sub-step B2, performing image segmentation on the visible light digital photo to obtain a defect area binary digital photo of the test surface of the training workpiece;

[0092] Sub-step B3, registering the defect area binary digital photo with the photo of the infrared thermal image to obtain the defect area digital mask image;

[0093] Sub-step B4, selecting a ring-shaped area outside the defect area of the defect area two-dimensional digital photo after registration as the non-defect area to obtain the non-defect area digital mask image.

[0094] It should be particularly noted that the non-defect area in the present application can be all the areas on the training workpiece that do not have defects. However, the applicant found that training the Bi-LSTM neural network with the "non-defect area" including all non-defect areas not only increases the amount of calculation, but more importantly, the training effect of the Bi-LSTM neural network is not ideal.

[0095] Therefore, in the preferred embodiment of the present application, a ring-shaped area outside the defect area is selected as the "non-defect area", and the pixel-related data included therein is used to train the Bi-LSTM neural network, greatly improving the training effect. However, it should be noted that the number of pixel points of the preferred "non-defect area" and "defect area" is preferably not significantly different. In summary, it needs to meet: 0.5S ≤ T ≤ 1.5S, where T is the number of pixel points of the non-defect area, and S is the number of pixel points of the defect area.

[0096] In the process of obtaining the non-defect data set, the influence of the lateral heat diffusion on the prediction of the defect size is considered, the temperature drop data sequence of the pixel points corresponding to the annular non-defect area adjacent to the flat-bottom hole defect is taken as the non-defect training data set, so that the network can better learn the difference between the pulse thermal imaging data signal corresponding to the defect area and the pulse thermal imaging data signal corresponding to the non-defect area, especially the pulse thermal imaging data signal corresponding to the defect edge area.

[0097] Step C, obtaining the pulse thermal imaging data sequence SEQ of each pixel point on the test surface of the training workpiece with N frames of infrared thermal images;

[0098] Specifically, step C specifically includes:

[0099] Sub-step C1, multiplying each frame of infrared thermal image with a defect area digital mask image to screen out each pixel point in the defect area, wherein the pixel value of the defect area in the defect area digital mask image is 1, and the pixel value of other areas is 0;

[0100] Sub-step C2, multiplying each frame of infrared thermal image with a non-defect area digital mask image to screen out each pixel point in the non-defect area, wherein the pixel value of the non-defect area in the non-defect area digital mask image is 1, and the pixel value of other areas is 0.

[0101] In addition, in step C, the pulse thermal imaging data sequence of each pixel point in the training workpiece is standardized to facilitate the feature extraction of the Bi-LSTM network to the signal. As shown in Figure 5 each pixel point corresponding to the pulse thermal imaging data sequence has a dimension of 1xN, and N represents the number of frames of the signal. The pulse thermal imaging data sequence is evenly divided into W segments of pulse thermal imaging data subsequences with a dimension of 1xN / W according to the time sequence, and W represents the number of segments manually divided by the pulse thermal signal. Among them, 2≤W≤5. Experiments prove that W=2 is particularly good.

[0102] It should be particularly noted that the present application adopts the general way in the art to represent vectors or matrices, for example: as above, the thermal imaging data sequence has a dimension of 1xN, that is, the thermal imaging data sequence is a vector with 1 row and N columns; and then the pulse thermal imaging data sequence is evenly divided into W segments of pulse thermal imaging subsequences with a dimension of 1xN / W, that is, the pulse thermal imaging subsequence is a vector with 1 row and N / W columns. In the subsequent training and detection process, the representation of vectors and matrices is the same as this.

[0103] Experiments show that the pulse thermal imaging data sequence is divided into different subsequences, and the long and short dependence between the signals of each subsequence is extracted by using a bidirectional LSTM network. Experiments show that the length of the subsequence has a certain sensitivity, and this way of dividing the subsequence can help improve the accuracy of defect inversion. In a specific embodiment of the present application, the length of the subsequence has a certain sensitivity, and in this experiment, the subsequence length is 1800, which is divided into two segments to obtain the best effect (each interval length is 900). It can be inferred that as the number of frames of the pulse thermal imaging data sequence increases, the number of subsequences should be appropriately increased.

[0104] Step D, constructing a defect training data set from the pulse thermal imaging data sequence of each pixel point in the defect area changing with time and the corresponding defect depth: Wherein, S is the number of pixel points in the M defect areas, s=1, 2, …, S, SEQ D,s is the pulse thermal imaging data sequence of the s-th pixel point in the defect area of the test surface of the workpiece under training in N frames of infrared thermal images changing with time, is the normalized defect depth of the s-th pixel point at the corresponding position of the workpiece under training:

[0105]

[0106] Wherein, D D,s is the physical depth of the s-th pixel point at the corresponding position of the workpiece under training, D mean , D max , D min represents the average, maximum and minimum of the depth corresponding to all thermal signals in the training set;

[0107] Those skilled in the art should understand that before applying the bidirectional LSTM network for training, the depth value D corresponding to the pulse thermal imaging data sequence of each pixel point is recorded, in order to train a robust depth regression model, the depth value D needs to be normalized to a value between 0 and 1, which is represented as y d , D mean , D max , D min represents the average, maximum and minimum of the depth corresponding to all thermal signals in the training set.

[0108] Step E, constructing a non-defect training data set from the pulse thermal imaging data sequence of each pixel point in the non-defect area changing with time and the corresponding defect depth:

[0109] T is the number of pixel points in the non-defect area, t=1, 2, …, T, SEQ D,tto train the pulse thermal imaging data sequence of the tthpixel point in the non-defect area of the workpiece test surface in N frames of infrared thermal images changing with time, to train the normalized defect depth of the non-defect area of the workpiece:

[0110]

[0111] wherein D U is the thickness of the non-defect area of the workpiece;

[0112] Step F, training the Bi-LSTM neural network regression model by using the data in the defect training data set and the non-defect training data set, and obtaining the trained Bi-LSTM neural network.

[0113] In each training of the step F, the pulse thermal imaging data sub-sequence with the dimension of 1×N / W of the W segments is independently input into the Bi-LSTM neural network for training.

[0114] The Bi-LSTM neural network automatically learns the front-back relationship dependence between the W segment signals in the single-point pulse thermal imaging data sequence. As shown in the formula 5. Figure 6 In the step E, each training in the step F includes:

[0115] Sub-step F1, the feature vector with the dimension of 1×H output by the forward LSTM network is O f , and the feature vector with the same dimension output by the backward LSTM network is O b The two vectors are spliced to obtain a feature vector with the dimension of 1×2H.

[0116] Sub-step F2, in the Bi-LSTM neural network training process, the feature vector with the dimension of 1×2H is multiplied by a learning matrix with the dimension of 2H×1 to obtain a normalized depth regression value y gp ;

[0117] Sub-step F3, in the training process, the distance loss F L1 is used to measure the gap between the true depth y g and the predicted depth value y gp . And in the test process, since the recorded depth value y g is normalized when the training sample is made, the predicted depth value y gp of the defect needs to be transformed to y gm in the real scale. The calculation method of y g transformed to y gm is shown in the formula 8:

[0118] F L1 = |ygp -y g | (7)

[0119] y gm =y gp ×(D max -D min )+D mean (8)

[0120] The distance loss F L1 is brought into the Bi-LSTM neural network regression model for continuous training until the distance loss F L1 is lower than a preset threshold.

[0121] In the above training process, the learning matrix is a matrix that needs to be updated in the training process of the deep learning network in the regression task. The specific learning process is to initialize the learning matrix at the beginning of the training, calculate the loss value by the loss function in each iteration during the training, and automatically learn the parameters in the optimization process according to the gradient descent method in the back propagation of the network. Among them, the gradient descent is the default method for updating the network parameters of the deep neural network. In simple terms, it is to calculate the loss once and update the network parameters once in each training iteration. The learning matrix updating process and the updating process of the Bi-LSTM main feature extraction network are the same. These are known to those skilled in the art and will not be described in detail.

[0122] At this point, the Bi-LSTM neural network regression model and the Bi-LSTM neural network classification model are obtained, and the training process is completed.

[0123] II. Detection part

[0124] It should be noted that some contents of the detection part are the same as those of the training part. In this part, these contents will not be described in detail, and the focus is on the special features of the detection part.

[0125] Please refer to Figure 3B The defect detection method based on the Bi-LSTM neural network of the present application comprises:

[0126] Step A', obtaining N frames of infrared thermal images of the test surface of the workpiece to be tested changing with time during the cooling process;

[0127] This step A' is generally similar to the previous step A, and further comprises:

[0128] Sub-step A'1, acquiring the background infrared thermal image of the test surface of the workpiece to be tested before heating by using the infrared thermal imager;

[0129] Sub-step A'2, heating the workpiece to be tested by using the high-energy flash lamp, and acquiring N frames of original infrared thermal images of the test surface of the workpiece to be tested changing with time during the cooling process by using the infrared thermal imager;

[0130] Sub-step A'3, subtracting the background infrared thermal image before heating from the N frames of original infrared thermal images in the cooling process to obtain N frames of infrared thermal images of the test surface of the workpiece to be tested changing with time in the cooling process.

[0131] Step C', obtaining a pulse thermal imaging data sequence SEQ' of each pixel point on the test surface of the workpiece to be tested changing with time from the N frames of infrared thermal images;

[0132] The pulse thermal imaging data sequence SEQ' is evenly divided into W segments of pulse thermal imaging data subsequences with a dimension of 1xN / W in time sequence. Preferably, the pulse thermal imaging data sequence of all pixel points on the test surface of the workpiece to be tested changing with time is obtained, so that the three-dimensional structure of the defect can be reconstructed subsequently.

[0133] Step G, bringing the pulse thermal imaging data sequence SEQ' of each pixel point changing with time into the trained Bi-LSTM neural network respectively to obtain the defect condition of the corresponding position of each pixel point.

[0134] In the step G, the step of bringing the pulse thermal imaging data sequence SEQ' of each pixel point changing with time into the trained Bi-LSTM neural network includes: bringing the W segment pulse thermal imaging data subsequences corresponding to the pulse thermal imaging data sequence SEQ' into the trained Bi-LSTM neural network.

[0135] In addition, in order to improve the depth of defect judgment as much as possible, the present application also sets a depth threshold. Specifically, step G further includes: when the defect depth of the corresponding position of the pixel point obtained is greater than the depth threshold h th , it is determined that the corresponding position of the pixel point is a non-defect, wherein h th ≥0.9h, h is the thickness of the corresponding position of the workpiece to be tested. In this embodiment, the method of combining the regression task and the depth threshold effectively shortens the time of reconstructing the three-dimensional structure of the defect.

[0136] Step H, using the mesh function in the Matlab software to reconstruct the three-dimensional structure of the defect of the workpiece to be tested from the defect conditions of all pixel points.

[0137] In the present application, after obtaining the defect depth data of all pixel points on the test surface of the workpiece to be tested, the three-dimensional structure of the internal defect of the workpiece to be tested is reconstructed more intelligently by combining the mesh function of Matlab, which facilitates observation and use. However, it should be understood by those skilled in the art that obtaining the three-dimensional structure of the defect is only more intuitive in expression, and if it is only used for subsequent data processing, the defect distribution condition of the workpiece to be tested obtained in step H can be ended.

[0138] As can be seen from the above description of the defect detection method of this invention, training the Bi-LSTM neural network using a training dataset of a training workpiece with the same thermal properties as the workpiece under test eliminates the need for prior information such as material thermal properties, reference regions, or feature times. This facilitates detection while ensuring the accuracy of each defect detection. Furthermore, training the Bi-LSTM neural network using pulsed thermal imaging data sequences during the cooling process requires no algorithmic processing (such as Fourier transform or principal component analysis) or manual extraction of feature data for network training. The Bi-LSTM neural network enables fully automated data analysis, making it more convenient to apply.

[0139] III. Computer Devices

[0140] According to another aspect of the present invention, a computer device is also provided. The computer device includes: a digital memory; and a processor electrically coupled to the digital memory, configured to execute the training method described above based on instructions stored in the digital memory.

[0141] According to another aspect of the present invention, a computer device is also provided. The computer device includes: a digital memory; and a processor electrically coupled to the digital memory, configured to execute the defect detection method described above based on instructions stored in the digital memory.

[0142] IV. Examples of Training and Defect Detection

[0143] Figure 7A This is a design drawing for a carbon fiber flat-bottom hole training workpiece. The dimensions of the training workpiece are 200mm in length, 16mm in width, and 10mm in height. Eight flat-bottom hole defects with different aspect ratios were pre-designed in the test piece. The width of the flat-bottom hole defects is 18mm, and the depths of the flat-bottom hole defects are 2.97mm, 2.81mm, 2.55mm, 2.19mm, 1.97mm, 1.64mm, 1.35mm, and 1.11mm, respectively.

[0144] Figure 7B This is the design drawing of the carbon fiber flat-bottom hole test piece. The dimensions of the test piece are the same as those of the training piece. Six flat-bottom holes with different width-to-depth ratios were pre-designed in the test piece. The width of the flat-bottom hole defects was 20 mm, and the depths of the flat-bottom hole defects were 2.5 mm, 2.22 mm, 1.94 mm, 1.67 mm, 1.35 mm, and 1.1 mm, respectively.

[0145] Please refer to Figure 4During the experiment, a high-energy flash lamp (flash duration 2ms, output energy 9.6KJ) and a FLIR SC3000 infrared thermal imager (wavelength range 8um-9um, pixel 320×240, temperature sensitivity 0.03℃) were placed on the same side of the workpiece. The thermal imager's acquisition frequency was set to 60Hz and the acquisition time to 30s. Before and after the flash excitation of the specimen surface, the thermal imager monitored and recorded the cooling data sequence of the specimen surface over time in real time. The cooling data sequences corresponding to areas with depths of 1.1mm, 1.64mm, 2.19mm, 2.81mm, and 10mm are shown below. Figure 8 As shown. Thermal images of the training workpiece and the workpiece under test 8 seconds after flash lamp excitation, as shown. Figure 9 and Figure 10 As shown.

[0146] Before training the Bi-LSTM network regression model, it is necessary to accurately obtain the flat-bottomed hole defect dataset and the non-defect dataset from the training workpiece as the training dataset. The details of the dataset preparation are as follows: We first obtain a visible light image of a carbon fiber flat-bottomed hole training workpiece, then process it using an image segmentation algorithm and register it with the infrared image to obtain the digital mask image A of the defect region, as shown below. Figure 11 As shown, the pixel value corresponding to the non-defect area is 0, and the pixel value corresponding to the defect area is 1. Each frame of the heat map of the training workpiece is multiplied with the digital mask A of the defect area to obtain the flat-bottom hole defect dataset of the training workpiece. From this dataset, datasets with depths of 1.11mm, 1.35mm, 1.64mm, 1.97mm, 2.19mm, 2.55mm, 2.81mm, and 2.97mm are classified, resulting in eight sets of defect datasets at different depths. In obtaining the non-defect dataset, considering the influence of lateral heat diffusion on the predicted defect size, we use the cooling data sequence of pixels corresponding to the non-defect areas adjacent to the flat-bottom hole defect as the non-defect training dataset. This allows the network to better learn the difference between the heat signals corresponding to the defect areas and the non-defect areas (especially the heat signals corresponding to the defect edge areas). Figure 12 A digital mask B for the non-defect region is shown, where the pixel value within the annular region is 1, and the pixel value in other regions is 0. Multiplying each frame of the heatmap of the training workpiece by the mask B yields a non-defect dataset with a depth of 10 mm. Finally, eight sets of defect datasets at different depths and one set of non-defect datasets are input together into a Bi-LSTM network for regression learning.

[0147] During the regression task, the data input parameters were set as follows: the number of frames for the pulse time-series signal N = 1800, and the number of segments for each group of pulse thermal imaging data signals W = 2. The hyperparameters of the Bi-LSTM network were set as follows: the feature dimension of the hidden layer nodes was 128, the number of layers in the Bi-LSTM network was 2, the random loss ratio was 0.5, the learning rate was 0.001, and the batch size was 64.

[0148] During testing, the cooling data sequence of the carbon fiber flat-bottomed hole workpiece was input into a trained Bi-LSTM regression model to generate a depth prediction map of the workpiece. To distinguish between defective and non-defective regions in the workpiece, a depth threshold of 9.9 mm was set; regions greater than the threshold were considered non-defective, and those less than the threshold were considered defective. To clearly display the depth distribution of the workpiece, the depth of non-defective regions was set to 0 in the depth prediction map, generating a depth prediction map for defects in the flat-bottomed holes. Similarly, the depth of defective regions in the flat-bottomed holes was set to 0 in the depth prediction map, generating a depth prediction map for non-defective regions, as shown below. Figure 13 and Figure 14 As shown. To evaluate the accuracy of defect classification, the F1-score index is introduced. The closer the F1-score is to 1, the more accurate the classification between defective and non-defective areas. Its calculation method is shown in formulas (9)-(11):

[0149]

[0150] in:

[0151]

[0152] TP represents the number of pixels belonging to the defect region and also identified as a defect region; FP represents the number of pixels belonging to the non-defect region but identified as a defect region; and FN represents the number of pixels belonging to the defect region but identified as a non-defect region. To count the actual number of defective pixels in the workpiece under test, we use the mask image C of the workpiece under test as the standard. Its fabrication process is the same as that of the training workpiece mask image A, as follows: Figure 15 As shown.

[0153] Table 1 shows the predicted evaluation of the depth of six defects and non-defect depths in the carbon fiber flat-bottomed hole test piece. Table 2 shows the classification evaluation of the six flat-bottomed hole defects and the lateral dimensional error of the defects in the test piece. Figure 13 Importing the data into Matlab software, the `mesh` function can be used to reconstruct the 3D structure of the flat-bottom hole defect in the carbon fiber workpiece under test, such as... Figure 16 As shown.

[0154] Table 1: Prediction and Evaluation Table of Defect Depth and Non-Defect Depth of Six Flat-Bottomed Holes in Carbon Fiber Flat-Bottomed Hole Test Workpiece

[0155] 6 flat-bottomed well defects and non-defect areas Predicted depth (mm) True depth (mm) Error (%) A 1.15 1.1 4.5％ B 1.31 1.35 -3.0％ C 1.69 1.67 1.2％ D 1.93 1.94 -0.5％ E 2.07 2.22 -6.8％ F 2.45 2.5 -2％ Non-defect area 9.98 10 -0.2％

[0156] Table 2: Classification and evaluation of six flat-bottom hole defects in carbon fiber flat-bottom hole test workpieces, and lateral dimensional errors of flat-bottom hole defects.

[0157]

[0158] Table 1 shows that the depth prediction errors for both flat-bottomed hole defects and non-defect predictions are within 6.8%. Table 2 shows that the F1-scores for the six flat-bottomed hole defects in the workpiece are all above 0.84, indicating good classification performance. The difference between the predicted diameter of the flat-bottomed hole defect and the actual diameter of the flat-bottomed hole defect is within 2 pixels, with a maximum defect diameter prediction error of 8.3%. Figure 16 It can be observed that there are some burrs at the edge of each flat-bottomed hole defect (the predicted depth corresponding to the burrs has a large error with the actual depth). This may be due to the fact that the training data for each flat-bottomed hole defect is lower than the training data for the non-defect area. However, it has little impact on the predicted average depth of the entire flat-bottomed hole defect area. In engineering applications, mean-averaging can be used before reconstructing the three-dimensional structure of the flat-bottomed hole defect.

[0159] This concludes the description of the embodiments of the present invention.

[0160] The embodiments of the present invention have now been described in detail with reference to the accompanying drawings. Based on the above description, those skilled in the art should have a clear understanding of the present invention.

[0161] In summary, this invention provides a defect detection method, training method, and apparatus based on a Bi-LSTM neural network. It fully utilizes the advantages of the Bi-LSTM neural network and proposes innovative methods such as data sequence segmentation, local defect regions, and partial data training regression models for practical applications of defect detection. In practical use, it has many advantages such as convenience and high accuracy, and has strong practical value.

[0162] It should be noted that for some implementation methods, if they are not the key content of this invention and are well known to those skilled in the art, they are not described in detail in the accompanying drawings or the main text of the specification. In this case, they can be understood by referring to the relevant prior art.

[0163] Furthermore, the above embodiments are provided merely to enable the invention to meet legal requirements, and the invention can be implemented in many different forms and should not be construed as limited to the embodiments set forth herein. In addition, the definitions of the elements and methods described above are not limited to the various specific structures, shapes, or methods mentioned in the embodiments, and those skilled in the art can easily modify or substitute them.

[0164] Unless explicitly stated otherwise, the numerical parameters in the specification and claims of this invention may be approximate values ​​and can be changed according to the content of this invention. Specifically, all figures used in the specification and claims to indicate the content of composition, reaction conditions, etc., should be understood to be modified by the term "about" in all cases, which means that there may be variations of ±10% in some embodiments, ±5% in some embodiments, ±1% in some embodiments, and ±0.5% in some embodiments.

[0165] Furthermore, unless specifically described or required to occur in a specific order, the order of the above steps is not limited to those listed above and can be varied or rearranged according to the desired design. Moreover, the above embodiments can be used in combination with each other or with other embodiments based on design and reliability considerations; that is, technical features from different embodiments can be freely combined to form more embodiments.

[0166] The algorithms and displays provided herein are not related to any particular computer, virtual system, or other inherent device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, this invention is not directed to any particular programming language. It should be understood that the contents of this invention can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of this invention.

[0167] This invention can be implemented using hardware comprising several different elements and a suitably programmed computer. Various component embodiments of the invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the related devices according to embodiments of the invention. The invention can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such programs implementing the invention can be stored on a computer-readable medium or can take the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0168] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features of the invention in this specification (including the claims, abstract, and drawings) and all processes or units of any method or device so invented. Unless expressly stated otherwise, each feature of the invention in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose. Furthermore, in the unit claims enumerating several means, several of these means may be embodied by the same hardware item.

[0169] Similarly, it should be understood that, in order to simplify the invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof. However, this approach should not be construed as reflecting an intention that the claimed invention requires more features than expressly recited in each claim. Rather, as reflected in the claims, the various inventive aspects consist of fewer than all the features of the preceding single embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0170] The specific embodiments described above have provided a detailed explanation of the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A training method based on a Bi-LSTM neural network, characterized in that, Used for training Bi-LSTM neural networks, including: Step A: Obtain N frames of infrared thermal images showing the change of the test surface of the training workpiece over time during the cooling process. The training workpiece and the workpiece to be tested have the same thermal properties. The training workpiece has M defect regions with different depths. The depth of each defect region is known, and M≥2. Step C: Obtain the pulsed thermal imaging data sequence SEQ of the time-varying pixel points on the test surface of the training workpiece from the N frames of infrared thermal images; Step D: Construct a defect training dataset from the pulsed thermal imaging data sequence of each pixel in the defect area over time and the corresponding defect depth; Step E: Construct a non-defect training dataset from the pulsed thermal imaging data sequence of each pixel in the non-defect region as a function of time and the corresponding defect depth. Step F: Use the data from the defective training dataset and the non-defective training dataset to train the Bi-LSTM neural network regression model and obtain the trained Bi-LSTM neural network. The defect training dataset is as follows: Where S is the number of pixels in the M defect regions, s = 1, 2, ..., S, SEQ D,s To train the pulsed thermal imaging data sequence of the s-th pixel within the defect region of the workpiece test surface in N frames of infrared thermal images as a function of time, The normalized defect depth of the s-th pixel at the corresponding position on the training workpiece is: Among them, D D,s Let D be the physical depth of the s-th pixel at the corresponding position on the training workpiece. mean D max D min This represents the average, maximum, and minimum values ​​of all thermal signals at the corresponding depth in the training set; The non-defect training dataset is: T is the number of pixels in the non-defect region, t = 1, 2, ..., T, SEQ D,t To train the pulsed thermal imaging data sequence of the t-th pixel in the non-defect region of the workpiece test surface in N frames of infrared thermal images as a function of time, To train the normalized defect depth in the non-defect region of the workpiece: Among them, D U To train the thickness of the non-defect area of ​​the workpiece; Each training step in step F includes: the forward LSTM network outputs a feature vector of dimension 1×H with a length of O. f The inverse LSTM network outputs a feature vector of the same dimension of O. b The two vectors are concatenated to obtain a feature vector O with a dimension of 1×2H; the feature vector O with a dimension of 1×2H is multiplied by the learning matrix with a dimension of 2H×1 to obtain the normalized deep regression value y. gp Normalize the deep regression value y gp y transformed to physical scale regression value gm y gm =y gp ×(D max -D min )+D mean ; The regression value based on the physical scale and the corresponding true defect depth value y g Calculate distance loss F L1 :F L1 =|y gp -y g |;The distance loss F L1 The data is fed into the Bi-LSTM neural network regression model and trained until the distance loss F is reached. L1 It is below the preset threshold.

2. The training method according to claim 1, characterized in that: The non-defect region is a ring-shaped region outside the defect region, and the number of pixels T in the non-defect region satisfies: 0.5S≤T≤1.5S; and / or Step C further includes: dividing the 1×N pulsed thermal imaging data sequence into W segments of 1×N / W pulsed thermal imaging data subsequences in chronological order, where 2≤W≤5; in each training step F, the W segments of 1×N / W pulsed thermal imaging data subsequences are independently input into the Bi-LSTM neural network for training.

3. The training method according to claim 1, characterized in that: Before step C, the procedure includes: Step B, acquiring a visible light digital photograph of the test surface of the training workpiece; performing image segmentation on the visible light digital photograph to obtain a binary digital photograph of the defect region on the test surface of the training workpiece; registering the binary digital photograph of the defect region with an infrared thermal image to obtain a digital mask of the defect region; selecting an annular region around the defect region in the registered binary digital photograph of the defect region as the non-defect region, and obtaining a digital mask of the non-defect region. Step C includes: multiplying each frame of infrared thermal image with the digital mask image of the defect area to filter out each pixel in the defect area, wherein in the digital mask image of the defect area, the pixel value of the defect area is 1 and the pixel value of other areas is 0; multiplying each frame of infrared thermal image with the digital mask image of the non-defect area to filter out each pixel in the non-defect area, wherein in the digital mask image of the non-defect area, the pixel value of the non-defect area is 1 and the pixel value of other areas is 0.

4. A training device based on a Bi-LSTM neural network, characterized in that, include: Digital memory; as well as A processor, electrically coupled to the digital memory, is configured to execute the training method as described in any one of claims 1 to 3 based on instructions stored in the digital memory.

Images