Impact damage identification method based on multi-modal data fusion and ensemble learning

Abstract

In order to improve the accuracy and reliability of impact damage identification based on machine learning, the invention provides impact damage identification based on multi-modal data fusion and ensemble learning. According to the method, multi-modal data including discrete impact damage characteristics, a time-domain impact signal sequence and a time-frequency image of an impact signal are adopted as input of a machine model, and impact damage information contained in an impact stress wave signal is fully mined; and the artificial neural network, the long-short-term memory network and the convolutional neural network are integrated by using a weight average technology, and impact damage is identified by fusing three distances including a cosine distance, a Manhattan distance and an Euclidean distance. The method is verified on a 400mm * 400mm composite material platy structure, and the result shows that the method can perform high-precision identification on the impact damage in the structure under the condition of using a small number of sensors.

Description

Technical Field

[0001] The present invention relates to the field of structural health monitoring, and in particular to an impact damage identification method based on multimodal data fusion and integrated learning. Background Art

[0002] Composite materials are widely used in engineering structures such as aerospace, rail transit and marine engineering due to their high specific strength and large specific modulus. However, during the manufacturing and service period, composite structures will inevitably suffer impact damage caused by unexpected conditions such as tool drops, sand and gravel impacts, and hail impacts, which significantly reduce the integrity and safety of composite structures. Therefore, real-time online impact damage monitoring of engineering composite structures is of great significance to maintaining the health and safety of the structures.

[0003] With the rapid development of computer technology and the rise of artificial intelligence, machine learning methods have become a research hotspot in various disciplines due to their super ability to solve various complex problems. In recent years, many impact damage identification methods based on machine learning have emerged. Commonly used machine learning models include artificial neural networks, support vector machines, extreme learning machines, convolutional neural networks, long short-term memory networks, and deep belief networks. However, the existing impact damage identification methods based on machine learning have the following problems: in terms of the selection of data sample sets, researchers basically use a single type of data (discrete impact damage characteristics, time domain impact signal sequences, or time-frequency images of impact signals) as the input of the model, which does not make full use of the rich impact damage information; in terms of model selection, researchers basically use a single machine learning model to identify impact damage, which has the risk of underfitting and overfitting, and does not consider the uncertainty of using different machine learning models for impact damage identification. Summary of the invention

[0004] The purpose of this application is to solve the problems existing in the prior art, improve the accuracy and reliability of impact damage identification, and propose an impact damage identification method based on data fusion and ensemble learning.

[0005] To this end, some embodiments of the present application propose an impact damage identification method based on multimodal data fusion and integrated learning, which includes step 1: collecting multiple non-destructive and different damage degree impact stress wave signals of the structure to be tested and labeling the impact stress wave signal according to the detection results of ultrasonic scanning; step 2: obtaining multiple categories of multimodal data of the impact stress wave signal and dividing the multimodal data into a training set and a test set; step 3: establishing different types of machine learning models for the multimodal data of each category; step 4: training the machine learning models of each type and using the relative accuracy index of the training results of the training sets of various multimodal data samples as the weights of each machine learning model, and integrating the three machine learning models according to the weights of each machine learning model to obtain an integrated learning model; step 5: inputting the test set into the integrated learning model, and classifying the damage by fusing at least two distances to measure the difference between the weighted average integrated prediction result and the actual label. ; The distance obtained is multiplied and fused to obtain a fusion distance, and the class with the smallest fusion distance is the impact damage identification result of the integrated learning model.

[0006] In some embodiments, the impact stress wave signal collection and labeling in step 1 specifically includes arranging a sparse piezoelectric sensor network on the structure to be tested, and collecting multiple non-destructive and impact stress wave signals of different damage degrees.

[0007] In some embodiments, the acquiring of multimodal data in step 2 specifically includes: the multimodal data includes discrete impact damage feature data, time domain impact signal sequence feature data or time-frequency image feature data of the impact signal.

[0008] In some embodiments, each type of multimodal data sample is divided into a training set and a test set in a ratio of 7:3.

[0009] In some embodiments, establishing multiple different types of machine learning models in step 3 specifically includes establishing an artificial neural network with discrete impact damage characteristics as input, a long short-term memory network with a time domain impact signal sequence as input, and a convolutional neural network with an impact signal time-frequency image as input.

[0010] In some embodiments, the machine learning model training and weighted average integration in step 4 specifically include training machine learning models with different training sets and Bayesian optimization algorithms, and performing weighted average integration on the machine learning models based on the error rate of the training results.

[0011] In some embodiments, the multi-distance measurement and damage classification in step 5 specifically includes inputting the test set into the integrated learning model, and performing damage classification by fusing the three distances of cosine distance, Manhattan distance and Euclidean distance to measure the difference between the integrated prediction result and the actual label.

[0012] Due to the adoption of the above technical solution, the beneficial technical effects brought about by the impact damage identification method based on data fusion and ensemble learning provided by the present invention include:

[0013] First, some embodiments of the present invention use multimodal data to identify impact damage, and fully exploit the impact damage information contained in the impact stress wave signal.

[0014] Second, some embodiments of the present invention integrate multiple machine learning models by weighted average to perform impact damage identification, which reduces the risk of underfitting and overfitting of a single model and has high accuracy.

[0015] Third, some embodiments of the present invention classify the final impact damage results by combining three distance measurements: cosine distance, Manhattan distance and Euclidean distance, and integrate these distances to improve the classification performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flowchart of an impact damage identification method based on multimodal data fusion and integrated learning according to an embodiment of the present invention.

[0018] Figure 2A Schematic diagram of an artificial neural network machine learning model according to an embodiment of the present invention.

[0019] Figure 2B Schematic diagram of a long short-term memory network machine learning model according to an embodiment of the present invention.

[0020] Figure 2C Schematic diagram of a convolutional neural network machine learning model according to an embodiment of the present invention.

[0021] Figure 3 Schematic diagram of a structure to be tested and a sensor network according to an embodiment of the present invention

[0022] Figure 4 a to Figure 4d are schematic diagrams of impact stress wave signals with different impact energies according to embodiments of the present invention.

[0023] Figure 5 a to Figure 5 d are schematic diagrams of detection results of ultrasonic C-scan according to an embodiment of the present invention.

[0024] Figure 6 a to Figure 6 d are schematic diagrams of time-frequency images of impact stress wave signals according to embodiments of the present invention.

[0025] Figure 7 is the error rate of different machine learning model training sets according to an embodiment of the present invention.

[0026] Figure 8 is the normalized confusion matrix of the test set according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the technical solutions and advantages of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention:

[0028] like Figure 1 The impact damage identification method based on multimodal data fusion and ensemble learning shown in FIG. specifically includes the following steps:

[0029] Step 1: collecting and marking impact stress wave signals;

[0030] Step 2: multimodal data acquisition and sample set division;

[0031] Step 3: Build multiple different types of machine learning models;

[0032] Step 4: Machine learning model training and weighted average integration;

[0033] Step 5, multi-distance measurement and damage classification;

[0034] The shock stress wave signal collection and labeling in step 1 specifically includes: arranging a sparse piezoelectric sensor network on the structure to be tested, and collecting shock stress wave signals under multiple different impact energies. The structure to be tested is scanned using an ultrasonic C-scan device, and the shock stress wave signal is labeled according to the detection result of the ultrasonic C-scan, for example: lossless is recorded as '[1 0 0 0]', slight damage is recorded as '[0 1 00]', moderate damage is recorded as '[0 0 1 0]', and severe damage is recorded as '[0 00 1]'.

[0035] The multimodal data acquisition and sample set division in step 2 specifically include: obtaining multimodal data samples: including, the first category: discrete impact damage feature data D100, such as fast Fourier transform to extract the center of gravity frequency, wavelet packet transform to extract the wavelet packet energy feature; the second category: the original time domain impact signal sequence feature data D200, and the third category: the time-frequency image feature data D300 of the impact signal obtained by continuous wavelet transform. For another example, the three types of data samples are divided into training set and test set according to the ratio of 7:3.

[0036] Step 3: Establish multiple different types of machine learning models, including: using Matlab software toolbox to establish Figure 2A As shown, the artificial neural network N100 that takes the first type of discrete impact damage feature data D100 as input will be Figure 2B The second type of time domain impulse signal sequence feature data D200 shown as the input long short-term memory network N200 and Figure 2C The convolutional neural network N300 uses the time-frequency image feature data D300 of the third type of impact signal as input.

[0037] Step 4, training and weighted average integration of the machine learning models specifically includes: inputting the training sets of various multimodal data samples D100, D200, and D300 into the input layers of the corresponding three machine learning models N100, N200, and N300, inputting the labels corresponding to each multimodal data sample D100, D200, and D300 into the output layer, and using the Bayesian optimization algorithm to train the three machine learning models N100, N200, and N300. After training, the three machine learning models are integrated using weighted average technology to obtain an integrated learning model, wherein the relative accuracy index of the training results of the training sets of various multimodal data samples D100, D200, and D300 is used as the weight of each machine learning model N100, N200, and N300, and the outputs of the three machine learning models N100, N200, and N300 are weighted averaged. The relative accuracy A of the kth machine learning model k The definitions are as follows (k=1 is the artificial neural network N100, k=2 is the long short-term memory network N200, k=3 is the convolutional neural network N300):

[0038]

[0039] Where: e max and e min is the maximum and minimum error rate of all machine learning models; e kis the error rate of the kth machine learning model, that is, the ratio of the number of samples with identification errors to the total number of samples. Therefore, the weight ω of each machine learning model k for:

[0040]

[0041] Assume x is the input sample, H k is the corresponding output of the k-th machine learning model, then the weighted average ensemble prediction result is as follows:

[0042]

[0043] The multi-distance measurement and damage classification of step 5 specifically includes: inputting the test set into the ensemble learning model, and classifying the damage by measuring the difference between the weighted average ensemble prediction result and the actual label by integrating the three distances of cosine distance, Manhattan distance and Euclidean distance. They represent the cosine distance, Manhattan distance, and Euclidean distance between the prediction result H and the jth damage category label y (j=1 is damageless, j=2 is slightly damaged, j=3 is moderately damaged, and j=4 is severely damaged), respectively. The calculation formula is as follows:

[0044] Cosine distance:

[0045] Manhattan distance:

[0046] Euclidean distance:

[0047] The above three distances are multiplied and fused to obtain the fusion distance F j , which can be expressed as follows:

[0048]

[0049] The category with the smallest fusion distance is the impact damage recognition result of the ensemble learning model.

[0050]

[0051] Example:

[0052] The monitoring object of this embodiment is as follows Figure 3The 400mm×400mm×3mm (length×width×thickness) T-300 reinforced resin-based composite material plate shown in the figure has a ply layer of [45 / -45 / 0 / 90] 2s. Four piezoelectric sensors P1, P2, P3, and P4 with a diameter of 16mm are pasted on the surface of the composite plate to monitor an area with a size of 200mm×200mm. The impact test was carried out using a drop hammer impact tester with a punch mass of 5.2kg. The impact energy was adjusted by adjusting the punch height. In the range of impact energy from 4J to 24J, the center of the plate was impacted every 2J. Figure 4 a. Figure 4 b. Figure 4 c and Figure 4 d shows the typical impact stress wave signals received by the P1 sensor under four different impact energies of 4J, 10J, 14J, and 18J. After each impact, the impact position of the composite plate is scanned using ultrasonic C-scanning, and the signal labels are annotated according to the detection results of the ultrasonic C-scanning, including four categories of labels: non-destructive, slightly damaged, moderately damaged, and severely damaged, as shown in Figure 4. Figure 5 a. Figure 5 b. Figure 5 c. Figure 5 After all the impact signals are obtained, the signal processing method described in the specific implementation manner is used to obtain a multimodal data set: including discrete feature data, the original time domain impact signal sequence and the impact signal time-frequency spectrum, such as Figure 6 a. Figure 6 b. Figure 6 c. Figure 6 d. In the experiment, 18 test pieces were used, and a total of 425 sets of data were obtained, including 175 sets of lossless data and 250 sets of lossy data. The training set and the test set were divided into a ratio of 7:3. Artificial neural networks, long short-term memory networks, and convolutional neural networks are established in the toolbox. Different types of feature data training sets are input into the corresponding machine learning models, and the output is the corresponding damage degree label. The three models are trained using the Bayesian optimization algorithm. The error rates of the three machine learning models after training are as follows: Figure 7 As shown in Figure 2, the weights of different machine learning models can be determined based on their error rates. The weighted average technique is used to integrate the three machine learning models. The normalized confusion matrix of the integrated test set is shown in Figure 2. Figure 8 As shown, the accuracy of distinguishing different degrees of damage is more than 90%, which shows that the method of the present invention can effectively identify impact damage with high accuracy.

[0053] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An impact damage identification method based on multimodal data fusion and ensemble learning, characterized in that: include Step 1: Collect multiple non-destructive and impact stress wave signals of different damage degrees of the structure to be tested and label the impact stress wave signals according to the detection results of ultrasonic scanning; Step 2: acquiring multimodal data of multiple categories of the impact stress wave signal and dividing the multimodal data into a training set and a test set; Step 3: Establish different types of machine learning models for the multimodal data of the above categories; Step 4: training the various types of machine learning models and using the relative accuracy index of the training results of the training sets of various types of multimodal data samples as the weights of the machine learning models, and integrating the three machine learning models according to the weights of the machine learning models to obtain an integrated learning model; Step 5: Input the test set into the ensemble learning model, and perform damage classification by fusing at least two distances to measure the difference between the weighted average ensemble prediction result and the actual label. The distances obtained are multiplied and fused to obtain the fusion distance, and the class with the smallest fusion distance is the impact damage recognition result of the ensemble learning model.

2. The impact damage identification method based on multimodal data fusion and ensemble learning according to claim 1 is characterized in that: The impact stress wave signal collection and labeling in step 1 specifically includes arranging a sparse piezoelectric sensor network on the structure to be tested, and collecting multiple non-destructive and impact stress wave signals of different damage degrees.

3. The impact damage identification method based on multimodal data fusion and ensemble learning according to claim 1 is characterized in that: Acquiring multimodal data in step 2 specifically includes: the multimodal data includes discrete impact damage feature data, time domain impact signal sequence feature data or time-frequency image feature data of the impact signal.

4. The impact damage identification method based on multimodal data fusion and ensemble learning according to claim 1 is characterized in that: The training set and test set are divided into different types of multimodal data samples in a ratio of 7:

3.

5. The impact damage identification method based on multimodal data fusion and ensemble learning according to claim 1 is characterized in that: The step 3 establishes multiple different types of machine learning models, specifically including establishing an artificial neural network with discrete impact damage characteristics as input, a long short-term memory network with a time domain impact signal sequence as input, and a convolutional neural network with a time-frequency image of the impact signal as input.

6. The impact damage identification method based on multimodal data fusion and ensemble learning according to claim 1 is characterized in that: The machine learning model training and weighted average integration in step 4 specifically include training machine learning models with different training sets and Bayesian optimization algorithms, and weighted average integration of the machine learning models based on the error rate of the training results.

7. The impact damage identification method based on multimodal data fusion and ensemble learning according to claim 1 is characterized in that: The multi-distance measurement and damage classification in step 5 specifically includes inputting the test set into the integrated learning model, and performing damage classification by fusing the three distances of cosine distance, Manhattan distance and Euclidean distance to measure the difference between the integrated prediction result and the actual label.

Images