Quantitative benchmarking of acoustic emission machine learning framework for damage mechanism identification

Abstract

A challenging opportunity of composite structure health monitoring is to classify acoustic emissions using a machine learning method according to the damage mechanism of the emitted signal. Various machine learning frameworks have been developed; however, in addition to limited overlap between experimental configurations, lack of ground live data sets has excluded any direct, quantitative benchmark for its accuracy. Here, we generate a ground live dataset consisting of pencil lead fractures at a known angle of incidence [theta]. A unique frequency spectrum is generated at each angle, and the frequency spectrum changes continuously along with theta, which may be similar to the attributes of acoustic emission signals generated in a fault process, such as those generated in a composite material. Five frames are then applied to the ground live data set and benchmark testing is performed according to their ability to differentiate two sets of signals with fixed [Delta] [Theta]. The performance related to feature selection is discussed, and a group of optimal practice guide for feature selection and standardized practice is provided.

Description

Technical Field

[0001] The present invention relates to a quantitative benchmark testing device and method for an acoustic emission machine learning framework for damage mechanism identification, and in particular to a damage identification framework technology in the field of learning. Background Art

[0002] A long-standing hypothesis investigated by the scientific community is that damage mechanisms in multiphase structural materials can be identified directly from the strain waves or acoustic emissions they generate. Developing this capability has broad implications for life prediction studies and operational monitoring of advanced structural materials. It will allow researchers to increase the flow of informed data for damage triangulation, life prediction, and high-resolution optical studies with complementary mechanisms.

[0003] However, directly mapping waveforms to their source mechanisms is not a trivial task. Factors that are difficult to capture in a single experiment, such as transducer contact, specimen geometry, and loading configuration, all affect the waveform structure. Due to these effects, it is not feasible to implement a bottom-up approach, in which the measured waveforms are directly compared to those generated by computational models. While these experimental factors are often difficult to capture in physics-based models, their impact on the acoustic signal propagating from the source to the sensor is constant. Therefore, it is more efficient to group the waveforms based on their differences and assign mechanisms to the groups. Therefore, unsupervised machine learning methods (i.e., pattern recognition techniques) that classify signals based on differences in their structure are an effective strategy for damage mechanism identification, and many such frameworks have been developed over the past two decades.

[0004] A general examination of these frameworks yields an important observation: there are no ground truth datasets, where mechanisms have been directly assigned to each independent signal, that are suitable for benchmarking the performance of acoustic emission-machine learning (AE-ML) frameworks. Previous studies have attempted to create ground truth datasets using various strategies, such as by designing loading configurations and specimen geometries to promote 1-2 damage mechanisms. Typically, such strategies produce datasets that are not useful for quantitative benchmarking because the ground truth remains unknown. For example, a geometry designed to promote composite fiber failure may contain a large number of signals from other mechanisms, such as interface damage, without visual confirmation.

[0005] Since the above datasets are not suitable for benchmarking accuracy, indirect methods for measuring the performance of the frameworks have been used. Metrics that have been studied include the tendency to fall into local minima, the compactness of clusters, and how well the average cluster characteristics match previous literature. However, these cannot be used as proxy metrics for accuracy because they do not measure accuracy or discrimination. Therefore, there is a need for datasets and methods that can be used for standardized, quantitative evaluation of AE-ML frameworks.

[0006] To generate a dataset that can be used to evaluate discrimination, pencil lead fractures provide a powerful solution. The frequency content of the signal emitted by the fracture can be controlled by varying the incident angle θ. Incremental increases in the incident angle Δθ result in incremental increases in the low-frequency (flexural wave) component of the acoustic emission signal.

[0007] Since the structure of a signal is uniquely determined by its frequency content, a set consisting of signals generated by angles θ and θ+Δθ can be used to quantitatively evaluate the ability of a framework to group signals according to their emission sources. A framework that can accurately distinguish between signals generated by θ and θ+Δθ when Δθ is small will have higher discrimination than a framework that cannot. Therefore, a dataset consisting of signals generated by known values ​​of θ can be used to quantitatively evaluate the discrimination of AE-ML frameworks, study how specific changes to the framework affect discrimination, and guide decisions to improve it.

[0008] To achieve the goal of generating a dataset that can be used to evaluate discrimination capabilities, pencil lead breaks are a powerful solution. The frequency content of the signal emitted by the PLB can be controlled by varying the incident angle θ. Incremental increases in the incident angle Δθ result in incremental increases in the low-frequency (fexural wave) component of the AE signal. Summary of the invention

[0009] Data Collection All pencil lead breaks were made with pentagonal 0.5 mm HB wire with a nominal free wire length of 4 mm. A Pentel GraphGear 500 mechanical drawing pencil was mounted on a custom-made, controlled displacement loading frame ( Figure 1 ). The loading frame consisted of a rotary table that allowed angular adjustment in 2° increments (corresponding to a manual angular measurement error of 1-2 measurement units, i.e. ±1°) and two precision-adjusted linear tables. The aluminum plate on which the PLB was performed had an unsupported span of 200.7 mm, a width of 51 mm, and a thickness of 1.2 mm.

[0010] Pencil breaks were recorded at 20°, 22°, 26°, 30°, 36°, and 40°. PLBs were generated by lowering the pencil on a linear Z stage until the lead broke on the aluminum plate. For each angle condition, the rotation stage was fixed using a set screw that was loosened to change the angle only. During the angle change, the linear X stage was adjusted to maintain a nominal distance of 25 mm from the PLB source to the sensor. Upon inspection of the collected AE signals, some were found to be reflections. These reflections appeared as a second low-amplitude signal that appeared immediately after the initial PLB and were excluded from the dataset. Due to the exclusion of these reflections, each angle had a different number of signals, ranging between 75-111. For the purpose of this analysis, the AE-ML framework only clustered the first 75 signals for each angle.

[0011] AE was recorded using a piezoelectric B1025 transducer (DigitalWave Corporation, Centennial, CO) with a broadband response of 50–2000 kHz. Figure 1 ). The threshold voltage was set to 0.1V, the number of pre-trigger points was set to 256, and the total length of the captured signal was 1024 points at a rate of 10MHz. The sensors were fixed to the aluminum plate using alligator clips using vacuum grease as a coupling agent. The sensors were not remounted at any time during the experiments, meaning that PLB at all angles was performed for a fixed sensor coupling. The authors note here that since the coupling remained constant throughout all data collection, an absolute calibration of the sensor was not required. Furthermore, all signals were collected over a single 3-hour span in a temperature-controlled laboratory where environmental factors that could affect the absolute sensor calibration (e.g., temperature) were assumed to be constant. The unsupervised framework groups the signals based on the differences in signal features rather than the absolute values ​​of those features. Since the absolute sensor calibration is constant, any differences in the signal features can be attributed to changes in the angle of incidence.

[0012] The signals collected at the reference angle θ0 = 20° and the signals collected at a single reference angle θb ∈ [22°, 26°, 30°, 36°, 40°] are clustered using each framework described in Section 1. The framework descriptions and accuracy metrics, as well as the relative discrimination capabilities are quantitatively evaluated using the procedure described below. Quantitative Benchmarks A permutation model of the Adjusted Rand Index (ARI) was used in the benchmarking framework. The ARI ranges from 0 to 1 and measures the accuracy of the ML-computed clustering compared to the ground truth in a label-agnostic manner.

[0013] It compares the membership similarity A of objects in the ML-computed clusters with the membership similarity B of objects in the ground truth clusters and assigns a higher number if the similarity is high. In the context of this work, signals from θ0 and θb are fed into the AE-ML framework. These signals are then assigned a label by the framework, either 0 or 1, depending on whether the framework believes the signal should be associated with θ0 or θb. The MLassigned label for each signal is then compared to the ground truth, which is the known angle at which the signal was collected. If the membership similarity of the ML-assigned label and the true angle are similar, the ARI will have a higher value.

[0014] ARI is an opportunity-adjusted version of the Rand Index (RI) and is calculated as:

[0015] Where N is the number of signals, N11 is the number of signal pairs grouped into the same cluster in A and B, and N00 is the number of signal pairs grouped into different clusters in both A and B. ARI can be calculated as:

[0016] where E[RI(A,B)] is the expected value of RI under a random model. ARI ranges between 0 and 1, with 0 corresponding to a random label assignment and 1 corresponding to a perfectly matched label. While many cluster similarity measures exist, ARI was chosen to compare partitions because it can be computed for any number of clusters (assuming an equal number of clusters in each partition) and it adapts to unbalanced cluster sizes. The signals from θ0 and θb are aggregated by each framework. The value Δθ=θb−θ0 where ARI vanishes represents the point at which the framework loses all discriminatory power and cannot identify the difference between two different sets of signals.

[0017] Framework Description and Accuracy Metrics The general AE-ML framework can be used Figure 2 After data collection ( Figure 2 a), the most important step in the framework is to select the feature set ( Figure 2 b). Waveforms can only be ranked according to their source mechanism if the feature set captures some basic information about the waveform-mechanism relationship. Features can be classified as belonging to the time domain, frequency domain, or time-frequency domain. However, there is little consensus on which category is best suited for damage mechanism identification. In fact, even if two frameworks utilize features within the same domain, their feature sets differ. Therefore, each framework uses a unique feature set where d relevant features are identified, extracted, and stored as a feature vector ( Figure 2 b) Reference can be made to the original survey for a discussion of why a particular feature set was chosen.

[0018] Next, the individual features of the feature vector can be rescaled or remapped using the transformation ( Figure 2 C).

[0019] Similar to the variation in feature sets, each framework uses a different set of preprocessing steps. Finally, an ML algorithm is applied to partition the feature vectors by assigning them to clusters where feature vectors in the same cluster are closest under a chosen distance metric ( Figure 2 D).

[0020] The frameworks described in the following subsections follow this workflow and are directly adopted from the literature. They are chosen to span the current space of different feature set types and ML algorithms. The key differences between the different frameworks lie in the choice of feature sets, preprocessing steps, and ML algorithms. The specific choices of feature sets, preprocessing steps, and ML algorithms are further summarized in Table 1. Results and Discussion,We provide key foundations and discuss the impact of feature selection.

[0021] Basic Framework We define a base framework relative to subsequent frameworks, which are variants on it (by exchanging feature sets, ML algorithms, or both). The framework adopts the time domain feature set as studied in: 1. Average frequency (number of counts / signal length) 2. Rise frequency (average frequency from the start of the signal to the maximum amplitude) 3. ln (energy) 4. ln (rise time / duration) 5. ln (amplitude / rise) 6. ln (amplitude / decay time) 7. ln (amplitude / average frequency) The start and end times of the experimental signal are determined by the first and last crossing of the floating 10% voltage threshold.

[0022] Using the MaxAbsScaler method, each feature is scaled by the maximum observed value of that feature so that they fall within the range [-1,1]. A principal component analysis (PCA) transformation is performed, retaining the principal components that contain ≥95% of the variance. The distance d between any two feature vectors x and y is calculated using the modified Euclidean metric:

[0023] where xi and yi are the i-th vector components of the eigenvectors in the PCA basis, and 휆i is the eigenvalue of the i-th PCA axis. Since the Scikit-learn implementation of k-means enforces the standard Euclidean metric, the eigenvectors need to be rescaled to fit the modified Euclidean metric (the proof is given in 6):

[0024] It should be noted that the distance metric in Eq. 3 differs from the standard PCA whitening method, where the distances along axes with large eigenvalues ​​are shrunk, rather than stretched.

[0025] K-means is then applied to the feature vectors. The reader is referred to for a detailed description of the k-means algorithm. Because k-means is not guaranteed to converge to the optimal solution, it is usually run multiple times and initialized with the lowest value of the loss function. To determine the number of reinitializations, a convergence check is performed by increasing the number of reinitializations until the loss remains unchanged. The minimum objective function did not change after 2×103 reinitializations. To conservatively ensure that the global optimum of the objective function is reached, the number of reinitializations is set to 2×104. Again, an error tolerance of 0.0001 and 300 iterations are sufficient to ensure that the local optimum of k-means is reached in one initialization.

[0026] Aggregation Framework The feature set used by the aggregation framework is: 1. Amplitude 2. Peak frequency The Agglomerative framework uses a hierarchical agglomerative approach instead of partitioning the feature vector by k-means as in the Base framework. In this approach, each data point is initially defined as a cluster. Clusters are then iteratively merged so that the chosen objective function (usually the sum of squared distances) reaches an extreme value. The reader is referred to for a discussion of this algorithm.

[0027] The linkage types, i.e. the parameters defining the pairwise distances between points, were not reported in the original work. Here, each linkage type was tested and no linkage type was found to outperform another.

[0028] Spectral framework The spectrum framework uses a partial power feature set. The i-th component of the feature vector is as follows:

[0029] where F[∗] is the Fourier transform operator, S(T) is the recorded signal, Ki and Ki−1 are the frequency bands over which the integration is performed, and d is the number of entries in the eigenvector. We set k0 = 200 kHz, kd = 800 kHz, and d = 23. The width of the integration bounds Ki−ki−1 is set equal for all i.

[0030] The sklearn implementation of spectral clustering was used to cluster the eigenvectors. The ARPACK eigensolver was used and the number of nearest neighbors was set to NN=5. To ensure that the cluster membership did not depend on the initialization parameters, convergence checks were performed on the error tolerance and maximum number of iterations. It was found that the cluster membership stabilized after 10 reinitializations. The number of reinitializations was set to 100 to conservatively ensure stability.

[0031] Frequency frame The frequency framework uses a feature set in the frequency domain: 1. Average frequency 2. Reverberation frequency 3. Rise frequency 4. Peak frequency 5. Frequency centroid 6. Weighted peak frequency 7. Partial power of 0–150kHz, 150–300kHz, 450–600kHz, 600–900kHz, and 900–1200kHz The features were independently normalized using a variance scaler, which centered the features to have zero mean and scaled them to unit variance. The feature vectors were then clustered using k-means. The same convergence checks as for the Base framework were performed and the same parameters were found to be sufficient for convergence.

[0032] Wavelet Packet Transform Framework The wavelet packet transform (WPT) framework extracts features by applying WPT. Three levels of WPT are applied to the waveform using the 2nd order Daubechies wavelet as the mother wavelet. The fractional energy carried by each node is calculated and the five most irrelevant values ​​are retained. These are used as features in addition to the waveform energy read by the AE software. The feature vectors are then normalized using a maximum scaler and PCA is performed. The principal components containing ≥95% of the variance are retained. The feature vectors are then partitioned by k-means using the modified Euclidean metric (Equation 3).

[0033] A convergence check was performed and it was found that the same parameters as the Base framework were sufficient for convergence. It should be noted that the algorithm described and used is k-means optimized by a genetic algorithm. Therefore, a fully converged k-means solution is indistinguishable from a fully converged genetic solution.

[0034] Our experimental setup enables precise control of the frequency content of the PLB signal by varying the incident angle. The resulting signal is consistent with expectations from plate wave theory; as the incident angle increases, the low-frequency components of the signal are observed to increase in power. Figure 3 In addition, almost no change in the PLB signal was observed under each angle condition. The average signal and its standard deviation envelope are shown in Figure 4 As shown. Due to the small changes between signals under a single angle condition, when the signal is represented in the feature space ( Figure 2 B), the standard deviation of these features is smaller than if the signal has a larger variation. If a benchmark set is constructed from more repeatable acoustic signals, then the signals at different angles will form tighter clusters in the feature space, and subsequently the ARI at each Δθ value will increase for each frame.

[0035] By comparing the ARI values ​​under different FX values ​​of Δθ, the relative discrimination ability of various frameworks can be quantitatively evaluated. The frameworks listed in Table 1 are grouped according to the procedure described in Section 3. Quantitative Benchmarks. The accuracy of each framework is plotted as a function of Δθ ( Figure 5). The discriminative power of each framework increased with increasing Δθ, which can be attributed to the increasing differences in signal structure as a function of Δθ. Frameworks that exhibit higher ARI values ​​at lower Δθ values, such as the spectral and frequency frameworks, have higher overall discriminative power and may be able to distinguish between damage mechanisms that emit similar signals.

[0036] The significance of the function Between the two FVE frameworks, no single set of features was used. While this is common across the broader AE-ML framework, the lack of consensus raises an important question: “What features should be used for AE signal recognition?” Resolving this question is critical because the discriminative power of the framework depends on how the signal is represented.

[0037] For signal recognition, both the exclusion of noisy features and the inclusion of useful features are necessary: ​​this principle is known as the ugly duckling theorem. To highlight the extent to which this principle affects the discriminability, features were parametrically excluded from both the frequency frame and the base frame. In the frequency frame, the ARI was maximized when clustering was performed using mean frequency, rising frequency, and partial power between 150 and 300 kHz. When the signal was encoded using these three optimal features, the ARI at Δθ = 2◦ increased from 0.681 to 0.973, representing a change from moderate to high discriminability. A similar procedure was performed for the base frame, where the ARI increased from 0.325 to 0.82 when only mean frequency and log(amplitude / mean frequency) were included. While such parametric studies can provide insight into which features are useful for a particular data set, they are less effective at identifying generally significant features. For this reason, it is necessary to consider the physical properties of the emission sources one by one, if possible. Figure 4 The average PLB signal and point-by-point standard deviation are given for each angle condition. Figure 1 The experimental plots shown produced signals that were found to be repeatable while still containing the variations that might be expected from signals collected during intraoperative health monitoring, excluding features that are dependent on external and uncontrollable factors unrelated to the source mechanism. For example, although the amplitude is correlated with the angle in this dataset ( Figure 6 A), but it should not be used to classify signals from multiphase materials because it is convolved with factors such as the crack area formed and the source-to-sensor distance, which are not related to the individual mechanisms. Similarly, although rise time is often used in the AE-ML framework, it is more related to the source-to-sensor distance due to the different velocities of the flexural and extensional wave components. Therefore, two signals emitted from similar locations will have similar rise times, even if the emission mechanisms are different ( Figure 6 B).

[0038] Limitations of the PLB Dataset Intuitively, a framework with a high ARI value is a promising candidate for damage mechanism identification when signals from different mechanisms are expected to be similar. However, it is unknown to what extent performance on the PLB dataset translates to performance under more realistic conditions and material systems. Specifically, the PLB signals in this dataset were collected under the most stringent conditions; the signals were from a single source-to-sensor distance, sensor coupling, and source type (e.g., pencil lead), eliminating factors that affect the signal such as dispersion, attenuation, and absolute frequency response. ML methods for mechanism identification must ultimately be robust to these effects. While this dataset represents a first step toward a quantitative benchmark, a comprehensive characterization of the framework's performance under realistic conditions is still needed.

[0039] Another limitation of the dataset we collected is the angular resolution; the ±1◦ tolerance of the rotation stage has an impact on the measured ARI values. For example, if the true Δθ between two angular conditions is smaller than the reported value, the signals generated at these angles will be more similar than expected due to the ±1◦ tolerance. Therefore, the measured ARI will be lower than the signal collected from the true angular condition with a larger Δθ value. The exact degree of change in the ARI is highly dependent on how each feature varies with θ and is also affected by any data-dependent preprocessing, such as PCA, which will further affect the framework performance.

[0040] Finally, the current experimental setup prevents the acquisition of signals from values ​​of Δθ < 2◦ for reasons of angular resolution. In order to properly benchmark the frameworks, there must be at least one Δθ value for which the ARI does not saturate at 1. For example, for Δθ = 20, the spectrum, frequency, and base frameworks all perform equally well, but Δθ = 6 allows comparison of the discrimination capabilities ( Figure 5 ). As the community continues to improve the discriminative power of frameworks, the ARI value will increase. Therefore, it is necessary to collect signal from values ​​of Δθ < 2◦, lower than what we allow in this study, to ensure that the performance of the frameworks can be separated.

[0041] Guidelines and Conclusions While many AE-ML frameworks have been developed and implemented, the lack of ground truth datasets has limited the discussion of their strengths and limitations, especially in terms of feature selection, and has hindered the development of standardized quantitative benchmarks. In this section, the amount of data in benchmark sets, the types of features that should be included in a framework, and transparent benchmarking practices are discussed.

[0042] The performance of unsupervised frameworks is inherently related to how well the sampled data represents its population distribution. In the context of the AE-ML benchmark dataset, it is critical to ensure that enough signal is collected to capture statistical variation. If too few waveforms are collected at any angle, the sampled distribution is unlikely to represent the population distribution of waveform features ( Figure 2B). Therefore, adding new waveforms will lead to spurious performance of the AE-ML framework. To ensure that there are enough signals in the benchmark set, the performance of the framework must be independent of the number of signals collected. For this benchmark set, 75 signals per angle proved to be sufficient to ensure that the ARI value we calculated is independent of the amount of data.

[0043] Feature selection is crucial to maximizing discriminative power. As shown in Section 1.1, feature saliency, including insignificant features is directly correlated to poor framework performance. Despite the importance of feature selection, there is little discussion in the literature about why certain features are selected. As a result, many modern frameworks continue to include insignificant features (e.g., rise time) that negatively impact framework performance.

[0044] To better select features, generic features should be preferred and, where possible, the choice of feature sets should be explicitly motivated. If a situation can be constructed where a given feature cannot reliably distinguish signals emitted by two unique sources, then the feature is likely convolved with factors unrelated to the source mechanism and is therefore not generic. The use of such non-generic features must be approached with caution. For example, while small amplitudes and large rise times are associated with Mode II cracks, these features are not generic because they are also strong functions of the source-to-sensor distance. This makes it possible to construct a dataset in which the features are not considered generic due to little or no statistical difference between the features ( Figure 6 ), only signals can look artificially similar.

[0045] Although the universally significant features do not change depending on the material system or loading configuration, the value of the feature may vary. For example, partial power appears to be a universally significant feature, but each frequency band does not provide the same discrimination capabilities. As indicated by the parameter removal of the frequency frame features, the frequency band of 150-300kHz is the most useful for signal discrimination. In this case, 150-300kHz is useful for distinguishing the two PLB signals; however, when the material system or loading conditions change, a different frequency band will be useful.

[0046] Finally, frameworks should be quantitatively benchmarked using publicly available standardized datasets. While these types of benchmark tools are common in other fields, they do not exist in the AE community. Developing and continuing to maintain these types of datasets will provide the necessary tools to evaluate the strengths and limitations of AE-ML frameworks and will allow for detailed discussions on the specific strengths and weaknesses of individual frameworks. This, in turn, will provide transparency and trust in the results obtained from these frameworks, promoting their wider use in scientific and engineering applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1Photo of the experimental setup. The mechanical pencil is attached to a rotating stage that controls the angle of incidence. A linear X stage is used to position the free lead tip at a consistent position on the aluminum plate. A linear Z stage is used to lower the pencil lead until it breaks. The final waveform generated is generated by a piezoelectric B1025 sensor; Figure 2 Workflow diagram of the AE-ML framework. a Waveforms are collected, b Relevant features are extracted from the waveform and then represented as vectors in feature space. c The feature vectors can then be rescaled and / or remapped before applying a clustering algorithm and labeling them. Each AE-ML framework follows this process Figure 3 Fourier transform (FFT) of the average signal at each angle condition Figure 4 The average PLB signal and point-by-point standard deviation for each angle condition. Figure 1 The experimental fixture shown generates signals that are found to be repeatable while still containing the variations that might be expected in signals collected during operational health monitoring. Figure 5 ARI for each framework as a function of Δθ. ARI values ​​over 0.4 are associated with good discrimination, while values ​​close to 0 correspond to no discrimination. The discriminatory power of each framework increases with increasing Δθ, and high ARI at low Δθ values ​​are more appropriate for clustering signals with less differentiation. The ability to directly compare accuracy between frameworks enables researchers to select the appropriate framework for their specific needs. Figure 6 a Mean maximum amplitude and b mean rise time θ of the signals produced for each incident angle. Error bars correspond to 1 standard deviation. There are no consistent differences between the values ​​of these two features. Since many groups of unique signals with indistinguishable amplitudes and rise times can be constructed, they should not be considered significant features and should be used with caution.

Claims

1. A quantitative benchmarking approach for an acoustic emission machine learning framework for damage mechanism identification characterized by The models adopted include data collection, quantitative benchmark, basic framework, aggregation framework, spectral framework and wavelet packet transform framework.

Images