Neuromorphic computing-enabled generalized multimodal data fusion method for aircraft structural health monitoring

Abstract

A neuromorphic computing-enabled generalized multimodal data fusion method for monitoring the structural health of aircraft structures first collects and standardizes multi-source signals from acoustic emission, guided waves, strain, and optical fibers, converting them into pulse sequences. Then, a multimodal teacher pulse convolutional neural network (TCNN) is constructed, with mean squared error loss as the optimization objective, outputting continuous damage indicators. Next, a domain generalization regularization term based on the maximum mean difference is introduced into the training loss to jointly optimize prediction accuracy and cross-domain invariance. Then, a lightweight student SCNN is trained using the teacher SCNN as soft label guidance, with a weighted sum of KL divergence loss and regression loss as the knowledge distillation objective. Finally, the student SCNN is pruned and quantized before being deployed on an embedded development board to achieve low-power, real-time structural health monitoring. This invention realizes intelligent, low-power, and deployable monitoring of aircraft structural health.

Description

Technical Field

[0001] This invention belongs to the field of aircraft structural health monitoring and intelligent diagnosis technology, specifically involving a neuromorphic computing-enabled generalized multimodal data fusion method for aircraft structural health monitoring. Background Technology

[0002] Aircraft structures are subjected to multiple factors over long periods, including cyclic loads, temperature variations, and environmental corrosion. Crack initiation, propagation, and overall stiffness degradation can accumulate gradually without being detected, ultimately leading to structural failure and even catastrophic consequences. Structural health monitoring is a core technology in aerospace engineering, and its accuracy directly impacts structural safety, operational reliability, and maintenance costs. Traditional health monitoring methods primarily rely on periodic manual inspections, which are costly and time-consuming.

[0003] With the rapid development of artificial intelligence technology, data-driven deep learning methods have been widely applied to damage evolution modeling. However, existing intelligent health monitoring schemes ([1] WANG Yu, HU Shuguang, XIONG Tao, HUANG Yongan, QIU Lei. Recent progress in aircraft smart skin for structuralhealth moni-toring [J]. Structural Health Monitoring, 2021, 21(5): 2453-2480.) still face the following key challenges: First, single-sensor modal methods cannot fully capture the multi-scale evolution process of material damage; second, although multi-modal fusion systems can improve accuracy, they have high computational complexity and high power consumption, making them difficult to deploy in real time on embedded platforms; third, the training data collected under laboratory conditions has a significant domain shift from the variable working conditions in engineering sites, resulting in insufficient model generalization ability.

[0004] Neuromorphic computing and spiking neural networks have become research hotspots in the field of structural health monitoring in recent years due to their event-driven and low-power computing characteristics. Spiking neural networks convert continuous sensor signals into discrete pulse sequences, which can significantly reduce energy consumption while efficiently capturing dynamic temporal features. However, existing research is mostly limited to software simulation ([2] ZUOLin, XU Fengjie, ZHANG Changhua, et al. A multi-layer spiking neural network-based approach to bearing fault diagnosis. Reliability Engineering & System Safety, 2022, 225: 108561.), and has not yet fully realized the engineering deployment at the hardware level, which restricts its application in actual aircraft health monitoring. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, the present invention aims to provide a neuromorphic computing-enabled generalized multimodal data fusion method for monitoring the structural health of aircraft structures. This method converts multi-source sensor signals into pulse representations, enhances cross-condition robustness by combining domain generalization strategies, and achieves efficient deployment of student networks on embedded development boards through knowledge distillation, pruning, and quantization, thus achieving a good balance between prediction accuracy, computational efficiency, and cross-domain generalization capability.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A neuromorphic computing-enabled method for monitoring the structural health of aircraft using generalized multimodal data fusion includes the following steps: Step 1: Collect four modal signals from the aircraft structural components: acoustic emission, guided wave, strain, and optical fiber, and perform normalization and filtering preprocessing on each modal signal; Step 2: Convert the preprocessed signal into a pulse sequence using a time-to-pulse coding scheme or an amplitude threshold coding scheme; Step 3: Construct a multimodal teacher spiking convolutional neural network, namely teacher SCNN, which includes four parallel feature extraction branches, a feature fusion module and a linear regression module, with mean squared error loss as the optimization objective, and outputs a continuous damage index; Step 4: Introduce a domain generalization regularization term based on the maximum mean difference into the training loss to jointly optimize prediction accuracy and cross-domain invariance; Step 5: Using the teacher's SCNN as soft label guidance, train a lightweight student SCNN, with the weighted sum of KL divergence loss and regression loss as the knowledge distillation objective; Step 6: After pruning and quantizing the student SCNN, deploy it on an embedded development board to achieve low-power real-time structural health monitoring.

[0007] In step 1, the first Modal signals Standardization preprocessing is performed using the following formula: In the formula, For the first One modal signal, and These are the mean and standard deviation of the modal signal, respectively. For the first The signal after modal standardization.

[0008] In step 2, the pulse excitation time of the pulse encoding is... for: In the formula, and Scaling parameters to control sensitivity; binary pulse output in amplitude threshold encoding scheme. for: In the formula, To trigger the threshold, For the first The modally standardized signal It outputs binary pulses.

[0009] Step 3 of the multimodal teacher pulse convolutional neural network includes three components: (1) four parallel feature extraction branches, corresponding to the four modes of acoustic emission, guided wave, strain and fiber, respectively. Each branch consists of two stacked convolution-IAF-pooling blocks, which gradually extract the spatiotemporal features specific to the mode; (2) a feature fusion module, which concatenates the outputs of the four branches and processes them through two shared convolution-IAF-pooling blocks to realize cross-modal feature interaction and joint damage representation learning; (3) a linear regression module, which flattens the fused features and outputs continuous damage indicators through a pulse fully connected regression layer. This index increases monotonically with the degree of damage, and the mean squared error loss function is used as the optimization objective: In the formula, For the sample i Predictive damage indicators; To calibrate reference values ​​for the experiment, N The total number of samples, This represents the mean square error loss.

[0010] Step 4 Domain generalization loss Based on the definition of the maximum mean difference: In the formula, For the sample i In the domain High-level pulse characteristics representation; For domain The number of samples, For the sample j In the domain High-level pulse characteristics representation; For domain The number of samples; the overall training objective function is defined as: In the formula, To balance the trade-off between classification accuracy and domain invariance, For the overall training objective.

[0011] The objective function for knowledge distillation in step 5 is: In the formula, The regression loss of the student SCNN relative to the real degenerate label; and These are the impairment metrics output by the teacher's SCNN and the student's SCNN, respectively. KL divergence measures the difference between the two distributions; The coefficient is used to balance the contributions of direct supervision and knowledge transfer.

[0012] In step 6, the number of parameters of the distilled student SCNN does not exceed 50% of that of the teacher SCNN, the power consumption does not exceed 2 mW, the energy consumption of a single inference does not exceed 1 mJ, and it supports real-time deployment on resource-constrained embedded development boards.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention proposes an event-driven neuromorphic computing framework that converts multi-source sensor signals into pulse sequences, reducing power consumption by approximately 39% compared to traditional deep neural networks while maintaining high accuracy. It effectively suppresses cross-condition feature distribution shifts through a domain generalization strategy, improving the model's generalization ability under different load conditions. Through knowledge distillation, pruning, and quantization, the model parameter size is compressed from 45 MB to 18 MB, power consumption is reduced from 90.27 mW to 1.42 mW, and single inference energy consumption is reduced from 87.56 mJ to 0.70 mJ, enabling real-time low-power deployment on embedded platforms and providing an engineering-applicable intelligent solution for aircraft structural health monitoring. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the overall framework of the aircraft structural health monitoring method according to an embodiment of the present invention.

[0015] Figure 2 This is a schematic diagram of the fatigue testing apparatus according to an embodiment of the present invention.

[0016] Figure 3 This is a comparison chart of the damage index prediction curves of various methods in the embodiments of the present invention. Detailed Implementation

[0017] The present invention will now be described in detail with reference to the embodiments and accompanying drawings.

[0018] Reference Figure 1 A neuromorphic computing-enabled method for monitoring the structural health of aircraft through generalized multimodal data fusion includes the following steps: Step 1: Multimodal signal acquisition and preprocessing. Four modal signals of the aircraft structural components during fatigue loading are acquired: acoustic emission signal, strain signal, fiber optic signal, and guided wave signal. Each modal signal provides complementary information: acoustic emission reflects microcrack initiation, guided waves characterize wave scattering and propagation changes, strain gauges measure local deformation, and fiber optic sensors monitor overall stiffness degradation. Let the first The modal signals are After normalization and filtering preprocessing, it is standardized as follows: In the formula, For the first One modal signal, and These are the mean and standard deviation of the modal signal, respectively. For the first The signal after modal standardization.

[0019] Step 2: Pulse Encoding. To adapt the preprocessed continuous signal to neuromorphic computation, a time-to-pulse encoding scheme is adopted. The activation time of the neuron is inversely proportional to the signal amplitude. Modally standardized signal Its pulse excitation time Defined as: In the formula, and Scaling parameters are used to control sensitivity; furthermore, amplitude-to-pulse encoding can be expressed as a threshold function: In the formula, To trigger the threshold, For the first The modally standardized signal The output is a binary pulse; the above pulse characterization effectively reduces redundant information while preserving the dynamic timing characteristics related to crack initiation and propagation. Step 3: Construct the teacher spiking convolutional neural network, i.e., the teacher SCNN. Spiking neural networks process information using discrete spiking events and have the characteristics of event-driven and energy-saving. In this embodiment, the Integrate-and-Fire (IAF) neuron is used as the basic computing unit. Based on the IAF neuron, the multimodal teacher spiking convolutional neural network model framework shown in Table 1 is constructed, which includes three components: (1) four parallel feature extraction branches, corresponding to the four modes of acoustic emission, guided wave, strain and optical fiber, respectively. Each branch consists of two stacked convolution-IAF-pooling blocks, which gradually extract the spatiotemporal features specific to the mode; (2) feature fusion module, which splices the outputs of the four branches and processes them through two shared convolution-IAF-pooling blocks to realize cross-modal feature interaction and joint damage representation learning; (3) linear regression module, which flattens the fused features and outputs continuous damage indicators through the spiking fully connected regression layer. This index increases monotonically with the degree of damage, and the mean squared error loss function is used as the optimization objective: In the formula, For the sample i Predictive damage indicators; To calibrate reference values ​​for the experiment, N The total number of samples, This is the mean square error loss; Table 1. Structure of the Teacher Spiral Convolutional Neural Network Model Step 4: Domain Generalization Training. Structural health monitoring data is typically collected under various operating conditions (different loads, temperatures, environmental noise). These variations introduce domain shifts, affecting the stability of damage indicators. Therefore, a domain generalization regularization term is introduced into the training loss. The maximum mean difference is used to quantify the differences in feature distributions between different domains: In the formula, For the sample i In the domain High-level pulse characteristics representation; For domain The number of samples, For the sample j In the domain High-level pulse characteristics representation; For domain The number of samples; the overall training objective function is defined as: In the formula, To balance the trade-off between classification accuracy and domain invariance, The overall training objective is to jointly optimize the two aspects mentioned above. The network simultaneously learns feature representations that are discriminative of damage states and robust to changes in operating conditions, ensuring the high generalization capability of the multimodal SCNN framework in real-world aviation environments.

[0020] Step 5: Knowledge Distillation. While the teacher's SCNN boasts high prediction accuracy, its deep structure and large parameter count make it unsuitable for embedded deployments. Therefore, a knowledge distillation strategy is employed to train a student SCNN with fewer parameters to approximate the teacher model's prediction behavior. The student SCNN retains the same multimodal input structure as the teacher's SCNN but significantly reduces the depth and width of the convolutional layers. During training, the student SCNN receives dual supervision from both the real degenerate labels and the teacher SCNN's output impairment metrics (soft labels). The distillation objective function is defined as: In the formula, The regression loss of the student SCNN relative to the real degenerate label; and These are the impairment metrics output by the teacher's SCNN and the student's SCNN, respectively. KL divergence measures the difference between the two distributions; The coefficient is used to balance the contributions of direct supervision and knowledge transfer.

[0021] Step 6: Embedded Deployment. The distilled student SCNN undergoes structural pruning (removing redundant parameters) and quantization (reducing numerical precision). The processed lightweight model is then deployed on an embedded development board to achieve low-power, real-time structural health monitoring.

[0022] This embodiment uses a fatigue test of a 7050 aluminum alloy specimen as an example to verify the effectiveness of the proposed method.

[0023] Experimental platform: (Refer to) Figure 2 The experiment was conducted using an electro-hydraulic servo fatigue testing machine (LF5105, rated load 100 kN). The specimen size was 320 mm × 40 mm × 2 mm. A sinusoidal cyclic loading mode was adopted, with a frequency of 5 Hz, a peak stress of 8~10 kN, a valley stress of 0.8~1 kN, and a stress ratio of approximately 0.1. Each experiment consisted of three stages: cyclic loading to induce fatigue damage, ramp unloading, and constant load holding. Four sensor modal signals were acquired simultaneously during the experiment. The experimental scheme is shown in Table 2.

[0024] Table 2 Experimental scheme and sensing modal parameters Training configuration: Teacher SCNN was implemented and trained on an NVIDIA RTX 4060 GPU (8 GB VRAM) using the PyTorch framework, employing the Adam optimizer with an initial learning rate of 1×10⁻⁶. -3 Batch size 64, weight decay factor 1×10 -5 The learning rate was reduced by 0.1 times every 50 epochs, and a total of 200 epochs were trained. The data of specimens 1, 2, and 3 were used as the training set, and the data of specimen 4 were used as the test set (under different stress conditions) to evaluate the domain generalization ability.

[0025] Comparison method settings: In order to fully evaluate the proposed method, it is compared with the following three methods: (1) DNN: traditional feedforward neural network without impulse dynamics and domain adaptation; (2) SCNN-Base: impulse convolutional neural network trained only with regression loss, without domain generalization strategy; (3) DNN-DG: deep neural network that introduces domain generalization optimization term to distinguish the contribution of impulse dynamics and domain generalization.

[0026] The comparison results are shown in Table 3 and Figure 3 The method of this invention consistently achieves the lowest MSE and the highest R. 2 The predicted curve closely matches the actual damage curve throughout the entire fatigue cycle; DNN exhibits significant fluctuations due to its lack of temporal modeling capabilities; SCNN-Base's predicted curve gradually deviates in the later fatigue stage; DNN-DG's accuracy is close to that of the method in this invention, but its power consumption and energy consumption are significantly higher.

[0027] Table 3 Experimental scheme and sensing modal parameters Furthermore, Table 4 shows the performance comparison of the model before and after deployment. Compared with the teacher SCNN, the student SCNN reduced the number of parameters from 45MB to 18MB, shortened the inference latency by approximately 50%, reduced power consumption from 90.27 mW to 1.42 mW, and reduced the energy consumption per inference from 87.56 mJ to 0.70 mJ. While maintaining acceptable prediction accuracy (MSE=0.019, R²=0.89), it met the real-time deployment requirements of resource-constrained platforms. After knowledge distillation, the overall performance of the model was further improved. These results demonstrate that the organic combination of impulse dynamics and domain generalization is key to achieving high-precision, low-power structural health monitoring.

[0028] Table 4 Performance Comparison of Teacher SCNN and Student SCNN Deployment

Claims

1. A method for monitoring the structural health of aircraft using generalized multimodal data fusion powered by neuromorphic computing, characterized in that, Includes the following steps: Step 1: Collect four modal signals from the aircraft structural components: acoustic emission, guided wave, strain, and optical fiber, and perform normalization and filtering preprocessing on each modal signal; Step 2: Convert the preprocessed signal into a pulse sequence using either a time-to-pulse coding scheme or an amplitude threshold coding scheme; Step 3: Construct a multimodal teacher spiking convolutional neural network, namely teacher SCNN, which includes four parallel feature extraction branches, a feature fusion module and a linear regression module, with mean squared error loss as the optimization objective, and outputs a continuous damage index; Step 4: Introduce a domain generalization regularization term based on the maximum mean difference into the training loss to jointly optimize prediction accuracy and cross-domain invariance; Step 5: Using the teacher's SCNN as soft label guidance, train a lightweight student SCNN, with the weighted sum of KL divergence loss and regression loss as the knowledge distillation objective; Step 6: After pruning and quantizing the student SCNN, deploy it on an embedded development board to achieve low-power real-time structural health monitoring.

2. The method according to claim 1, characterized in that: In step 1, the first Modal signals Standardization preprocessing is performed using the following formula: In the formula, For the first One modal signal, and These are the mean and standard deviation of the modal signal, respectively. For the first The signal after modal standardization.

3. The method according to claim 1, characterized in that: In step 2, the pulse excitation time of the pulse encoding is... for: In the formula, and To control the scaling parameters for sensitivity, For the first The modally standardized signal; binary pulse output in the amplitude threshold coding scheme. for: In the formula, To trigger the threshold, For the first The modally standardized signal It outputs binary pulses.

4. The method according to claim 1, characterized in that: Step 3 of the multimodal teacher pulse convolutional neural network comprises three components: 1) four parallel feature extraction branches, corresponding to the four modes of acoustic emission, guided wave, strain, and optical fiber, respectively. Each branch consists of two stacked convolution-IAF-pooling blocks, progressively extracting mode-specific spatiotemporal features; 2) a feature fusion module, which concatenates the outputs of the four branches and processes them through two shared convolution-IAF-pooling blocks to achieve cross-modal feature interaction and joint damage representation learning; and 3) a linear regression module, which flattens the fused features and passes them through a pulsed fully connected regression layer to output continuous damage indices. This index increases monotonically with the degree of damage, and the mean squared error loss function is used as the optimization objective: In the formula, For the sample i Predictive damage indicators; To calibrate reference values ​​for the experiment, N The total number of samples, This represents the mean square error loss.

5. The method according to claim 4, characterized in that: Step 4 Domain generalization loss Based on the definition of the maximum mean difference: In the formula, For the sample i In the domain High-level pulse characteristics representation; For domain The number of samples, For the sample j In the domain High-level pulse characteristics representation; For domain The number of samples; the overall training objective function is defined as: In the formula, To balance the trade-off between classification accuracy and domain invariance, For the overall training objective.

6. The method according to claim 1, characterized in that: The objective function for knowledge distillation in step 5 is: In the formula, The regression loss of the student SCNN relative to the real degenerate label; and These are the impairment metrics output by the teacher's SCNN and the student's SCNN, respectively. KL divergence measures the difference between the two distributions; The coefficient is used to balance the contributions of direct supervision and knowledge transfer.

7. The method according to claim 1, characterized in that: In step 6, the number of parameters of the distilled student SCNN does not exceed 50% of that of the teacher SCNN, the power consumption does not exceed 2 mW, the energy consumption of a single inference does not exceed 1 mJ, and it supports real-time deployment on resource-constrained embedded development boards.

Images