Multi-modal data fusion damage identification system and method for unmanned aerial vehicle structural health management

Abstract

The application relates to the technical field of unmanned aerial vehicle (UAV) structure health monitoring, in particular to a multi-modal data fusion damage identification system and method for UAV structure health management, which comprises a lightweight distributed sensing network, an on-board edge computing module, a self-adaptive monitoring area dynamic division module, the on-board edge computing module, a multi-algorithm fusion diagnosis core module and a health state evaluation and decision support module. The scheme aims to solve the technical problem that the prior art cannot satisfy real-time damage monitoring of the UAV in orbit, and realizes early discovery and accurate positioning of micro damage.

Description

Technical Field

[0001] The embodiments of this application relate to the field of unmanned aerial vehicle (UAV) structural health monitoring technology, and in particular to a multimodal data fusion damage identification system and method for UAV structural health management. Background Technology

[0002] Unmanned aerial vehicles (UAVs) are widely used in reconnaissance, logistics transportation, and infrastructure inspection due to their lightweight structure and high maneuverability. However, under complex loads (such as severe maneuvers and turbulent impacts), their structures are prone to fatigue damage, especially in stress concentration areas such as wing roots and joints. Traditional damage monitoring technologies rely on post-mission data analysis, comparing vibration signals with a baseline to determine damage; alternatively, they employ piezoelectric sensor arrays and Lamb wave analysis, which are sensitive to pore-like damage; or they use high-definition cameras and deep learning to identify surface damage.

[0003] However, traditional damage detection technologies have obvious limitations, such as monitoring lag, inability to achieve real-time early warning during flight, leading to damage accumulation and catastrophic failures; furthermore, sensor networks have high power consumption, making them unsuitable for small drones; or, high-definition cameras are easily affected by light and weather, with reliability dropping sharply in rainy or nighttime environments; and, drones have limited onboard computing power, making it difficult for existing deep learning models (such as convolutional neural networks) to achieve real-time edge computing. Summary of the Invention

[0004] In view of this, embodiments of this application propose a multimodal data fusion damage identification system and method for UAV structural health management, aiming to solve the technical problem that the existing technology cannot meet the requirements of real-time on-orbit damage monitoring of UAVs, and to achieve early detection and accurate location of minor damage.

[0005] To achieve the above objectives, embodiments of this application propose a multimodal data fusion damage identification system for UAV structural health management, comprising: A lightweight distributed sensor network is deployed on the key load-bearing structure of the UAV to collect multimodal sensor data of the UAV structure in real time and simultaneously record flight status parameters from the UAV flight control system. The multimodal sensor data includes multi-point strain data, vibration data, and temperature data. The total weight of the lightweight distributed sensor network does not exceed 0.5% of the maximum takeoff weight of the UAV. An airborne edge computing module is integrated into the UAV flight control system and connected to a lightweight distributed sensor network for preprocessing multimodal strain data and flight status parameters, real-time feature extraction, and rapid anomaly detection. The adaptive monitoring area dynamic division module is interconnected with the UAV flight control system and is used to dynamically adjust the monitoring parameters of the monitoring area based on real-time flight status parameters and a preset structural finite element model. The airborne edge computing module is also used to compute onboard feature parameters in real time using its integrated feature extraction engine in a sliding window manner; the feature parameters include time domain, frequency domain and time-frequency domain feature parameters; The multi-algorithm fusion diagnostic core module adopts an edge-cloud collaborative architecture, integrating airborne rapid anomaly detection algorithms and ground station precise diagnostic algorithms, and performs decision-level fusion of diagnostic results through information fusion methods; The health status assessment and decision support module is used to comprehensively assess the structural safety margin based on damage information, generate a maintenance report with annotations of a three-dimensional digital twin model, and send operational restriction suggestions to the UAV flight control system via a data link.

[0006] Optionally, the lightweight distributed sensor network includes at least one of fiber optic grating sensors and microelectromechanical system strain sensors, and is arranged in an array at the wing root, beam-rib joint, or fuselage connection.

[0007] Optionally, the airborne edge computing module communicates with the lightweight distributed sensor network via a CAN bus or RS-485 bus and collects multimodal sensor data and flight status parameters in real time; the airborne edge computing module integrates moving average filtering and temperature compensation algorithms for preprocessing; the flight status parameters include airspeed, overload, attitude and rudder deflection angle.

[0008] Optionally, the monitoring parameters include monitoring weights, boundaries, and sampling frequency; the adaptive monitoring area dynamic partitioning module is used to: determine the flight state of the UAV based on flight state parameters; under high-load flight conditions, and by calling the pre-set finite element model of the UAV structure, simultaneously obtain the theoretical stress distribution cloud map corresponding to the key load-bearing structure under high-load flight conditions, and map the stress concentration area to the corresponding physical monitoring area sensor array; based on the theoretical stress distribution cloud map, dynamically calculate the weight coefficient of each monitoring area using a fuzzy logic algorithm; wherein, the weight coefficient of each monitoring area is positively correlated with the stress concentration coefficient and overload value of that monitoring area; based on the weight coefficient of each monitoring area, automatically load the matching damage identification algorithm parameter set; wherein, the damage identification algorithm parameter set includes: the sensitivity threshold of the anomaly detection algorithm, the sliding window size of feature extraction, and the weight allocation in the data fusion decision rules.

[0009] Optionally, the multi-algorithm fusion diagnostic core module is used to deploy a fast anomaly detection algorithm on the airborne side and transmit the detected suspicious data to the ground station, as well as deploy an autoencoder or convolutional neural network on the ground station side for diagnosis, and use DS evidence theory to perform decision-level fusion of the diagnostic results of the airborne and ground stations.

[0010] This application proposes a multimodal data fusion damage identification system for UAV structural health management. The system includes a lightweight distributed sensor network deployed on the UAV's critical load-bearing structure, an airborne edge computing module integrated into the UAV flight control system, an adaptive monitoring area dynamic partitioning module interconnected with the UAV flight control system, a multi-algorithm fusion diagnostic core module, and a health status assessment and decision support module. Since the total weight of the lightweight distributed sensor network does not exceed 0.5% of the UAV's maximum takeoff weight, this addresses the technical limitation of its application on small UAVs. The multi-algorithm fusion diagnostic core module adopts an edge-cloud collaborative architecture, integrating airborne rapid anomaly detection algorithms and ground station precise diagnostic algorithms. It also performs decision-level fusion of diagnostic results through information fusion methods, thus distributing computational tasks between the airborne (edge) and ground station (cloud) domains, thereby improving data efficiency. The system offers timeliness and accuracy. Through an adaptive monitoring area dynamic division module, monitoring parameters can be dynamically adjusted based on real-time flight status parameters and a pre-set structural finite element model. This allows for real-time perception of flight status changes and dynamic adjustment of monitoring strategies, maintaining high reliability across different flight phases and effectively suppressing false alarms caused by environmental interference. The health status assessment and decision support module comprehensively evaluates structural safety margins based on damage information, generates a maintenance report with annotations from a 3D digital twin model, and sends operational restriction suggestions to the UAV flight control system via data link. This avoids unnecessary disassembly of the UAV, reducing downtime, minimizing manpower and material costs, and extending the UAV's lifespan. Therefore, this solution aims to address the technical limitations of existing technologies in providing real-time on-orbit damage monitoring for UAVs, enabling early detection and precise location of minor damage.

[0011] To achieve the above objectives, embodiments of this application propose a multimodal data fusion damage identification method for UAV structural health management, the method comprising the following steps: A lightweight distributed sensor network is deployed on the key load-bearing structure of the UAV to collect multimodal sensor data of the UAV structure in real time and simultaneously record the flight status parameters of the UAV flight control system; among which, the multimodal sensor data includes multi-point strain data, vibration data and temperature data. Based on flight status parameters, a preset finite element model is invoked to dynamically adjust the weight of the monitoring area and the parameters of the damage identification algorithm. On the airborne edge computing module, after preprocessing the multimodal strain data and flight state parameters, the feature extraction engine calculates the onboard feature parameters in real time using a sliding window method; the feature parameters include time domain, frequency domain and time-frequency domain feature parameters. It adopts an edge-cloud collaborative architecture, integrates airborne rapid anomaly detection algorithms and ground station accurate diagnostic algorithms, and performs decision-level fusion of diagnostic results through information fusion methods; The system comprehensively assesses the structural safety margin based on damage information, generates a maintenance report including annotations from a 3D digital twin model, and sends operational limitation recommendations to the UAV flight control system via data link.

[0012] Optionally, based on flight state parameters, a preset finite element model is invoked to dynamically adjust the weights of the monitoring areas and the parameters of the damage identification algorithm. This includes: continuously reading flight state parameters; when an overload value is detected in the flight state parameters and continuously exceeds a preset threshold, and the UAV is detected to be in a specific maneuver, it is determined to be a high-load flight state; based on the high-load flight state, a preset finite element model of the UAV structure is invoked, and the theoretical stress distribution cloud map corresponding to the key load-bearing structure under the high-load flight state is obtained, and the stress concentration area is mapped to the corresponding physical monitoring area sensor array; based on the theoretical stress distribution cloud map, a fuzzy logic algorithm is used to dynamically calculate the weight coefficients of each monitoring area; wherein, the weight coefficients of each monitoring area are positively correlated with the stress concentration coefficient and overload value of the detection area; based on the weight coefficients of each monitoring area, a matching set of damage identification algorithm parameters is automatically loaded; wherein, the set of damage identification algorithm parameters includes: the sensitivity threshold of the anomaly detection algorithm, the sliding window size of feature extraction, and the weight allocation in the data fusion decision rules.

[0013] Optionally, an edge-cloud collaborative architecture is adopted, integrating airborne rapid anomaly detection algorithms and ground station precise diagnostic algorithms, and the diagnostic results are fused at the decision level through information fusion methods, including: On the airborne edge computing module, a fast anomaly detection algorithm is continuously running to monitor the feature parameters extracted in real time; the fast anomaly detection algorithm includes isolated forest and / or a type of support vector machine. When the characteristic parameter exceeds the threshold set based on the health baseline for a continuous preset period, an alarm is triggered, and the raw data corresponding to the abnormal time segment is transmitted to the ground station via the UAV data link; When the ground station receives the raw data corresponding to the downlinked abnormal time segment, it obtains the ground depth analysis result based on the ground station's accurate diagnostic algorithm; the ground depth analysis result includes damage type identification result and location information; By using information fusion methods, the preliminary anomaly diagnosis results from the airborne edge side are fused with the ground depth analysis results at the decision level to calculate the final damage confidence, thereby completing the identification, classification, and location of the damage.

[0014] To achieve the above objectives, embodiments of this application also propose an electronic device, including a processor and a memory, wherein the memory stores instructions executable by the processor, and the processor is configured to execute the instructions such that the electronic device can implement the multimodal data fusion damage identification method for UAV structural health management as described above.

[0015] To achieve the above objectives, embodiments of this application also propose a computer-readable storage medium storing a computer program that, when executed by a processor, enables a multimodal data fusion damage identification method for UAV structural health management as described above. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies of this application will be briefly introduced below. Obviously, the following drawings are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings described herein are only used to explain this application and are not intended to limit this application.

[0017] Figure 1 This is a schematic diagram of a multimodal data fusion damage identification system for UAV structural health management provided in one embodiment of this application; Figure 2 This is a schematic diagram of deployment on a drone platform provided in one embodiment of this application; Figure 3 This is a flowchart of a diagnostic mechanism for an edge-cloud collaborative architecture provided in one embodiment of this application; Figure 4 This is a flowchart of a multimodal data fusion damage identification method for UAV structural health management provided in another embodiment of this application; Figure 5 This is a detailed flowchart of a multimodal data fusion damage identification method for UAV structural health management provided in another embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. Those skilled in the art will understand that many technical details have been presented in the embodiments of this application to facilitate better understanding. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of this application. The following embodiments can be combined with and referenced by each other without contradiction.

[0019] Unmanned aerial vehicles (UAVs) are widely used in reconnaissance, logistics transportation, and infrastructure inspection due to their lightweight structure and high maneuverability. However, under complex loads (such as severe maneuvers and turbulent impacts), their structures are prone to fatigue damage, especially in stress concentration areas such as wing roots and joints. Traditional damage monitoring technologies rely on post-mission data analysis, comparing vibration signals with a baseline to determine damage; alternatively, they employ piezoelectric sensor arrays and Lamb wave analysis, which are sensitive to pore-like damage; or they use high-definition cameras and deep learning to identify surface damage.

[0020] However, traditional damage detection technologies have obvious limitations, such as monitoring lag, inability to achieve real-time early warning during flight, leading to damage accumulation and catastrophic failures; furthermore, sensor networks have high power consumption, making them unsuitable for small drones; or, high-definition cameras are easily affected by light and weather, with reliability dropping sharply in rainy or nighttime environments; and, drones have limited onboard computing power, making it difficult for existing deep learning models (such as convolutional neural networks) to achieve real-time edge computing.

[0021] In view of this, embodiments of this application propose a multimodal data fusion damage identification system and method for UAV structural health management, aiming to provide an intelligent damage monitoring system that can adapt to the characteristics of UAV platforms and achieve lightweight, low power consumption, and high real-time performance through edge-cloud collaboration and adaptive algorithms.

[0022] First, the embodiments of this application propose a multimodal data fusion damage identification system for UAV structural health management, that is, a damage identification system that integrates lightweight sensing, adaptive region division and efficient multi-algorithm fusion, which aims to solve the technical problem that the existing technology cannot meet the real-time damage monitoring of UAVs in orbit, and realize the early detection and accurate location of minor damage.

[0023] like Figure 1As shown, embodiments of this application propose a multimodal data fusion damage identification system for UAV structural health management. The system includes: a lightweight distributed sensor network 110, an airborne edge computing module 120, an adaptive monitoring area dynamic partitioning module 130, a multi-algorithm fusion diagnostic core module 140, and a health status assessment and decision support module 150.

[0024] The lightweight distributed sensor network 110 is deployed on the key load-bearing structure of the UAV to collect multimodal sensor data of the UAV structure in real time and simultaneously record the flight status parameters of the UAV flight control system.

[0025] The multimodal sensing data includes multi-point strain data, vibration data, and temperature data; the total weight of the lightweight distributed sensor network does not exceed 0.5% of the maximum takeoff weight of the UAV.

[0026] For example, the key load-bearing structure of a drone can be a critical part of the drone, such as the wing root area, the beam-rib joint area, the fuselage connection, etc.

[0027] For example, such as Figure 2 As shown, Figure 2 This is a schematic diagram illustrating a deployment on a drone platform according to this application. The lightweight distributed sensor network 110 may include a left wing sensor network, a fuselage sensor network, and a right wing sensor network. Data from the left wing sensor network, fuselage sensor network, right wing sensor network, and the drone flight control system is sent to the airborne edge computing module 120, and after processing by other modules, it is sent to the ground control station via a data link.

[0028] For example, multimodal data can be strain data, vibration data, temperature data, etc.

[0029] For example, to achieve the goal of the total weight not exceeding 0.5% of the maximum takeoff weight of the UAV, the following approach is adopted: taking a medium-sized fixed-wing UAV (maximum takeoff weight 200kg) as an example: the maximum weight of the sensor network is allowed to be 1.0kg.

[0030] In one possible embodiment, the lightweight distributed sensor network 110 includes at least one of fiber optic grating sensors and microelectromechanical system strain sensors, and is arranged in an array at the wing root, beam-rib joint, or fuselage connection.

[0031] For example, a fiber Bragg grating (FBG) sensor can be based on the wavelength modulation mechanism of a fiber Bragg grating, where strain changes cause a shift in the reflected wavelength. Microelectromechanical system (MEMS) strain sensors are chip-scale packaged, with individual sensor dimensions (length × width × height) as low as 3 mm × 3 mm × 1 mm, and low power consumption.

[0032] For example, fiber optic grating (FBG) sensors or microelectromechanical system (MEMS) strain sensors are arranged in arrays at key load-bearing structures such as the wing root, beam-rib joints, and fuselage connections of the UAV, with the total weight not exceeding 0.5% of the UAV's maximum takeoff weight.

[0033] For example, the lightweight distributed sensor network 110 can be deployed in the root region of the wing, such as 30% of the wingspan outward from the junction of the wing and fuselage, and sensors at the junction of the wing ribs and skin to detect delamination and delamination damage. A row of sensors can be deployed at the leading edge and the trailing edge to monitor the effects of aerodynamic loads.

[0034] Sensor arrays can also be deployed in beam-rib joint areas, such as at the intersection of wing beams and wing ribs, using a cross-shaped arrangement to monitor bidirectional stress states, and focusing on monitoring stress concentration in bolted connection areas.

[0035] They can also be placed at fuselage joints, such as a ring-shaped sensor array around the wing-fuselage joint, shock-resistant monitoring sensors in the landing gear mounting area, and vibration monitoring sensors at the tail fin connection.

[0036] The airborne edge computing module 120 is integrated into the UAV flight control system and connected to the lightweight distributed sensor network 110 for preprocessing multimodal strain data, real-time feature extraction, and rapid anomaly detection.

[0037] For example, the airborne edge computing module 120 can be connected to the lightweight distributed sensor network 110 via a CAN bus or an RS-485 bus. Through the communication connection with the lightweight distributed sensor network 110, the airborne edge computing module 120 can acquire multimodal strain data and flight status parameters in real time.

[0038] In one possible embodiment, the airborne edge computing module 120 communicates with a lightweight distributed sensor network via a CAN bus or RS-485 bus and collects multimodal sensor data and flight status parameters in real time; the airborne edge computing module integrates moving average filtering and temperature compensation algorithms for preprocessing.

[0039] The flight status parameters include airspeed, overload, attitude, and rudder deflection.

[0040] For example, in the data preprocessing stage, the airborne edge computing module 120 can perform moving average filtering on the received multimodal sensor data to effectively filter out noise signals such as environmental vibration and electromagnetic interference; at the same time, it can perform temperature compensation in combination with the collected temperature data to eliminate the influence of temperature changes on sensor signals and ensure data accuracy.

[0041] For example, in the real-time feature extraction stage, the airborne edge computing module 120 extracts basic feature parameters adapted to the airborne computing power from the preprocessed data, providing data support for rapid anomaly detection.

[0042] For example, during the rapid anomaly detection phase, the airborne edge computing module 120 can run a rapid anomaly detection algorithm with low computational complexity, such as an isolated forest or a support vector machine, to screen the extracted feature parameters in real time, so as to detect abnormal trends in the data in a timely manner and trigger the start-up conditions for subsequent accurate diagnosis.

[0043] For example, taking a UAV performing a climb maneuver as an example, at time T=0ms, the onboard edge computing module 120 starts the sensors to collect data via the CAN bus, and synchronously reads flight status parameters (e.g., airspeed 85m / s, overload 1.2g, pitch angle 15°, rudder deflection +5°); at time T=10ms, the data collection of 120 sensor channels is completed and the moving average filtering process is started; at time T=15ms, the filtering process is completed, and temperature compensation calculation begins, during which the temperature at the wing root is detected to drop from 25 degrees Celsius (°C) to 18°C; at time T=20ms, the data preprocessing is completed, clean strain data is generated and packaged for subsequent feature extraction.

[0044] The airborne edge computing module 120 is also used to compute onboard feature parameters in real time via its integrated feature extraction engine in a sliding window manner.

[0045] The feature parameters include time-domain, frequency-domain, and time-frequency-domain feature parameters.

[0046] Understandably, the airborne edge computing module 120 integrates a lightweight feature extraction engine. Its core function is to use a sliding window to calculate the onboard adaptive feature parameters from the preprocessed multimodal sensor data in real time, providing comprehensive and accurate feature input for multi-algorithm fusion diagnosis.

[0047] For example, the window size in the sliding window method can be dynamically adjusted according to the flight status. In a typical scenario, the window duration is set to 1 second, corresponding to 500 sampling points. The sliding window method ensures that the system can continuously monitor changes in structural state while maintaining computational efficiency.

[0048] Time-domain feature parameters can include mean, variance, peak value, kurtosis, root mean square, peak factor, etc., reflecting the statistical characteristics of data in the time dimension.

[0049] Frequency domain characteristic parameters are obtained by fast Fourier transform to obtain the frequency domain characteristics of the data, which may include FT energy distribution, main frequency offset, spectral peaks, etc., and the frequency domain characteristics of the data are obtained by Fourier transform.

[0050] Time-frequency domain feature parameters, combining time-domain and frequency-domain characteristics, are specifically designed to capture the non-stationary variation patterns of data. These parameters can include wavelet packet energy entropy, etc., and by combining time-domain and frequency-domain characteristics, they accurately capture the non-stationary variation patterns of data. The adaptive monitoring area dynamic division module 130 is interconnected with the UAV flight control system and is used to dynamically adjust the monitoring parameters of the monitoring area based on real-time flight status parameters and a preset structural finite element model.

[0051] Understandably, the adaptive monitoring area dynamic division module 130 can dynamically adjust the monitoring parameters of the monitoring area based on real-time flight status parameters and a preset structural finite element model, so that the system can adapt to the monitoring needs under different flight scenarios.

[0052] In one possible embodiment, the monitoring parameters include monitoring weights, boundaries, and sampling frequency; the adaptive monitoring area dynamic division module is used to: determine the flight state of the UAV based on flight state parameters; under high load flight state, and call the preset finite element model of the UAV structure, while obtaining the theoretical stress distribution cloud map corresponding to the key load-bearing structure under high load flight state, and mapping the stress concentration area to the corresponding physical monitoring area sensor array; based on the theoretical stress distribution cloud map, use fuzzy logic algorithm to dynamically calculate the weight coefficient of each monitoring area.

[0053] The weight coefficient of each monitoring area is positively correlated with the stress concentration coefficient and overload value of that monitoring area. Based on the weight coefficient of each monitoring area, the damage identification algorithm parameter set that matches it is automatically loaded. The damage identification algorithm parameter set includes: the sensitivity threshold of the anomaly detection algorithm, the sliding window size of feature extraction, and the weight allocation in the data fusion decision rule.

[0054] For example, the adaptive monitoring area dynamic division module 130 incorporates a finite element model of the UAV structure. This model can pre-obtain the stress distribution patterns of each region under different flight conditions through structural mechanics simulation. During operation, this module collects flight state parameters in real time (such as climb, cruise, and high-g-load maneuvers like steep turns) and dynamically adjusts the monitoring strategy based on the stress distribution data from the pre-set finite element model. For instance, it adjusts the weights of different monitoring areas to prioritize high-stress areas, optimizes monitoring area boundaries to focus on key load-bearing components, and adjusts sensor sampling frequencies to increase sampling density in high-risk scenarios. This ensures that the system can accurately focus on high-risk areas under complex flight conditions, effectively improving monitoring efficiency and sensitivity.

[0055] For example, the adaptive monitoring area dynamic division module 130 can determine the UAV's flight status based on real-time flight status parameters. When the system detects an overload value continuously exceeding 3g and identifies the UAV as performing a steep turn, it automatically determines it to be in a high-load flight state. At this time, a preset finite element model is invoked to obtain the theoretical stress distribution cloud map corresponding to the key load-bearing structure under this high-load state. The theoretical stress distribution cloud map is used to show that there is a significant stress concentration phenomenon in the wing root region.

[0056] Then, the theoretical stress concentration areas are precisely mapped to the sensor arrays of the physical monitoring areas, particularly the twenty fiber optic grating sensor arrays located at the wing root. A fuzzy logic algorithm is then used to dynamically calculate the weight coefficients for each monitoring area. These weight coefficients are positively correlated with the stress concentration coefficient and overload value of the detected area. For example, the weight coefficient for the wing root area is increased from a baseline of 1.0 to 1.8, reflecting the high-risk level of this area under the current flight conditions. Based on the calculated weight coefficients, the system can automatically load a matching damage identification algorithm parameter set. This parameter set includes three key parameters: the sensitivity threshold of the anomaly detection algorithm is reduced from 150% to 120% of the normal state; the sliding window size for feature extraction is adjusted from 100 milliseconds to 50 milliseconds to improve response speed; and the weight assigned to the data in this area in the data fusion decision rule is increased from 0.5 to 0.8. Through this series of adaptive adjustments, the system achieves enhanced monitoring and accurate diagnosis capabilities for high-risk areas.

[0057] The multi-algorithm fusion diagnostic core module 140 adopts an edge-cloud collaborative architecture, integrating airborne rapid anomaly detection algorithms and ground station precise diagnostic algorithms, and performs decision-level fusion of diagnostic results through information fusion methods.

[0058] In one possible embodiment, the multi-algorithm fusion diagnostic core module is used to deploy a fast anomaly detection algorithm on the airborne side and transmit the detected suspicious data to the ground station, and to deploy an autoencoder or convolutional neural network on the ground station side for diagnosis, and to use DS evidence theory to perform decision-level fusion of the diagnostic results of the airborne and ground stations.

[0059] For example, on the airborne side, fast anomaly detection algorithms with low computational complexity can be deployed to ensure real-time monitoring with limited computing power. Combined with... Figure 3 The flowchart of the edge-cloud collaborative architecture diagnostic mechanism is shown below, using the Isolation Forest algorithm as an example: The isolated forest algorithm is configured as follows: 50 trees are set to balance detection accuracy and computational cost, the number of subsamples is 256 samples to adapt to embedded storage limitations, and the anomaly score threshold is 0.65. Any value exceeding this threshold is considered a suspicious anomaly.

[0060] When the UAV conducts high-altitude reconnaissance missions, the airborne edge computing component continuously monitors characteristic parameters. Assume that at time T, the system detects that the root mean square (RMS) values ​​of strain data from three adjacent sensors at the third rib of the right wing exceed the healthy baseline threshold by 150% for three consecutive cycles. The Isolation Forest algorithm calculates the path length anomaly scores for these data points, with scores of 0.68, 0.72, and 0.70 respectively, all exceeding the threshold of 0.65. The system immediately activates a level-two alert and automatically identifies the abnormal time period. The algorithm records the anomaly start time, duration, relevant sensor numbers, and anomaly score information. Simultaneously, a data compression process is initiated, transmitting the suspicious data to the ground station via a data link.

[0061] On the ground station side, after receiving the downlinked abnormal data, the system initiates a computationally complex, precise diagnostic algorithm for in-depth analysis. The ground station deploys a pre-trained stacked autoencoder model, which consists of an encoder and a decoder. The encoder compresses the input data from 1024 dimensions to a 32-dimensional latent feature space, and the decoder attempts to reconstruct the original data from the latent features. The model is trained on a large number of normal samples to learn the data distribution patterns of healthy states.

[0062] Taking the received abnormal data from the right wing as an example: The ground station system sequentially performs data preprocessing steps to decompress and standardize the downlink compressed data. Then, it performs reconstruction error calculation by inputting the abnormal data into the automatic encoder and calculating the reconstruction error between the output and the original input. Next, it performs pattern matching to compare the reconstruction error pattern with the historical damage database. Finally, it identifies the damage type based on the error distribution characteristics. The damage types include various damage modes such as cracks, degumming, and corrosion.

[0063] The analysis showed that the reconstruction error reached 0.15, which is significantly higher than the 0.05 threshold of normal samples, and the error pattern matched the historical microcrack damage data by 85%. Therefore, the damage type can be confirmed as a first-order microcrack, and it can be precisely located at the junction of the right wing 3rd rib and the lower skin.

[0064] For example, the DS evidence theory can be used to perform decision-level fusion of diagnostic results from airborne and ground stations. For instance, a basic probability allocation setting can be performed first: Basic probability distribution of airborne detection results: , ; Basic probability distribution of ground station diagnostic results: , ; According to Dempster's combination rule, calculate the joint basic probability assignment of the two sources of evidence: Conflict coefficient as follows: ; Confidence of damage after fusion as follows: ; The system can set the fusion confidence threshold to 0.85. Since the calculated fusion confidence score of 0.955 is greater than the fusion confidence threshold of 0.85, the system finally confirms the existence of damage, thereby effectively reducing the risk of misjudgment by a single algorithm and significantly improving the reliability of the diagnostic results.

[0065] The health status assessment and decision support module 150 is used to comprehensively assess the structural safety margin based on damage information, generate a maintenance report containing annotations of a three-dimensional digital twin model, and send operational restriction suggestions to the UAV flight control system via a data link.

[0066] For example, the health status assessment and decision support module 150 can receive damage information output by the multi-algorithm fusion diagnostic core module 140; the damage information includes damage location, damage type, severity, etc.; at the same time, combined with the current flight mission profile, such as the remaining mission duration, flight area environment, load distribution, etc., the remaining structural safety margin is calculated through the structural safety margin assessment model.

[0067] Maintenance reports annotated with 3D digital twin models can be used to accurately mark the location, size, and severity of damage on the model, while providing specific maintenance recommendations, such as flaw detection methods, repair procedures, and component replacement cycles. For example, operational restriction suggestions can be sent to the UAV flight control system via data link to provide real-time decision support for flight safety. Suggestions could include: "Limit roll rate to no more than 30 degrees / second (° / s)," "High-G maneuvers are prohibited," and "Immediate return to base is recommended."

[0068] For example, based on the diagnostic results and the remaining 1 hour of the current mission, the health status assessment and decision support module 150 assesses the remaining structural safety margin as 85%. At this point, it can be determined that the current mission can be completed, but high-G maneuvers need to be limited. A maintenance report is then generated, and the damage location is accurately marked on the three-dimensional digital twin model. At the same time, the maintenance suggestion of "perform ultrasonic testing on the 3rd rib of the right wing after this mission is completed" is automatically written, and an operation restriction instruction of "limiting the roll rate to no more than 30° / s" is sent to the UAV flight control system via data link to ensure flight safety and the accuracy of subsequent maintenance.

[0069] This application proposes a multimodal data fusion damage identification system for UAV structural health management. The system includes a lightweight distributed sensor network deployed on the UAV's critical load-bearing structure, an airborne edge computing module integrated into the UAV flight control system, an adaptive monitoring area dynamic partitioning module interconnected with the UAV flight control system, a multi-algorithm fusion diagnostic core module, and a health status assessment and decision support module. Since the total weight of the lightweight distributed sensor network does not exceed 0.5% of the UAV's maximum takeoff weight, this addresses the technical limitation of its application on small UAVs. The multi-algorithm fusion diagnostic core module adopts an edge-cloud collaborative architecture, integrating airborne rapid anomaly detection algorithms and ground station precise diagnostic algorithms. It also performs decision-level fusion of diagnostic results through information fusion methods, distributing computational tasks between the airborne (edge) and ground station (cloud) domains, thereby improving data real-time performance. Accuracy: Through the adaptive monitoring area dynamic division module, the monitoring parameters of the monitoring area can be dynamically adjusted based on real-time flight status parameters and a preset structural finite element model. This enables real-time perception of changes in flight status and dynamic adjustment of monitoring strategies, maintaining high reliability at different flight stages and effectively suppressing false alarms caused by environmental interference. Through the health status assessment and decision support module, the structural safety margin is assessed by comprehensively evaluating damage information, generating a maintenance report with annotations of a 3D digital twin model, and sending operational restriction suggestions to the UAV flight control system via data link. This avoids unnecessary disassembly of the UAV, thereby reducing downtime, lowering manpower and material consumption, and extending the service life of the UAV. Based on this, this solution provides an intelligent damage monitoring system that adapts to the characteristics of the UAV platform and achieves lightweight, low power consumption, and high real-time performance through edge-cloud collaboration and adaptive algorithms.

[0070] Another embodiment of this application proposes a multimodal data fusion damage identification method for UAV structural health management, applied to an electronic device, wherein the electronic device can be a terminal or a server. This embodiment and the following embodiments will use a server as an example for description. The implementation details of the multimodal data fusion damage identification method for UAV structural health management proposed in this embodiment are described below. The following implementation details are provided for ease of understanding and are not essential for implementing this solution.

[0071] The specific process of the multimodal data fusion damage identification method for UAV structural health management proposed in this embodiment can be described as follows: Figure 4 As shown, it includes: Step 101: Through a lightweight distributed sensor network deployed on the key load-bearing structure of the UAV, multimodal sensor data of the UAV structure are collected in real time, and flight status parameters of the UAV flight control system are recorded simultaneously.

[0072] The multimodal sensing data includes multi-point strain data, vibration data, and temperature data; the total weight of the lightweight distributed sensor network does not exceed 0.5% of the maximum takeoff weight of the UAV.

[0073] Step 102: Based on real-time flight status parameters and a preset structural finite element model, dynamically adjust the monitoring parameters of the monitoring area.

[0074] Step 103: On the airborne edge computing module, after preprocessing the multimodal strain data and flight state parameters, the feature extraction engine calculates the onboard feature parameters in real time using a sliding window method.

[0075] The feature parameters include time-domain, frequency-domain, and time-frequency-domain feature parameters.

[0076] Step 104: An edge-cloud collaborative architecture is adopted, integrating the airborne fast anomaly detection algorithm and the ground station's accurate diagnostic algorithm, and the diagnostic results are fused at the decision level through information fusion methods.

[0077] Step 105: Assess the structural safety margin based on comprehensive damage information, generate a maintenance report including annotations of a three-dimensional digital twin model, and send operational restriction recommendations to the UAV flight control system via data link.

[0078] For a detailed description of steps 101 to 105, please refer to the description of a multimodal data fusion damage identification system for UAV structural health management in the above embodiments, which will not be repeated here.

[0079] like Figure 5 As shown, Figure 5 A detailed flowchart of a multimodal data fusion damage identification method for UAV structural health management is provided for another embodiment of this application.

[0080] First, raw multimodal sensor data is collected through a lightweight sensor network. Next, the system synchronizes data acquisition with flight status to ensure precise temporal alignment between sensor data and state parameters such as airspeed and overload provided by the flight control system. Then, through flight status recognition, the system determines in real time whether the UAV is in different states such as climb, cruise, or maneuvering. Once recognition is complete, a pre-set model corresponding to this state (e.g., a finite element stress distribution model) is invoked, and adaptive monitoring area division is performed accordingly, dynamically adjusting the monitoring weights and parameters for each structural component. On the airborne side, the process enters the real-time processing stage. First, airborne lightweight feature extraction is performed, efficiently calculating key features in the time and frequency domains from the raw data. Then, airborne rapid anomaly monitoring is initiated, using lightweight algorithms for preliminary feature screening. This is a critical decision point: if no anomaly is detected, the system returns to normal monitoring; if an anomaly is detected, the system triggers an alarm and compresses and transmits the relevant feature data to the ground station. At the ground station, upon receiving the downlink data, in-depth diagnostics and confirmation are initiated. More complex models and algorithms are used for precise analysis to determine damage location and health status assessment, identifying the specific location, type, and severity level of the damage. Ultimately, the system generates decision recommendations and visualization reports, such as marking damage on a 3D digital twin model and providing maintenance suggestions like limiting flight payloads, thus completing a full closed loop from perception to decision-making.

[0081] The steps described above are for clarity only. In implementation, they can be combined into one step, or some steps can be broken down into multiple steps, as long as they involve the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0082] Another embodiment of this application provides an electronic device, such as Figure 6 As shown, it includes a processor 31 and a memory 32. The memory 32 stores instructions that the processor 31 can execute. When the processor 31 is configured to execute the instructions, the electronic device can realize a multimodal data fusion damage identification method for UAV structural health management as described in the above method embodiment.

[0083] The memory and processor are connected via a bus, which includes any number of interconnecting buses and bridges, connecting various circuits of one or more processors and the memory. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0084] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0085] Another embodiment of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, can implement a multimodal data fusion damage identification method for UAV structural health management as described in the above method embodiments.

[0086] That is, those skilled in the art will understand that all or part of the steps in the above method embodiments can be implemented by a program instructing related hardware. The program is stored in a storage medium and includes several instructions to cause a device (such as a microcontroller, chip, etc.) or processor to execute all or part of the steps of the method described in the method embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0087] Those skilled in the art will understand that the above embodiments are specific implementations of this application, and in practical applications, various changes can be made in form and detail without departing from the spirit and scope of this application. For those skilled in the art, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A multimodal data fusion damage identification system for structural health management of unmanned aerial vehicles (UAVs), characterized in that, The system includes: A lightweight distributed sensor network is deployed on the key load-bearing structure of the UAV to collect multimodal sensor data of the UAV structure in real time and simultaneously record flight status parameters from the UAV flight control system. The multimodal sensor data includes multi-point strain data, vibration data, and temperature data. The total weight of the lightweight distributed sensor network does not exceed 0.5% of the maximum takeoff weight of the UAV. An airborne edge computing module is integrated into the UAV flight control system and connected to a lightweight distributed sensor network for preprocessing multimodal strain data and flight status parameters, real-time feature extraction, and rapid anomaly detection. The adaptive monitoring area dynamic division module is interconnected with the UAV flight control system and is used to dynamically adjust the monitoring parameters of the monitoring area based on real-time flight status parameters and a preset structural finite element model. The airborne edge computing module is also used to compute onboard feature parameters in real time using its integrated feature extraction engine in a sliding window manner; the feature parameters include time domain, frequency domain and time-frequency domain feature parameters; The multi-algorithm fusion diagnostic core module adopts an edge-cloud collaborative architecture, integrating airborne rapid anomaly detection algorithms and ground station precise diagnostic algorithms, and performs decision-level fusion of diagnostic results through information fusion methods; The health status assessment and decision support module is used to comprehensively assess the structural safety margin based on damage information, generate a maintenance report with annotations of a three-dimensional digital twin model, and send operational restriction suggestions to the UAV flight control system via a data link.

2. The method according to claim 1, characterized in that, The lightweight distributed sensing network includes at least one of fiber optic grating sensors and microelectromechanical system strain sensors, and is arranged in an array at the wing root, beam-rib joint, or fuselage connection.

3. The system according to claim 1, characterized in that, The airborne edge computing module communicates with the lightweight distributed sensor network via CAN bus or RS-485 bus and collects multimodal sensor data and flight status parameters in real time. The airborne edge computing module integrates moving average filtering and temperature compensation algorithms for preprocessing. Flight status parameters include airspeed, overload, attitude and rudder deflection angle.

4. The system according to claim 1, characterized in that, Monitoring parameters include monitoring weights, boundaries, and sampling frequency; the adaptive monitoring area dynamic division module is used for: The flight status of the UAV is determined based on flight status parameters; Under high load flight conditions, the system calls the pre-set finite element model of the UAV structure, obtains the theoretical stress distribution cloud map of the key load-bearing structure under high load flight conditions, and maps the stress concentration area to the corresponding physical monitoring area sensor array. Based on the theoretical stress distribution cloud map, a fuzzy logic algorithm is used to dynamically calculate the weight coefficient of each monitoring area; the weight coefficient of each monitoring area is positively correlated with the stress concentration factor and overload value of that monitoring area. Based on the weight coefficients of each monitoring area, the matching damage identification algorithm parameter set is automatically loaded; the damage identification algorithm parameter set includes: the sensitivity threshold of the anomaly detection algorithm, the sliding window size of feature extraction, and the weight allocation in the data fusion decision rules.

5. The system according to claim 1, characterized in that, The multi-algorithm fusion diagnostic core module is used to deploy a fast anomaly detection algorithm on the airborne side and transmit the detected suspicious data to the ground station. It also deploys an autoencoder or convolutional neural network on the ground station side for diagnosis and uses DS evidence theory to perform decision-level fusion of the diagnostic results from the airborne and ground stations.

6. A multimodal data fusion damage identification method for structural health management of unmanned aerial vehicles (UAVs), characterized in that, include: A lightweight distributed sensor network is deployed on the key load-bearing structure of the UAV to collect multimodal sensor data of the UAV structure in real time and simultaneously record flight status parameters in the UAV flight control system. The multimodal sensor data includes multi-point strain data, vibration data, and temperature data. The total weight of the lightweight distributed sensor network does not exceed 0.5% of the maximum takeoff weight of the UAV. Based on real-time flight status parameters and a preset structural finite element model, the monitoring parameters of the monitoring area are dynamically adjusted. On the airborne edge computing module, after preprocessing the multimodal strain data and flight state parameters, the feature extraction engine calculates the onboard feature parameters in real time using a sliding window method; the feature parameters include time domain, frequency domain and time-frequency domain feature parameters. It adopts an edge-cloud collaborative architecture, integrates airborne rapid anomaly detection algorithms and ground station accurate diagnostic algorithms, and performs decision-level fusion of diagnostic results through information fusion methods; The system comprehensively assesses the structural safety margin based on damage information, generates a maintenance report including annotations from a 3D digital twin model, and sends operational limitation recommendations to the UAV flight control system via data link.

7. The method according to claim 6, characterized in that, The process of dynamically adjusting the weights of the monitoring area and the parameters of the damage identification algorithm based on flight state parameters and a pre-set finite element model includes: The system continuously reads flight status parameters. When an overload value is detected in the flight status parameters and it continues to exceed a preset threshold, and the drone is detected to be in a specific maneuver, it is determined to be a high-load flight state. Based on the high-load flight state, the pre-set finite element model of the UAV structure is called, and the theoretical stress distribution cloud map of the key load-bearing structure under the high-load flight state is obtained. The stress concentration area is then mapped to the corresponding physical monitoring area sensor array. Based on the theoretical stress distribution cloud map, a fuzzy logic algorithm is used to dynamically calculate the weight coefficient of each monitoring area; the weight coefficient of each monitoring area is positively correlated with the stress concentration factor and overload value of that monitoring area. Based on the weight coefficients of each monitoring area, the matching damage identification algorithm parameter set is automatically loaded; the damage identification algorithm parameter set includes: the sensitivity threshold of the anomaly detection algorithm, the sliding window size of feature extraction, and the weight allocation in the data fusion decision rules.

8. The method according to claim 6, characterized in that, The aforementioned architecture employs an edge-cloud collaborative approach, integrating airborne rapid anomaly detection algorithms and ground-based precise diagnostic algorithms. It also utilizes information fusion methods to perform decision-level fusion of the diagnostic results, including: On the airborne edge computing module, a fast anomaly detection algorithm is continuously running to monitor the feature parameters extracted in real time; the fast anomaly detection algorithm includes isolated forest and / or a type of support vector machine. When the characteristic parameter exceeds the threshold set based on the health baseline for a continuous preset period, an alarm is triggered, and the raw data corresponding to the abnormal time segment is transmitted to the ground station via the UAV data link; When the ground station receives the raw data corresponding to the downlinked abnormal time segment, it obtains the ground depth analysis result based on the ground station's accurate diagnostic algorithm; the ground depth analysis result includes damage type identification result and location information; By using information fusion methods, the preliminary anomaly diagnosis results from the airborne edge side are fused with the ground depth analysis results at the decision level to calculate the final damage confidence, thereby completing the identification, classification, and location of the damage.

9. An electronic device, characterized in that, include: The processor and memory, wherein the memory stores instructions executable by the processor, and the processor is configured to, when executing the instructions, enable the electronic device to implement a multimodal data fusion damage identification method for unmanned aerial vehicle structural health management as described in any one of claims 6 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can implement a multimodal data fusion damage identification method for unmanned aerial vehicle structural health management as described in any one of claims 6 to 8.

Images