An unmanned aerial vehicle multi-end cooperative intelligent system and method for detecting defects of cultural relics

Abstract

The application provides a kind of unmanned plane multi-end coordination intelligent system and method for cultural relic flaw detection, related to unmanned plane detection and cultural relic protection technical field, the intelligent system includes airborne end, ground end and cloud end, airborne end, ground end, cloud end is realized two-way encrypted data interaction by 5G / industrial Wi-Fi / POE+, forms the closed loop system of multi-end coordination.This application designs the depth reinforcement learning fault-tolerant controller of cultural relic safety priority, combines ST-GCN early fault prediction and micro air cushion dynamic protection, realizes millisecond level response when fault, preferentially ensures that unmanned plane and cultural relic are not in contact, avoids secondary damage, solves the problem that traditional unmanned plane protection takes itself as the core.This application also proposes cross-modal attention fusion and interpretation network, realizes the accurate alignment and fusion of multispectral, ultrasonic, thermal image multi-modal features, combines multi-task decoding head, realizes the pixel level segmentation, fine-grained classification and quantization of cultural relic defects, and the defect detection precision and quantization ability are greatly improved.

Description

Technical Field

[0001] This application relates to the fields of unmanned aerial vehicle (UAV) detection and cultural relic protection technology, and in particular to a multi-terminal collaborative intelligent system and method for UAV flaw detection of cultural relics. Background Technology

[0002] Cultural relic surveying is a prerequisite for cultural relic protection. Traditional manual surveying methods suffer from low efficiency, limited survey range, and the risk of contact damage to cultural relics. Unmanned aerial vehicle (UAV) flaw detection technology has become an important development direction for cultural relic surveying. However, existing UAV cultural relic surveying technology still has many shortcomings:

[0003] 1. Drone flight safety protection focuses primarily on the protection of the drone itself, failing to prioritize the safety of cultural relics. This results in delayed response to malfunctions and increases the risk of drones colliding with cultural relics, causing secondary damage.

[0004] 2. Flaw detection data mostly uses single-mode acquisition and analysis, and the fusion degree of multi-mode data such as multispectral, ultrasonic, and thermal imaging is low, resulting in insufficient defect detection accuracy and quantification capabilities;

[0005] 3. The lack of a dynamic scheduling strategy for flaw detection equipment, the fixed flaw detection parameters, and the inability to adaptively adjust according to the shape of the cultural relics and the needs of the survey make it difficult to balance flaw detection accuracy and energy consumption.

[0006] 4. Battery management systems are mostly simple displays of remaining power, lacking refined energy consumption allocation and fault-linked power control, which can easily lead to problems such as insufficient power or unreasonable power distribution during the survey process.

[0007] 5. The prediction of the health status of cultural relics is mostly based on simple time series data without taking into account the physical mechanism of the cultural relics materials. The reliability and interpretability of the prediction results are poor, making it difficult to support scientific maintenance decisions. Summary of the Invention

[0008] To overcome the shortcomings of the existing technology, this invention provides a multi-terminal collaborative intelligent flaw detection and safety protection system and method for cultural relic survey drones. Through multi-terminal hardware architecture design and multi-algorithm collaboration, it achieves drone fault tolerance with priority to cultural relic safety, high-precision fusion analysis of multimodal flaw detection data, adaptive dynamic scheduling of flaw detection devices, intelligent battery management, and prediction of cultural relic health evolution based on digital twins, thereby improving the safety, accuracy, and intelligence level of cultural relic surveys.

[0009] To achieve the above objectives, this application provides the following technical solution:

[0010] A multi-terminal collaborative intelligent system for unmanned aerial vehicle (UAV) flaw detection for cultural relics includes an airborne terminal, a ground terminal, and a cloud terminal. The airborne terminal, ground terminal, and cloud terminal achieve bidirectional encrypted data interaction through 5G / industrial Wi-Fi / POE+, forming a closed-loop system for multi-terminal collaboration.

[0011] Preferably, the airborne terminal is based on a high-payload, high-stability UAV platform, and is equipped with a carbon fiber composite material protective fuselage, a multi-sensor fusion module, a micro air cushion dynamic protection module, a flaw detection device, and an airborne computing unit;

[0012] The ground terminal includes a control and analysis system, which comprises a multi-core hardware platform, a dual-frequency encrypted access module, a GPU-accelerated data processing unit, and a human-machine interface.

[0013] The cloud includes a data management and prediction system, which comprises a distributed storage module, an AI cloud computing module, a digital twin modeling module, and a task scheduling module.

[0014] Preferably, the airborne computing unit is the core control unit, which deploys all airborne algorithms to achieve real-time fault prediction, fault-tolerant control, flaw detection scheduling, and battery management.

[0015] This application also provides a method for intelligent flaw detection and safety protection of cultural relic survey drones, using a multi-terminal collaborative intelligent drone system for cultural relic flaw detection as described in any of the above claims.

[0016] Preferably, the protection method comprises the following steps:

[0017] S1: The airborne drone initializes, the battery intelligent management algorithm starts and monitors the power status in real time, the multi-sensor fusion module collects drone flight data, and the flaw detection device completes self-test and preheating.

[0018] S2: The ST-GCN-based early health status prediction model for UAVs analyzes flight data, predicts potential UAV failures, and outputs failure confidence and hazard evolution time.

[0019] S3: The drone flies to the target cultural relic area. The dynamic scheduling algorithm of the detection device adaptively adjusts the action of the telescopic drive mechanism and the working parameters of the flaw detection unit according to the shape of the cultural relic and the survey requirements to complete the multi-modal flaw detection data acquisition.

[0020] S4: If the prediction model detects an anomaly in the drone, the DRL-based cultural relic safety priority fault-tolerant controller triggers the fault-tolerant control strategy, and combines the micro air cushion dynamic protection module to realize drone attitude adjustment, emergency evacuation or safe landing, prioritizing the protection of the cultural relic itself.

[0021] S5: The airborne terminal encrypts and transmits multimodal flaw detection data to the ground terminal. CM-AFINet performs fusion analysis on the data to achieve pixel-level segmentation, fine-grained classification and quantification of cultural relic defects.

[0022] S6: The ground end uploads structured defect data to the cloud, and the cloud constructs a digital twin of the cultural relic. Through a hybrid prediction model of physical information neural network and long short-term memory network, the evolution of the health status of the cultural relic is predicted and the maintenance decision is simulated.

[0023] S7: The cloud feeds back the prediction results and optimized exploration strategies to the airborne and ground terminals, realizing multi-terminal data closure and dynamic optimization of exploration strategies.

[0024] Preferably, the battery intelligent management algorithm in S1 includes remaining power estimation and dynamic energy consumption allocation, and the calculation method is as follows:

[0025] 1.1 The remaining battery capacity (SOC) is estimated based on the fusion of the ampere-hour integral method and the open-circuit voltage method. The calculation formula is as follows:

[0026]

[0027] in, For the battery's rated capacity, This is the charging and discharging current. For correction factor, This is the battery open-circuit voltage;

[0028] 1.2 Based on the UAV fault prediction results and the flaw detection mission progress, dynamic energy consumption allocation is performed, and an energy consumption allocation coefficient is defined. ,in These are the UAV flight control system, flaw detection device, communication module, and protection module, respectively, to meet the requirements. The energy consumption allocation formula is:

[0029]

[0030] in, For the total output power of the battery, when or fault prediction At the same time, increase the energy consumption distribution coefficient of the flight control and protection modules and reduce the energy consumption coefficient of the flaw detection device.

[0031] Preferably, the method for constructing and calculating the ST-GCN early prediction model for UAV health status in S2 is as follows:

[0032] 2.1 Constructing the Unmanned Aerial Vehicle (UAV) Subsystem Diagram Structure , where nodes For the UAV's power, sensing, flight control, payload and other subsystems, side Define node characteristics for physical connections, data flows, or energy flows between subsystems. No. Each node in the time window within Time-series data from dimensional sensors;

[0033] 2.2 Temporal convolution and spatial graph convolution are fused to the node features. Temporal convolution uses 1D-CNN to capture temporal features, and the spatial graph convolution is calculated as follows:

[0034]

[0035] in, For nodes The set of neighboring nodes, For the first Layer graph convolution weights, For bias terms, For activation functions;

[0036] 2.3 After fusing spatiotemporal features, the fault prediction results, including fault type, are output through a fully connected layer. Fault confidence Time it takes for a fault to develop to a dangerous level The calculation formula is:

[0037]

[0038] in, For the last layer of spatial graph convolution features, For the last layer of temporal convolution features, It is a fully connected layer.

[0039] Preferably, the calculation method of the dynamic scheduling algorithm for the detection device in S3 is as follows:

[0040] 3.1 Constructing a partial morphological model of cultural relics based on UAV RTK localization and visual recognition results Extracting the surface curvature of cultural relics Priority of exploration areas ;

[0041] 3.2 Define the set of operating parameters for the flaw detection device ,in The extension speed of the telescopic push rod. This refers to the ultrasonic detection frequency. For multi-spectral LED brightness, To detect dwell time;

[0042] 3.3 A multi-objective optimization function is constructed with the optimization objectives of maximizing flaw detection accuracy and minimizing energy consumption:

[0043]

[0044] in, For the survey area, This is the flaw detection accuracy coefficient. , , These are the power consumptions of the LED, ultrasonic unit, and telescopic drive mechanism, respectively.

[0045] 3.4 The NSGA-II algorithm is used to solve the multi-objective optimization problem, and the optimal operating parameters in the Pareto optimal solution set are obtained. And adjust it in real time according to the appearance of the cultural relics.

[0046] Preferably, the decision-making method of the cultural relic safety priority fault-tolerant controller of the DRL in S4 is as follows:

[0047] 4.1 Defining the State Space ,in This includes the drone's own status, attitude, health prediction results, and other relevant information. The environmental status of the 3D point cloud of cultural relics and the annotation of sensitive areas. This indicates the current progress of the flaw detection mission.

[0048] 4.2 Define the action space A, including fine-grained control commands such as motor power fine-tuning, flight path adjustment, emergency retraction of flaw detection device, and pre-inflation of micro air cushion;

[0049] 4.3 Design a reward function that prioritizes the safety of cultural relics The calculation formula is:

[0050]

[0051] in, As a safety reward; As a reward for completing the task; For efficiency rewards; α, β, γ are weighting coefficients, and ;

[0052] 4.4 The agent and policy network are trained using the DDPG algorithm. Output continuous control actions, value network The value of actions is evaluated, and the strategy is optimized through experience playback and target network updates, ultimately outputting the optimal fault-tolerant control strategy.

[0053] Preferably, the multimodal flaw detection data fusion analysis method of CM-AFINet in S5 is as follows:

[0054] 5.1 Extracting optical features from multispectral images respectively Acoustic characteristics of ultrasound B-scan / C-scan images Thermodynamic characteristics of infrared thermal imaging sequences ,in , For feature map size, , Ct represents the number of feature channels;

[0055] 5.2 Self-attention enhancement is applied to single-modal features, calculated as follows:

[0056]

[0057] Among them, MSA is a multi-head self-attention mechanism;

[0058] 5.3 Design a cross-modal deformable cross-attention module, using acoustic features as query features and optical and thermodynamic features as key-value features, to achieve accurate alignment and fusion of cross-modal features. The fused feature calculation formula is as follows:

[0059]

[0060] 5.4 Construct a multi-task decoding head, including a segmentation head, a classification head, and a quantization head, which respectively output defect pixel-level masks. Defect types Defect depth With area The defect depth is calculated based on the ultrasonic echo time difference:

[0061]

[0062] in, The speed at which ultrasound propagates in cultural relics. This is the time difference between ultrasonic wave transmission and echo reception.

[0063] Compared with the prior art, this application has at least the following beneficial effects:

[0064] 1. This application designs a deep reinforcement learning fault-tolerant controller that prioritizes the safety of cultural relics. It combines ST-GCN early fault prediction with dynamic protection of micro air cushions to achieve millisecond-level response in case of faults. It prioritizes ensuring that the drone does not come into contact with the cultural relics and avoids secondary damage, thus solving the problem of traditional drone protection being centered on itself.

[0065] 2. This application proposes a cross-modal attention fusion and interpretation network to achieve accurate alignment and fusion of multi-spectral, ultrasonic, and thermal imaging multi-modal features. Combined with a multi-task decoding head, it enables pixel-level segmentation, fine-grained classification, and quantification of cultural relic defects, significantly improving defect detection accuracy and quantification capabilities.

[0066] 3. This application designs a dynamic scheduling algorithm for the detection device, which adaptively adjusts the flaw detection parameters according to the shape of the cultural relics and the needs of the survey, and constructs a multi-objective optimization model to achieve a balance between flaw detection accuracy and energy consumption, thus solving the problem of fixed flaw detection parameters in traditional methods.

[0067] 4. This application integrates the ampere-hour integral method and the open-circuit voltage method to achieve accurate estimation of remaining power, and combines fault prediction and mission progress to dynamically allocate energy consumption, ensuring the power supply for UAV flight and safety protection, and improving the UAV's endurance and safe return capability.

[0068] 5. This application constructs a PINN-LSTM hybrid prediction model, which integrates the physical mechanism of cultural relic materials with historical data to achieve long-term prediction of the evolution of cultural relic defects. Combined with digital twins, it simulates maintenance schemes to provide scientific maintenance decisions for cultural relic protection and solves the problem of traditional prediction lacking physical support.

[0069] 6. This application constructs a multi-terminal collaborative technical system of airborne terminal, ground terminal, and cloud terminal, realizing two-way encrypted data interaction and collaborative algorithm work, forming a closed-loop system of "collection-analysis-prediction-optimization", and improving the intelligent level of the entire process of cultural relic survey. Attached Figure Description

[0070] Figure 1 The image shows a SOLIDWORKS model of the UAV used for artifact flaw detection.

[0071] Figure 2 This is a diagram of the "machine-ground-cloud" three-terminal collaborative architecture for the UAV used for cultural relic flaw detection;

[0072] Figure 3 The flowchart of the intelligent flaw detection and safety protection method for cultural relics is as follows. Detailed Implementation

[0073] Please see Figure 1 and Figure 2 This application provides a multi-terminal collaborative intelligent flaw detection and safety protection system for cultural relic survey drones, including an airborne terminal, a ground terminal, and a cloud terminal. The airborne terminal, ground terminal, and cloud terminal achieve bidirectional encrypted data interaction through 5G / industrial Wi-Fi / POE+, forming a closed-loop system for multi-terminal collaboration.

[0074] In one embodiment, the airborne terminal uses a high-payload, high-stability UAV as a platform and is equipped with a carbon fiber composite material protective fuselage, a multi-sensor fusion module, a micro air cushion dynamic protection module, a flaw detection device, and an airborne computing unit. The multi-sensor fusion module includes an IMU, a visual sensor, ultrasonic sensors, radar, etc.; the flaw detection device includes a multispectral imaging unit, an ultrasonic flaw detection unit, and a telescopic drive mechanism.

[0075] The airborne computing unit is the core control unit, which deploys all airborne algorithms to realize real-time fault prediction, fault-tolerant control, flaw detection scheduling and battery management. Specifically, the airborne computing unit deploys an early prediction model of UAV health status based on spatiotemporal graph convolutional networks, a cultural relic safety priority fault-tolerant controller based on deep reinforcement learning, a dynamic scheduling algorithm for detection devices, and a battery intelligent management algorithm.

[0076] The ground terminal includes a control and analysis system, which comprises a multi-core hardware platform, a dual-frequency encrypted access module, a GPU-accelerated data processing unit, and a human-machine interface.

[0077] In one embodiment, the ground terminal is equipped with a cross-modal attention fusion and interpretation network to achieve real-time fusion analysis of multimodal flaw detection data, defect detection and automated report generation, while also supporting manual remote control of UAVs and emergency fault handling.

[0078] The cloud platform includes a data management and prediction system. This system is built on a distributed storage technology (HDFS / Ceph) to construct a data warehouse, equipped with an AI cloud computing platform and a digital twin modeling module. It deploys a hybrid prediction model of physical information neural network-long short-term memory network to achieve long-term storage of cultural relic defect data, construction of digital twins of cultural relics, prediction of the evolution of the health status of cultural relics, and simulation of maintenance decisions. It also supports cross-regional scheduling of multiple survey tasks.

[0079] In one embodiment, the data management and prediction system includes a distributed storage module, an AI cloud computing module, a digital twin modeling module, and a task scheduling module.

[0080] Based on the above model, this application also provides a method for intelligent flaw detection and safety protection of cultural relic survey drones, such as... Figure 2 As shown, the steps of the method are as follows:

[0081] S1: System Initialization: The airborne UAV is powered on, the battery intelligent management algorithm is activated, and the remaining battery power and health status are monitored in real time; the multi-sensor fusion module and the flaw detection device complete self-checks and preheating, and the telescopic drive mechanism, multispectral imaging unit, and ultrasonic flaw detection unit of the flaw detection device enter standby mode; the ground end and the cloud end complete communication connection and enter the data receiving and processing state.

[0082] The battery intelligent management algorithm integrates the ampere-hour integral method and the open-circuit voltage method to accurately estimate the remaining power. Combined with UAV fault prediction results and flaw detection mission progress, it dynamically allocates energy consumption across multiple modules to ensure power supply for UAV flight and safety protection.

[0083] In one embodiment, the battery intelligent management algorithm includes remaining power estimation and dynamic energy consumption allocation, and the calculation method is as follows:

[0084] 1.1 The remaining battery capacity (SOC) is estimated based on the fusion of the ampere-hour integral method and the open-circuit voltage method. The calculation formula is as follows:

[0085]

[0086] in, For the battery's rated capacity, This is the charging and discharging current. For correction factor, This is the battery open-circuit voltage;

[0087] 1.2 Based on the UAV fault prediction results and the flaw detection mission progress, dynamic energy consumption allocation is performed, and an energy consumption allocation coefficient is defined. ,in These are the UAV flight control system, flaw detection device, communication module, and protection module, respectively, to meet the requirements. The energy consumption allocation formula is:

[0088]

[0089] in, For the total output power of the battery, when (Low battery threshold) or fault prediction When the (safe time threshold) is reached, increase the energy consumption allocation coefficient of the flight control and protection modules and reduce the energy consumption coefficient of the flaw detection device to ensure the safe return of the UAV.

[0090] S2: Real-time prediction of UAV health status: The UAV health status early prediction model based on spatiotemporal graph convolutional network analyzes flight data, predicts potential UAV faults, and outputs fault confidence and danger evolution time. Specifically, the airborne multi-sensor fusion module continuously collects operational data of UAV subsystems such as power, sensing, flight control, and payload. The ST-GCN fault prediction model performs spatiotemporal feature fusion analysis on the data and outputs fault type, confidence, and danger evolution time in real time. If no abnormality is detected, the UAV flies to the target cultural relic area according to the preset path. If a potential abnormality is detected, an early warning message is sent to the ground and the fault-tolerant controller is provided with decision-making basis.

[0091] In one embodiment, the UAV health status early prediction model based on spatiotemporal graph convolutional network (ST-GCN) models each subsystem of the UAV as a graph structure, integrates the temporal characteristics of multi-sensor time series data with the spatial characteristics between subsystems, realizes early prediction of potential UAV faults, and outputs fault type, confidence level and danger evolution time, providing quantitative basis for fault-tolerant control.

[0092] Specifically, the construction and calculation method of the UAV health status early prediction model based on spatiotemporal graph convolutional network (ST-GCN) is as follows:

[0093] 2.1 Constructing the Unmanned Aerial Vehicle (UAV) Subsystem Diagram Structure , where nodes For the UAV's power, sensing, flight control, payload and other subsystems, side Define node characteristics for physical connections, data flows, or energy flows between subsystems. No. Each node in the time window within Time-series data from dimensional sensors;

[0094] 2.2 Temporal convolution and spatial graph convolution are fused to the node features. Temporal convolution uses 1D-CNN to capture temporal features, and the spatial graph convolution is calculated as follows:

[0095]

[0096] in, For nodes The set of neighboring nodes, For the first Layer graph convolution weights, For bias terms, For activation functions;

[0097] 2.3 After fusing spatiotemporal features, the fault prediction results, including fault type, are output through a fully connected layer. Fault confidence Time it takes for a fault to develop to a dangerous level The calculation formula is:

[0098]

[0099] in, For the last layer of spatial graph convolution features, For the last layer of temporal convolution features, It is a fully connected layer.

[0100] S3: Dynamic scheduling of flaw detection equipment and multimodal data acquisition: After the UAV flies to the target cultural relic area, it constructs a local morphological model of the cultural relic based on RTK positioning and visual recognition results, and extracts the surface curvature of the cultural relic and the priority of the exploration area; the dynamic scheduling algorithm of the detection equipment solves the optimal flaw detection parameters according to the morphological model, and adaptively adjusts the extension speed of the telescopic push rod, the ultrasonic detection frequency, the brightness of the multispectral LED and the detection dwell time; the telescopic drive mechanism extends the ultrasonic probe to the preset position according to the scheduling command, and the multispectral imaging unit, the ultrasonic flaw detection unit and the infrared thermal imager work synchronously to complete the multimodal flaw detection data acquisition.

[0101] In one embodiment, the dynamic scheduling algorithm of the detection device is based on the cultural relic morphology model and the priority of the survey area. It constructs a multi-objective optimization problem with the goal of maximizing the flaw detection accuracy and minimizing energy consumption. The optimal flaw detection parameters are solved by the NSGA-II algorithm to realize the adaptive scheduling of the telescopic drive mechanism and the flaw detection unit.

[0102] Specifically, the calculation method of the dynamic scheduling algorithm for the detection device is as follows:

[0103] 3.1 Constructing a partial morphological model of cultural relics based on UAV RTK localization and visual recognition results Extracting the surface curvature of cultural relics Priority of exploration areas ;

[0104] 3.2 Define the set of operating parameters for the flaw detection device ,in The extension speed of the telescopic push rod. This refers to the ultrasonic detection frequency. For multi-spectral LED brightness, To detect dwell time;

[0105] 3.3 A multi-objective optimization function is constructed with the optimization objectives of maximizing flaw detection accuracy and minimizing energy consumption:

[0106]

[0107] in, For the survey area, This is the flaw detection accuracy coefficient. , , These are the power consumptions of the LED, ultrasonic unit, and telescopic drive mechanism, respectively.

[0108] 3.4 The non-dominated sorting genetic algorithm (NSGA-II) is used to solve the multi-objective optimization problem, and the optimal working parameters in the Pareto optimal solution set are obtained. And adjust it in real time according to the appearance of the cultural relics.

[0109] S4: Fault-tolerant control prioritizing cultural relic safety: If the fault hazard evolution time predicted by the ST-GCN model is less than the safety threshold, the deep reinforcement learning fault-tolerant controller immediately triggers the fault-tolerant control strategy. Based on the UAV's own state, the state of the cultural relic environment, and the mission progress, it outputs the optimal control commands, including motor power fine-tuning, flight path adjustment, emergency retrieval of the flaw detection device, and pre-inflation of the micro air cushion. The micro air cushion dynamic protection module completes instantaneous inflation within 5ms, enabling the UAV to land safely or evacuate in an emergency, prioritizing no contact between the UAV and the cultural relic throughout the entire process.

[0110] In one embodiment, a cultural relic safety-first fault-tolerant controller based on deep reinforcement learning (DRL) is used: a reward function is designed with cultural relic safety as the highest priority, and an agent is trained through a deep deterministic policy gradient algorithm to achieve fault-tolerant control such as drone attitude adjustment, emergency evacuation, and safe landing in case of failure. Combined with a micro air cushion dynamic protection module, it avoids drone collisions with cultural relics.

[0111] Specifically, the decision-making method of the cultural relic safety priority fault-tolerant controller based on deep reinforcement learning (DRL) is as follows:

[0112] 4.1 Defining the State Space ,in This includes the drone's own status, attitude, health prediction results, and other relevant information. The environmental status of the 3D point cloud of cultural relics and the annotation of sensitive areas. This indicates the current progress of the flaw detection mission.

[0113] 4.2 Define the action space A, including fine-grained control commands such as motor power fine-tuning, flight path adjustment, emergency retraction of flaw detection device, and pre-inflation of micro air cushion;

[0114] 4.3 Design a reward function that prioritizes the safety of cultural relics The calculation formula is:

[0115]

[0116] in, For safety rewards, a high negative reward is given when a drone intrudes into the cultural relic space, a medium negative reward is given when it enters the safety buffer zone, and a positive reward is given when it maintains a safe distance; As a task reward, a positive reward will be taken when the flaw detection path point scan is completed; For efficiency rewards, positive rewards are given for reducing ineffective flights and energy consumption; α, β, and γ are weighting coefficients, and Ensuring the safety of cultural relics is the highest priority.

[0117] 4.4 The agent and policy network are trained using the Deep Deterministic Policy Gradient (DDPG) algorithm. Output continuous control actions, value network The value of actions is evaluated, and the strategy is optimized through experience playback and target network updates, ultimately outputting the optimal fault-tolerant control strategy.

[0118] S5: Ground-based Multimodal Data Fusion and Defect Analysis: The airborne terminal transmits the collected multimodal flaw detection data to the ground terminal via 5G / Industrial Wi-Fi dual-mode encryption. The ground-based GPU-accelerated data processing unit activates the cross-modal attention fusion and interpretation network (CM-AFINet) to perform self-attention enhancement and cross-modal fusion on multispectral, ultrasonic, and thermal imaging features. The multi-task decoding head outputs the pixel-level mask, type, depth, and area of ​​the cultural relic defects. Simultaneously, the ground terminal automatically generates a flaw detection report, supporting online expert annotation and collaborative analysis. In one embodiment, the cross-modal attention fusion and interpretation network (CM-AFINet) achieves precise alignment and fusion of multispectral, ultrasonic, and thermal imaging features through self-attention and cross-modal deformable cross-attention mechanisms, constructing a multi-task decoding head to achieve pixel-level segmentation, fine-grained classification, and quantification of cultural relic defects.

[0119] Specifically, the multimodal flaw detection data fusion and analysis method of the cross-modal attention fusion and interpretation network (CM-AFINet) is as follows:

[0120] 5.1 Extracting optical features from multispectral images respectively Acoustic characteristics of ultrasound B-scan / C-scan images Thermodynamic characteristics of infrared thermal imaging sequences ,in , For feature map size, , Ct represents the number of feature channels;

[0121] 5.2 Self-attention enhancement is applied to single-modal features, calculated as follows:

[0122]

[0123] Among them, MSA is a multi-head self-attention mechanism;

[0124] 5.3 Design a cross-modal deformable cross-attention module, using acoustic features as query features and optical and thermodynamic features as key-value features, to achieve accurate alignment and fusion of cross-modal features. The fused feature calculation formula is as follows:

[0125]

[0126] 5.4 Construct a multi-task decoding head, including a segmentation head, a classification head, and a quantization head, which respectively output defect pixel-level masks. Defect types Defect depth With area The defect depth is calculated based on the ultrasonic echo time difference:

[0127]

[0128] in, The speed at which ultrasound propagates in cultural relics. This is the time difference between ultrasonic wave transmission and echo reception.

[0129] S6: Cloud-based Digital Twin Modeling and Health Evolution Prediction: Structured defect data, environmental monitoring data, and flaw detection reports are uploaded to the cloud from the ground. The cloud uses distributed storage technology to archive the data and construct a digital twin of the cultural relic. A hybrid prediction model combining a physical information neural network (PINN) and a long short-term memory (LSTM) network is used to predict the evolution of the cultural relic's health status and simulate maintenance decisions. This PINN-LSTM hybrid prediction model integrates the physical mechanisms of the cultural relic's materials with historical data to predict the rate of defect expansion and the probability of new defect emergence. Simultaneously, it simulates the implementation effects of different maintenance schemes within the digital twin and constructs an evaluation function to solve for the optimal maintenance decision.

[0130] In one embodiment, the Physical Information Neural Network-Long Short-Term Memory Network Hybrid Prediction Model (PINN-LSTM Hybrid) integrates the PINN branch driven by physical mechanisms and the attention LSTM branch driven by data to construct a digital twin of cultural relics, realize long-term prediction of the evolution of defects in cultural relics, simulate the effect of maintenance schemes in the twin, and output the optimal maintenance decision.

[0131] Specifically, the method for predicting the health evolution of cultural relics using the Physical Information Neural Network-Long Short-Term Memory Network Hybrid Prediction Model (PINN-LSTMHybrid) is as follows:

[0132] 6.1 Constructing a digital twin of cultural relics ,in For the initial three-dimensional model of the cultural relic, These are the mechanical parameters of the material. For structured defect data, This includes environmental monitoring data such as temperature, humidity, and micro-vibrations.

[0133] 6.2 Construct the PINN branch, embedding the elasticity equations and heat conduction equations of the cultural relic materials as soft constraints into the neural network. The physical constraint loss function is:

[0134]

[0135] in, For physical governing equations, Responding to the structure of cultural relics;

[0136] 6.3 Construct an attention-based LSTM branch, inputting historical defect data and environmental data. Using an attention mechanism, focus on the impact of sudden environmental changes on defect evolution, and output data-driven defect evolution characteristics. ;

[0137] 6.4 Physical characteristics of fused PINN branches Data analysis characteristics of the LSTM branch , obtain fusion features The fully connected layer outputs the prediction results of artifact defect evolution, including the defect propagation rate. Probability of new defects emerging ;

[0138] 6.5 Simulate the implementation effects of different maintenance schemes in a digital twin and construct an evaluation function for the maintenance schemes. ,in As to the degree of defect improvement, To maintain costs, To maintain the project schedule, find the optimal maintenance plan.

[0139] S7: Multi-terminal data closed loop and strategy optimization: The cloud feeds back the cultural relic health prediction results and the optimized exploration strategy to the airborne terminal and the ground terminal; the airborne terminal adjusts the subsequent flaw detection path and parameters according to the feedback results, and the ground terminal updates the flaw detection task plan, realizing the closed-loop interaction of multi-terminal data and dynamic optimization of exploration strategy. If a second exploration is required, the UAV will execute the subsequent flaw detection task according to the optimized strategy.

[0140] The above content will be explained in conjunction with simulation experiments:

[0141] This embodiment uses the survey of ancient architectural stone carvings as an example to verify the system and method of the present invention.

[0142] 1. System Setup: The airborne terminal uses a multi-rotor UAV with a payload of ≥2kg, equipped with a carbon fiber composite fuselage, RTK positioning module, IMU sensor, visual obstacle avoidance sensor, and micro air cushion dynamic protection module. The flaw detection device integrates a Sony IMX428 multispectral sensor, a Verasonics L11-5v ultrasonic probe, and a Firgelli L12-50-12-1 electric actuator. The ground terminal uses an Intel i9 multi-core processor, dual 4K high-resolution displays, a 5G / industrial Wi-Fi dual-band communication module, and an NVIDIA RTX 4090 GPU acceleration unit. The cloud terminal is based on the HDFS distributed storage system and deploys the TensorFlow / PyTorch AI cloud computing framework to build a digital twin of the stone carving cultural relic.

[0143] 2. Algorithm Training and Deployment: The ST-GCN fault prediction model is trained using historical UAV fault data and normal operation data, with a training set to test set ratio of 8:2, and the model's fault prediction accuracy is ≥95%; the DRL fault-tolerant controller constructs an ancient building stone carving cultural relic scene in the Unity simulation environment and conducts millions of training runs to ensure a decision-making strategy that prioritizes the safety of cultural relics; the CM-AFINet network is trained using defect annotation data of ancient building stone carving cultural relics, with defect segmentation mIoU ≥90% and classification accuracy ≥92%; the PINN-LSTM hybrid prediction model is trained using the mechanical parameters of stone carving cultural relic materials and historical flaw detection data, with a defect evolution prediction error ≤8%.

[0144] 3. On-site survey implementation: After the UAV takes off, the ST-GCN model monitors the UAV's status in real time, and the battery intelligent management algorithm estimates the remaining power to be 90%, with energy consumption evenly distributed; the UAV flies to the ancient building stone carving area, visual recognition constructs a stone carving shape model, and the detection device dynamic scheduling algorithm solves for the optimal parameters: telescopic push rod extension speed 5mm / s, ultrasonic detection frequency 5MHz, multispectral LED brightness 80%, and detection dwell time 0.5s; the flaw detection device completes multimodal data acquisition according to the parameters, and the airborne terminal transmits the data to the ground terminal.

[0145] 4. Defect Analysis and Prediction: The ground-based CM-AFINet network performs fusion analysis on the data, detecting two shallow cracks and one internal micro-cavity on the stone carving surface, accurately outputting the defect location, depth (0.5-2cm), and area; the cloud-based defect data is imported into the digital twin of the stone carving, and the PINN-LSTM hybrid prediction model predicts that under natural conditions, the annual expansion rate of shallow cracks is about 0.1cm, and the internal cavity does not expand significantly. Simulation shows that the optimal maintenance solution is to fill the shallow cracks with nanomaterials, which has low maintenance cost, short construction period, and good repair effect.

[0146] 5. Fault Simulation Verification: During the survey, a minor overheating fault of the UAV motor was simulated. The ST-GCN model predicted a fault confidence level of 0.92 and a danger evolution time of 10 seconds. The DRL fault-tolerant controller immediately triggered the fault-tolerant strategy, reduced the energy consumption allocation coefficient of the flaw detection device, increased the energy consumption coefficient of the flight control and protection modules, and controlled the UAV to retreat away from the stone carving. The micro air cushion was pre-inflated, and the UAV landed smoothly in a safe area without contacting the cultural relic. This verified the fault tolerance and cultural relic protection capabilities of the present invention.

[0147] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

Claims

1. A multi-terminal collaborative intelligent system for unmanned aerial vehicle (UAV) flaw detection of cultural relics, characterized in that: It includes airborne terminals, ground terminals, and cloud terminals. The airborne terminals, ground terminals, and cloud terminals achieve bidirectional encrypted data interaction through 5G / industrial Wi-Fi / PoE+, forming a closed-loop system with multi-terminal collaboration.

2. The UAV multi-terminal collaborative intelligent system for cultural relic flaw detection according to claim 1, characterized in that: The airborne terminal uses a high-payload, high-stability UAV as a platform and is equipped with a carbon fiber composite material protective fuselage, a multi-sensor fusion module, a micro air cushion dynamic protection module, a flaw detection device, and an airborne computing unit. The ground terminal includes a control and analysis system, which comprises a multi-core hardware platform, a dual-frequency encrypted access module, a GPU-accelerated data processing unit, and a human-machine interface. The cloud includes a data management and prediction system, which comprises a distributed storage module, an AI cloud computing module, a digital twin modeling module, and a task scheduling module.

3. The UAV multi-terminal collaborative intelligent system for cultural relic flaw detection according to claim 2, characterized in that: The airborne computing unit is the core control unit, which deploys all airborne algorithms to achieve real-time fault prediction, fault-tolerant control, flaw detection scheduling, and battery management.

4. A method for intelligent flaw detection and safety protection using unmanned aerial vehicles (UAVs) for cultural relic surveying, characterized in that: The UAV multi-terminal collaborative intelligent system for cultural relic flaw detection as described in any one of claims 1-3.

5. The method for intelligent flaw detection and safety protection of cultural relic survey drones according to claim 4, characterized in that: The protection method steps are as follows: S1: The airborne drone initializes, the battery intelligent management algorithm starts and monitors the power status in real time, the multi-sensor fusion module collects drone flight data, and the flaw detection device completes self-test and preheating. S2: An early prediction model for the health status of UAVs based on ST-GCN analyzes flight data, predicts potential UAV failures, and outputs the failure confidence and hazard evolution time. S3: The drone flies to the target cultural relic area. The dynamic scheduling algorithm of the detection device adaptively adjusts the action of the telescopic drive mechanism and the working parameters of the flaw detection unit according to the shape of the cultural relic and the survey requirements to complete the multi-modal flaw detection data acquisition. S4: If the prediction model detects an anomaly in the drone, the DRL-based cultural relic safety priority fault-tolerant controller triggers the fault-tolerant control strategy, and combines the micro air cushion dynamic protection module to realize drone attitude adjustment, emergency evacuation or safe landing, prioritizing the protection of the cultural relic itself. S5: The airborne terminal encrypts and transmits multimodal flaw detection data to the ground terminal. CM-AFINet performs fusion analysis on the data to achieve pixel-level segmentation, fine-grained classification and quantification of cultural relic defects. S6: The ground end uploads structured defect data to the cloud, and the cloud constructs a digital twin of the cultural relic. Through a hybrid prediction model of physical information neural network and long short-term memory network, the evolution of the health status of the cultural relic is predicted and maintenance decision is simulated. S7: The cloud feeds back the prediction results and optimized exploration strategies to the airborne and ground terminals, realizing multi-terminal data closure and dynamic optimization of exploration strategies.

6. The intelligent flaw detection and safety protection method for cultural relic survey drones according to claim 4, characterized in that: The battery intelligent management algorithm in S1 includes remaining power estimation and dynamic energy consumption allocation, and the calculation method is as follows: 1.1 The remaining battery capacity (SOC) is estimated based on the fusion of the ampere-hour integral method and the open-circuit voltage method. The calculation formula is as follows: in, For the battery's rated capacity, This is the charging and discharging current. For correction factor, This is the battery open-circuit voltage; 1.2 Based on the UAV fault prediction results and the flaw detection mission progress, dynamic energy consumption allocation is performed, and an energy consumption allocation coefficient is defined. ,in These are the UAV flight control system, flaw detection device, communication module, and protection module, respectively, to meet the requirements. The energy consumption allocation formula is: in, For the total output power of the battery, when or fault prediction At the same time, increase the energy consumption distribution coefficient of the flight control and protection modules and reduce the energy consumption coefficient of the flaw detection device.

7. The intelligent flaw detection and safety protection method for cultural relic survey drones according to claim 4, characterized in that: The construction and calculation method of the ST-GCN UAV health status early prediction model in S2 is as follows: 2.1 Constructing the Unmanned Aerial Vehicle (UAV) Subsystem Diagram Structure , where nodes For the UAV's power, sensing, flight control, payload and other subsystems, side Define node characteristics for physical connections, data flows, or energy flows between subsystems. No. Each node in the time window within Time-series data from dimensional sensors; 2.2 Temporal convolution and spatial graph convolution are fused to the node features. Temporal convolution uses 1D-CNN to capture temporal features, and the spatial graph convolution is calculated as follows: in, For nodes The set of neighboring nodes, For the first Layer graph convolution weights, For bias terms, For activation functions; 2.3 After fusing spatiotemporal features, the fault prediction results, including fault type, are output through a fully connected layer. Fault confidence Time it takes for a fault to develop to a dangerous level The calculation formula is: in, For the last layer of spatial graph convolution features, For the last layer of temporal convolution features, It is a fully connected layer.

8. The intelligent flaw detection and safety protection method for cultural relic survey drones according to claim 4, characterized in that: The calculation method of the dynamic scheduling algorithm for the detection device in S3 is as follows: 3.1 Constructing a partial morphological model of cultural relics based on UAV RTK localization and visual recognition results Extracting the surface curvature of cultural relics Priority of exploration areas ; 3.2 Define the set of operating parameters for the flaw detection device ,in The extension speed of the telescopic push rod. This refers to the ultrasonic detection frequency. For multi-spectral LED brightness, To detect dwell time; 3.3 A multi-objective optimization function is constructed with the optimization objectives of maximizing flaw detection accuracy and minimizing energy consumption: in, For the survey area, This is the flaw detection accuracy coefficient. , , These are the power consumptions of the LED, ultrasonic unit, and telescopic drive mechanism, respectively. 3.4 The NSGA-II algorithm is used to solve the multi-objective optimization problem, and the optimal operating parameters in the Pareto optimal solution set are obtained. And adjust it in real time according to the appearance of the cultural relics.

9. The intelligent flaw detection and safety protection method for cultural relic survey drones according to claim 4, characterized in that: The decision-making method of the cultural relic safety priority fault-tolerant controller of the DRL in S4 is as follows: 4.1 Defining the State Space ,in This includes the drone's own status, such as position, attitude, and health prediction results. The environmental status of the 3D point cloud of cultural relics and the annotation of sensitive areas. This indicates the current progress of the flaw detection mission. 4.2 Define the action space A, including fine-grained control commands such as motor power fine-tuning, flight path adjustment, emergency retraction of flaw detection device, and pre-inflation of micro air cushion; 4.3 Design a reward function that prioritizes the safety of cultural relics The calculation formula is: in, As a safety reward; As a reward for completing the task; For efficiency rewards; α, β, γ are weighting coefficients, and ; 4.4 The agent and policy network are trained using the DDPG algorithm. Output continuous control actions, value network The value of actions is evaluated, and the strategy is optimized through experience playback and target network updates, ultimately outputting the optimal fault-tolerant control strategy.

10. A method for intelligent flaw detection and safety protection of cultural relic survey drones according to claim 4, characterized in that: The multimodal flaw detection data fusion analysis method of CM-AFINet in S5 is as follows: 5.1 Extracting optical features from multispectral images respectively Acoustic characteristics of ultrasound B-scan / C-scan images Thermodynamic characteristics of infrared thermal imaging sequences ,in , For feature map size, , Ct represents the number of feature channels; 5.2 Self-attention enhancement is applied to single-modal features, calculated as follows: Among them, MSA is a multi-head self-attention mechanism; 5.3 Design a cross-modal deformable cross-attention module, using acoustic features as query features and optical and thermodynamic features as key-value features, to achieve accurate alignment and fusion of cross-modal features. The fused feature calculation formula is as follows: 5.4 Construct a multi-task decoding head, including a segmentation head, a classification head, and a quantization head, which respectively output defect pixel-level masks. Defect types Defect depth With area The defect depth is calculated based on the ultrasonic echo time difference: in, The speed at which ultrasound propagates in cultural relics. This is the time difference between ultrasonic wave transmission and echo reception.

Images