A small UAV inspection data management method

Abstract

The present invention discloses a small-scale unmanned aerial vehicle inspection data management method, relates to the technical field of unmanned aerial vehicle inspection, and is used to solve the problem of unclear unmanned aerial vehicle inspection data management; the present invention associates the multimodal data collected by the unmanned aerial vehicle with the equipment position through high-precision positioning technology and timestamp synchronization, realizes the spatiotemporal consistency of the inspection data, and performs multimodal feature fusion and dimensionality reduction processing to generate a unified feature representation, improves the accuracy and efficiency of data analysis, combines historical data with real-time collected data, monitors the equipment status in real time, identifies and classifies fault modes and assesses risks, predicts fault development trends through a time series prediction model, generates maintenance suggestions, improves the timeliness of fault diagnosis, dynamically optimizes inspection task priorities and path planning based on fault detection results, generates optimal inspection routes, and corrects flight paths, thereby improving the intelligence and accuracy of inspection tasks and significantly improving the efficiency and safety of unmanned aerial vehicle inspection systems.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle inspection technology, and more specifically, to a small unmanned aerial vehicle inspection data management method. Background Art

[0002] Drone inspection is an intelligent inspection method that uses drones equipped with multiple sensors (such as high-definition cameras, temperature and humidity sensors, vibration sensors, etc.) to conduct real-time detection, data collection and fault diagnosis of target areas. Traditional inspection methods usually rely on manual or fixed equipment, which is not only inefficient, but also has inspection blind spots, manual errors and safety hazards. Especially in complex or dangerous scenes such as substations, power lines, petrochemicals, and transportation facilities, traditional methods are difficult to meet the needs of efficient, safe and accurate inspections.

[0003] Deficiencies in existing technologies: The multimodal data processing method collected during drone inspections is too simple, making it difficult to effectively integrate images, sensors, and location information, resulting in an incomplete analysis of equipment status and limited fault detection capabilities. Especially when identifying complex fault modes, it is often difficult to make accurate judgments, and there is a lack of predictions on fault development trends, making it impossible to provide maintenance recommendations in a timely manner. At the same time, the path planning and scheduling of inspection tasks are usually fixed and cannot be dynamically adjusted according to real-time data, resulting in low utilization efficiency of inspection resources, limiting the intelligent development and actual application effects of drone inspections. Summary of the invention

[0004] In order to overcome the above-mentioned defects of the prior art, there is a solution as follows to solve the problem of unclear management of drone inspection data in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A small UAV inspection data management method comprises the following steps:

[0007] Obtain the spatial position and collected data of the drone during the inspection mission, synchronize the data in time and space through timestamps, and associate the collected data with the equipment location;

[0008] The collected data is subjected to noise removal, standardization and multimodal feature extraction, and multimodal feature fusion is performed through a deep learning network. The fused features are subjected to dimensionality reduction processing to generate feature representation;

[0009] Based on historical data and currently collected data, the equipment in the inspection area is monitored in real time, equipment faults are identified and classified, equipment risks are assessed based on the identified fault modes and severity scores, and fault development trends are analyzed through time series prediction models to generate equipment maintenance recommendations;

[0010] Based on real-time fault detection results, the priority of inspection tasks is dynamically adjusted, and the inspection path is optimized by combining flight path length, energy consumption and equipment risk score to generate the optimal inspection route, and the path is dynamically corrected during UAV mission execution.

[0011] In a preferred embodiment, the spatial position and collected data of the drone in the inspection task are obtained, and the data is synchronized in time and space through timestamps to associate the collected data with the device position. The specific steps are as follows:

[0012] Obtain the coordinates of the drone at each time point, and perform position correction based on the correction signal transmitted between the base station coordinates and the mobile terminal coordinates to determine the spatial position;

[0013] Synchronize all the sensor data collected by the drone with the drone’s location information, and unify the collected data into the same time dimension through timestamps. The sensor data includes image data, environmental data, and sensor data;

[0014] The image data collected by the drone is fused with the environmental data using a multimodal deep fusion network to generate a dataset with location and time labels.

[0015] In a preferred embodiment, noise removal, standardization and multimodal feature extraction are performed on the collected data, and multimodal feature fusion is performed through a deep learning network, and the fused features are subjected to dimensionality reduction processing to generate feature representation. The specific steps include:

[0016] Use Kalman filtering to smooth the collected data and perform standardization and normalization processing;

[0017] A convolutional neural network is used to extract features from image data, and an autoregressive model is used to extract time series features from sensor data. The drone position data is obtained by calculating the velocity, acceleration, and rate of change of the drone trajectory, and the extracted time domain and frequency domain features are used as trajectory features.

[0018] A deep neural network is used to fuse the features of different modalities to obtain a unified multimodal feature vector. The principal component analysis method is used to reduce the dimension of the fused multimodal features, which are then used as feature representation after dimensionality reduction.

[0019] In a preferred embodiment, based on historical data and currently collected data, real-time status monitoring is performed on the equipment in the inspection area to identify and classify equipment failures. The specific steps are as follows:

[0020] Determine sensor data anomalies, monitor temperature, humidity and vibration sensor data in real time, use isolation forests to identify anomalies based on the preset normal range, detect image data anomalies, process high-definition camera data in real time, determine abnormal features in the image based on edge detection and feature point matching, and obtain abnormal fluctuations or deviations in the currently collected sensor data and image data;

[0021] Based on historical data and calibrated fault categories, the classification model is trained using support vector machine learning method;

[0022] The trained model outputs a predicted failure mode by analyzing real-time sensor data and image data. If the predicted failure mode shows that the device is in a faulty state, an alarm is triggered and the subsequent fault handling procedure is entered. Each failure mode corresponds to a set of sensor features and image features.

[0023] In a preferred embodiment, the equipment risk is assessed based on the identified failure modes and severity scores, and the failure development trend is analyzed through a time series prediction model to generate equipment maintenance recommendations, including the following steps:

[0024] Based on the identified failure modes, the severity of the failure is evaluated, and each failure mode is assigned a severity score;

[0025] The fault severity score expression is: , where R(t) is the severity score of the fault at time t, is the i-th fault mode identified at the current moment, is the weight of the ith failure mode, Is the failure mode The extent of the impact on the system;

[0026] Use time series prediction methods to predict the development of faults and quantitatively assess future risks;

[0027] Integrate historical data, real-time fault scores, and environmental characteristics to build a time series data set as prediction input, fault severity score R(t) and real-time risk score As the key features of the time series input, the time series matrix X(t) is constructed, and each line is the multimodal feature of each time point, expressed as: , in, , …, is the 1st, 2nd, …, mth modal sensor data at time point t, n is the length of the time series, and m represents the total number of sensors;

[0028] Divide the time series into windows of fixed length, each of which contains the features of the past n time points, and is used to predict the future Failure score at each time point;

[0029] The ADF test method is used to determine whether the data is stationary. For non-stationary sequences, difference or logarithmic transformation is performed: ;

[0030] Statistical feature extraction is performed on the data in each time window, including maximum value, minimum value, and peak change rate;

[0031] Perform dynamic correlation analysis between modalities, analyze the dynamic correlation between modal data of different sensors, calculate the correlation between each modality through the cross-correlation function, and form cross-modal correlation features: , where i, j are mode numbers, k is the lag time, , are the data of sensor i and j at time t respectively; , are the means of sensor i and sensor j respectively, and N represents the total length of the time series;

[0032] Identify causal relationships or temporal dependencies between multimodal data by analyzing the correlation between sensor modalities i and j at different lag times k;

[0033] To predict the failure risk at a future time point, a long short-term memory network is used to process time series data. The input layer is a time series X(t) of length n. The long short-term memory network hidden layer extracts the dynamic pattern in the time series. The fully connected output layer predicts the failure score at a future time point. , the prediction formula is: ;

[0034] Combine the fault score prediction with the fault mode prediction and convert the prediction results into specific risk quantification indicators. The distribution of risks is divided into low, medium and high risks, as follows: The risk is low when The risk is medium when High risk, among which, is a low risk threshold, For high risk thresholds, maintenance recommendations are made based on the fault development trend.

[0035] In a preferred embodiment, based on the real-time fault detection results, the inspection task priority is dynamically adjusted, the inspection path is optimized in combination with the flight path length, energy consumption and equipment risk score, the optimal inspection route is generated, and the path is dynamically corrected during the UAV task execution, including the following steps:

[0036] The priority of the task is determined based on the failure mode, severity score and risk prediction of the equipment. The priority of the task is expressed as: ,in, is the priority of the task, is the weight of the device, is the weight of future risk;

[0037] Based on the inspection task priority and real-time fault detection results, the inspection path is dynamically planned, and particle swarm optimization is used to optimize the path when there are multiple target points;

[0038] The objective function is designed to minimize the flight distance: ,in, For flight from arrive The flight distance between

[0039] Minimize energy consumption: ,in, For flight from arrive The energy consumption required between

[0040] Maximize task priority: ,in, For inspection point The task priority weights, high priority points have higher weights;

[0041] The multi-objective optimization is transformed into a single-objective problem through a weighted method: ,in, , is the weight coefficient, which is used to balance the impact of different objectives;

[0042] Define particles, each particle represents an arrangement of inspection paths , i.e. the order of visiting inspection points;

[0043] Initialize N particles, and for each particle, randomly generate the initial path and the change direction of the path;

[0044] Use the objective function F(P) to calculate the fitness of each particle;

[0045] In each iteration, the particle velocity is updated according to the individual optimal solution and the global optimal solution;

[0046] Adjust the particle's path based on the updated velocity;

[0047] Calculate the value of the objective function according to the new particle path, update the fitness, and compare the current fitness with the historical optimal fitness for each particle to update the global optimal solution;

[0048] When the maximum number of iterations is reached or the global optimal solution converges without significant changes in consecutive k iterations, the particle position corresponding to the global optimal fitness value is output as the optimal inspection path.

[0049] The technical effects and advantages of a small UAV inspection data management method of the present invention are as follows:

[0050] The present invention uses high-precision positioning technology and timestamp synchronization to associate the multimodal data collected by the drone with the equipment location, achieve temporal and spatial consistency of the inspection data, and perform multimodal feature fusion and dimensionality reduction processing to generate a unified feature representation, thereby improving the accuracy and efficiency of data analysis. It combines historical data with real-time collected data to monitor equipment status in real time, identify and classify fault modes and assess risks, predict fault development trends through time series prediction models, generate maintenance recommendations, and improve the timeliness of fault diagnosis. Based on the fault detection results, it dynamically optimizes the inspection task priority and path planning, generates the optimal inspection route, and corrects the flight path, thereby improving the intelligence and accuracy of the inspection task and significantly improving the efficiency and safety of the drone inspection system. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 The present invention is a flowchart of a small UAV inspection data management method. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] In order to achieve the above objectives, Figure 1 A structural schematic diagram of a small UAV inspection data management method of the present invention is given, which specifically includes the following steps:

[0054] Obtain the spatial position and collected data of the drone during the inspection mission, synchronize the data in time and space through timestamps, and associate the collected data with the equipment location;

[0055] The collected data is subjected to noise removal, standardization and multimodal feature extraction, and multimodal feature fusion is performed through a deep learning network. The fused features are subjected to dimensionality reduction processing to generate feature representation;

[0056] Based on historical data and currently collected data, the equipment in the inspection area is monitored in real time, equipment faults are identified and classified, equipment risks are assessed based on the identified fault modes and severity scores, and fault development trends are analyzed through time series prediction models to generate equipment maintenance recommendations;

[0057] Based on real-time fault detection results, the priority of inspection tasks is dynamically adjusted, and the inspection path is optimized by combining flight path length, energy consumption and equipment risk score to generate the optimal inspection route, and the path is dynamically corrected during UAV mission execution.

[0058] Step 1: Data collection and real-time positioning are performed to ensure spatial and temporal synchronization of data, as well as data accuracy and consistency. That is, data collected by different sensors (such as high-definition cameras and temperature, humidity, and vibration sensors) can be associated with specific inspection tasks and equipment locations. The specific steps are as follows:

[0059] The high-precision positioning system (RTK) is used to accurately position the drone. During the flight, the drone will obtain its precise position through the collaboration of satellites and ground base stations, and mark its coordinates at every time point;

[0060] Assume that the position of the drone at a certain time point t is P(t)=(x(t), y(t), z(t)), where x(t), y(t), and z(t) are the coordinates of the drone in three-dimensional space. In order to obtain a high-precision position, RTK positioning uses differential GPS signals to reduce satellite positioning errors. and mobile terminal coordinates The correction signal is transmitted between them, and the differential technique is used to finally obtain the position correction function: ,in, It represents the positioning error compensation obtained by differential correction, which can usually be calculated by the relative distance, relative speed and error model between the base station and the mobile terminal;

[0061] Through dynamic correction, the signals of satellites and base stations are combined to eliminate errors caused by factors such as the atmosphere and building shielding, ensuring that the drone can always maintain centimeter-level positioning accuracy during flight. The precise location of the drone is the basis for subsequent data collection and analysis, especially during equipment inspections, ensuring that the spatial relationship between data and equipment is accurate.

[0062] Synchronize all the sensor data collected by the drone (including image data, environmental data, equipment data, etc.) with the drone location information, and use timestamps to ensure the consistency of each data in the same time dimension. Suppose the sensor data is ,in, is the collected value of the i-th sensor (such as temperature sensor, vibration sensor, etc.) at time t. The location information of the drone is P(t)=(x(t), y(t), z(t)). Each time data is collected, all sensor data will be attached with a unified timestamp t to ensure the time alignment of sensor data and location information.

[0063] In the process of time synchronization, if there is a difference between the acquisition time of sensor data and the timestamp of location data, the time alignment is performed through interpolation algorithm. is the collection time of sensor data, is the collection time of the position data, we can assume that the time difference between the two , you can use linear interpolation or more precise spline interpolation for synchronization: , where T represents the sampling interval. Through the interpolation method, it can be guaranteed that Sensor data collected at all times Ability to accurately map to location data ;

[0064] Inaccurate time synchronization can lead to inconsistent data, which in turn affects subsequent tasks such as fault diagnosis and path planning. The interpolation method can pull data at different time points back to a unified timeline, thereby ensuring that each data point can reflect the device status and drone position at the correct time.

[0065] The image data collected by the drone camera is fused with the data of other sensors (such as temperature and vibration in environmental data) to generate a unified data set with location and time tags. Let the image data be I(t) and the sensor data be Through feature extraction, the image data I(t) is converted into vector features , sensor data is converted into vector features The fusion of image and sensor data can be achieved through a multimodal deep fusion network. Suppose the fused feature vector is , which is expressed as: ,here, It is a fusion function of a deep neural network, usually a combination of a multi-layer perceptron (MLP) or a convolutional neural network (CNN). In this way, different types of data are fused through network learning to obtain a global feature representation that not only contains the visual information of the image, but also incorporates the device status data measured by the sensor;

[0066] Perform accurate location calibration to ensure that all data (including images, sensor data, and location data) can be accurately stored and accurately bound to the device's geographic location. To this end, data calibration is required. Due to the dynamic changes in the location data and sensor data during the inspection process of the drone, the drone location data is P(t)=(x(t), y(t), z(t)), and the sensor data is , for each timestamp t, the data can be accurately bound to the location through the following location calibration function: , where S(t) is a complete data set containing location and sensor data. During storage, all sensor data will be marked with a specific geographic location to facilitate subsequent retrieval and data analysis. In order to ensure data security, encrypted storage and blockchain technology are used so that each piece of data has an unalterable history record. By using encryption algorithms (such as AES) and blockchain technology, each piece of data will be given a unique identity to ensure that it cannot be modified or tampered with;

[0067] Through high-precision positioning, time synchronization, multimodal data fusion and data calibration, drones can accurately collect and associate device locations and status.

[0068] Step 2: Multimodal data fusion and preprocessing, that is, fusing and preprocessing data from different sensors and devices. Data fusion and preprocessing remove redundant information and improve data expression capabilities. The specific steps are as follows:

[0069] Remove noise and redundant information from the collected raw data, that is, remove measurement errors and external environmental noise. Although the RTK positioning system of the drone is highly accurate, there may still be small deviations in the case of signal obstruction or temporary loss of connection. Therefore, the position data is dynamically corrected through Kalman filtering to reduce the impact of external noise on the positioning results; sensors (such as temperature, humidity, vibration, etc.) may be affected by environmental changes, causing measurement errors. Kalman filtering is used to further smooth these data and eliminate short-term fluctuations;

[0070] Image data may be disturbed by factors such as light changes, sensor failure or reflection. Image data should be denoised, such as using median filtering or bilateral filtering to remove noise in the image to ensure that the image is clear and detailed.

[0071] Kalman filtering is used to smooth the position data and sensor data. , apply Kalman filtering to remove noise: ,in, represents the sensor data after filtering, K(t) is the Kalman gain, me(t) is the sensor measurement value at the current moment. Through Kalman filtering, the system dynamically adjusts the gain according to the difference between the predicted and observed values ​​to remove unnecessary noise;

[0072] It should be noted that the Kalman filter can combine the prediction model and the measurement value in the dynamic system to provide an optimal estimate. For UAVs in flight, it is particularly suitable for filtering sensor data such as position and speed to ensure the smoothness and consistency of the data;

[0073] Data standardization and normalization are performed to convert sensor data collected from different sources into a unified scale. That is, since different types of sensors (such as temperature and humidity, vibration, image sensors, etc.) are used in drone inspections, the dimensions and value ranges of these sensor data may vary greatly, so the data needs to be standardized and normalized. , the standardization process can be adopted by the following formula: ,in, is the normalized data of sensor i, is the mean value of sensor i, is the standard deviation of sensor i. Through standardization, all sensor data will be converted to a standard scale with zero mean and unit variance, so that different types of data have similar scales, avoiding the dominance of certain sensors (such as image sensors) in the analysis process;

[0074] Perform multimodal data feature extraction, that is, extract representative features from data of different modalities (such as images, sensor data, and location data) to form a unified feature vector;

[0075] Different feature extraction methods are used according to the characteristics of different modal data. For example, for image data, convolutional neural networks (CNNs) can be used to extract deep features; for sensor data, time series analysis (such as autoregressive models) can be used to extract time series features, as follows:

[0076] Image data feature extraction: The features of image data can be extracted through convolutional neural networks through the following process: ,in, is the extracted image feature, I(t) represents the image data collected at time t, that is, the image taken by the drone during the inspection process, which may be an image frame at a specific time point, or a frame of image extracted from a video sequence;

[0077] Sensor data feature extraction: For sensor data , use the autoregressive model (AR) to extract time series features, the expression is: ,in, is the extracted sensor timing characteristics;

[0078] Position data feature extraction: For the drone position data, the feature vectors in the time domain and frequency domain can be obtained by calculating the drone trajectory speed, acceleration and its conversion rate. These feature vectors include the time series characteristics of position change, motion speed, acceleration and periodic characteristics in the frequency domain, forming a comprehensive trajectory feature vector for further analysis and processing, that is, the trajectory feature is calibrated as ;

[0079] Perform multimodal data fusion to fuse the features extracted from different modal sensors (images, sensor data, position data) to obtain a unified feature representation;

[0080] The purpose of data fusion is to organically combine data from different modalities to maximize the advantages of each data form. For example, image data provides visual information outside the device, sensor data provides status data of the environment and the device, and location data provides spatial information for the device status. Fusion of these data can provide a more comprehensive understanding of the device status and improve the accuracy of analysis;

[0081] Use deep neural networks (DNN) to fuse features of different modalities. The fusion network can extract the most representative features by learning the relationship between features of different modalities: ,in, is the fused multimodal feature, A deep learning network (such as a multi-layer perceptron (MLP) or an autoencoder network) that learns the optimal fusion representation by co-optimizing features of different modalities;

[0082] The fused multimodal feature is a unified multimodal feature vector, which integrates the key information of different modal data (such as images, sensors, locations, etc.) to form a global feature that comprehensively describes the equipment status and inspection environment;

[0083] Multimodal data fusion is not just a simple concatenation or weighted average, but a collaborative optimization of features of different modalities through a deep learning network, so that the data of each modality can contribute important information to the final feature representation.

[0084] After feature extraction and fusion, the principal component analysis data dimensionality reduction method is used to process the fused multimodal features and use them as subsequent feature representations;

[0085] Through the cleaning, standardization, feature extraction, fusion and dimensionality reduction of multimodal data, the data collected by inspections from different sources can be effectively integrated, so that it can have efficient and accurate information expression in subsequent analysis and decision-making.

[0086] Step 3: Perform real-time fault detection and status assessment during the drone inspection process. Based on the processed and fused multimodal data, perform real-time fault detection and equipment status assessment. Based on historical data and current sensor information, use intelligent algorithms to determine whether there are faults, potential problems or risks in the equipment in the inspection area. The specific steps are as follows:

[0087] During drone inspections, sensor data is often affected by environmental changes, equipment aging, or external interference, causing data changes to deviate from normal patterns. Abnormal changes (progressive failures, hidden failures, etc.) are identified through feature anomaly detection.

[0088] Sensor data anomaly determination, that is, real-time monitoring of sensor data such as temperature, humidity, and vibration, combined with the preset normal range, to determine whether there is an anomaly. Using isolation forest for anomaly determination, it is possible to identify abnormal data points based on each data point. These data points represent behaviors that are significantly different from the surrounding data, usually meaning that the device status has deviated. The expression is: ,in, For real-time sensor data, is the expected data from history or model prediction, is the allowable error threshold of sensor i. Exceeding the allowable error threshold means an anomaly is detected;

[0089] Detect image data anomalies, process high-definition camera data in real time, and combine computer vision technology (such as edge detection, feature point matching, etc.) to analyze abnormal features in the image. Possible anomalies in the image include cracks on the device surface, dirt accumulation, smoke or other visual anomalies;

[0090] This can be analyzed using deep learning methods such as convolutional neural networks (CNNs), which are able to automatically learn features from images and identify anomalies: ,in, is the difference between the current moment and the expected image, is the normal image feature predicted by the model;

[0091] Identify unusual fluctuations or deviations in sensor and image data through anomaly detection;

[0092] Perform fault mode identification and classification. Based on the comparison of real-time data and historical data, accurately identify the fault mode of the current equipment and classify it. The specific steps are as follows:

[0093] Based on historical data and calibrated fault categories, the support vector machine (SVM) learning method is used to train the classification model. The fault category is a summary or labeling result of the equipment fault, which is used to indicate the type or attribution of the fault. The fault mode refers to the specific behavior or characteristic pattern exhibited by the equipment when it fails. Each fault mode will correspond to a set of specific sensor features and image features. For example, for an equipment overheating fault, it may be accompanied by a sharp rise in temperature sensor data and a sharp fluctuation in vibration data. The fault mode is the basis of the fault category, and each fault category can contain multiple fault modes.

[0094] The trained model outputs a predicted failure mode by analyzing real-time sensor data and image data. For example, the status of the device may be classified as "normal", "overheating failure", "abnormal vibration" or "equipment failure", etc. The expression is: ,in, is the predicted category of the current state of the device, It is a feature vector obtained by fusing multiple sensors and image data;

[0095] By inputting real-time data, the classification model evaluates the equipment status in real time. If the model prediction results show that the equipment is in a faulty state, an alarm is triggered and the subsequent fault handling procedure is initiated;

[0096] For example, in a drone inspection, the data collected by the high-definition camera and sensors in real time are as follows: the temperature collected by the temperature sensor continues to rise and exceeds the normal range, the vibration frequency of the vibration sensor exceeds the preset threshold, and the image data collected shows that there are significant cracks on the surface of the device. After these data are processed, a fusion feature vector is generated and input into the classification model. The model receives the fusion feature vector and recognizes the following patterns: abnormal temperature (high feature weight) and image features show cracks, which are consistent with the historical "overheating failure" pattern. The model outputs the result, and the device is marked as "overheating failure" according to the result;

[0097] Furthermore, in actual scenarios, the equipment may have multiple failure modes at the same time (such as abnormal vibration accompanied by overheating). In this case, multi-label classification can be used to output a probability value for each failure mode separately to indicate the probability of the equipment belonging to a certain failure mode.

[0098] According to the identified fault mode, the severity of the fault is evaluated. Each fault mode is assigned a severity score, which is quantified by expert knowledge or historical data. The fault severity score can be a weighted average, in which different sensor features and image features are assigned different weights according to their contribution to the fault impact. The fault severity score expression is: , where R(t) is the severity score of the fault at time t, is the i-th fault mode identified at the current time t, is the weight of the ith failure mode, Is the failure mode The extent of the impact on the system;

[0099] Use time series prediction methods to predict the development of faults and prevent the expansion of potential equipment problems. Combine historical data with real-time fault scores to quantitatively assess future risks. The specific steps are as follows:

[0100] Integrate historical data, real-time fault scores, and related environmental features to build a time series data set as input for subsequent predictions, that is, extract time series features from sensor data, including parameters such as temperature, vibration, humidity, fault severity score R(t), and real-time risk score As the key features of the time series input, the time series matrix X(t) is constructed, and each line is the multimodal feature at a certain time point, expressed as: , in, , …, The 1st, 2nd, ..., mth modal sensor data (such as temperature, humidity, pressure data, etc.) at time point t, where n is the length of the time series and m represents the total number of sensors;

[0101] Divide the time series into windows of fixed length, each of which contains the features of the past n time points, and is used to predict the future Failure score at each time point;

[0102] The ADF test method is used to determine whether the data is stationary. For non-stationary series, difference or logarithmic transformation is performed: ;

[0103] Statistical features are extracted from the data in each time window, including maximum value, minimum value, peak change rate, etc. Autoregressive (AR) or autoregressive moving average (ARIMA) models are used to fit the historical trends of sensor data and fault scores and extract dynamic features.

[0104] Perform dynamic correlation analysis between modal data and analyze the dynamic correlation between modal data of different sensors, such as the correlation between temperature and vibration. The correlation between each modality is calculated through the cross-correlation function (CCF) to form cross-modal correlation features: , where i, j are mode numbers, k is the lag time, , are the data of sensor i and j at time t respectively; , are the means of sensor i and sensor j, respectively, and N represents the total length of the time series (i.e., the number of sampling points);

[0105] By analyzing the correlation between sensor modalities i and j at different lag times k, we can identify the causal relationship or temporal dependency between multimodal data. For example, if the temperature sensor The rising trend of vibration sensors A high positive correlation at k = 5 may indicate that temperature anomalies can lead to a delayed response of vibration anomalies;

[0106] To predict the failure risk at a future time point, a long short-term memory network (LSTM) is used to process time series data. The input layer accepts a time series X(t) of length n, the LSTM hidden layer extracts the dynamic pattern in the time series, and the fully connected output layer predicts the failure score at a future time point. , the prediction formula is: ;

[0107] Combine the fault score prediction with the specific fault mode prediction and use joint learning to optimize the model performance. The total loss function is: ,in, The prediction error (e.g., mean squared error) for scoring faults, is the failure mode classification error (such as cross entropy loss), , is the weight coefficient used to balance the losses of the two tasks;

[0108] Convert the prediction results into specific risk quantification indicators and score them according to future failures The distribution of risks is divided into multiple levels (such as low, medium and high risks), as follows: The risk is low when The risk is medium when High risk, among which, is a low risk threshold, is a high risk threshold;

[0109] Maintenance recommendations based on the fault development trend (risk quantification indicator), including emergency inspections, preventive maintenance or parts replacement, etc. For example, immediately arrange for the drone to return to the fault point for additional inspections, check the equipment contact points and heat dissipation devices, and recommend that maintenance personnel arrive at the site within 3 hours;

[0110] For example, during a substation inspection, sensor data and real-time scores show that a device has a minor overheating fault. The current score R(t)=0.6. X(t) includes the temperature, vibration, and humidity data of the past 24 hours, as well as the severity score R(t). The LSTM model predicts the fault score output within the next 6 hours: , , (High Risk), the prediction shows that the risk score rises rapidly and reaches a high risk status after 6 hours, marking the equipment as "Priority Maintenance" and recommending that an overhaul be scheduled immediately.

[0111] During drone inspections, real-time data analysis may reveal new fault points or equipment in areas that have been inspected historically as potential high-risk areas. The equipment in these areas may require further detailed inspections. Even if the drone has collected data, the health of the equipment may deteriorate rapidly during the inspection due to changes in the environment or status. At this time, the task sequence needs to be adjusted immediately, and in-depth inspections of risk points should be prioritized.

[0112] Step 4: Perform dynamic inspection path optimization to ensure that the drone can complete the inspection task quickly and effectively, maximize inspection efficiency and ensure equipment safety. The specific steps are as follows:

[0113] Based on the results of real-time fault detection and status assessment, the inspection tasks are prioritized to determine which equipment needs to be inspected first to avoid potential equipment downtime or fault expansion;

[0114] In the inspection task, the priority of the task will be determined according to the equipment's failure mode, severity score and risk prediction. For faulty or high-risk equipment, the inspection task needs to be performed first. For equipment without faults, the inspection task can be postponed or carried out as planned;

[0115] Task priority evaluation is based on the following factors: fault severity R(t), the severity score of the fault. The higher the fault score, the higher the priority of the task;

[0116] Equipment Importance : Weight scoring is performed based on the importance of the equipment in the power grid system, with key equipment having a higher weight;

[0117] Failure risk prediction : Predict the score of future risks. Devices with higher future risks need to be processed first. The priority of the task can be expressed as: ,in, is the priority of the task, is the weight of the device, is the weight of the failure risk prediction. A high priority task means that the equipment has a higher failure or potential risk;

[0118] Through priority assessment, inspection tasks can be reasonably arranged according to the current status and future risks of the equipment, and the most critical equipment can be inspected first;

[0119] Based on the inspection task priority and real-time fault detection results, the inspection route is dynamically planned to complete the task in the shortest time and with the lowest flight cost;

[0120] When there are multiple target points in the inspection task, particle swarm optimization is used for path optimization. The specific steps are as follows:

[0121] The objective function is designed to minimize the flight distance: ,in, For flight from arrive The flight distance between

[0122] Minimize energy consumption (based on distance and load): ,in, For flight from arrive The energy consumption required between

[0123] Maximize task priority: ,in, For inspection point The task priority weights, high priority points have higher weights;

[0124] The multi-objective optimization is transformed into a single-objective problem through a weighted method: ,in, , is the weight coefficient, which is used to balance the impact of different objectives;

[0125] Define particles, each particle represents an arrangement of inspection paths , i.e. the order of visiting inspection points;

[0126] Initialize N particles (path arrangement), and for each particle, randomly generate the initial position (path) and speed (indicating the direction of path change);

[0127] The objective function F(P) is used to calculate the fitness of each particle to evaluate the quality of the particle;

[0128] In each iteration, the particle velocity is updated according to the individual optimal solution (pBest) and the global optimal solution (gBest);

[0129] Adjust the particle's position (path) according to the updated velocity;

[0130] Calculate the value of the objective function F(P) according to the new particle position (path), update the fitness, and compare the current fitness with the historical optimal fitness for each particle to update the global optimal solution;

[0131] When the maximum number of iterations is reached or the global optimal solution converges without significant changes in consecutive k iterations, the particle position corresponding to the global optimal fitness value is output as the optimal inspection path.

[0132] Dynamic inspection path optimization and task scheduling are key links in realizing small UAV inspection data management methods. The core of data management lies in the efficient collection, storage, processing and analysis of multimodal data. Through dynamic path planning, UAVs can flexibly adjust the flight sequence and inspection point coverage based on the results of real-time fault detection and status assessment, ensuring that data of high-priority equipment is collected first, improving data integrity and quality, avoiding inefficient repeated collection and waste of resources, and ensuring the timeliness and effectiveness of key data.

[0133] It should be noted that the threshold information in this embodiment is pre-set by professionals and will not be explained in detail here. Some parameter English letters in the embodiments have the same situation, but different meanings are explained when used, which will not be explained one by one here.

[0134] The present invention uses high-precision positioning technology and timestamp synchronization to accurately associate the multimodal data collected by the drone with the equipment location, achieving the spatiotemporal consistency of the inspection data. Through noise removal, standardization, and multimodal feature fusion and dimensionality reduction processing of deep learning networks, a unified feature representation is generated to improve the accuracy and efficiency of data analysis. The historical data and real-time collected data are combined to monitor the equipment status in real time, identify and classify fault modes, evaluate equipment risks, and predict the development trend of faults through time series prediction models. Accurate maintenance suggestions are generated to improve the timeliness of fault diagnosis. Based on the fault detection results, the inspection task priority and path planning are dynamically optimized to generate the optimal inspection route, and the flight path is dynamically corrected to achieve efficient resource utilization, thereby improving the intelligence and accuracy of the inspection task and significantly improving the efficiency and safety of the drone inspection system.

[0135] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0136] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product.

[0137] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0138] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0139] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0140] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A small UAV inspection data management method, characterized by: The steps include: Obtain the spatial position and collected data of the drone during the inspection mission, synchronize the data in time and space through timestamps, and associate the collected data with the equipment location; The collected data is subjected to noise removal, standardization and multimodal feature extraction, and multimodal feature fusion is performed through a deep learning network. The fused features are subjected to dimensionality reduction processing to generate feature representation; Based on historical data and currently collected data, the equipment in the inspection area is monitored in real time, equipment faults are identified and classified, equipment risks are assessed based on the identified fault modes and severity scores, and fault development trends are analyzed through time series prediction models to generate equipment maintenance recommendations; Based on real-time fault detection results, the inspection task priority is dynamically adjusted, the inspection path is optimized by combining flight path length, energy consumption and equipment risk score, the optimal inspection route is generated, and the path is dynamically corrected during the UAV mission execution; Obtain the spatial position and collected data of the drone during the inspection mission, synchronize the data in time and space through timestamps, and associate the collected data with the device location. The specific steps are as follows: Obtain the coordinates of the drone at each time point, and perform position correction based on the correction signal transmitted between the base station coordinates and the mobile terminal coordinates to determine the spatial position; Synchronize all sensor data collected by the drone with the drone’s location information, and unify the collected data into the same time dimension through timestamps. The sensor data includes image data, environmental data, and sensor data; The image data collected by the drone is fused with the environmental data using a multimodal deep fusion network to generate a dataset with location and time tags; Based on the real-time fault detection results, the inspection task priority is dynamically adjusted, the inspection path is optimized by combining the flight path length, energy consumption and equipment risk score, the optimal inspection route is generated, and the path is dynamically corrected during the UAV task execution, including the following steps: The priority of the task is determined based on the failure mode, severity score and risk prediction of the equipment. The priority of the task is expressed as: ,in, is the priority of the task, is the weight of the device, is the weight of future risk; Based on the inspection task priority and real-time fault detection results, the inspection path is dynamically planned, and particle swarm optimization is used to optimize the path when there are multiple target points; The objective function is designed to minimize the flight distance: ,in, For flight from arrive The flight distance between them, n represents the total number of inspection points; Minimize energy consumption: ,in, For flight from arrive The energy consumption required between Maximize task priority: ,in, For inspection point The task priority weights; The multi-objective optimization is transformed into a single-objective problem through a weighted method: ,in, , is the weight coefficient, which is used to balance the impact of different objectives.

2. A small UAV inspection data management method according to claim 1, characterized in that: The collected data is subjected to noise removal, standardization, and multimodal feature extraction. Multimodal feature fusion is performed through a deep learning network, and the fused features are subjected to dimensionality reduction to generate feature representation. The specific steps include: Use Kalman filtering to smooth the collected data and perform standardization and normalization processing; Convolutional neural networks are used to extract features from image data, and autoregressive models are used to extract time series features from sensor data. The drone position data is obtained by calculating the speed, acceleration, and rate of change of the drone trajectory, and the extracted time domain and frequency domain features are used as trajectory features. A deep neural network is used to fuse the features of different modalities to obtain a unified multimodal feature vector. The principal component analysis method is used to reduce the dimension of the fused multimodal features and use them as feature representation after dimensionality reduction.

3. A small UAV inspection data management method according to claim 2, characterized in that: Based on historical data and current collected data, real-time status monitoring of equipment in the inspection area is performed to identify and classify equipment failures. The specific steps are as follows: Determine sensor data anomalies, monitor temperature, humidity and vibration sensor data in real time, use isolation forests to identify anomalies based on the preset normal range, detect image data anomalies, process high-definition camera data in real time, determine abnormal features in the image based on edge detection and feature point matching, and obtain abnormal fluctuations or deviations in the currently collected sensor data and image data; Based on historical data and calibrated fault categories, the classification model is trained using support vector machine learning method; The trained model outputs a predicted failure mode by analyzing real-time sensor data and image data. If the predicted failure mode shows that the device is in a faulty state, an alarm is triggered and the subsequent fault handling procedure is entered. Each failure mode corresponds to a set of sensor features and image features.

4. A small UAV inspection data management method according to claim 3, characterized in that: The equipment risk is assessed based on the identified failure modes and severity scores, and the failure development trend is analyzed through the time series prediction model to generate equipment maintenance recommendations, including the following steps: Based on the identified failure modes, the severity of the failure is evaluated, and each failure mode is assigned a severity score; The fault severity score expression is: , where R(t) is the severity score of the fault at time t, is the i-th fault mode identified at the current time t, is the weight of the ith failure mode, Is the failure mode The extent of the impact on the system; Use time series prediction methods to predict the development of faults and quantitatively assess future risks; Integrate historical data, real-time fault scores, and environmental characteristics to build a time series data set as prediction input, fault severity score R(t) and real-time risk score As the key features of the time series input, the time series matrix X(t) is constructed, and each line is the multimodal feature of each time point, expressed as: ,in, , …, is the 1st, 2nd, …, mth modal sensor data at time point t, n is the length of the time series, and m represents the total number of sensors; Divide the time series into windows of fixed length, each of which contains the features of the past n time points, and is used to predict the future Failure score at each time point; The ADF test method is used to determine whether the data is stationary. For non-stationary sequences, difference or logarithmic transformation is performed: ; Statistical feature extraction is performed on the data in each time window, including maximum value, minimum value, and peak change rate; Perform dynamic correlation analysis between modalities, analyze the dynamic correlation between modal data of different sensors, calculate the correlation between each modality through the cross-correlation function, and form cross-modal correlation features: , where i, j are mode numbers, k is the lag time, , are the data of sensor i and j at time t respectively; , are the means of sensor i and sensor j respectively, and N represents the total length of the time series; Identify causal relationships or temporal dependencies between multimodal data by analyzing the correlation between sensor modalities i and j at different lag times k; To predict the failure risk at a future time point, a long short-term memory network is used to process time series data. The input layer is a time series X(t) of length n. The long short-term memory network hidden layer extracts the dynamic pattern in the time series. The fully connected output layer predicts the failure score at a future time point. , the prediction formula is: ； Combine the fault score prediction with the fault mode prediction and convert the prediction results into specific risk quantification indicators. The distribution of risks is divided into low, medium and high risks, as follows: The risk is low when The risk is medium when High risk, among which, is a low risk threshold, For high risk thresholds, maintenance recommendations are made based on the fault development trend.

5. A small UAV inspection data management method according to claim 4, characterized in that: Based on the inspection task priority and the results of real-time fault detection, the inspection path is dynamically planned, and when there are multiple target points, particle swarm optimization is used for path optimization, including the following steps: Define particles, each particle represents an arrangement of inspection paths , i.e. the order of visiting inspection points; Initialize N particles, and for each particle, randomly generate the initial path and the change direction of the path; Use the objective function F(P) to calculate the fitness of each particle; In each iteration, the particle velocity is updated according to the individual optimal solution and the global optimal solution; Adjust the particle's path based on the updated velocity; Calculate the value of the objective function according to the new particle path, update the fitness, and compare the current fitness with the historical optimal fitness for each particle to update the global optimal solution; When the maximum number of iterations is reached or the global optimal solution converges without significant changes in consecutive k iterations, the particle position corresponding to the global optimal fitness value is output as the optimal inspection path.

Images