Fault identification method and device for unmanned aerial vehicle inspection, equipment and storage medium

Abstract

The embodiment of the invention discloses a fault identification method and device for unmanned aerial vehicle inspection, equipment and a storage medium. The method comprises the following steps: collecting infrared and visible light image data of a target area under different environmental conditions through an unmanned aerial vehicle; constructing a multi-modal data set according to the collected image data; performing labeling processing on a fault type label of the image data by using the multi-modal data set to obtain a labeled multi-modal data set; training a fault recognition model based on the labeled multi-modal data set; integrating the fault identification model into an unmanned aerial vehicle inspection system of the unmanned aerial vehicle to realize real-time monitoring and intelligent diagnosis of the fault type of the target area; and periodically updating the marked multi-modal data set and the fault identification model. Through the above method, multi-modal data acquisition, adaptive model training and dynamic updating technologies are introduced, efficient inspection and intelligent diagnosis of the target area are realized, and the accuracy of fault identification and the long-term reliability of the system are improved.

Description

Technical Field

[0001] The present invention relates to the field of drone inspection technology, and in particular to a fault identification method and device, equipment and storage medium for drone inspection. Background Art

[0002] At present, most drone inspection systems use traditional image processing and machine learning methods to identify faults in collected infrared and visible light images. Although the existing drone inspection systems can meet basic inspection needs to a certain extent, there are still some obvious defects. First, traditional image processing methods are difficult to cope with complex and changeable environmental conditions and fault types, resulting in low accuracy and robustness of fault identification. Secondly, existing fault identification models usually require a large amount of training data and a long training time, and the generalization ability of the model is poor when facing new environments and new fault types. Finally, the existing drone inspection system lacks an effective adaptive mechanism and cannot dynamically update image matching rules and algorithm models according to environmental changes and technological advances, making it difficult to ensure the long-term accuracy and reliability of the system. Summary of the invention

[0003] The main purpose of the present invention is to provide a method and device, equipment and storage medium for fault identification of drone inspection, which can solve the problem that the accuracy of fault identification of drone inspection in the prior art needs to be improved.

[0004] To achieve the above object, the present invention provides a first aspect of a fault identification method for unmanned aerial vehicle inspection, the method comprising:

[0005] On a preset flight path, a drone equipped with at least an infrared camera and a visible light camera is used to synchronously inspect the target area, and infrared and visible light image data of the target area under different environmental conditions are collected;

[0006] Constructing a multimodal data set based on the collected image data, the multimodal data set including the image data and environmental parameters of environmental conditions corresponding to the image data, the environmental parameters including at least a timestamp, a geographic location, and a meteorological parameter;

[0007] Using the multimodal data set to label the fault type labels of the image data, to obtain a labeled multimodal data set;

[0008] Based on the annotated multimodal data set, a fault recognition model is trained, wherein the fault recognition model is used to automatically predict the fault type in the new infrared and visible light images when receiving new infrared and visible light image inputs, and to give corresponding warning signals;

[0009] Integrating the fault identification model into the drone inspection system of the drone to achieve real-time monitoring and intelligent diagnosis of the fault type in the target area;

[0010] The annotated multimodal data set and the fault identification model are updated regularly to obtain an updated fault identification model.

[0011] To achieve the above-mentioned object, the second aspect of the present invention provides a fault identification device for unmanned aerial vehicle inspection, the device comprising:

[0012] Data collection module: used to synchronously inspect the target area on a preset flight path by using a drone equipped with at least an infrared camera and a visible light camera to collect infrared and visible light image data of the target area under different environmental conditions;

[0013] A collection construction module: used to construct a multimodal data set based on the collected image data, wherein the multimodal data set includes the image data and environmental parameters of environmental conditions corresponding to the image data, wherein the environmental parameters include at least a timestamp, a geographic location, and a meteorological parameter;

[0014] A label marking module: used to use the multimodal data set to perform labeling processing on the fault type labels of the image data to obtain a labeled multimodal data set;

[0015] Model training module: used for training a fault recognition model based on the annotated multimodal data set, wherein the fault recognition model is used for automatically predicting the fault type in the new infrared and visible light images when receiving new infrared and visible light image inputs, and giving corresponding warning signals;

[0016] Fault monitoring module: used to integrate the fault identification model into the drone inspection system of the drone to achieve real-time monitoring and intelligent diagnosis of the fault type of the target area;

[0017] Model updating module: used to regularly update the annotated multimodal data set and fault identification model to obtain an updated fault identification model.

[0018] To achieve the above-mentioned purpose, the third aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform the steps of the method shown in the first aspect.

[0019] To achieve the above objectives, the fourth aspect of the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method shown in the first aspect.

[0020] The embodiments of the present invention have the following beneficial effects:

[0021] The present invention provides a method for fault identification of unmanned aerial vehicle inspection, the method comprising: on a preset flight path, a target area is synchronously inspected by a unmanned aerial vehicle equipped with at least an infrared camera and a visible light camera, and infrared and visible light image data of the target area under different environmental conditions are collected; a multimodal data set is constructed according to the collected image data, the multimodal data set comprising image data and environmental parameters of environmental conditions corresponding to the image data, the environmental parameters comprising at least a timestamp, a geographical location and meteorological parameters; a fault type label of the image data is annotated by using the multimodal data set to obtain an annotated multimodal data set; a fault identification model is trained based on the annotated multimodal data set, the fault identification model is used to automatically predict the fault type in the new infrared and visible light images when receiving new infrared and visible light image inputs, and to give corresponding warning signals; the fault identification model is integrated into the unmanned aerial vehicle inspection system of the unmanned aerial vehicle to realize real-time monitoring and intelligent diagnosis of the fault type of the target area; the annotated multimodal data set and the fault identification model are regularly updated to obtain an updated fault identification model. Through the above method, multimodal data acquisition, adaptive model training and dynamic update technology are introduced to achieve efficient inspection and intelligent diagnosis of the target area, improve the accuracy of fault identification and the long-term reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] in:

[0024] Figure 1 This is a flow chart of a method for fault identification during unmanned aerial vehicle inspection in an embodiment of the present invention;

[0025] Figure 2 This is a structural block diagram of a fault identification device for unmanned aerial vehicle inspection in an embodiment of the present invention;

[0026] Figure 3 4 is a structural block diagram of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] See also Figure 1 , Figure 1 Flow chart of a method for fault identification of drone inspection in an embodiment of the present invention. The method can be applied to both a terminal and a server. The terminal can be a desktop terminal or a mobile terminal. The mobile terminal can be at least one of a mobile phone, a tablet computer, a laptop computer, etc. The server can be implemented by an independent server or a server cluster composed of multiple servers. This embodiment is described by applying to a terminal as an example. Figure 1 The method comprises the following steps:

[0029] 101. On a preset flight path, a drone equipped with at least an infrared camera and a visible light camera is used to synchronously inspect the target area, and infrared and visible light image data of the target area under different environmental conditions are collected;

[0030] It should be noted that on the preset flight path, a drone equipped with at least an infrared camera and a visible light camera conducts synchronous inspections of the target area, and collects infrared images and visible light images of the target area under different environmental conditions. The drone conducts inspections according to the predetermined flight path to ensure that all key locations in the target area are covered. The infrared camera and the visible light camera collect image data synchronously. The infrared camera is used to detect temperature anomalies, while the visible light camera is used to capture visible light images of the target area. At the same time, the drone records the timestamp, geographic location information, and meteorological parameters of each image. This information is crucial for subsequent data analysis and fault identification. The target area can be a substation, transmission line, and other places related to electricity. The drone is used to perform intelligent inspections of power places. The flight path is related to the inspection requirements, and is used for the drone to fly along the path to collect data.

[0031] 102. Construct a multimodal data set based on the collected image data, wherein the multimodal data set includes the image data and environmental parameters of environmental conditions corresponding to the image data, wherein the environmental parameters include at least a timestamp, a geographic location, and a meteorological parameter;

[0032] Furthermore, a multimodal dataset can be constructed based on the collected infrared images and visible light images. The multimodal dataset contains not only the original image information, but also the timestamp, geographic location, and meteorological parameters recorded by the drone. The constructed multimodal dataset is the basis for subsequent analysis. Each record in the dataset contains infrared images, visible light images, timestamps, geographic location information, and meteorological parameters. These data will be used to train and verify the fault identification model and provide a basis for subsequent fault detection.

[0033] In a feasible implementation, in order to avoid the impact of different environments on the collected images, the present application also takes environmental parameters into consideration, that is, constructs a set of multimodal data including image data and environmental parameters as the basis for subsequent fault recognition model training to reduce environmental impact and improve the accuracy of fault recognition, wherein step 102 may include steps A01 to A02:

[0034] A01. Using real-time environmental perception technology, using sensors carried by the drone to monitor in real time the environmental parameters of the drone's flight environment, the environmental parameters including timestamp, meteorological parameters and geographic location;

[0035] By introducing real-time environmental perception technology, the drone's onboard sensors monitor the flight environment in real time and make dynamic adjustments based on the preset path. When a sudden obstacle is detected ahead, the drone can automatically re-plan the path and bypass the obstacle to continue the mission.

[0036] Furthermore, for all kinds of data collected, the data collected by different sensors are comprehensively processed through multi-sensor data fusion technology. Multi-sensor data is fused using algorithms such as Kalman filtering or particle filtering to improve the accuracy and reliability of the data. The fused data can provide more comprehensive and accurate environmental perception information. An intelligent path planning algorithm is designed by combining the preset path and real-time environmental perception data. The algorithm can dynamically adjust the flight path according to the current environmental conditions and the status of the drone. The optimal path is generated using the A* algorithm and Dijkstra algorithm path planning technology.

[0037] A02. Perform multimodal fusion of image data and environmental parameters to construct a multimodal dataset.

[0038] Furthermore, the image data and environmental parameter data are multimodally fused to construct a comprehensive data set. The image data is feature extracted using a deep learning model, and the environmental parameter data is feature extracted using a time series analysis model. Through multimodal data fusion, the system can more comprehensively understand the relationship between the environmental conditions of the inspection task and the image quality; the convolutional neural network is used to extract features from infrared images and visible light images to capture texture features, edge features, color features, etc. in the image. At the same time, a recursive neural network or a long short-term memory network is used to perform time series analysis on the environmental parameter data to extract the changing trends and periodic features of the environmental parameters. Multi-layer fully connected layers and nonlinear activation functions are used to further fuse and represent the comprehensive feature vector.

[0039] In a feasible implementation, the method shown in the present application further includes steps A03 to A04 for path planning:

[0040] A03. Record the spatiotemporal distribution of the environmental parameters and establish a three-dimensional spatiotemporal model of the environmental parameters;

[0041] After obtaining the environmental parameters, the application will also record the spatiotemporal distribution of the environmental parameters, that is, the changes in environmental parameters in different time periods and different geographical locations. Through high-precision timestamp and geographical location information, a three-dimensional spatiotemporal model of environmental parameters is established to provide richer background information for subsequent data analysis and fault identification.

[0042] Further, in this application, the sensor samples data at a high frequency (such as multiple times per second) to ensure that the recorded data has a high temporal resolution. Through high-frequency sampling, it is possible to capture the instantaneous changes of environmental parameters and provide detailed data support for subsequent analysis. The collected environmental parameters are organized and managed using a spatiotemporal data model. The spatiotemporal data model not only contains timestamps and geographic location information, but also contains multi-dimensional attributes of environmental parameters. Through high-precision timestamps and geographic location information, each data point is accurately located in three-dimensional space. The inspection area is divided into multiple spatiotemporal grids, each grid containing environmental parameter data within a certain time period and geographical range. Through spatiotemporal grid division, a large amount of data is organized into an orderly structure for subsequent analysis and processing. Through spatiotemporal correlation analysis, the relationship between different environmental parameters is studied. Using the Pearson correlation coefficient and mutual information method, the correlation between different environmental parameters is quantified to reveal the inherent mechanism of environmental parameter changes.

[0043] A04. When an obstacle is detected in front of the UAV, a path is replanned based on the three-dimensional space-time model and the flight path to obtain an obstacle avoidance path to bypass the obstacle.

[0044] Furthermore, by recording the spatiotemporal distribution of environmental parameters, that is, the changes in environmental parameters in different time periods and different geographical locations, the present application can achieve high-precision environmental perception and dynamic spatiotemporal modeling, and provide comprehensive and reliable environmental data. Multimodal sensor data acquisition and spatiotemporal data fusion technology, combined with intelligent data analysis and prediction, generate high-precision spatiotemporal distribution maps of environmental parameters, providing rich background information and scientific basis for drone inspection tasks. This not only improves the accuracy and timeliness of fault identification, but also when an obstacle is detected in front of the drone, the path is replanned based on the three-dimensional spatiotemporal model and the flight path to obtain an obstacle avoidance path to bypass the obstacle. It also ensures that the drone can complete the inspection task safely and efficiently in a complex environment through intelligent path planning and dynamic obstacle avoidance strategies, significantly improving the inspection efficiency and the overall performance of the system.

[0045] 103. Using the multimodal data set to perform labeling of fault type labels of image data to obtain a labeled multimodal data set;

[0046] It should be noted that after obtaining a set of multimodal data at each time point, each multimodal data can be labeled. Specifically, a fault type label is assigned through the feature expression of the image data. The fault type label includes but is not limited to temperature abnormality, texture abnormality, color abnormality, etc., such as abnormal temperature increase, color change, texture fracture, etc.

[0047] Exemplarily, step 103 includes: performing registration processing on the infrared image and the visible light image in the image data to obtain registered image data; performing annotating processing on the fault type label using the registered image data to obtain the fault type label of the image data, and the annotated multimodal data set includes image data, fault type label, timestamp, geographic location and meteorological parameters.

[0048] The registration process includes analyzing and determining the corresponding relationship between the infrared image and the visible light image, so that the registered image data contains not only visible light information but also temperature information.

[0049] It should be noted that the multimodal dataset is used to analyze and determine the correspondence between infrared images and visible light images, especially for image feature performance under specific types of faults or abnormal conditions, to form a set of image matching rules that can effectively identify the fault type. Through manual annotation and feature comparison analysis, the characteristic performance in the fault image is identified, such as abnormal temperature increase, color change, texture fracture, etc. Based on these features, image matching rules are generated, which include but are not limited to temperature anomaly rules, color change rules, texture fracture rules, and multimodal feature combination rules. These rules will be used to guide the development and training of fault recognition models.

[0050] Furthermore, the data can be filtered to improve data accuracy, and the image data and environmental parameters can be filtered out in multiple dimensions using advanced filtering algorithms to improve the signal-to-noise ratio of the data and ensure the accuracy of subsequent analysis. That is, before step 103, steps B01 to B03 are also included to filter the data:

[0051] B01, filtering the image data using a preset wavelet filtering algorithm to obtain filtered image data;

[0052] B02. Filtering the environmental parameters using a preset extended Kalman filter model to obtain filtered environmental parameters;

[0053] It is understandable that in the process of processing complex data streams, an advanced signal processing technology is first used. This technology is based on multi-dimensional filtering theory and aims to extract useful information from the mixed raw data and reduce unnecessary interference. This process is not just a simple noise reduction, but through a series of carefully designed mathematical operations, the ratio of the target signal to the background noise is enhanced, thereby improving the overall quality of the data and laying a solid foundation for subsequent data analysis and processing.

[0054] For image data, the image data is filtered using a preset wavelet filtering algorithm to obtain filtered image data, and further, the denoised subbands are reassembled using an inverse wavelet transform to restore the original structure of the image. By optimizing the reconstruction algorithm, the clarity and detail of the image are further improved. By using a reverse engineering method, namely the inverse wavelet transform technology, the various components in the data stream that have been preliminarily processed are reassembled to restore their original structural characteristics. This method can not only restore the basic form of the data, but also introduce specific optimization strategies, such as adaptive threshold setting, on this basis, to further improve the clarity and detail level of the data, so that the final result is closer to the original state, while having higher information density and resolution.

[0055] For environmental data, an extended Kalman filter model is constructed based on the nonlinear characteristics of environmental parameters. This model can handle the state estimation problem in nonlinear systems and improve the filtering accuracy. Considering the ubiquitous nonlinear relationship in environmental parameters, the extended Kalman filter model is introduced to solve the state estimation problem. This model is developed on the basis of the classic Kalman filter and is specifically used to deal with complex systems that cannot be directly expressed by linear equations. By cleverly adjusting the parameter settings within the model and the real-time response mechanism to changes in the external environment, the extended Kalman filter can effectively capture the changing trend of the system state while maintaining high computational efficiency, thereby achieving the purpose of improving filtering accuracy.

[0056] B03. Acquire multimodal features from the filtered image data and environmental parameters, and dynamically optimize the weights of each modal feature using an adaptive weight adjustment algorithm to obtain optimized image data and environmental parameters, wherein the multimodal data set includes the optimized image data and environmental parameters.

[0057] Furthermore, the weight of each modal feature is dynamically adjusted according to the importance and reliability of the multimodal features. The adaptive weight adjustment algorithm is used to automatically adjust the weight according to the variance and correlation of the features to optimize the filtering effect. Specifically, feature vectors of multiple dimensions are obtained from image data and environmental parameters, and the weights of each modal feature are dynamically optimized using the adaptive weight adjustment algorithm. The algorithm is based on the variance and correlation of the features and evaluates the relationship between different features by calculating the covariance matrix of the feature vector. By introducing an adaptive threshold and a dynamic adjustment factor, the algorithm can automatically adjust the weight of each feature according to the variance size and correlation strength of the features. This process can not only highlight important features, but also suppress noise and redundant information, thereby improving the filtering effect. By introducing a feedback loop, the algorithm can dynamically adjust the weight according to the real-time evaluation results of the system performance to ensure that the system is always in the best working state. By adjusting the weights of each modal feature, the adaptive weight adjustment algorithm can effectively optimize the filtering effect of multimodal data and improve the overall performance of the system.

[0058] Furthermore, adaptive filtering algorithms can be introduced to dynamically adjust the filter parameters according to the changes in real-time data. Extended Kalman filtering and particle filtering can be used to handle the state estimation problem of nonlinear systems and improve the robustness and adaptability of the filter.

[0059] Among them, the adaptive adjustment factor w k and v k They are process noise and observation noise respectively, assuming that they have zero mean and noise covariance matrix Q k and R k Gaussian distribution, state vector X k , state transfer matrix F k , observation matrix H k , λ, δ, γ and μ are hyperparameters, is the variance of the particle state, is the observed value z k and state x k The covariance of the state vector X k , state transfer matrix F k , observation matrix H k , P k Current state covariance, P k-1 is the state covariance of the previous step;

[0060]

[0061] P k∣k =(IK k H k ) k∣k-1

[0062] Through the above formula, the adaptive filtering algorithm can dynamically adjust the parameters of the filter according to the changes in real-time data, handle the state estimation problem of the nonlinear system, and significantly improve the robustness and adaptability of the filter.

[0063] 104. Based on the annotated multimodal data set, a fault recognition model is trained, wherein the fault recognition model is used to automatically predict the fault type in the new infrared and visible light images when receiving new infrared and visible light image inputs, and to give corresponding warning signals;

[0064] Specifically, step 104 includes steps C01 to C04:

[0065] C01. Using the annotated multimodal dataset as a training dataset, each training sample of the training dataset includes image data, environmental parameters, timestamp, geographic location, and fault type label;

[0066] First, prepare the training data. Use the labeled fault images and normal images as training data to generate a training data set. Each training sample contains image data, environmental parameters, timestamp, geographic location information, and fault labels. The generated training data set contains not only rich image information, but also detailed environmental parameters and time and space information, providing a high-quality data foundation for subsequent model training.

[0067] C02. Using the training data set to train a preset deep learning model to generate a fault recognition model;

[0068] Then the model training can be performed, and the training data set can be trained using the deep learning model to generate a fault recognition model. The model parameters are optimized through multiple rounds of iterations to improve the accuracy and generalization ability of the model. The fused feature vector is input into the deep learning model for training, wherein the fused feature vector can be obtained through subsequent steps D01 to D05, which will not be described here. The model parameters are optimized through multiple rounds of iterations, and the model weights are updated using the back propagation algorithm and optimizer. During the training process, the performance indicators of the model are monitored in real time, such as the loss function value, accuracy, recall rate, F1 score, etc., and the hyperparameters such as the learning rate and batch size are dynamically adjusted according to the monitoring results to improve the training effect of the model. The generated fault recognition model can automatically evaluate whether there are predefined fault modes in the image when receiving new infrared and visible light image inputs, and give corresponding warning signals.

[0069] C03. Verifying the fault identification model using a preset verification data set and evaluating the performance indicators of the fault identification model;

[0070] C04. Optimizing the model parameters of the fault identification model according to the performance index to obtain an optimized fault identification model.

[0071] Finally, the model is validated by using an independent validation dataset to validate the trained model, and the model performance indicators are evaluated. The model parameters are adjusted according to the validation results to further optimize the model performance.

[0072] Prepare an independent validation dataset that contains faulty images and normal images that were not used in the training, as well as the corresponding environmental parameters, timestamps, geographic location information, and fault labels. Use the confusion matrix to analyze the classification effect of the model in detail and identify the weaknesses and deficiencies of the model. The confusion matrix can be used to understand the performance of the model in different categories. The structure of the confusion matrix assumes that we have a binary classification problem with positive and negative categories. The structure of the confusion matrix is ​​as follows:

[0073]

[0074] In summary, by using the confusion matrix to analyze the classification effect of the model in detail and identify the weaknesses and deficiencies of the model, this method can significantly improve the performance and reliability of the fault identification model. Specifically, the confusion matrix provides a wealth of performance indicators, such as accuracy, precision, recall, F1 score, specificity, false positive rate, false negative rate and AUC value, which can comprehensively evaluate the performance of the model in different categories. By analyzing these indicators, the false alarm rate and false negative rate of the model when predicting positive and negative samples can be found, and then the model parameters and structure can be adjusted to optimize the model training process. For example, by increasing the number of positive or negative samples, adjusting the threshold, or retraining the model, the precision and recall of the model can be effectively improved, and false alarms and false negatives can be reduced. Finally, the optimized model can more accurately identify fault modes when receiving new infrared and visible light image inputs, provide reliable early warning signals, significantly improve the fault identification accuracy and robustness of the UAV inspection system, and improve the efficiency and safety of the inspection task.

[0075] In a feasible implementation, step C02 includes steps D01 to D08:

[0076] D01. Use multi-scale convolutional neural networks to extract image features of different scales from infrared images and visible light images; use multi-band recurrent neural networks to extract environmental features of different frequency bands from environmental parameters;

[0077] It should be noted that multi-scale convolutional neural networks are used to extract features of different scales from infrared images and visible light images. Through convolution kernels of different sizes, local and global features in the image are captured to ensure the diversity and richness of features. Multi-band recurrent neural networks are used to extract features of different frequency bands from environmental parameters, capture the trend of changes on the time scale, and enhance the temporal feature representation of environmental parameters.

[0078] D02. Dynamically adjust the weights of the image features and the environmental features through an adaptive feature splicing algorithm to generate a high-dimensional comprehensive feature vector, and enhance the comprehensive feature vector using an autoencoder;

[0079] Furthermore, the adaptive feature splicing algorithm dynamically adjusts the weights of each modal feature to generate a high-dimensional comprehensive feature vector, and uses the autoencoder to enhance the robustness and expressiveness of the feature. The adaptive feature splicing algorithm uses the variance and covariance matrices of the features to evaluate the importance of different modal features and adjusts the weights based on the evaluation results. The generated high-dimensional comprehensive feature vector not only contains image features and environmental parameter features, but also ensures the robustness and expressiveness of the features through adaptive adjustment.

[0080] D03. Introducing a cross-modal interaction module, through an attention mechanism and a gating mechanism, enables information exchange and complementation between features of different modalities of the comprehensive feature vector to obtain a comprehensive feature vector after interaction;

[0081] For the enhanced comprehensive feature vector, a cross-modal interaction module is introduced to enable information exchange and complementation between features of different modalities through attention mechanism and gating mechanism. This enables features of different modalities to influence and complement each other. The attention mechanism dynamically adjusts the weights of features of each modality by calculating the correlation between features of different modalities to highlight important features. The gating mechanism controls the flow of information through gating units to ensure the temporal consistency of features.

[0082] D04. Design a multimodal feature fusion network, and further fuse and represent the integrated feature vector after the interaction through multiple layers of fully connected layers and nonlinear activation functions to obtain a fused integrated feature vector;

[0083] Design and use a multimodal feature fusion network to further fuse and represent the features after interaction through multiple layers of fully connected layers and nonlinear activation functions. Use multiple layers of fully connected layers and nonlinear activation functions to further fuse and represent the features after interaction. The multimodal feature fusion network can capture the complex relationship between features of different modalities and generate richer feature representations.

[0084] D05. Use a dynamic feature selection algorithm to perform dimensionality reduction processing on the fused comprehensive feature vector to obtain a comprehensive feature vector after dimensionality reduction;

[0085] The dynamic feature selection algorithm is used to dynamically select the most relevant feature subset, reduce the feature dimension, improve the computational efficiency and generalization ability of the model, and obtain the comprehensive feature vector after dimensionality reduction.

[0086] D06. Use the comprehensive feature vector after dimensionality reduction and the deep learning model to predict the fault type and obtain a prediction result;

[0087] Among them, independent deep learning models can be used to train data of different modalities to generate their own prediction results, and the prediction results of each modality can be fused through a multimodal prediction result fusion algorithm and an adaptive weight adjustment mechanism to ensure the accuracy and robustness of the final prediction result. Specifically, the weighted average method can calculate the confidence of the prediction results of each modality and dynamically adjust the weights. The voting mechanism can select the final prediction result by majority voting. The stacking method can construct a multi-layer model and use the prediction results of different modalities as input to generate the final prediction result.

[0088] An adaptive weight adjustment mechanism can also be introduced to dynamically adjust the weights according to the confidence and consistency of the prediction results. This mechanism dynamically adjusts the weights according to the confidence and consistency of the prediction results of each modality. Specifically, by calculating the confidence and consistency of the prediction results of each modality, evaluating their reliability and relevance, and dynamically adjusting the weights, the final prediction results are ensured to have the highest accuracy and robustness.

[0089] In summary, the present application can realize multimodal feature extraction, cross-modal interaction, multimodal feature fusion and prediction result fusion of drone inspection data, which significantly improves the overall performance of the system. Specifically, multi-scale convolutional neural networks and multi-band recurrent neural networks are used to extract features of different scales from infrared images and visible light images, and features of different frequency bands are extracted from environmental parameters to ensure the diversity and richness of features. Through adaptive feature splicing algorithms and autoencoders, feature weights are dynamically adjusted and the robustness and expression ability of features are enhanced to generate high-dimensional comprehensive feature vectors. Cross-modal interaction modules and multimodal feature fusion networks are introduced to enable information exchange and complementation between features of different modalities, generate richer feature representations, and reduce feature dimensions through dynamic feature selection algorithms to improve the computational efficiency and generalization ability of the model. Use independent deep learning models to train data of different modalities, generate their own prediction results, and fuse the prediction results of each modality through multimodal prediction result fusion algorithms and adaptive weight adjustment mechanisms to ensure the accuracy and robustness of the final prediction results. After the final training is completed, the fused prediction results can be integrated into the decision support system to provide a visual interface to display image data, environmental parameters, fault identification results and warning signals, and optimize the accuracy and robustness of the model through a real-time feedback mechanism, significantly improving the efficiency and reliability of UAV inspection tasks.

[0090] D07. Determine whether the deep learning model converges according to the prediction result and the fault type label;

[0091] D08. If converged, the training of the deep learning model is terminated to obtain a fault identification model.

[0092] It can be understood that the prediction result can be a weighted fusion prediction result, and whether the deep learning model converges is determined by the fault type and fault type label of the prediction result. For example, by calculating the loss value between the predicted value and the true value, whether it converges is judged. When it converges, the model training is completed to obtain the fault identification model. If it does not converge, the model parameters are adjusted and the deep learning model training is returned to continue until it converges or the preset number of iterations is reached.

[0093] 105. Integrate the fault identification model into the drone inspection system of the drone to achieve real-time monitoring and intelligent diagnosis of the fault type of the target area;

[0094] Furthermore, the fault identification model (i.e., algorithm model) is integrated into the UAV inspection system to achieve real-time monitoring and intelligent diagnosis of the target area, while supporting remote access and data sharing to facilitate multi-user collaborative work and decision support. The integrated system can receive image data and environmental parameters transmitted by the UAV in real time, automatically evaluate whether there are predefined fault modes in the image through the fault identification model, and generate corresponding warning signals. The warning signal is transmitted to the ground control center and the user's terminal device in real time through the wireless communication module to notify the operator to take corresponding preventive measures. The system provides a visual interface to display image data, environmental parameters, fault identification results and warning signals, supports multi-user simultaneous access and collaborative work, and improves work efficiency and decision-making quality. In addition, the system supports data sharing, and users can export or share the collected data and generated reports with other users and teams to ensure data security and privacy protection.

[0095] 106. Regularly update the annotated multimodal data set and fault identification model to obtain an updated fault identification model.

[0096] Finally, the image matching rules and algorithm models will be updated regularly to ensure that they continue to adapt to environmental changes and technological advances and maintain the accuracy and reliability of the system. New annotated multimodal data sets can be obtained by regularly updating the matching rules, and new algorithm models can be obtained using the new annotated multimodal data sets. New inspection data, including infrared images, visible light images, timestamps, geographic location information, and environmental parameters, are regularly collected and annotated and preprocessed. The fault recognition model is retrained and optimized using new data to generate new image matching rules. Through the real-time feedback mechanism, the operating status and performance indicators of the system are monitored to detect and solve problems in a timely manner. Operators can feed back the actual processing results to the system for further optimization of models and rules. The overall performance of the system is regularly evaluated, and the models and rules are continuously optimized according to environmental changes and technological advances to ensure the accuracy and reliability of the system.

[0097] This application can achieve efficient and accurate inspection of the target area, significantly improving the accuracy and reliability of fault identification. The system can monitor the target area in real time, automatically detect and warn of potential faults or abnormal conditions, support multi-user collaboration and remote decision support, and greatly improve inspection efficiency and response speed. Regularly update image matching rules and algorithm models to ensure that the system continues to adapt to environmental changes and technological advances, maintain long-term accuracy and reliability, and thus provide strong technical support for drone inspection tasks.

[0098] In a feasible implementation, the step 106 specifically includes the following steps E01 to E04:

[0099] E01. Use multi-band recurrent neural network to extract features of different frequency bands from environmental parameters. Through multi-step time series analysis, capture the changing trends of environmental parameters on different time scales.

[0100] Among them, the use of multi-band recurrent neural networks can not only process high-frequency changes, but also capture low-frequency changes. Through multi-level time windows, the characteristics of environmental parameters can be extracted from different time scales from micro to macro, and high-dimensional time series feature vectors can be generated. These feature vectors not only contain the dynamic change information of environmental parameters, but also reflect the periodicity and trend at different time scales, providing rich background information for subsequent feature fusion.

[0101] E02. Align the extracted image features and environmental parameter features through an adaptive feature alignment algorithm. This algorithm calculates the similarity matrix between features and adjusts the dimension and order of feature vectors to ensure that features of different modalities have similar structures and distributions before splicing.

[0102] Among them, the adaptive feature alignment algorithm first calculates the similarity matrix between image features and environmental parameter features, evaluates the correlation between different features through cosine similarity, Euclidean distance or other similarity measurement methods, and dynamically adjusts the dimension and order of feature vectors according to the similarity matrix to ensure that the features of different modalities have similar structures and distributions before splicing. Through this adaptive alignment, not only can the inconsistency between features of different modalities be eliminated, but also the complementarity and complementarity of features can be enhanced, providing more consistent and coordinated input for subsequent feature fusion.

[0103] E03. Use dimensionality reduction techniques such as principal component analysis or linear discriminant analysis to reduce the dimension of the comprehensive feature vector, thereby improving the computational efficiency and generalization ability of the model. The feature vector after dimensionality reduction not only retains the main information of the original feature, but also reduces noise and redundant information.

[0104] In order to reduce the dimension of the feature vector and improve the computational efficiency and generalization ability of the model, this method uses dimensionality reduction techniques such as principal component analysis or linear discriminant analysis to reduce the dimension of the comprehensive feature vector. PCA generates a reduced-dimensional feature vector by finding the main components of the feature vector, retaining the direction of the maximum variance, removing redundant information and noise. LDA retains the discriminant information of the feature vector by maximizing the inter-class distance and minimizing the intra-class distance, and generates a more discriminative feature vector. The reduced-dimensional feature vector not only retains the main information of the original feature, but also reduces the dimension of the feature, improving the computational efficiency and generalization ability of the model. Through dimensionality reduction processing, not only can the training speed of the model be improved, but also the risk of overfitting can be reduced, and the robustness and generalization ability of the model can be improved.

[0105] E04, multi-modal feature fusion network, further fuses and represents the concatenated comprehensive feature vector through multiple layers of fully connected layers and non-linear activation functions. This network can capture the interactive information between different modal features and generate richer feature representations.

[0106] In order to generate richer feature representations, the present application designs a multimodal feature fusion network. The network further fuses and represents the spliced ​​comprehensive feature vector through multiple layers of fully connected layers and nonlinear activation functions. The multimodal feature fusion network first maps the reduced feature vector to a high-dimensional space through multiple layers of fully connected layers to capture the interactive information between different modal features. Then, through the nonlinear activation function, a nonlinear transformation is introduced to enhance the expressiveness and distinguishability of the features. The multimodal feature fusion network can not only capture the linear relationship between different modal features, but also capture the nonlinear relationship to generate richer feature representations. Through this multi-layer nonlinear transformation, not only the robustness and generalization ability of the features can be improved, but also the predictive performance of the model can be enhanced, providing a more accurate and reliable basis for the final fault identification and early warning.

[0107] In summary, through multi-band feature extraction, adaptive feature alignment, dimensionality reduction processing and multimodal feature fusion, this method not only improves the environmental perception and data processing capabilities of the UAV inspection system, but also significantly improves the overall performance of the system. These innovative methods are not only profound in theory, but also demonstrate strong practical value in practical applications, providing strong technical support for UAV inspection tasks and significantly improving inspection efficiency and fault identification accuracy.

[0108] The present invention provides a method for fault identification of unmanned aerial vehicle inspection, the method comprising: on a preset flight path, a target area is synchronously inspected by a unmanned aerial vehicle equipped with at least an infrared camera and a visible light camera, and infrared and visible light image data of the target area under different environmental conditions are collected; a multimodal data set is constructed according to the collected image data, the multimodal data set comprising image data and environmental parameters of environmental conditions corresponding to the image data, the environmental parameters comprising at least a timestamp, a geographical location and meteorological parameters; a fault type label of the image data is annotated by using the multimodal data set to obtain an annotated multimodal data set; a fault identification model is trained based on the annotated multimodal data set, the fault identification model is used to automatically predict the fault type in the new infrared and visible light images when receiving new infrared and visible light image inputs, and to give corresponding warning signals; the fault identification model is integrated into the unmanned aerial vehicle inspection system of the unmanned aerial vehicle to realize real-time monitoring and intelligent diagnosis of the fault type of the target area; the annotated multimodal data set and the fault identification model are regularly updated to obtain an updated fault identification model. Through the above method, multimodal data acquisition, adaptive model training and dynamic update technology are introduced to achieve efficient inspection and intelligent diagnosis of the target area, improve the accuracy of fault identification and the long-term reliability of the system.

[0109] See also Figure 2 , Figure 2 FIG. 1 is a structural block diagram of a fault identification device for unmanned aerial vehicle inspection in an embodiment of the present invention. Figure 2 The device shown comprises:

[0110] Data collection module 201: used to perform synchronous inspection of a target area on a preset flight path by using a drone equipped with at least an infrared camera and a visible light camera, and collect infrared and visible light image data of the target area under different environmental conditions;

[0111] A collection construction module 202 is used to construct a multimodal data set based on the collected image data, wherein the multimodal data set includes the image data and environmental parameters of environmental conditions corresponding to the image data, wherein the environmental parameters include at least a timestamp, a geographic location, and a meteorological parameter;

[0112] The label marking module 203 is used to mark the fault type labels of the image data using the multimodal data set to obtain a marked multimodal data set;

[0113] Model training module 204: used to train a fault recognition model based on the annotated multimodal data set, wherein the fault recognition model is used to automatically predict the fault type in the new infrared and visible light images when receiving new infrared and visible light image inputs, and to give corresponding warning signals;

[0114] Fault monitoring module 205: used to integrate the fault identification model into the drone inspection system of the drone to achieve real-time monitoring and intelligent diagnosis of the fault type of the target area;

[0115] Model updating module 206: used for regularly updating the annotated multimodal data set and fault identification model to obtain an updated fault identification model.

[0116] It should be noted that Figure 2 The functions of each module in the device shown are Figure 1 The contents of each step in the method shown are similar, and are not described here to avoid repetition. For details, please refer to Figure 1 The content of each step in the method shown.

[0117] The present invention provides a fault identification device for unmanned aerial vehicle inspection, the device comprising: Figure 2 , Figure 2 FIG. 1 is a structural block diagram of a fault identification device for unmanned aerial vehicle inspection in an embodiment of the present invention. Figure 2 The device shown includes: a data collection module: used for synchronously inspecting a target area on a preset flight path by using a drone equipped with at least an infrared camera and a visible light camera, and collecting infrared and visible light image data of the target area under different environmental conditions; a collection construction module: used for constructing a multimodal data set based on the collected image data, the multimodal data set including image data and environmental parameters of environmental conditions corresponding to the image data, the environmental parameters including at least timestamp, geographic location and meteorological parameters; a label annotation module: used for annotating fault type labels of image data using the multimodal data set to obtain annotated multimodal data set; a model training module: used for training a fault recognition model based on the annotated multimodal data set, the fault recognition model being used for automatically predicting the fault type in the new infrared and visible light images when receiving new infrared and visible light image inputs, and giving corresponding warning signals; a fault monitoring module: used for integrating the fault recognition model into the drone inspection system of the drone, so as to realize real-time monitoring and intelligent diagnosis of the fault type of the target area; a model updating module: used for regularly updating the annotated multimodal data set and the fault recognition model to obtain an updated fault recognition model. Through the above-mentioned device, multimodal data acquisition, adaptive model training and dynamic update technology are introduced to achieve efficient inspection and intelligent diagnosis of the target area, improve the accuracy of fault identification and the long-term reliability of the system.

[0118] Figure 3 FIG. 1 shows an internal structure diagram of a computer device in an embodiment. The computer device may be a terminal or a server. Figure 3As shown, the computer device includes a processor, a memory and a network interface connected via a system bus. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program. When the computer program is executed by the processor, the processor can implement the above method. The internal memory may also store a computer program. When the computer program is executed by the processor, the processor can implement the above method. Those skilled in the art can understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0119] In one embodiment, a computer device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the following steps: Figure 1 The steps of the method are shown.

[0120] In one embodiment, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the processor performs the following steps: Figure 1 The steps of the method are shown.

[0121] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0122] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0123] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for fault identification during drone inspection, characterized in that: The method comprises: On a preset flight path, a drone equipped with at least an infrared camera and a visible light camera is used to synchronously inspect the target area, and infrared and visible light image data of the target area under different environmental conditions are collected; Constructing a multimodal data set based on the collected image data, the multimodal data set including the image data and environmental parameters of environmental conditions corresponding to the image data, the environmental parameters including at least a timestamp, a geographic location, and a meteorological parameter; Using the multimodal data set to perform labeling of fault type labels of image data to obtain a labeled multimodal data set; Based on the annotated multimodal data set, a fault recognition model is trained, wherein the fault recognition model is used to automatically predict the fault type in the new infrared and visible light images when receiving new infrared and visible light image inputs, and to give corresponding warning signals; Integrating the fault identification model into the drone inspection system of the drone to achieve real-time monitoring and intelligent diagnosis of the fault type in the target area; The annotated multimodal data set and the fault identification model are updated regularly to obtain an updated fault identification model.

2. The method according to claim 1, characterized in that: The method of using the multimodal data set to label the fault type labels of the image data to obtain the labeled multimodal data set includes: Performing registration processing on the infrared image and the visible light image in the image data to obtain registered image data; The registered image data is used to perform fault type labeling processing to obtain the fault type label of the image data, and the labeled multimodal data set includes image data, fault type label, timestamp, geographic location and meteorological parameters.

3. The method according to claim 1, characterized in that: The multimodal data set is constructed based on the collected image data, including: By using real-time environmental perception technology, the sensors carried by the drone are used to monitor the environmental parameters of the flight environment of the drone in real time, wherein the environmental parameters include timestamp, meteorological parameters and geographic location; The image data and environmental parameters are multimodally fused to construct a multimodal dataset.

4. The method according to claim 1 or 3, characterized in that: The method further comprises: Recording the spatiotemporal distribution of the environmental parameters and establishing a three-dimensional spatiotemporal model of the environmental parameters; When an obstacle is detected in front of the UAV, a path is replanned based on the three-dimensional space-time model and the flight path to obtain an obstacle avoidance path to bypass the obstacle.

5. The method according to claim 1, characterized in that: The method of using the multimodal dataset to label the fault type labels of the image data to obtain the labeled multimodal dataset also includes: Filtering the image data using a preset wavelet filtering algorithm to obtain filtered image data; Filtering the environmental parameters using a preset extended Kalman filter model to obtain filtered environmental parameters; Multimodal features are obtained from filtered image data and environmental parameters, and the weights of each modal feature are dynamically optimized using an adaptive weight adjustment algorithm to obtain optimized image data and environmental parameters. The multimodal data set includes the optimized image data and environmental parameters.

6. The method according to claim 1, characterized in that: The training of the fault recognition model based on the annotated multimodal data set includes: The labeled multimodal dataset is used as a training dataset, wherein each training sample of the training dataset includes image data, environmental parameters, timestamp, geographic location, and fault type label; Using the training data set to train a preset deep learning model to generate a fault recognition model; Using a preset verification data set to verify the fault identification model and evaluate the performance indicators of the fault identification model; The model parameters of the fault identification model are optimized according to the performance index to obtain an optimized fault identification model.

7. The method according to claim 6, characterized in that: The method of using the training data set to train a preset deep learning model to generate a fault recognition model includes: Use multi-scale convolutional neural networks to extract image features of different scales from infrared images and visible light images; use multi-band recurrent neural networks to extract environmental features of different frequency bands from environmental parameters; Dynamically adjusting the weights of the image features and the environmental features through an adaptive feature concatenation algorithm to generate a high-dimensional comprehensive feature vector, and enhancing the comprehensive feature vector using an autoencoder; A cross-modal interaction module is introduced to enable information exchange and complementation between features of different modalities of the comprehensive feature vector through an attention mechanism and a gating mechanism, thereby obtaining a comprehensive feature vector after interaction; Designing a multimodal feature fusion network, further fusing and representing the interactive comprehensive feature vector through multiple layers of fully connected layers and nonlinear activation functions, to obtain a fused comprehensive feature vector; The fused comprehensive feature vector is subjected to dimensionality reduction processing by using a dynamic feature selection algorithm to obtain a comprehensive feature vector after dimensionality reduction; Using the comprehensive feature vector after dimensionality reduction and the deep learning model to predict the fault type, and obtain a prediction result; Determining whether the deep learning model converges according to the prediction result and the fault type label; If converged, the training of the deep learning model is terminated to obtain a fault identification model.

8. A fault identification device for drone inspection, characterized in that: The device comprises: Data collection module: used to synchronously inspect the target area on a preset flight path by using a drone equipped with at least an infrared camera and a visible light camera to collect infrared and visible light image data of the target area under different environmental conditions; A collection construction module: used to construct a multimodal data set based on the collected image data, wherein the multimodal data set includes the image data and environmental parameters of environmental conditions corresponding to the image data, wherein the environmental parameters include at least a timestamp, a geographic location, and a meteorological parameter; A label marking module is used to mark the fault type labels of the image data using the multimodal data set to obtain a marked multimodal data set; Model training module: used for training a fault recognition model based on the annotated multimodal data set, wherein the fault recognition model is used for automatically predicting the fault type in the new infrared and visible light images when receiving new infrared and visible light image inputs, and giving corresponding warning signals; Fault monitoring module: used to integrate the fault identification model into the drone inspection system of the drone to achieve real-time monitoring and intelligent diagnosis of the fault type of the target area; Model updating module: used to regularly update the annotated multimodal data set and fault identification model to obtain an updated fault identification model.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 7.

10. A computer device comprising a memory and a processor, characterized in that: The memory stores a computer program, and when the computer program is executed by the processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 7.

Images