Analysis method and system based on multi-mode conversion protocol for unmanned aerial vehicle
Through the automated drone inspection and multimodal data fusion, an abnormal feature recognition model is built, which solves the equipment status error problem during photovoltaic power station inspection, improves diagnostic accuracy and efficiency, and extends the equipment life.
Patent Information
- Application Number
- CN202411426073.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-14
- Publication Date
- 2024-11-15
AI Technical Summary
The existing photovoltaic power station inspection technology relies on empirical value formulas or simplified models, resulting in large errors in capturing and characterizing the real operating status of the equipment, and may cause excessive maintenance or insufficient maintenance, affecting the operation and maintenance effect.
Using a multimodal conversion protocol-based analysis method for drones, through automated patrols of drones, thermal imaging, equipment power and acoustic data, a multimodal abnormality feature recognition model is built to improve the accuracy and efficiency of abnormal diagnosis. .
It improves the accuracy and efficiency of abnormal diagnosis of photovoltaic power generation equipment, reduces potential failure risk, extends the service life of the equipment, and predicts faults through in-depth analysis, reduces noise interference, and improves the adaptability and robustness of the model.
Smart Images

Figure CN118965276A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of unmanned aerial vehicles, and in particular to an analysis method and system based on a multi-modal conversion protocol for unmanned aerial vehicles. Background Art
[0002] As an efficient, flexible and safe inspection method, UAV photovoltaic inspection technology has been widely used in the inspection of photovoltaic power stations in recent years. Compared with traditional manual inspection, UAV inspection has the advantages of convenience, efficiency and safety, which can greatly reduce the inspection cost and obtain more comprehensive and accurate data information. Therefore, the photovoltaic power station inspection system based on UAV has great research significance.
[0003] The Chinese patent with publication number CN118115591A discloses a power prediction model training method, photovoltaic power station operation and maintenance method and related devices. The neural network model is trained by collecting thermal infrared images, photovoltaic string power generation and weather data of the photovoltaic power station, and using data information from different sources to complement each other, which can provide a more comprehensive perspective to monitor and evaluate the status of the photovoltaic power station; after collecting the thermal infrared image, it is aligned to locate the position of the photovoltaic module in the thermal infrared image, and on this basis, the photovoltaic string power and weather data are integrated to realize the power generation prediction of each photovoltaic module. However, the evaluation methods adopted in the above applications mostly rely on empirical formulas or simplified model calculations, which makes the capture and characterization of the actual operating status of the equipment have large errors, and may lead to over-maintenance or under-maintenance, affecting the overall operation and maintenance effect. Therefore, it is very necessary to provide an analysis method and system based on a multi-modal conversion protocol for drones to improve the accuracy and efficiency of abnormal diagnosis. Summary of the invention
[0004] In view of this, the present invention proposes an analysis method and system based on a multimodal conversion protocol for drones. The use of drones for automated inspections overcomes the limitations of manual inspections, greatly improves inspection coverage and efficiency, and enhances safety. By fusing multi-source heterogeneous data and applying multimodal fusion technology, a more accurate equipment anomaly detection model is constructed, which improves the accuracy and efficiency of anomaly diagnosis.
[0005] The present invention provides an analysis method based on a multi-modal conversion protocol for unmanned aerial vehicles, the method comprising: S1, according to the geographic information of the photovoltaic power generation equipment to be inspected, the ground inspection system plans the inspection path of the drone and the corresponding photovoltaic power generation equipment to be inspected; S2, collecting initial multi-source heterogeneous data on the inspection path, and preprocessing the initial multi-source heterogeneous data to obtain integrated multi-source heterogeneous data, wherein the initial multi-source heterogeneous data includes thermal imaging data, equipment power data, and acoustic data; S3, extracting features corresponding to the thermal imaging data, the equipment power data, and the acoustic data in the integrated multi-source heterogeneous data, and screening out a key feature subset through a feature selection method; S4, normalizing the key feature subset, integrating the normalized key feature subset into an anomaly detection feature vector, and constructing a multimodal anomaly feature recognition model; S5: Perform anomaly detection on the multi-source heterogeneous data based on the anomaly detection feature vector and the multimodal anomaly feature recognition model.
[0006] On the basis of the above technical solution, preferably, the method of enabling the ground inspection system to plan the inspection path of the drone and the corresponding photovoltaic power generation equipment to be inspected according to the location information of the photovoltaic power generation equipment to be inspected specifically includes: Determine a path starting point, a path end point, and a plurality of intermediate stop points according to the geographic information of the photovoltaic power generation equipment to be inspected and the ground inspection system, wherein the plurality of intermediate stop points are determined according to the location of the photovoltaic power generation equipment to be inspected; According to the geographic information of the photovoltaic power generation equipment to be inspected and the ground inspection system, the sub-inspection paths corresponding to the multiple intermediate stop points are determined, and the starting point and the end point of any sub-inspection path are the intermediate stop points corresponding to the sub-inspection path; The inspection path is generated according to the path starting point, the path end point, the multiple intermediate stop points, and the sub-inspection paths respectively corresponding to the multiple intermediate stop points.
[0007] On the basis of the above technical solution, preferably, the collecting of initial multi-source heterogeneous data on the inspection path and preprocessing of the initial multi-source heterogeneous data specifically include: Collecting initial multi-source heterogeneous data on the inspection path according to the built-in sensor array of the drone; The initial multi-source heterogeneous data is subjected to noise removal and data normalization processing to obtain optimized multi-source heterogeneous data.
[0008] More preferably, the acquiring and integrating multi-source heterogeneous data specifically includes: Based on a predefined multimodal conversion protocol, the optimized multi-source heterogeneous data is parsed, format converted, and semantically mapped to obtain integrated multi-source heterogeneous data, where the integrated multi-source heterogeneous data represents the optimized multi-source heterogeneous data after data unification.
[0009] More preferably, extracting features corresponding to the thermal imaging data, equipment power data and acoustic data in the integrated multi-source heterogeneous data and screening out key feature subsets by a feature selection method specifically includes: Extracting temperature variation features from the thermal image data, power fluctuation features from the device power data, and device vibration features from the acoustic data; Respectively calculating the correlation coefficients of the temperature change characteristic, the power fluctuation characteristic, and the equipment vibration characteristic with the corresponding target variables to obtain a first correlation coefficient, a second correlation coefficient, and a third correlation coefficient, wherein the first correlation coefficient represents the correlation coefficient of the temperature change characteristic with the temperature target variable, the second correlation coefficient represents the correlation coefficient of the power fluctuation characteristic with the power target variable, and the third correlation coefficient represents the correlation coefficient of the equipment vibration characteristic with the vibration target variable; Based on the absolute values of the first correlation coefficient, the second correlation coefficient and the third correlation coefficient, the temperature change characteristics, the power fluctuation characteristics and the equipment vibration characteristics are sorted in descending order, and the top K features with the highest correlation with the target variable are selected as the key feature subset.
[0010] More preferably, the expression of the first correlation coefficient is: ; ; ; Among them, r represents the initial correlation coefficient, x pq represents the qth temperature change feature in the pth sample, y i represents the temperature target variable value in the pth sample, represents the average value of the temperature variation characteristic, represents the average value of the temperature target variable, t represents the test statistic, R represents the first correlation coefficient, N represents the sample size, and p represents the sample size.
[0011] Further preferably, the multimodal abnormal feature recognition model includes a hypergraph convolutional neural network, and the hypergraph convolutional neural network introduces a multimodal attention mechanism to assign weights to different vertices in the hypergraph convolutional neural network.
[0012] More preferably, the expression of the hypergraph convolutional neural network is: ; ; ; ; ; in, Represents the first l The output function in the hidden layer is represents the sigmoid function, Indicates l The weight coefficients of the hidden layers are Indicates l The bias of the hidden layer, v i Represents the first i Vertices, v j represents the jth vertex in the hypergraph, Represents a hypergraph vertex v i and v j The attention coefficient when it intersects with the hyperedge, T represents the transformation matrix, represents the normalized exponential function, represents the rectified linear function, represents the attention parameter vector, represents the attention weight matrix, Represents the first i The fusion representation of vertices, N( i ) represents the first i The modal set of connected vertices, Indicates l The fusion output function of the combined layers is Indicates l The weight coefficient of the combined layer, Indicates l The bias term of the combined layer, represents the classification function, represents the weight coefficient of the output layer, Represents the bias term of the output layer.
[0013] More preferably, performing anomaly detection on the multi-source heterogeneous data based on the anomaly detection feature vector and the multimodal anomaly feature recognition model specifically includes: Inputting the abnormality detection feature vector into the multimodal abnormality feature recognition model to perform abnormality detection on the photovoltaic power generation equipment to be inspected; When it is detected that the working state of the photovoltaic power generation equipment to be inspected is abnormal, the knowledge graph reasoning and case reasoning methods are used to perform fault tracing analysis, and the abnormal diagnosis results and fault analysis report are output; The abnormal diagnosis result and the fault analysis report are sent to the terminal device of the relevant personnel for archiving, and the relevant personnel are prompted to inspect and repair the photovoltaic power generation equipment to be inspected.
[0014] In a second aspect of the present application, an analysis system based on a multimodal conversion protocol for unmanned aerial vehicles is provided, wherein the analysis system comprises a data acquisition module, a frequency modulation optimization module and a deviation frequency modulation module connected in sequence, wherein: The data acquisition module is used to enable the ground inspection system to plan the inspection path of the drone and the corresponding photovoltaic power generation equipment to be inspected according to the geographic information of the photovoltaic power generation equipment to be inspected, collect the initial multi-source heterogeneous data on the inspection path, and pre-process the initial multi-source heterogeneous data to obtain and integrate the multi-source heterogeneous data, wherein the initial multi-source heterogeneous data includes thermal image data, equipment power data and acoustic data; The frequency modulation optimization module is used to extract features corresponding to the thermal imaging data, equipment power data and acoustic data in the integrated multi-source heterogeneous data, and screen out a key feature subset through a feature selection method; The deviation frequency modulation module is used to normalize the key feature subset, integrate the normalized key feature subset into an anomaly detection feature vector, and construct a multimodal anomaly feature recognition model, and perform anomaly detection on the multi-source heterogeneous data based on the anomaly detection feature vector and the multimodal anomaly feature recognition model.
[0015] The analysis method based on multimodal conversion protocol for unmanned aerial vehicles provided by the present invention has the following beneficial effects compared with the prior art: (1) By planning the inspection route of drones, the coverage and efficiency of inspections can be greatly improved, and the operation data of photovoltaic power generation equipment can be collected in an all-round and high-precision manner. At the same time, by fusing multi-source heterogeneous data such as thermal imaging, power, and acoustics, and using multi-modal fusion technology to build a multi-modal abnormal feature recognition model, it can fully explore the inherent correlation between different data sources and improve the accuracy of abnormal diagnosis. By conducting in-depth analysis of monitoring data, the accuracy and efficiency of abnormal diagnosis are greatly improved. At the same time, through the in-depth analysis of monitoring data by the multi-modal abnormal feature recognition model, the potential fault risk is predicted, which further reduces the risk of photovoltaic power generation equipment operation and increases the service life of photovoltaic power generation equipment; (2) By calculating the correlation coefficients between different features and the target variable and sorting them, the features with the greatest impact on the target variable can be screened out, thereby improving the performance and accuracy of the prediction model. In the feature screening process, only the features with the highest correlation with the target variable are retained, which reduces the data dimension and improves the efficiency of data processing and analysis. The features that are closely related to the target variable are screened out, which facilitates the interpretation and understanding of the model, thereby enabling better analysis and optimization of system performance. At the same time, by selecting the most relevant features, the interference of noise can be reduced, and the adaptability and robustness of the model to different scenarios can be improved. Finally, the combination of hypergraph convolutional neural network and multimodal attention mechanism is adopted, so that the combined network can utilize both the expressiveness of Hopfield neural network and the identifiability of attention mechanism, thereby improving the accuracy of anomaly identification and the interpretability of multimodal anomaly feature recognition model; (3) Multi-source heterogeneous data are integrated into anomaly detection feature vectors, and a multimodal anomaly feature recognition model is used for anomaly detection. Compared with the detection of a single data source, this method can identify the abnormal status of equipment more comprehensively and accurately. At the same time, knowledge graph reasoning and case reasoning methods are used for fault tracing analysis, which can quickly locate the cause of the anomaly and output detailed anomaly diagnosis results and fault analysis reports, greatly improving the efficiency of anomaly diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0017] Figure 1 A schematic diagram of a flow chart of an analysis method based on a multi-modal conversion protocol for drones provided by the present invention; Figure 2 This is a schematic diagram of the structure of the analysis system provided by the present invention.
[0018] Explanation of the accompanying drawings: 1. Analysis system; 11. Data acquisition module; 12. Frequency modulation optimization module; 13. Deviation frequency modulation module. DETAILED DESCRIPTION
[0019] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0020] The present application embodiment discloses an analysis method based on a multi-modal conversion protocol for drones, such as Figure 1 As shown, the method comprises steps S1 to S5.
[0021] Step S1, according to the geographic information of the photovoltaic power generation equipment to be inspected, the ground inspection system plans the inspection path of the drone and the corresponding photovoltaic power generation equipment to be inspected.
[0022] In this embodiment, the ground inspection system includes a BIM model, and the inspection path of the drone is optimized through the BIM model, without the need to use various complex path planning algorithms. In addition, the optimization of the inspection path is associated with the BIM model of photovoltaic power generation equipment detection, so that an inspection path that is more suitable for the inspection needs of the engineering scene can be obtained. Therefore, this technical solution can realize automatic drone inspection path optimization based on BIM modeling, and improve the efficiency and accuracy of path optimization.
[0023] The BIM model can be understood as an application of three-dimensional modeling. The three-dimensional model constructed by the BIM model can be set as a photovoltaic power generation equipment information model, which can be understood as a model that characterizes the modeling of multiple photovoltaic power generation equipment, the positional relationship between multiple photovoltaic power generation equipment, and the equipment information of photovoltaic power generation equipment.
[0024] It is understandable that the geometric shape and size of photovoltaic modules can be accurately established in the BIM model by inputting the geographic information and shape information of the photovoltaic power generation equipment, and can also include the appearance, installation angle and orientation of the photovoltaic power generation equipment.
[0025] In the photovoltaic inspection scenario, a drone and a terminal device may be included, and the drone and the terminal device are in communication connection. The number of the drones may be one or more.
[0026] In some embodiments, the terminal device can send an inspection path to the drone so that the drone performs inspections according to the inspection path and transmits the obtained inspection data back to the terminal device. On the one hand, the terminal device can determine the inspection results based on the inspection data and feed back the inspection results to relevant personnel. On the other hand, the inspection path of the drone can be optimized based on the inspection data so that the drone performs optimized inspections according to the optimized inspection path and transmits the obtained inspection data back to the terminal device. Thus, the executor of the technical solution provided in the embodiment of the present application can be a terminal device, which is used to optimize the inspection path of the drone.
[0027] This step also includes steps S11 to S13.
[0028] Step S11, determining the path starting point, path end point and multiple intermediate stop points according to the geographic information of the photovoltaic power generation equipment to be inspected and the ground inspection system, wherein the multiple intermediate stop points are determined according to the location of the photovoltaic power generation equipment to be inspected.
[0029] In this step, based on the BIM model in the ground inspection system, the path starting point, path end point and multiple intermediate stop points are determined. For example, the path starting point can be a certain edge position of the BIM model in the ground inspection system, and correspondingly, the path end point is another edge position of the BIM model in the ground inspection system. Each intermediate stop point can correspond to a preset type of photovoltaic power generation equipment, and the intermediate stop point can be an inspection position of a preset type of photovoltaic power generation equipment, such as the top, left, right, etc. of the preset type of photovoltaic power generation equipment.
[0030] Step S12, according to the geographic information of the photovoltaic power generation equipment to be inspected and the ground inspection system, determine the sub-inspection paths corresponding to the multiple intermediate stop points, and the starting point and the end point of any sub-inspection path are the intermediate stop points corresponding to the sub-inspection path.
[0031] Furthermore, for each intermediate stop point, a corresponding sub-inspection path needs to be determined. For any sub-inspection path, the starting point of the sub-inspection path and the end point of the sub-inspection path are both intermediate stop points corresponding to the sub-inspection path, and the path length of the sub-inspection path is determined according to the shape parameters of the preset type of photovoltaic power generation equipment where the intermediate stop point corresponding to the sub-inspection path is located. The shape parameters may include: length, width, height, diameter (radius), etc. For example, if determined according to the width / diameter, the path length is greater than 2 times the width / diameter. Or it can be understood that the path length can achieve a circle around the preset type of photovoltaic power generation equipment.
[0032] Step S13, generating an inspection path according to the path starting point, the path end point, the plurality of intermediate stop points, and the sub-inspection paths corresponding to the plurality of intermediate stop points.
[0033] In addition, multiple obstacle avoidance points need to be determined, and the multiple obstacle avoidance points may be the locations where obstacles are about to be reached.
[0034] In this implementation, the inspection path can be generated according to some preset path planning algorithms, such as some curve generation algorithms. For example, the inspection path between two points can be generated according to the curve generation algorithm based on two points each time. For the sub-inspection paths corresponding to multiple intermediate stop points, the sub-inspection paths can be generated according to the curve generation algorithm based on the path length and the intermediate stop points.
[0035] Step S2, collecting initial multi-source heterogeneous data on the inspection path, and preprocessing the initial multi-source heterogeneous data to obtain integrated multi-source heterogeneous data, wherein the initial multi-source heterogeneous data includes thermal imaging data, equipment power data, and acoustic data.
[0036] This step also includes steps S21 to S23.
[0037] Step S21, collecting initial multi-source heterogeneous data on the inspection path according to the sensor array built into the drone.
[0038] In this step, the drone collects multi-source heterogeneous data on the photovoltaic power generation equipment through the multi-modal sensor array it carries. The multi-modal sensor array includes one or more of a light intensity sensor, a temperature and humidity sensor, and a sound sensor.
[0039] Furthermore, drones can carry special power sensor modules, which can directly connect to the cables or interfaces of photovoltaic power generation equipment to collect real-time power parameter data such as voltage, current, and power. Wireless communication technologies such as Bluetooth and WIFI can also be used to exchange data with the monitoring unit of photovoltaic power generation equipment and obtain power parameter data remotely. Drones can be equipped with environmental monitoring equipment such as light intensity sensors and temperature and humidity sensors to collect real-time environmental status data of photovoltaic power stations during flight. These environmental parameter data are helpful for analyzing the actual power generation performance and abnormal conditions of photovoltaic battery groups. Drones can be equipped with highly sensitive sound sensors to collect real-time sound signals of photovoltaic power generation equipment during flight inspections. By analyzing these sound data, equipment abnormalities such as wear of mechanical parts and electrical failures can be detected.
[0040] Step S22, performing noise removal and data normalization processing on the initial multi-source heterogeneous data to obtain optimized multi-source heterogeneous data.
[0041] Step S23, based on a predefined multimodal conversion protocol, the optimized multi-source heterogeneous data is parsed, format converted, and semantically mapped to obtain integrated multi-source heterogeneous data, and the integrated multi-source heterogeneous data represents the optimized multi-source heterogeneous data after data unification.
[0042] In this step, a unified multimodal data conversion protocol is developed to specify the standard format of various sensor or device data, and a common data model is defined in the protocol, including standards such as data type, data field, unit, encoding method, etc. At the same time, semantic standards are defined to map and associate data fields with actual meanings.
[0043] For example, for different types of sensors and devices, such as temperature sensors, pressure sensors, power equipment, etc., a unified data model standard is defined to develop a complete data model for each device type, define the data type as any of integer, floating point, and string, and define the data field (such as temperature, pressure, voltage, current, etc.), and add other extended attributes (such as timestamp, device ID, etc.) to establish semantic labels and descriptions for each data model field.
[0044] For example, the "temp" field is mapped to the semantic meaning of "device temperature", and the "pressure" field is mapped to the semantic meaning of "device pressure" to ensure the correspondence between data fields and actual business concepts, providing semantic support for subsequent data analysis applications.
[0045] In one example, a metadata management mechanism can be established in the ground control system to maintain metadata information of various sensors or equipment, such as model, range, unit, etc. During the data conversion process, this metadata information is used to calibrate and convert units of the original data to ensure the integrity and interpretability of the data.
[0046] Furthermore, when the original heterogeneous data is received, the data is first parsed using the parsing rules specified in the protocol. It is understandable that a data adaptation layer is set up in the ground control system, which is responsible for receiving various types of raw data and converting them into a standardized format. The adaptation layer can use a predefined conversion template or a dynamic parsing method to automatically identify and adapt raw data in different formats. According to the data model, the raw data is mapped to a unified data structure, such as JSON, XML and other formats. And during the parsing process, the data fields need to be identified and corresponded to the standard fields defined in the protocol.
[0047] It is understandable that for different raw data formats, unified conversion rules to standard data models are defined, including conversion of encoding methods, such as converting hexadecimal to decimal, unified conversion of units such as converting Celsius to Fahrenheit, unified processing of precision (such as retaining 2 decimal places) and standardization of time formats (such as YYYY-MM-DD HH:mm:ss), to ensure that all input data can be converted into a unified standard format.
[0048] Furthermore, after the data of different modes (such as vision, thermal imaging, electricity, etc.) are converted through the above steps, they are associated and fused. Complete multimodal equipment operation status information is constructed to provide a unified data foundation for subsequent analysis applications. The spatiotemporal correlation between data needs to be considered in the fusion process to ensure the consistency and integrity of the data. By pre-defining multimodal conversion protocols and parsing, format conversion, and semantic mapping of heterogeneous data in accordance with protocol standards, a unified representation of multimodal data can be finally achieved.
[0049] Step S3, extracting and integrating the features corresponding to the thermal imaging data, equipment power data and acoustic data in the multi-source heterogeneous data, and screening out the key feature subsets through the feature selection method.
[0050] By calculating the correlation coefficients between different features and the target variable and sorting them, the features that have the greatest impact on the target variable can be screened out, thereby improving the performance and accuracy of the prediction model. In the feature screening process, only the features with the highest correlation with the target variable are retained, which reduces the data dimension and improves the efficiency of data processing and analysis. The features that are closely related to the target variable are screened out, which facilitates the interpretation and understanding of the model, thereby better analyzing and optimizing system performance. At the same time, by selecting the most relevant features, the interference of noise can be reduced, and the adaptability and robustness of the model to different scenarios can be improved.
[0051] This step also includes steps S31 to S33.
[0052] Step S31, extracting temperature change features in thermal image data, power fluctuation features in device power data, and device vibration features in acoustic data.
[0053] In this step, for the extraction of temperature change features, the temperature value of the device surface at each moment can be extracted from the thermal image sequence to form temperature time series data, and the statistical characteristics of the temperature time series can be calculated. The statistical characteristics of the temperature time series include the average value, standard deviation, maximum value, and minimum value of the temperature; the number of temperature peaks, peak amplitude, and peak frequency; the first-order and second-order derivative characteristics such as the slope and curvature of the temperature change. The temperature time series is smoothed and filtered to remove measurement noise, extract the trend change of temperature, and use methods such as Fourier transform or wavelet transform to analyze the spectral characteristics of the temperature signal, and extract the frequency domain characteristics such as the main frequency, frequency distribution, and spectral power of the temperature signal.
[0054] To extract the power fluctuation characteristics in the equipment power data, the real-time power data can be extracted from the equipment operation records to form a power time series, and the statistical characteristics of the power time series can be calculated. The statistical characteristics of the power time series include the average value, standard deviation, maximum value, minimum value of power; the number of power peaks, peak amplitude, peak frequency; the slope, fluctuation amplitude, fluctuation frequency of power changes, etc. The power time series is smoothed to remove measurement interference and noise. The power spectrum density analysis method is used to extract the spectrum characteristics of the power signal. The extracted power fluctuation characteristics include the main frequency components, spectrum entropy, and power spectrum concentration.
[0055] To extract equipment vibration features from acoustic data, the vibration time series can be extracted from the acoustic signal collected by the equipment vibration sensor, and the time domain statistical features of the vibration signal can be calculated. The time domain statistical features of the vibration signal include the mean, standard deviation, and peak value of the vibration amplitude; the skewness, kurtosis and other high-order moment features of the vibration signal; the root amplitude and root mean square value of the vibration signal. The vibration signal is analyzed in the frequency domain to extract the spectrum features including the main frequency components, frequency center, frequency band energy, power spectrum density, harmonic distortion and other frequency domain parameters. Finally, the time-frequency analysis method is combined to extract the time-frequency features of the vibration signal (such as the time-frequency features of short-time Fourier transform and wavelet transform).
[0056] Step S32, respectively calculate the correlation coefficients between the temperature change characteristics, power fluctuation characteristics and equipment vibration characteristics and the corresponding target variables, and obtain the first correlation coefficient, the second correlation coefficient and the third correlation coefficient, wherein the first correlation coefficient represents the correlation coefficient between the temperature change characteristics and the temperature target variable, the second correlation coefficient represents the correlation coefficient between the power fluctuation characteristics and the power target variable, and the third correlation coefficient represents the correlation coefficient between the equipment vibration characteristics and the vibration target variable.
[0057] In this step, the expression of the first correlation coefficient is: ; ; ; Among them, r represents the initial correlation coefficient, x pq represents the qth temperature change feature in the pth sample, y i represents the temperature target variable value in the pth sample, represents the average value of the temperature variation characteristics, represents the average value of the temperature target variable, t represents the test statistic, R represents the first correlation coefficient, N represents the sample size, and p represents the sample size.
[0058] The expressions of the second correlation coefficient and the third correlation coefficient are consistent with those of the first correlation coefficient and will not be elaborated here.
[0059] Step S33, based on the absolute values of the first correlation coefficient, the second correlation coefficient and the third correlation coefficient, the temperature change characteristics, the power fluctuation characteristics and the equipment vibration characteristics are sorted in descending order, and the top K characteristics with the highest correlation with the target variable are selected as the key feature subset.
[0060] In this step, the absolute values of the first correlation coefficient, the second correlation coefficient and the third correlation coefficient are sorted by the size of the correlation coefficient, and the following sorting results are obtained: the X1th correlation coefficient corresponds to the X1th temperature change feature, the X2th correlation coefficient corresponds to the X2th power fluctuation feature, the X3th correlation coefficient corresponds to the X3th vibration feature... until the correlation coefficients of all features are sorted.
[0061] According to the above sorting results, the first K features with the highest correlation are selected as the key feature subset, where the value of K can be adjusted according to actual needs. Usually, the first 3-5 features are taken, and the original feature indicator names corresponding to these K features are recorded.
[0062] Step S4, normalizing the key feature subsets, integrating the normalized key feature subsets into anomaly detection feature vectors, and constructing a multimodal anomaly feature recognition model.
[0063] In this step, the multimodal abnormal feature recognition model includes a hypergraph convolutional neural network, and the hypergraph convolutional neural network introduces a multimodal attention mechanism to assign weights to different vertices in the hypergraph convolutional neural network.
[0064] The Hypergraph Convolutional Neural Network (HCNN) focuses on utilizing the non-Euclidean relationship between modalities to represent data as a hypergraph, where each modality is a node and the intricate relationship between them forms a hypergraph. On the other hand, the Multimodal Attention Fusion Network (MAFN) emphasizes an attention mechanism that can identify the key connections between modalities and ensure that related data is effectively fused. By combining the Hypergraph Convolutional Neural Network and the Multimodal Attention Mechanism, the combined network can utilize both the expressiveness of the Hopfield Neural Network and the identifiability of the attention mechanism, thereby improving the accuracy and interpretability of the model.
[0065] In this embodiment, given a data set containing n vertices (patterns), it can be understood that the hypergraph can be represented as H =( V , E ),in, V represents a vertex set or a pattern set,V ={ v 1, v 2, ..., v n}, E represents a vertex set or a pattern set, E ={ e 1, e 2, ..., e m}, m represents the number of supergrids.
[0066] The expression of the hypergraph convolutional neural network is: ; ; ; ; ; in, Represents the first l The output function in the hidden layer is represents the sigmoid function, Indicates l The weight coefficients of the hidden layers are Indicates l The bias of the hidden layer, v i Represents the first i Vertices, v j represents the jth vertex in the hypergraph, Represents a hypergraph vertex v i and v j The attention coefficient when it intersects with the hyperedge, T represents the transformation matrix, represents the normalized exponential function, represents the rectified linear function, represents the attention parameter vector, represents the attention weight matrix, Represents the first i The fusion representation of vertices, N( i ) represents the first i The modal set of connected vertices, Indicates l The fusion output function of the combined layers is Indicates l The weight coefficient of the combined layer, Indicates lThe bias term of the combined layer, represents the classification function, represents the weight coefficient of the output layer, Represents the bias term of the output layer.
[0067] Here, it can be seen from the above expression that the hyperedge ek refers to the edge connecting different vertices (modalities) in the hypergraph, that is, the hyperedge represents the association or connection relationship between different modalities in the hypergraph. This hyperedge plays the role of an attention coefficient in the hypergraph convolutional neural network, which is used to give different weights to the relationship between different modalities.
[0068] Step S5, performing anomaly detection on multi-source heterogeneous data based on anomaly detection feature vectors and a multimodal anomaly feature recognition model.
[0069] This step also includes steps S51 to S53.
[0070] Step S51, inputting the abnormality detection feature vector into the multimodal abnormality feature recognition model to perform abnormality detection on the photovoltaic power generation equipment to be inspected.
[0071] In this step, you can use historical normal working data and abnormal working data for supervised training, adjust the model's hyperparameters, such as network structure, learning rate, etc., optimize the model performance, evaluate the model's anomaly detection accuracy, recall rate and other indicators on the validation set, and continuously iterate and optimize until the performance of the multimodal anomaly feature recognition model meets the requirements. Input the real-time multimodal feature vector of the device to be detected into the trained anomaly detection model, and the model output is the judgment result of whether the current device status is abnormal.
[0072] Step S52: When it is detected that the working state of the photovoltaic power generation equipment to be inspected is abnormal, the knowledge graph reasoning and case reasoning methods are used to perform fault tracing analysis, and the abnormal diagnosis results and fault analysis report are output.
[0073] In this step, expert knowledge on failure modes, causes, and manifestations of photovoltaic power generation equipment is collected, and this knowledge is organized into an abnormal diagnosis knowledge graph in the form of entity-relationship. The entities in the knowledge graph include: fault types (such as battery failure, inverter failure), fault causes (such as aging, short circuit, overvoltage, etc.), fault symptoms (such as power drop, abnormal noise, etc.), etc., and the relationships include: semantic relationships such as cause, manifest, and diagnosis. The knowledge graph is constructed using ontology construction tools (such as Protégé) or graph databases (such as Neo4j), and all entities and their attributes, relationships, etc. are modeled into the graph database to ensure that the semantic relationships between entities in the graph are complete and accurate.
[0074] Furthermore, when an abnormal working state of the equipment is detected, the knowledge graph is activated for fault analysis. According to the detected abnormal features, semantic search and rule reasoning are performed in the knowledge graph to find the fault mode and possible fault causes that best match the current abnormal features. The results of graph reasoning are used to give preliminary abnormal diagnosis results. At the same time, the fault cases most similar to the current abnormal features are retrieved from the historical fault case library, and the fault causes, maintenance measures and other information of these historical cases are analyzed. Combined with the current abnormal features, a more credible fault analysis is given through case comparison and reasoning. The fault analysis results obtained by knowledge graph reasoning and case reasoning are integrated to give a comprehensive abnormal diagnosis result, including possible fault causes, fault manifestations, etc. At the same time, a corresponding fault analysis report is generated for reference by maintenance personnel.
[0075] Step S53, sending the abnormal diagnosis result and the fault analysis report to the terminal device of the relevant personnel for archiving, and prompting the relevant personnel to inspect and repair the photovoltaic power generation equipment.
[0076] In this step, the abnormal diagnosis results obtained in step S52 are standardized and formatted, wherein the abnormal diagnosis results include key information such as fault type, fault cause, and fault manifestation. The diagnosis results are integrated into a structured fault report. When the system detects an equipment abnormality and completes the abnormal diagnosis, it automatically triggers the generation and push of the report. The report clearly describes the symptoms, possible causes, recommended maintenance measures, etc. of the fault, and attaches supporting information such as relevant abnormal data and fault analysis process. Language and charts can be used to facilitate maintenance personnel to understand and handle.
[0077] By planning the inspection routes of drones, the coverage and efficiency of inspections can be greatly improved, and the operating data of photovoltaic power generation equipment can be collected in an all-round and high-precision manner. At the same time, by integrating multi-source heterogeneous data such as thermal imaging, power, and acoustics, and using multimodal fusion technology to build a multimodal abnormal feature recognition model, the inherent correlation between different data sources can be fully explored, which improves the accuracy of abnormal diagnosis. By conducting in-depth analysis of monitoring data, the accuracy and efficiency of abnormal diagnosis are greatly improved. At the same time, through the multimodal abnormal feature recognition model, the in-depth analysis of monitoring data can predict potential fault risks, further reduce the risk of photovoltaic power generation equipment operation, and increase the service life of photovoltaic power generation equipment.
[0078] Based on the above method, the embodiment of the present application discloses an analysis system based on a multi-modal conversion protocol for drones, referring to Figure 2 The analysis system 1 includes a data acquisition module 11, a frequency modulation optimization module 12 and a deviation frequency modulation module 13 connected in sequence, wherein: The data acquisition module 11 is used to enable the ground inspection system to plan the inspection path of the drone and the corresponding photovoltaic power generation equipment to be inspected according to the geographic information of the photovoltaic power generation equipment to be inspected, collect the initial multi-source heterogeneous data on the inspection path, and pre-process the initial multi-source heterogeneous data to obtain and integrate the multi-source heterogeneous data, wherein the initial multi-source heterogeneous data includes thermal image data, equipment power data and acoustic data; The frequency modulation optimization module 12 is used to extract and integrate the features corresponding to the thermal imaging data, equipment power data and acoustic data in the multi-source heterogeneous data, and screen out the key feature subset through the feature selection method; The deviation frequency modulation module 13 is used to normalize the key feature subsets, integrate the normalized key feature subsets into anomaly detection feature vectors, and construct a multimodal anomaly feature recognition model. Based on the anomaly detection feature vectors and the multimodal anomaly feature recognition model, anomaly detection is performed on multi-source heterogeneous data.
[0079] In one example, the data acquisition module 11 is used to determine the starting point, the end point and multiple intermediate stop points of the path based on the geographic information of the photovoltaic power generation equipment to be inspected and the ground inspection system, and the multiple intermediate stop points are determined according to the location of the photovoltaic power generation equipment to be inspected; according to the geographic information of the photovoltaic power generation equipment to be inspected and the ground inspection system, the sub-inspection paths corresponding to the multiple intermediate stop points are determined, and the starting point and end point of any sub-inspection path are the intermediate stop points corresponding to the sub-inspection path; according to the starting point, the end point, the multiple intermediate stop points and the sub-inspection paths corresponding to the multiple intermediate stop points, an inspection path is generated.
[0080] In one example, the data acquisition module 11 is used to collect initial multi-source heterogeneous data on the patrol path based on the sensor array built into the drone; and perform noise removal and data normalization on the initial multi-source heterogeneous data to obtain optimized multi-source heterogeneous data.
[0081] In one example, the data acquisition module 11 is used to parse, format convert, and semantically map optimized multi-source heterogeneous data based on a predefined multimodal conversion protocol, obtain integrated multi-source heterogeneous data, and integrate the multi-source heterogeneous data to represent the optimized multi-source heterogeneous data after data unification.
[0082] In one example, the frequency modulation optimization module 12 is used to extract temperature change characteristics in thermal imaging data, power fluctuation characteristics in equipment power data, and equipment vibration characteristics in acoustic data; respectively calculate the correlation coefficients between the temperature change characteristics, power fluctuation characteristics, and equipment vibration characteristics and the corresponding target variables, and obtain the first correlation coefficient, the second correlation coefficient, and the third correlation coefficient, wherein the first correlation coefficient represents the correlation coefficient between the temperature change characteristics and the temperature target variable, the second correlation coefficient represents the correlation coefficient between the power fluctuation characteristics and the power target variable, and the third correlation coefficient represents the correlation coefficient between the equipment vibration characteristics and the vibration target variable; based on the absolute values of the first correlation coefficient, the second correlation coefficient, and the third correlation coefficient, the temperature change characteristics, power fluctuation characteristics, and equipment vibration characteristics are sorted in descending order, and the top K characteristics with the highest correlation with the target variable are selected as the key feature subset.
[0083] In one example, the expression of the first correlation coefficient is: ; ; ; Among them, r represents the initial correlation coefficient, x pq represents the qth temperature change feature in the pth sample, y p represents the temperature target variable value in the pth sample, represents the average value of the temperature variation characteristics, represents the average value of the temperature target variable, t represents the test statistic, R represents the first correlation coefficient, N represents the sample size, and p represents the sample size.
[0084] In one example, the multimodal abnormal feature recognition model includes a hypergraph convolutional neural network, and the hypergraph convolutional neural network introduces a multimodal attention mechanism to assign weights to different vertices in the hypergraph convolutional neural network.
[0085] In an example, the expression of the hypergraph convolutional neural network is: ; ; ; ; ; in, Represents the first l The output function in the hidden layer is represents the sigmoid function, Indicates l The weight coefficients of the hidden layers are Indicates l The bias of the hidden layer, v i Represents the first i Vertices, v j represents the jth vertex in the hypergraph, Represents a hypergraph vertex v i and v j The attention coefficient when it intersects with the hyperedge, T represents the transformation matrix, represents the normalized exponential function, represents the rectified linear function, represents the attention parameter vector, represents the attention weight matrix, Represents the first i The fusion representation of vertices, N( i ) represents the first i The modal set of connected vertices, Indicates l The fusion output function of the combined layers is Indicates l The weight coefficient of the combined layer, Indicates l The bias term of the combined layer, represents the classification function, represents the weight coefficient of the output layer, Represents the bias term of the output layer.
[0086] In one example, the deviation frequency modulation module 13 is used to input the anomaly detection feature vector into the multimodal anomaly feature recognition model to perform anomaly detection on the photovoltaic power generation equipment to be inspected; when it is detected that the working state of the photovoltaic power generation equipment to be inspected is abnormal, the knowledge graph reasoning and case reasoning methods are used to perform fault tracing analysis, and output the abnormal diagnosis results and fault analysis report; the abnormal diagnosis results and the fault analysis report are sent to the terminal device of the relevant personnel for archiving, and the relevant personnel are prompted to repair the photovoltaic power generation equipment to be inspected.
[0087] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. An analysis method based on multimodal conversion protocol for unmanned aerial vehicles, characterized in that: The method comprises: According to the geographic information of the photovoltaic power generation equipment to be inspected, the ground inspection system plans the inspection path of the drone and the corresponding photovoltaic power generation equipment to be inspected; Collecting initial multi-source heterogeneous data on the inspection path, and preprocessing the initial multi-source heterogeneous data to obtain integrated multi-source heterogeneous data, wherein the initial multi-source heterogeneous data includes thermal imaging data, equipment power data, and acoustic data; Extracting features corresponding to the thermal imaging data, equipment power data, and acoustic data in the integrated multi-source heterogeneous data, and screening out a key feature subset through a feature selection method; Normalizing the key feature subsets, integrating the normalized key feature subsets into anomaly detection feature vectors, and constructing a multimodal anomaly feature recognition model; Based on the anomaly detection feature vector and the multimodal anomaly feature recognition model, anomaly detection is performed on the multi-source heterogeneous data.
2. The method according to claim 1, characterized in that The method of enabling the ground inspection system to plan the inspection path of the drone and the corresponding photovoltaic power generation equipment to be inspected according to the location information of the photovoltaic power generation equipment to be inspected specifically includes: Determine a path starting point, a path end point, and a plurality of intermediate stop points according to the geographic information of the photovoltaic power generation equipment to be inspected and the ground inspection system, wherein the plurality of intermediate stop points are determined according to the location of the photovoltaic power generation equipment to be inspected; According to the geographic information of the photovoltaic power generation equipment to be inspected and the ground inspection system, the sub-inspection paths corresponding to the multiple intermediate stop points are determined, and the starting point and the end point of any sub-inspection path are the intermediate stop points corresponding to the sub-inspection path; The inspection path is generated according to the path starting point, the path end point, the multiple intermediate stop points, and the sub-inspection paths respectively corresponding to the multiple intermediate stop points.
3. The method according to claim 1, characterized in that The collecting of initial multi-source heterogeneous data on the inspection path and preprocessing of the initial multi-source heterogeneous data specifically includes: Collecting initial multi-source heterogeneous data on the inspection path according to the built-in sensor array of the drone; The initial multi-source heterogeneous data is subjected to noise removal and data normalization processing to obtain optimized multi-source heterogeneous data.
4. The method according to claim 3, characterized in that The acquisition and integration of multi-source heterogeneous data specifically includes: Based on a predefined multimodal conversion protocol, the optimized multi-source heterogeneous data is parsed, format converted, and semantically mapped to obtain integrated multi-source heterogeneous data, where the integrated multi-source heterogeneous data represents the optimized multi-source heterogeneous data after data unification.
5. The method according to claim 1, characterized in that The extracting of features corresponding to the thermal image data, the equipment power data and the acoustic data in the integrated multi-source heterogeneous data, and screening out a key feature subset by a feature selection method, specifically includes: Extracting temperature variation features from the thermal image data, power fluctuation features from the device power data, and device vibration features from the acoustic data; Respectively calculating the correlation coefficients of the temperature change characteristic, the power fluctuation characteristic, and the equipment vibration characteristic with the corresponding target variables to obtain a first correlation coefficient, a second correlation coefficient, and a third correlation coefficient, wherein the first correlation coefficient represents the correlation coefficient of the temperature change characteristic with the temperature target variable, the second correlation coefficient represents the correlation coefficient of the power fluctuation characteristic with the power target variable, and the third correlation coefficient represents the correlation coefficient of the equipment vibration characteristic with the vibration target variable; Based on the absolute values of the first correlation coefficient, the second correlation coefficient and the third correlation coefficient, the temperature change characteristics, the power fluctuation characteristics and the equipment vibration characteristics are sorted in descending order, and the top K features with the highest correlation with the target variable are selected as the key feature subset.
6. The method according to claim 5, characterized in that The expression of the first correlation coefficient is: ; ; ; Among them, r represents the initial correlation coefficient, x pq represents the qth temperature change feature in the pth sample, y p represents the temperature target variable value in the pth sample, represents the average value of the temperature variation characteristic, represents the average value of the temperature target variable, t represents the test statistic, R represents the first correlation coefficient, N represents the sample size, and p represents the sample size.
7. The method according to claim 1, characterized in that The multimodal abnormal feature recognition model includes a hypergraph convolutional neural network, and the hypergraph convolutional neural network introduces a multimodal attention mechanism to assign weights to different vertices in the hypergraph convolutional neural network.
8. The method according to claim 7, characterized in that The expression of the hypergraph convolutional neural network is: ; ; ; ; ; in, Represents the first l The output function in the hidden layer is represents the sigmoid function, Indicates l The weight coefficients of the hidden layers are Indicates l The bias of the hidden layer, v i Represents the first i Vertices, v j represents the jth vertex in the hypergraph, Represents a hypergraph vertex v i and v j The attention coefficient when it intersects with the hyperedge, T represents the transformation matrix, represents the normalized exponential function, represents the rectified linear function, represents the attention parameter vector, represents the attention weight matrix, Represents the first i The fusion representation of vertices, N( i ) represents the first i The modal set of connected vertices, Indicates l The fusion output function of the combined layers is Indicates l The weight coefficient of the combined layer, Indicates l The bias term of the combined layer, represents the classification function, represents the weight coefficient of the output layer, Represents the bias term of the output layer.
9. The method according to claim 1, characterized in that The performing anomaly detection on the multi-source heterogeneous data based on the anomaly detection feature vector and the multimodal anomaly feature recognition model specifically includes: Inputting the abnormality detection feature vector into the multimodal abnormality feature recognition model to perform abnormality detection on the photovoltaic power generation equipment to be inspected; When it is detected that the working state of the photovoltaic power generation equipment to be inspected is abnormal, the knowledge graph reasoning and case reasoning methods are used to perform fault tracing analysis, and the abnormal diagnosis results and fault analysis report are output; The abnormal diagnosis result and the fault analysis report are sent to the terminal device of the relevant personnel for archiving, and the relevant personnel are prompted to inspect and repair the photovoltaic power generation equipment to be inspected.
10. An analysis system based on a multimodal conversion protocol for drones, characterized in that: The analysis system (1) comprises a data acquisition module (11), a frequency modulation optimization module (12) and a deviation frequency modulation module (13) which are connected in sequence, wherein: The data acquisition module (11) is used to enable the ground inspection system to plan the inspection path of the drone and the corresponding photovoltaic power generation equipment to be inspected according to the geographic information of the photovoltaic power generation equipment to be inspected, collect initial multi-source heterogeneous data on the inspection path, and pre-process the initial multi-source heterogeneous data to obtain integrated multi-source heterogeneous data, wherein the initial multi-source heterogeneous data includes thermal image data, equipment power data and acoustic data; The frequency modulation optimization module (12) is used to extract features corresponding to the thermal imaging data, equipment power data and acoustic data in the integrated multi-source heterogeneous data, and screen out a key feature subset through a feature selection method; The deviation frequency modulation module (13) is used to normalize the key feature subset, integrate the normalized key feature subset into an anomaly detection feature vector, and construct a multimodal anomaly feature recognition model, and perform anomaly detection on the multi-source heterogeneous data based on the anomaly detection feature vector and the multimodal anomaly feature recognition model.
Citation Information
Patent Citations
Power grid intelligent operation and maintenance method and system based on machine patrol and AI fusion
CN117332347A
Intelligent box-type substation with railway power monitoring device as platform and monitoring method
CN118157321A
Cited By
Environment monitoring service method and system based on LSTM model
CN119719204A
Marine operation decision-making method and device based on big data and computer equipment
CN121169144A
Big Data-Based Marine Operations Decision-Making Methods, Devices, and Computer Equipment
CN121169144B
