Multi-source and multi-modal data fusion method for electric power operation site

Abstract

The invention discloses a multi-source and multi-modal data fusion method for an electric power operation site, which comprises the following steps: a, inherent heterogeneous data fusion: the inherent heterogeneous data have different formats, units, dimensions and other isomerities, and need to be fused to support the unified management of a power transmission system; b, multi-type inspection data fusion: visible light, infrared videos and image data acquired by manual inspection and unmanned aerial vehicle inspection are combined with other sensor data; c, fusing machine inspection data and manual inspection data; the fusion of the machine inspection data and the manual inspection data aims to optimize the inspection process through the complementation of automation and manual work, and ensure the mutual complementation of the data at the same time. According to the method, real-time processing of the unmanned aerial vehicle inspection data, the manual inspection data, the sensor data and the operation and maintenance data is realized through edge calculation, time delay of uploading to a remote server or a cloud is remarkably reduced, the real-time performance is higher, and the requirements of field operation on real-time decision making and quick response are met.

Description

Technical fields:

[0001] This invention relates to a power operation site management and control technology, and in particular to a method for fusing multi-source, multi-modal data from power operation sites. Background technology:

[0002] The earliest power industry inspections can be traced back to the early 20th century, relying on manual inspection and maintenance. At that time, the inspection of power lines and equipment mainly depended on on-site personnel periodically conducting manual inspections of transmission lines, using simple tools such as telescopes to detect faults or potential hazards. This method was not only inefficient and labor-intensive, but also extremely inconvenient for inspecting power facilities in some remote areas. Furthermore, due to the complex working environment, manual inspections carried a high risk of safety hazard and false alarms.

[0003] With the expansion of power systems and technological advancements, automated inspection technologies have been gradually introduced. In the 1960s, rudimentary automated monitoring equipment, such as temperature sensors and voltage and current monitoring devices, began to be used in substations and transmission lines to help maintenance personnel monitor equipment status in real time. These automation technologies improved the accuracy and timeliness of inspections, but still relied on fixed monitoring equipment, had limited coverage, and were insufficient to meet the monitoring needs of large-scale transmission lines.

[0004] With the development of unmanned aerial vehicle (UAV) technology, especially after the turn of the 21st century, UAVs have gradually emerged in the field of power line inspection. The introduction of UAVs has drastically changed the traditional methods of power line inspection. UAVs can remotely inspect high-voltage transmission lines and power facilities in remote mountainous areas, greatly reducing manpower and improving inspection efficiency and safety. In the mid-2000s, domestic and international power companies gradually began to apply UAV technology to power line inspection. UAVs can be equipped with high-definition cameras, thermal imagers, and other equipment to provide more detailed and comprehensive line monitoring data. This technology significantly reduces labor costs and safety risks, resulting in a substantial increase in inspection efficiency.

[0005] Currently, in routine drone inspections in China, the small screens of drone controllers make it impossible to identify defects on-site or import relevant videos and defect maps into various databases. The typical work pattern is as follows: power transmission maintenance personnel spend half a day conducting aerial inspections and the other half a day manually interpreting images or analyzing videos and manually entering data into relevant databases. Although there are platforms for machine image interpretation, the recognition rate of pin-level defects is still only around 70%.

[0006] Furthermore, current power operation sites cannot integrate existing physical data, manual inspection data, sensor data, and machine inspection data, resulting in various cumbersome operations or long hours of manual work. This requires intelligent and automated methods to replace these methods, automatically organizing and classifying key data on-site, reducing the manual data organization and analysis work of grassroots teams, and significantly improving work efficiency. Summary of the Invention:

[0007] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a method for fusing multi-source and multi-modal data at power operation sites that is reasonably designed, has good fusion effect and fast response speed.

[0008] The technical solution of this invention is:

[0009] A method for fusing multi-source, multi-modal data from power operation sites includes the following steps:

[0010] a. Integration of inherent heterogeneous data: The inherent heterogeneous data includes physical information of transmission line equipment, key parameters of towers and conductors, laser point cloud data, auxiliary facility information, etc. These data sources have different formats, units, dimensions and other heterogeneities, and need to be integrated to support the unified management of the transmission system.

[0011] b. Multi-type inspection data fusion: Combine visible light and infrared video and image data collected by manual inspection and drone inspection, as well as other sensor data, to improve the ability to perceive the status of power transmission lines.

[0012] c. Integration of machine inspection and manual inspection data; Machine inspection includes drones, robots, sensor networks, etc. The integration of machine inspection and manual inspection data aims to optimize the inspection process and reduce the workload of manual labor through the complementarity of automation and manual labor, while ensuring that the data complement each other.

[0013] d. Employ edge computing to deploy computing power close to the data source, directly process and analyze data on-site for emergency response and dynamic decision-making; adopt an intelligent RPA management system to automatically process videos, images and infrared spectra captured by drones, automatically identify equipment problems, generate inspection reports, and organize relevant data into standardized output formats.

[0014] Further: In step a, the inherent heterogeneous data fusion step includes:

[0015] a1. Data Standardization and Cleaning: Standardize data from different sources, including format conversion, unit consistency, and handling of missing values, to ensure that different data are comparable under the same standard;

[0016] a2. Spatial alignment and coordinate transformation: Point cloud data is often used to reflect the three-dimensional structure of power transmission lines. It needs to be spatially aligned and coordinate transformed with equipment information and geographic information system (GIS) data to ensure that various types of equipment can be analyzed in the same spatial coordinate system.

[0017] a3. Feature Extraction and Multimodal Analysis: For laser point cloud data, geometric structure features are extracted and combined with equipment parameters to perform multimodal analysis in order to evaluate the overall condition of the line.

[0018] Further: In step a, the inherent heterogeneous data fusion is based on the DS evidence theory, introduces a distance function to redistribute focal element weights to obtain a new BPA, uses average evidence to replace conflicting evidence, and then constructs a confidence matrix to complete the multi-sensor information fusion perception;

[0019] First, calculate the average value of each piece of evidence for the k-th focal element, as shown in the following formula:

[0020]

[0021] The distance from each piece of evidence to the average value of that focal element is calculated as follows:

[0022]

[0023] Where d i This represents the basic probability function value of the i-th piece of evidence for the j-th focal element. It can be seen that the higher the similarity between the two pieces of evidence, the higher the probability function value. i The smaller the value;

[0024] The weights of each piece of evidence and the average value of new evidence for the focal element are shown in (3-1) and (3-4), respectively:

[0025]

[0026] After obtaining the weights, the BPA values ​​of the conflicting evidence are recalculated. The main process is as follows: Construct the confidence matrix M, as shown in the following formula:

[0027]

[0028] Where M is the basic probability assignment value of evidence i to focus element j, and the summation of each row of the matrix in the above formula satisfies the following equation:

[0029] m i1 +m i1 +...+m im =1, (i = 1, 2, ..., n) (3-6)

[0030] Transpose the i-th row of matrix M, then multiply it by the j-th row, i.e.

[0031]

[0032] This yields a new matrix A (m*m dimensional).

[0033]

[0034] As can be seen from the formula, the product of the elements on the diagonal represents the fusion result of evidence i and evidence j, while the sum of all other elements in the matrix is ​​the uncertainty coefficient of the fusion result, as shown in the following formula:

[0035] K ' =Σ P≠q m ip ×m jq (p,q=1,2,...,m) (3-9)

[0036] Finally, multiple state information are fused and perceived.

[0037]

[0038] Further: In step b, the step of fusing multi-type inspection data includes:

[0039] b1. Multimodal feature extraction and matching: For multimodal data generated by UAVs and sensors, feature information such as line damage and hot spots is extracted through computer vision technology or image processing algorithms; for manual inspection data, key information is extracted through natural language processing technology, and compared and verified with machine inspection data to ensure data consistency and complementarity.

[0040] b2. Time and space alignment: The data collection time and spatial location of different inspection methods may be inconsistent; technically, by using time synchronization, GPS positioning and data time and space alignment technology, the data information of different inspection sources can be unified, so that different data can be analyzed collaboratively to form a comprehensive perception of the line status.

[0041] b3. Data Fusion and Anomaly Detection: By using deep learning and machine learning algorithms, various types of inspection data are fused to automatically identify anomalies and perform fault detection and diagnosis.

[0042] Furthermore: A fusion algorithm is used to jointly calibrate the three sensor coordinate systems and time. The main steps are as follows: Establish a camera coordinate system (Oc, Xc, Yc, Zc), where the coordinates of a point in the world coordinate system are [xw, yw, zw]. Transform to the camera coordinate system, where its position is represented as [xc, yc, zc]. The transformation formula between the two coordinate systems is:

[0043]

[0044] The homogeneous coordinate form is shown in the following equation.

[0045]

[0046] M1 is the overall transformation matrix, which is the rotation and translation matrix between two coordinate systems, including the orthogonal rotation matrix R and the translation matrix T. The object in the world coordinate system is refracted by the lens to form the image coordinate system (OI,XI,YI,ZI), which is parallel to the center plane of the lens.

[0047] The coordinates of the sensor's spatial point P[l,a] in the world coordinate system are given by the following formula:

[0048]

[0049] The formula for the correspondence between a spatial point P in the camera's pixel coordinate system [x, y] and the world coordinate system is:

[0050]

[0051] The above transformation relationship does not take into account lens distortion, so it is necessary to correct the distortion of the camera. Combining the distortion coefficient k of the image sensor itself, the corrected image coordinates P'[x',y'] are obtained:

[0052]

[0053] Where k is the radial distortion coefficient and p is the tangential distortion coefficient.

[0054] Further: In step c, the fusion of machine inspection and manual inspection data includes the following steps:

[0055] c1. Consistency Verification: Machine inspection provides continuous and objective data, while manual inspection relies on human experience and judgment. By using spatiotemporal consistency and data matching algorithms, the detection results of the two are compared at the same time and location to ensure the accuracy of machine detection data.

[0056] c2. Complementarity analysis: Machine inspection is good at covering a wide range of routine tasks, while manual inspection is suitable for handling complex and difficult-to-standardize tasks. Through the fusion mechanism, the anomalies detected by machine inspection are fed back to manual inspection for on-site verification. At the same time, the experience of manual inspection is fed back to the machine inspection model to optimize its fault detection algorithm.

[0057] c3. Feedback closed-loop mechanism: After the machine inspection finds potential problems, it automatically generates a report and notifies the human inspectors to confirm on-site, forming a closed-loop process of "machine detection - human confirmation - result feedback" to improve the accuracy of data and inspection efficiency.

[0058] Furthermore: In step d, when the RPA system can automatically process data, the weight of each attribute will greatly affect the evaluation result. The maximum deviation method is used to determine the attribute weights in the multi-attribute evaluation problem, where the evaluation information is represented by numerical values.

[0059] A model is constructed using the maximum deviation method to solve the problem of determining attribute weights in hesitant and fuzzy environments. The model is as follows:

[0060]

[0061] By solving the above model, we can obtain:

[0062]

[0063] For ease of calculation, let:

[0064]

[0065] By standardizing wj so that their sum is 1, we can obtain:

[0066]

[0067] In a hesitant and fuzzy environment, A+ represents the positive ideal solution, and A- represents the hesitant and fuzzy negative ideal solution, that is:

[0068]

[0069] Using Euclidean distance, each evaluation scheme is compared with A. + and A - The distances are represented as di. + and di - :

[0070]

[0071] The relative similarity between schemes A+ and A- is:

[0072]

[0073] 0≤c(Ai)≤1, i=1,2,…,n. As can be seen from the above formula, when c(Ai) is closer to 1, the schemes Ai+ and A+ are closer, and the schemes are farther away from A-. The selection of the scheme can be determined based on the relative proximity c(Ai). Using this method to combine edge computing with RPA system, it is possible to automatically analyze the video and images collected during the drone inspection process, classify and label the detected equipment problems, and generate a visual report for on-site personnel to refer to.

[0074] The beneficial effects of this invention are:

[0075] 1. This invention, through its portable hardware design and powerful software functions, becomes an ideal tool for power operation sites. It can efficiently interface with drones and various sensors to achieve the fusion of multi-source and multi-modal data. It has a good ergonomic design, which makes it easy for operators to respond quickly in complex environments and comprehensively improve the efficiency of power transmission operations and the level of intelligent inspection.

[0076] 2. This invention enables real-time processing of UAV inspection data, manual inspection data, sensor data, and operation and maintenance data through edge computing, significantly reducing the time delay of uploading to remote servers or the cloud, making it more real-time and meeting the needs of on-site operations for real-time decision-making and rapid response.

[0077] 3. The manual inspection data of this invention extracts key information through natural language processing (NLP) technology, and compares and verifies it with machine inspection data to ensure data consistency and complementarity.

[0078] 4. This invention unifies data information from different inspection sources through time synchronization, GPS positioning, and spatiotemporal alignment technologies, enabling collaborative analysis of different data to form a comprehensive perception of the line status.

[0079] 5. This invention is not only applicable to the inspection and maintenance of power transmission lines, but can also be extended to other areas of the power system, forming a wider range of industry-wide promotion channels. It can even be applied across industries, such as infrastructure industries like oil, natural gas, and transportation, industrial manufacturing and smart factories, and municipal engineering and public facility maintenance.

[0080] 6. This invention uses the ternary connection coefficient from set pair analysis theory when calculating the connection degree matrix, which can more accurately quantify the degree of support of the measurement values ​​of each sensor at the same time; at the same time, it comprehensively considers the consistency of the measurement values ​​and the reliability of the sensors when calculating the weighting coefficients.

[0081] 7. The machine inspection of this invention provides continuous and objective data, while manual inspection relies on human experience and judgment. By using spatiotemporal consistency and data matching algorithms, the detection results of the two are compared at the same time and location to ensure the accuracy of machine detection data. For example, manual confirmation of abnormal equipment temperature or malfunction.

[0082] 8. After the machine inspection finds potential problems, the present invention automatically generates a report and notifies the human inspectors to confirm on-site, forming a closed-loop process of "machine detection - human confirmation - result feedback" to improve the accuracy of data and inspection efficiency.

[0083] 9. The edge device of this invention can share the computing tasks of the cloud and improve computing efficiency by utilizing a distributed architecture, thereby improving the overall response speed of the system. It is especially suitable for scenarios with high real-time requirements, such as inspection and emergency response in power operations. It is easy to promote and implement and has good economic benefits. Attached image description:

[0084] Figure 1 This is a system architecture diagram of a method for fusing multi-source, multi-modal data from power operation sites.

[0085] Figure 2 This is a diagram illustrating the overall architecture of a multi-source, multi-modal data fusion method for power operation sites.

[0086] Figure 3 This is a schematic diagram illustrating the principle of multi-source, multi-modal data fusion.

[0087] Figure 4 A diagram illustrating the data cleaning steps;

[0088] Figure 5 A diagram showing spatial alignment and coordinate transformation;

[0089] Figure 6 For feature extraction and multimodal analysis;

[0090] Figure 7 This is a multimodal fusion analysis diagram;

[0091] Figure 8 This is a schematic diagram of the sensor data fusion principle.

[0092] Figure 9 A schematic diagram illustrating the transformation between the image sensor coordinate system and the world coordinate system shared by multiple sensors;

[0093] Figure 10 The flowchart of the correlation analysis algorithm for verifying the correlation degree of set pair analysis. Detailed implementation method:

[0094] Example: See Figure 1 -- Figure 10 In the picture:

[0095] A method for fusing multi-source, multi-modal data from power operation sites includes the following steps:

[0096] a. Integration of inherent heterogeneous data: The inherent heterogeneous data includes physical information of transmission line equipment, key parameters of towers and conductors, laser point cloud data, auxiliary facility information, etc. These data sources have different formats, units, dimensions and other heterogeneities, and need to be integrated to support the unified management of the transmission system.

[0097] b. Multi-type inspection data fusion: Combine visible light and infrared video and image data collected by manual inspection and drone inspection, as well as other sensor data, to improve the ability to perceive the status of power transmission lines.

[0098] c. Integration of machine inspection and manual inspection data; Machine inspection includes drones, robots, sensor networks, etc. The integration of machine inspection and manual inspection data aims to optimize the inspection process and reduce the workload of manual labor through the complementarity of automation and manual labor, while ensuring that the data complement each other.

[0099] d. Employ edge computing to deploy computing power close to the data source, directly process and analyze data on-site for emergency response and dynamic decision-making; adopt an intelligent RPA management system to automatically process videos, images and infrared spectra captured by drones, automatically identify equipment problems, generate inspection reports, and organize relevant data into standardized output formats.

[0100] Preferred solution: In step a, the inherent heterogeneous data fusion step includes:

[0101] a1. Data Standardization and Cleaning: Standardize data from different sources, including format conversion, unit consistency, and handling of missing values, to ensure that different data are comparable under the same standard;

[0102] a2. Spatial alignment and coordinate transformation: Point cloud data is often used to reflect the three-dimensional structure of power transmission lines. It needs to be spatially aligned and coordinate transformed with equipment information and geographic information system (GIS) data to ensure that various types of equipment can be analyzed in the same spatial coordinate system.

[0103] a3. Feature Extraction and Multimodal Analysis: For laser point cloud data, geometric structure features are extracted and combined with equipment parameters to perform multimodal analysis in order to evaluate the overall condition of the line.

[0104] Preferred solution: In step a, the inherent heterogeneous data fusion is based on the DS evidence theory. A distance function is introduced to redistribute the focal element weights to obtain a new BPA. Average evidence is used to replace conflicting evidence. Then, a confidence matrix is ​​constructed to complete the multi-sensor information fusion perception.

[0105] First, calculate the average value of each piece of evidence for the k-th focal element, as shown in the following formula:

[0106]

[0107] The distance from each piece of evidence to the average value of that focal element is calculated as follows:

[0108]

[0109] Where di This represents the basic probability function value of the i-th piece of evidence for the j-th focal element. It can be seen that the higher the similarity between the two pieces of evidence, the higher the probability function value. i The smaller the value;

[0110] The weights of each piece of evidence and the average value of new evidence for the focal element are shown in (3-1) and (3-4), respectively:

[0111]

[0112] After obtaining the weights, the BPA values ​​of the conflicting evidence are recalculated. The main process is as follows: Construct the confidence matrix M, as shown in the following formula:

[0113]

[0114] Where M is the basic probability assignment value of evidence i to focus element j, and the summation of each row of the matrix in the above formula satisfies the following equation:

[0115] m i1 +m i1 +...+m im =1, (i = 1, 2, ..., n) (3-6)

[0116] Transpose the i-th row of matrix M, then multiply it by the j-th row, i.e.

[0117]

[0118] This yields a new matrix A (m*m dimensional).

[0119]

[0120] As can be seen from the formula, the product of the elements on the diagonal represents the fusion result of evidence i and evidence j, while the sum of all other elements in the matrix is ​​the uncertainty coefficient of the fusion result, as shown in the following formula:

[0121] K'=Σ P≠q m ip ×m jq (p,q=1,2,...,m) (3-9)

[0122] Finally, multiple state information are fused and perceived.

[0123]

[0124] Preferred solution: In step b, the steps for fusing multi-type inspection data include:

[0125] b1. Multimodal feature extraction and matching: For multimodal data generated by UAVs and sensors, feature information such as line damage and hot spots is extracted through computer vision technology or image processing algorithms; for manual inspection data, key information is extracted through natural language processing technology, and compared and verified with machine inspection data to ensure data consistency and complementarity.

[0126] b2. Time and Space Alignment: Data collection times and spatial locations may differ between different inspection methods. Technically, time synchronization, GPS positioning, and data time and space alignment technologies are used to unify data information from different inspection sources, enabling collaborative analysis of different data and forming a comprehensive understanding of the line status.

[0127] b3. Data fusion and anomaly detection: By using deep learning and machine learning algorithms, various types of inspection data are fused to automatically identify anomalies and perform fault detection and diagnosis.

[0128] Preferred solution: Use a fusion algorithm to jointly calibrate the three sensor coordinate systems and time. The main steps are as follows: Establish a camera coordinate system (Oc, Xc, Yc, Zc). The coordinates of a point in the world coordinate system space are [xw, yw, zw]. Transform to the camera coordinate system, and its position is represented as [xc, yc, zc]. The transformation relationship between the two coordinate systems is:

[0129]

[0130] The homogeneous coordinate form is shown in the following equation.

[0131]

[0132] M1 is the overall transformation matrix, that is, the rotation and translation matrix between two coordinate systems, including the orthogonal rotation matrix R and the translation matrix T. For example... Figure 9 As shown, objects in the world coordinate system are refracted through the lens to form an image coordinate system (OI, XI, YI, ZI), which is parallel to the center plane of the lens.

[0133] The coordinates of the sensor's spatial point P[l,a] in the world coordinate system are given by the following formula:

[0134]

[0135] The formula for the correspondence between a spatial point P in the camera's pixel coordinate system [x, y] and the world coordinate system is:

[0136]

[0137] The above transformation relationship does not take into account lens distortion, so it is necessary to correct the distortion of the camera. Combining the distortion coefficient k of the image sensor itself, the corrected image coordinates P'[x',y'] are obtained:

[0138]

[0139] Where k is the radial distortion coefficient and p is the tangential distortion coefficient.

[0140] Beyond spatial transformation, this invention also includes multimodal data (such as visible light, infrared images, and temperature sensor data). Computer vision technology or image processing algorithms are used to extract feature information such as line damage and overheating. Manual inspection data is processed using natural language processing (NLP) to extract key information, which is then compared and verified with machine inspection data to ensure data consistency and complementarity. The main approach involves defining sensor data processing threads and data synchronization threads. Time registration is performed based on the timestamps of the sensor data in the data processing thread, and the data synchronization thread sends the registered data to a buffer for subsequent fusion and invocation.

[0141] To conduct cross-validation analysis, this invention takes the inverse-type opposition as its starting point to study the IDO relationship between two sets in the set pairs formed by multi-sensor data processing. Since multi-sensor information processing often requires real-time, accurate, and reliable processing of target data detected by each sensor at each moment, the connection status and degree of connection between the number pairs formed by the homogeneous sensors at each moment, because a number pair is a specific set pair with an IDO relationship, needs to be quantitatively characterized by IDO.

[0142] Assuming the target data detected by the sensor is within the range of [0, ∞], the following derivations are made based on the proposed formulas for opposition, identity, and difference:

[0143] s in a single complementary interval of t The proportion of internal is:

[0144]

[0145] s in The formula for the degree of incompatibility with t is expressed as:

[0146]

[0147] The formula for the degree of identity between s and t is expressed as:

[0148]

[0149] Therefore, the formula for the degree of difference of the pairs [s,t] is expressed as:

[0150]

[0151] Analysis yields the following expression for the IDO association degree (i.e., ternary association number) u of the pair [s,t]:

[0152]

[0153] In the above formula, s[0,∞], t∈[0,∞] and 1≤s≤t are satisfied. i is the dissimilarity label, which needs to take a value within [j,1]. j is the label for the opposition type relation, which can take a value according to different opposition types. When the IDO relation is a positive-negative opposition relation, j = -1 is taken, in which case i∈[-1,1]; when the IDO relation is a reciprocal opposition relation... R is the smallest number among the n pairs of relations discussed, and

[0154] The relationship matrix and dimension expansion process are discussed. Consider a multi-sensor system consisting of n independent sensors located at different positions, all measuring the same parameter. With unknown observation noise, the measurement equation for parameter x is:

[0155] z i (k)=x+v i (k), (i = 1, 2, ..., n) (3-21)

[0156] Here, zi(k) is the observation value of the i-th sensor at time k, vi(k) is the observation noise at time k, and the prior knowledge of E[vi] and D[vi] is unknown. zi(k) and zj(k) represent the measurements of sensors i and j at time k, respectively. If the values ​​of the two sensors differ significantly, it indicates low mutual support between the two sensors, or even mutual incompatibility; if the values ​​of the two sensors are close, it indicates high support between sensors i and j, and the higher the authenticity of the observation value.

[0157] To quantify the degree of support between measurements from different sensors at the same time, the concept of correlation degree in set pair analysis is introduced, and a correlation degree matrix is ​​constructed. First, the correlation degree between observations from different sensors at time k is calculated using the ternary correlation coefficient method, obtaining zi(k) and zj(k), which are taken as observations from any two sensors at time k, and (zi(k)≤zj(k)) are treated as pairs [zi(k),zj(k)]. Using the inverse characteristic function of similarity and difference in set pair analysis theory, i.e., the ternary correlation coefficient, the correlation degree between sensor observations is quantified. This avoids the absoluteness of either 1 or 0 in traditional methods, and thus the correlation degree matrix between sensors at time k can be calculated.

[0158] This invention employs membership functions and maximum / minimum proximity from fuzzy set theory to measure the degree of correlation among data. It also performs a correlation coefficient processing on the fuzzy membership degree. Applying the correlation coefficient makes the uncertain information represented by the fuzzy membership degree more complete, systematic, and objective, and the results will provide new insights for the research and application of fuzzy set theory.

[0159] The correlation matrix between any two sensor data points at time 1k is defined as follows:

[0160]

[0161] In the matrix above, the elements in the i-th row are accumulated as follows: If the cumulative value of this row is large, it indicates that the measurement value of sensor i at this moment is close to that of most sensors and has good consistency; conversely, if the observation value of sensor i differs greatly from that of most sensors, the reliability of its observation value is not high.

[0162] The above matrix reflects the correlation between different sensors at a certain moment. To improve the correlation of fused data from different moments, the fusion estimation results from previous moments are taken into account when calculating the observation correlation at the next moment. This results in an expanded-dimensional correlation matrix, where the fusion estimation result from the previous moment is added as a hypothetical measurement value to the calculation at the next moment. The expanded-dimensional correlation matrix U n+1 (k):

[0163]

[0164] The expanded-dimensional connectivity matrix not only utilizes the observation data from previous observations, but also reflects the degree of interrelationship between the observed values ​​throughout the entire observation interval.

[0165] Definition 1 reflects the degree of correlation between the observations of sensor i and j at a certain observation time, but it does not consider the correlation between sensor i and the measurements of all sensors (including itself) at that time. This necessitates defining a consistency measure to address this issue.

[0166] Define the consistency measure between the measurement of sensor i and the measurements of all sensors at time 2k as follows:

[0167]

[0168] Consistent reliability measures, which measure the consistency across all observation times, can effectively reflect the reliability of a sensor. Starting from the reliability level across the entire observation interval, we employ the sample mean and variance from statistical theory. First, we define the consistency mean and consistency variance.

[0169] Define the mean of observation consistency of sensor i at time 3k as follows:

[0170]

[0171] The variance of the observation consistency of the i-th sensor at time 4k is defined as follows:

[0172]

[0173] To reduce computational load, a recursive formula is applied to the fusion algorithm when calculating the consistency reliability measure.

[0174]

[0175]

[0176] Sensor reliability is determined by its consistency mean and variance. Higher reliability results in a greater weight being assigned to the sensor's measurement data. Reliability is measured by the signal-to-noise ratio (signal-to-noise ratio, the ratio of mean to variance).

[0177] wi(k) is the weighting coefficient of the measurement value of sensor i, which is positively correlated with the mean of consistency and negatively correlated with the variance of consistency, that is:

[0178]

[0179] After normalization, the weighting coefficients of each sensor measurement at time k are obtained:

[0180]

[0181] In summary, the data fusion formula based on connectivity is:

[0182]

[0183] The specific steps of the corroborative association analysis algorithm based on set pair analysis of association degree are as follows:

[0184] Step 1: After obtaining the sensor's measurement values, use the formula to obtain the connection degree matrix U, and then expand the dimensions to obtain U';

[0185] Step 2: Calculate the consistency measure r, and calculate the consistency measure at different times for different cases;

[0186] Step 3: Calculate the consistency mean and consistency variance, obtain the weighting coefficient W, and normalize the weighting coefficient.

[0187] Step 4: Calculate the fusion result.

[0188] As can be seen from the above steps, the algorithm proposed in this paper uses the ternary connection number in set pair analysis theory when calculating the connection degree matrix, which can more accurately quantify the degree of support of the measurement values ​​of each sensor at the same time; at the same time, it comprehensively considers the consistency of the measurement values ​​and the reliability of the sensors when calculating the weighting coefficients.

[0189] Preferred solution: In step c, the fusion of machine inspection and manual inspection data includes the following steps:

[0190] c1. Consistency Verification: Machine inspection provides continuous and objective data, while manual inspection relies on human experience and judgment. By using spatiotemporal consistency and data matching algorithms, the detection results of the two are compared at the same time and location to ensure the accuracy of machine detection data.

[0191] c2. Complementarity analysis: Machine inspection is good at covering a wide range of routine tasks, while manual inspection is suitable for handling complex and difficult-to-standardize tasks. Through the fusion mechanism, the anomalies detected by machine inspection are fed back to manual inspection for on-site verification. At the same time, the experience of manual inspection is fed back to the machine inspection model to optimize its fault detection algorithm.

[0192] c3. Feedback closed-loop mechanism: After the machine inspection finds potential problems, it automatically generates a report and notifies the human inspectors to confirm on-site, forming a closed-loop process of "machine detection - human confirmation - result feedback" to improve the accuracy of data and inspection efficiency.

[0193] Preferred solution: In step d,

[0194] A comprehensive optimization model for point selection schemes, combining AHP (Analytic Hierarchy Process) and TOPSIS algorithms, is employed. First, a scaling method is used to construct the judgment matrix. Let aξj be the ratio of index ξ to index j, and let n be the matrix order. Through pairwise comparisons between factors at each index level, the judgment matrix A is obtained, and its calculation formula is:

[0195] A = (a ξj ) n×n (3-32)

[0196] Let λmax be the largest eigenvalue of A. The consistency index Ic of A is obtained from the average consistency indices, and its calculation formula is:

[0197]

[0198] If λmax = n, then Ic = 0, and A is consistent. The larger Ic is, the greater the inconsistency of A. Further calculation of the consistency ratio RC is given by the following formula:

[0199] R C =I c / I R (3-34)

[0200] If RC < 0.1, then A is considered to have passed the consistency test. The steps for calculating the weights using the geometric mean method are as follows: ① Multiply the elements of A by row to obtain a new column vector; ② Take the nth root of each component of the new column vector, and let k be the k-th row of the proportional vector; ③ Dimensionally unify the column vector to obtain the weight φξ of the ξ-th index; ④ Calculate the relative weights of the criterion layer and the index layer respectively using this method, and finally obtain the total weight.

[0201] A normalized matrix Z is formed by p evaluation schemes and q evaluation indicators (already normalized), where Zij is the j-th indicator value of the i-th evaluation scheme. The normalized matrix Z is denoted as Y, and the formula for calculating each element yij in Y is:

[0202]

[0203] Finally, let Γi be the evaluation vector composed of the closeness values ​​between the i-th evaluation scheme and the positive optimal solution, and its calculation formula is:

[0204]

[0205] The larger Γi is, the closer the corresponding evaluation scheme is to the optimal solution. By combining the closeness values ​​of the evaluation schemes with the weights obtained from the analytic hierarchy process (AHP) and ranking them, a comprehensive evaluation of multiple schemes can be achieved. Let the final comprehensive evaluation vector of the model be S, and its calculation formula is:

[0206]

[0207] The above algorithms can comprehensively consider various factors to form an optimal multi-source, multi-modal data fusion mechanism system, covering inherent physical data, manual inspection data, sensor data, and machine inspection data. It can be queried and displayed in a unified manner, ensuring the accurate relevance of different data sources.

[0208] When an RPA system can automate data processing, the weight of each attribute will greatly affect the evaluation result. The maximum deviation method is used to determine the attribute weights in multi-attribute evaluation problems, where the evaluation information is represented by numerical values.

[0209] A model is constructed using the maximum deviation method to solve the problem of determining attribute weights in hesitant and fuzzy environments. The model is as follows:

[0210]

[0211] By solving the above model, we can obtain:

[0212]

[0213] For ease of calculation, let:

[0214]

[0215] By standardizing wj so that their sum is 1, we can obtain:

[0216]

[0217] In a hesitant and fuzzy environment, A+ represents the positive ideal solution, and A- represents the hesitant and fuzzy negative ideal solution, that is:

[0218]

[0219] The Euclidean distance is used. Each evaluation scheme is compared with A. + and A - The distances are represented as di. + and di - :

[0220]

[0221] The relative similarity between schemes A+ and A- is:

[0222]

[0223] 0≤c(Ai)≤1, i=1,2,…,n. As can be seen from the above formula, when c(Ai) is closer to 1, the schemes Ai+ and A+ are closer, and the schemes are farther away from A-. The selection of the scheme can be determined based on the relative proximity c(Ai). Using this method to combine edge computing with RPA system, it is possible to automatically analyze the video and images collected during the drone inspection process, classify and label the detected equipment problems, and generate a visual report for on-site personnel to refer to.

[0224] This invention addresses the pain points of basic operations in power systems:

[0225] Pain point 1 at the grassroots level: Defect images from drone inspections still need to be returned to the base station for manual image analysis and manual entry into the relevant database, resulting in poor timeliness and a very large workload.

[0226] Pain Point 2 at the Grassroots Level: With the comprehensive application of drone inspection images and videos, the storage space for massive amounts of data is enormous. For each tower undergoing lean drone inspection, there are approximately 24-80 images; however, the video volume for corridor inspections is even larger. Taking the Zhengzhou company as an example, the total amount of these inspection images and videos each year is no less than 30TB. Currently, manually managing this massive amount of data is inefficient, labor-intensive, and has low query utilization, resulting in significant data waste.

[0227] Pain point 3 at the grassroots level: The grassroots level lacks a query and association mechanism that is directly related to drone inspection data and integrates various inherent data and operation and maintenance data of power transmission lines.

[0228] In power operation sites, wide-area sensors are used to monitor the operating status of transmission lines. However, the management and analysis of existing sensor data mainly rely on centralized data centers, resulting in insufficient real-time performance and accuracy in field operations. For the fusion and practical application of multi-source, multi-modal data in complex environments, a mature integrated solution has not yet been developed, leading to slow data flow and impacting on-site emergency response efficiency.

[0229] Pain point 4 at the grassroots level: During on-site operation and maintenance, the operation of various operation and maintenance data is cumbersome and needs to be replaced by intelligent and automated means.

[0230] The equipment used in a method for multi-source multimodal data fusion at power operation sites includes a carrying case. The carrying case consists of a case body and a lid. The case body and lid are rotatably connected on one side and connected by a lock on the other side. The case body is equipped with a handle. The case body contains a computer host, a keyboard, and a display screen. The computer host includes a data acquisition module, a communication module, a position module, a power supply module, a storage module, and a computing control module. It is also equipped with USB 3.0, HDMI, RJ45, Ethernet interfaces, and various sensor interfaces.

[0231] The data acquisition module is used to collect images and videos of power facilities, including data acquisition from inherent heterogeneous data, manual inspection data, and drone data.

[0232] The communication module is used to transmit collected data and receive standard operating procedures and specifications issued by the cloud or server.

[0233] The computational control module is used to integrate, analyze, and process multi-source, multi-modal data from different sources to obtain more accurate and comprehensive information. Then, it automatically identifies anomalies through deep learning and learning algorithms to perform fault detection and diagnosis.

[0234] The computing control module uses edge computing to deploy computing power close to the data source, directly processing and analyzing data on-site. This reduces the time latency of uploading to remote servers or the cloud, resulting in stronger real-time performance, making it suitable for emergency response and dynamic decision-making.

[0235] Preferred solution: The inherent heterogeneous data includes physical information of transmission line equipment, key parameters of towers and conductors, laser point cloud data, auxiliary facilities and other information.

[0236] Preferred option: The computer host is equipped with a high-performance Intel i7 or Ryzen processor, no less than 16GB of memory and 512GB of SSD storage to ensure that it can process and save large amounts of data in real time and support efficient data analysis and multitasking.

[0237] The display screen is a high-definition touch screen of no less than 15 inches with a resolution of no less than 1920x1080. It has good viewing angles and brightness, and the color and saturation ensure that data and images can be clearly displayed under different lighting conditions.

[0238] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications made based on the technical essence of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A method for fusing multi-source, multi-modal data from power operation sites, comprising the following steps: a. Integration of inherent heterogeneous data: The inherent heterogeneous data includes physical information of transmission line equipment, key parameters of towers and conductors, laser point cloud data, auxiliary facility information, etc. These data sources have different formats, units, dimensions and other heterogeneities, and need to be integrated to support the unified management of the transmission system. b. Multi-type inspection data fusion: Combine visible light and infrared video and image data collected by manual inspection and drone inspection, as well as other sensor data, to improve the ability to perceive the status of power transmission lines. c. Integration of machine inspection and manual inspection data; Machine inspection includes drones, robots, sensor networks, etc. The integration of machine inspection and manual inspection data aims to optimize the inspection process and reduce the workload of manual labor through the complementarity of automation and manual labor, while ensuring that the data complement each other. d. Employ edge computing to deploy computing power close to the data source, directly process and analyze data on-site for emergency response and dynamic decision-making; adopt an intelligent RPA management system to automatically process videos, images and infrared spectra captured by drones, automatically identify equipment problems, generate inspection reports, and organize relevant data into standardized output formats.

2. The method for fusing multi-source, multi-modal data at a power operation site according to claim 1, characterized in that: in In step a, the inherent heterogeneous data fusion step includes: a1. Data Standardization and Cleaning: Standardize data from different sources, including format conversion, unit consistency, and handling of missing values, to ensure that different data are comparable under the same standard; a2. Spatial alignment and coordinate transformation: Point cloud data is often used to reflect the three-dimensional structure of power transmission lines. It needs to be spatially aligned and coordinate transformed with equipment information and geographic information system (GIS) data to ensure that various types of equipment can be analyzed in the same spatial coordinate system. a3. Feature Extraction and Multimodal Analysis: For laser point cloud data, geometric structure features are extracted and combined with equipment parameters to perform multimodal analysis in order to evaluate the overall condition of the line.

3. The method for fusing multi-source, multi-modal data at a power operation site according to claim 2, characterized in that: In step a, the inherent heterogeneous data fusion is based on the DS evidence theory. A distance function is introduced to redistribute the focal element weights to obtain a new BPA. Average evidence is used to replace conflicting evidence. Then, a confidence matrix is ​​constructed to complete the multi-sensor information fusion perception. First, calculate the average value of each piece of evidence for the k-th focal element, as shown in the following formula: The distance from each piece of evidence to the average value of that focal element is calculated as follows: Where d i This represents the basic probability function value of the i-th piece of evidence for the j-th focal element. It can be seen that the higher the similarity between the two pieces of evidence, the higher the probability function value. i The smaller the value; The weights of each piece of evidence and the average value of new evidence for the focal element are shown in (3-1) and (3-4), respectively: After obtaining the weights, the BPA values ​​of the conflicting evidence are recalculated. The main process is as follows: Construct the confidence matrix M, as shown in the following formula: Where M is the basic probability assignment value of evidence i to focus element j, and the summation of each row of the matrix in the above formula satisfies the following equation: m i1 +m i1 +...+m im =1,(i=1,2,...,n) (3-6) Transpose the i-th row of matrix M, then multiply it by the j-th row, i.e. This yields a new matrix A (m*m dimensional). As can be seen from the formula, the product of the elements on the diagonal represents the fusion result of evidence i and evidence j, while the sum of all other elements in the matrix is ​​the uncertainty coefficient of the fusion result, as shown in the following formula: K ' =Σ P≠q m ip ×m jq (p,q=1,2,...,m) (3-9) Finally, multiple state information are fused and perceived.

4. The method for fusing multi-source, multi-modal data at a power operation site according to claim 1, characterized in that: Step b, the step of fusing multi-type inspection data includes: b1. Multimodal feature extraction and matching: For multimodal data generated by UAVs and sensors, feature information such as line damage and hot spots is extracted through computer vision technology or image processing algorithms; Manual inspection data is used to extract key information through natural language processing technology, and then compared and verified with machine inspection data to ensure data consistency and complementarity. b2. Time and space alignment: The data collection time and spatial location of different inspection methods may be inconsistent; Technically, time synchronization, GPS positioning, and spatiotemporal alignment of data are used to unify data from different inspection sources, enabling collaborative analysis of different data and forming a comprehensive perception of the line status. b3. Data Fusion and Anomaly Detection: By using deep learning and machine learning algorithms, various types of inspection data are fused to automatically identify anomalies and perform fault detection and diagnosis.

5. The method for fusing multi-source, multi-modal data at a power operation site according to claim 4, characterized in that: The joint calibration of three sensor coordinate systems and time is performed using a fusion algorithm. The main steps are as follows: Establish a camera coordinate system (Oc, Xc, Yc, Zc). In the world coordinate system space, the coordinates of a point are [xw, yw, zw]. Transform to the camera coordinate system, where its position is represented as [xc, yc, zc]. The transformation formula between the two coordinate systems is: The homogeneous coordinate form is shown in the following equation. M1 is the overall transformation matrix, which is the rotation and translation matrix between two coordinate systems, including the orthogonal rotation matrix R and the translation matrix T. The object in the world coordinate system is refracted by the lens to form the image coordinate system (OI,XI,YI,ZI), which is parallel to the center plane of the lens. The coordinates of the sensor's spatial point P[l,a] in the world coordinate system are given by the following formula: The formula for the correspondence between a spatial point P in the camera's pixel coordinate system [x, y] and the world coordinate system is: The above transformation relationship does not take into account lens distortion, so it is necessary to correct the distortion of the camera. Combining the distortion coefficient k of the image sensor itself, the corrected image coordinates P'[x',y'] are obtained: Where k is the radial distortion coefficient and p is the tangential distortion coefficient.

6. The method for fusing multi-source, multi-modal data at a power operation site according to claim 1, characterized in that: In step c, the fusion of machine inspection and manual inspection data includes the following steps: c1. Consistency Verification: Machine inspection provides continuous and objective data, while manual inspection relies on human experience and judgment. By using spatiotemporal consistency and data matching algorithms, the detection results of the two are compared at the same time and location to ensure the accuracy of machine detection data. c2. Complementarity analysis: Machine inspection is good at covering a wide range of routine tasks, while manual inspection is suitable for handling complex and difficult-to-standardize tasks. Through the fusion mechanism, the anomalies detected by machine inspection are fed back to manual inspection for on-site verification. At the same time, the experience of manual inspection is fed back to the machine inspection model to optimize its fault detection algorithm. c3. Feedback closed-loop mechanism: After the machine inspection finds potential problems, it automatically generates a report and notifies the human inspectors to confirm on-site, forming a closed-loop process of "machine detection - human confirmation - result feedback" to improve the accuracy of data and inspection efficiency.

7. The method for fusing multi-source, multi-modal data at a power operation site according to claim 1, characterized in that: In step d, when the RPA system can automatically process data, the weight of each attribute will greatly affect the evaluation result. The maximum deviation method is used to determine the attribute weights in the multi-attribute evaluation problem, where the evaluation information is represented by numerical values. A model is constructed using the maximum deviation method to solve the problem of determining attribute weights in hesitant and fuzzy environments. The model is as follows: By solving the above model, we can obtain: For ease of calculation, let: By standardizing wj so that their sum is 1, we can obtain: In a hesitant and fuzzy environment, A+ represents the positive ideal solution, and A- represents the hesitant and fuzzy negative ideal solution, that is: Using Euclidean distance, each evaluation scheme is compared with A. + and A - The distances are represented as di. + and di - : The relative similarity between schemes A+ and A- is: 0≤c(Ai)≤1, i=1,2,…,n. As can be seen from the above formula, when c(Ai) is closer to 1, the schemes Ai+ and A+ are closer, and the schemes are farther away from A-. The selection of the scheme can be determined based on the relative proximity c(Ai). Using this method to combine edge computing with RPA system, it is possible to automatically analyze the video and images collected during the drone inspection process, classify and label the detected equipment problems, and generate a visual report for on-site personnel to refer to.

Images