A multi-clustering method for large-scale multi-source heterogeneous data

Abstract

The present invention discloses a multi-clustering method for large-scale multi-source heterogeneous data, which relates to the technical field of data processing, including: pre-processing heterogeneous data from different sources through an ETL tool and converting them into a unified target data format; then classifying them according to voltage level, equipment type, and acquisition measurement type; constructing a topology analysis engine for the classified multi-source heterogeneous data sets according to the mutual correlation between the distribution network and network elements; eliminating unsatisfactory data sets based on topological analysis to obtain data sets to be fused; performing observation coefficient analysis on the data sets to be fused, and allocating a corresponding number of processing terminals to fuse the data sets to be fused, thereby improving data fusion efficiency and realizing cross-composite in-depth analysis of distribution network historical data and quasi-real-time data; outputting data fusion results for distribution network staff to study and analyze, providing guidance for refined energy management and user services, and timely achieving fault warnings and improving power safety.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a multi-clustering method for large-scale multi-source heterogeneous data. Background Art

[0002] With the in-depth development of smart grid construction, the business systems of the distribution network have different professional directions, different construction times, and different architectures. During the operation process, a large amount of measurement data, business form data, ledger information data, etc. are generated with diverse structures, complex sources, non-uniform time scales, and multi-source heterogeneous data of different spatial scales. According to statistics, a medium-sized distribution network will generate hundreds of TB of data each year. These data are independent of each other in their respective business systems and have not been effectively integrated. The data efficiency has not been fully mined and utilized. As a result, when data abnormalities occur in power equipment, relevant information cannot be pushed to relevant staff in a timely and accurate manner to provide fault warnings.

[0003] Based on this, some research on the above-mentioned problems has been conducted in the prior art. For example, patent application CN109241169A discloses a method for integrating a database of multi-source heterogeneous data fusion of distribution network operation information, which accesses different business subsystems of the power grid according to demand to obtain a target data set, selects a data set that meets certain conditions in the target data set based on a topological analysis engine, and constructs a data fusion model based on a regularized residual search method to remove bad data in the target data set after topological analysis processing, and then fuses the data. Patent application CN114238464A discloses a method for heterogeneous fusion of multi-energy data, which pre-processes heterogeneous data from different sources and then fuses the data. However, the existing multi-source heterogeneous data clustering analysis system cannot intelligently allocate the number of terminals for clustering analysis of the corresponding distribution network production and operation data, and the resource utilization is low, and the data analysis efficiency is low. Summary of the invention

[0004] To solve the above technical problems, the present invention provides a multi-clustering method for large-scale multi-source heterogeneous data, performs observation coefficient analysis on the fused data set, and intelligently adjusts the number of allocated processing terminals according to the observation coefficient to improve data processing efficiency.

[0005] The multi-clustering method for large-scale multi-source heterogeneous data described in the present invention comprises the following steps:

[0006] Step 1: During the selected time period, different business subsystems connected to the distribution network are continuously collected online according to demand to form a multi-source heterogeneous data set;

[0007] Step 2: Preprocess heterogeneous data from different sources through ETL tools to convert various formats of original data into a unified target data format; preprocessing includes data screening and data repair;

[0008] Step 3: Classify the preprocessed multi-source heterogeneous data sets according to voltage level, equipment type, and acquisition measurement type; construct a topology analysis engine for the classified multi-source heterogeneous data sets according to the interrelationships between the distribution network and network elements;

[0009] Step 4: Based on the analysis of the topology analysis engine, select the data sets from the multi-source heterogeneous data sets that meet the same time section, the same voltage, and the KCL law for current and power, and remove the data sets that do not meet the requirements to obtain the data sets to be fused; the data sets to be fused carry the time section;

[0010] Step 5: Perform observation coefficient GF analysis on the data set to be fused, and allocate a corresponding number of processing terminals according to the observation coefficient GF to fuse the data set to be fused, wherein the fusion is based on the HFCM clustering algorithm;

[0011] Step 6: Output the data fusion results for distribution network staff to study and analyze, and provide guidance for refined energy management and user services; the data fusion results carry time sections.

[0012] Furthermore, the observation coefficient GF analysis is performed on the fused data set. The specific analysis steps are as follows:

[0013] Obtain the time section corresponding to the data set to be fused, and retrieve the research attraction value YG of the corresponding time section;

[0014] The data size of the data set to be fused is counted as D1; ​​the distribution network corresponding to the data set to be fused is obtained, and the scale value GM and the fault coefficient GZ of the corresponding distribution network are retrieved;

[0015] The observation coefficient GF of the data set to be fused is calculated using the formula GF=YG×g1+D1×g2+GM×g3+GZ×g4; where g1, g2, g3, and g4 are coefficient factors.

[0016] Furthermore, a corresponding number of processing terminals are allocated according to the observation coefficient GF to fuse the data set to be fused, specifically including:

[0017] The database stores a comparison table of observation coefficient ranges and allocation quantity thresholds; first determine the observation coefficient range corresponding to the observation coefficient GF, then determine the allocation quantity threshold corresponding to the observation coefficient range and mark it as L1, that is, allocate L1 processing terminals to fuse the data set to be fused.

[0018] Furthermore, it also includes: monitoring the access to the data fusion results, and performing research attraction value YG analysis based on the access records. The specific analysis steps are:

[0019] Within a preset time, obtaining access records of data fusion results, wherein the access records include access start time and access end time; obtaining a time section corresponding to the data fusion result;

[0020] For the same time section, the number of visits to the time section is counted as C1; the visit duration of each visit is accumulated to obtain the total visit duration ZT; the research attraction value YG of the time section is calculated using the formula YG=C1×a1+ZT×a2, where a1 and a2 are coefficient factors.

[0021] Furthermore, it also includes: performing a scale value GM analysis on the distribution network, specifically:

[0022] Obtain the power supply area of ​​the distribution network; count the length of the power supply line in the power supply area as DL, the number of households powered as HL, and the average power consumption per household as VL; use the formula GM=DL×a3+HL×a4+VL×a5 to calculate the scale value GM of the distribution network, where a3, a4, and a5 are coefficient factors.

[0023] Furthermore, it also includes: carrying out maintenance tracking on the distribution network, and evaluating the fault coefficient GZ of the distribution network according to the maintenance information; specifically:

[0024] Obtain all maintenance information of the distribution network within a preset time period; the maintenance information includes faulty network components, maintenance duration, and maintenance level;

[0025] The number of maintenance times of the distribution network is counted as G1; the number of faulty network components in each maintenance information is marked as GL, the maintenance time is marked as GT, and the maintenance level is marked as GD; the maintenance value JXi is calculated using the formula JXi=GL×b1+GT×b2+GD×b3, where b1, b2, and b3 are coefficient factors;

[0026] Compare the maintenance value JXi with the maintenance threshold; count the number of times JXi is greater than the maintenance threshold as G2. When JXi is greater than the maintenance threshold, obtain the difference between JXi and the maintenance threshold and sum them to get the over-inspection value CJ; use the formula CP = G2 × b4 + CJ × b5 to calculate the over-inspection coefficient CP, where b4 and b5 are coefficient factors; use the formula The fault coefficient GZ is calculated, where f1 and f2 are coefficient factors.

[0027] Furthermore, in step three, the voltage levels are divided into: 35kV, 20kV, 10kV; the equipment types are divided into: transformers, switch cabinets, and lines according to different business subsystems; the collected measurement types are divided into: state quantities and analog quantities, real-time data, and non-real-time data.

[0028] The beneficial effects of the present invention are as follows: the present invention combines the data set to be fused, performs access monitoring on the data fusion result, and performs research attraction value YG analysis based on the access records; performs scale value GM analysis on the distribution network to obtain the power supply area of ​​the distribution network; calculates the scale value GM of the distribution network in combination with the length of the power supply line, the number of households powered, and the average household power consumption in the power supply area; performs maintenance tracking on the distribution network, and evaluates the fault coefficient GZ of the distribution network based on the maintenance information, and calculates the observation coefficient GF of the data set to be fused; and then allocates a corresponding number of processing terminals to fuse the data set to be fused based on the observation coefficient GF, that is, observation coefficients in different intervals correspond to different numbers of processing terminals; more processing terminals can be allocated to the data set to be fused with a high observation coefficient GF for data fusion, so as to maximize resource utilization and improve data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 The present invention is a principle block diagram of a multi-clustering method for large-scale multi-source heterogeneous data. DETAILED DESCRIPTION

[0030] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] like Figure 1 As shown, a multi-clustering method for large-scale multi-source heterogeneous data includes:

[0032] Step 1: During the selected time period, different business subsystems connected to the distribution network are continuously collected online according to demand to form a multi-source heterogeneous data set;

[0033] Step 2: Preprocess heterogeneous data from different sources through ETL tools to convert the various formats of the original data into a unified target data format; perform data screening and data repair through ETL tools to unify the format of multi-source heterogeneous data; the purpose is to preliminarily organize the data for more accurate data mining;

[0034] Step 3: Classify the preprocessed multi-source heterogeneous data sets; classify the multi-source heterogeneous data sets according to voltage level, equipment type, and acquisition measurement type; voltage levels are divided into: 35kV, 20kV, 10kV; equipment types are divided into: transformers, switch cabinets, and lines according to different business subsystems; acquisition measurement types are divided into: state quantity and analog quantity, real-time data, and non-real-time data;

[0035] Step 4: Construct a topology analysis engine for the classified multi-source heterogeneous data sets according to the interrelationships between the distribution network and network elements. The establishment of the topology analysis engine relies on the following principles:

[0036] 1) According to the connection relationship of the distribution network, establish dependency relationships for circuit breakers and switches in the distribution network;

[0037] 2) Information about the interdependence of network components: switch position information, relationship between collected quantity measurements;

[0038] 3) The dependency of network components on data, including temporary and fixed dependencies under different operation modes, the dependency of switch positions on collected quantity measurements, and the dependency on historical data;

[0039] 4) Information dependency between network components and alarm events, SOE events, and protection systems in the distribution network operation information;

[0040] Based on the analysis of the topology analysis engine, select the data sets from the multi-source heterogeneous data sets that meet the same time section, have consistent voltage, and satisfy the KCL law for current and power, and remove the data sets that do not meet the requirements to obtain the data sets to be fused; wherein the data sets to be fused carry the corresponding time sections;

[0041] In this embodiment, within 24 hours a day, the operation of the distribution network in different time periods is different, which is divided into peak period, valley period and stable period; therefore, the data efficiency and data value of the production and operation data generated by the distribution network in different periods are different;

[0042] Step 5: Analyze the observation coefficient of the fused data set, and allocate a corresponding number of processing terminals to fuse the fused data set according to the observation coefficient GF. The fusion is based on the HFCM clustering algorithm, which is used to explore the value of multi-source heterogeneous data and realize the interconnection, exchange and sharing of multi-source heterogeneous data. The specific analysis steps are:

[0043] Obtain the time section corresponding to the data set to be fused, and retrieve the research attraction value of the corresponding time section as YG; and calculate the data size of the data set to be fused as D1;

[0044] Get the distribution network corresponding to the data set to be fused; call the scale value of the corresponding distribution network as GM, and call the fault coefficient of the corresponding distribution network as GZ; use the formula GF = YG × g1 + D1 × g2 + GM × g3 + GZ × g4 to calculate the observation coefficient GF of the data set to be fused; where g1, g2, g3, and g4 are coefficient factors;

[0045] The number of processing terminals to be allocated is determined to be L1 according to the observation coefficient GF. Specifically, a comparison table of the observation coefficient range and the allocation number threshold is stored in the database; first, the observation coefficient range corresponding to the observation coefficient GF is determined, and then the allocation number threshold corresponding to the observation coefficient range is determined and marked as L1; that is, L1 processing terminals are allocated to fuse the data set to be fused;

[0046] Step 6: Output the data fusion results for research and analysis by the staff of the supply and distribution network, and provide guidance for refined energy management and user services, so as to achieve interaction between users and the power grid supply and demand; the data fusion results carry the corresponding time sections.

[0047] The present invention classifies and topologically analyzes a large amount of multi-source heterogeneous data generated by the distribution network, removes bad data, and then fuses them, solving the extraction, integration and data quality improvement of the multi-source heterogeneous operation information of the distribution network, and realizing the cross-composite in-depth analysis of the historical data and quasi-real-time data of the distribution network. When data abnormalities occur in power equipment, relevant information can be pushed to relevant staff in a timely and accurate manner to achieve fault warning and improve power safety.

[0048] In addition, since the distribution network operation data is massive, in order to complete the above data processing tasks, an efficient data processing framework is required. First, based on the analysis of the topology analysis engine, select the data sets from the multi-source heterogeneous data sets that meet the same time section, the voltage is consistent, and the current and power meet the KCL law. Eliminate the data sets that do not meet the requirements to obtain the data sets to be fused. Then, according to the observation coefficient GF of the data set to be fused, allocate the corresponding number of processing terminals to fuse the data sets to maximize resource utilization, thereby improving data processing efficiency.

[0049] The method further includes: monitoring the access to the data fusion results, and conducting research attraction analysis based on the access records. The specific analysis steps are:

[0050] Within a preset time, obtain access records of data fusion results, the access records including the access start time and the access end time; obtain the time section corresponding to the data fusion results;

[0051] For the same time section, the number of visits to the time section is counted as C1; the visit duration of each visit is accumulated to obtain the total visit duration ZT; the research attraction value YG of the time section is calculated using the formula YG=C1×a1+ZT×a2, where a1 and a2 are coefficient factors;

[0052] The method further includes: performing scale value analysis on the distribution network, and the specific analysis steps are:

[0053] Obtain the power supply area of ​​the distribution network; count the length of the power supply line in the power supply area as DL, the number of households powered as HL, and the average power consumption per household as VL; use the formula GM=DL×a3+HL×a4+VL×a5 to calculate the scale value GM of the distribution network, where a3, a4, and a5 are coefficient factors;

[0054] The method further includes: tracking the maintenance of the distribution network, and when it is detected that the distribution network is under maintenance, recording the maintenance information and evaluating the fault coefficient of the distribution network according to the maintenance information; specifically:

[0055] Obtain all maintenance information of the distribution network within a preset time period; maintenance information includes faulty network components, maintenance duration, and maintenance level; the maintenance level is evaluated by the maintenance personnel based on the maintenance resources invested after the maintenance is completed; the more maintenance resources invested, the higher the maintenance level;

[0056] The number of maintenance times of the distribution network is counted as G1; the number of faulty network components in each maintenance information is marked as GL, the maintenance time is marked as GT, and the maintenance level is marked as GD; the maintenance value JXi is calculated using the formula JXi=GL×b1+GT×b2+GD×b3, where b1, b2, and b3 are coefficient factors;

[0057] Compare the maintenance value JXi with the maintenance threshold; count the number of times JXi is greater than the maintenance threshold as G2. When JXi is greater than the maintenance threshold, obtain the difference between JXi and the maintenance threshold and sum them to get the over-inspection value CJ; use the formula CP = G2 × b4 + CJ × b5 to calculate the over-inspection coefficient CP, where b4 and b5 are coefficient factors; use the formula The fault coefficient GZ is calculated, where f1 and f2 are coefficient factors.

[0058] The above formulas are all calculated by removing dimensions and taking numerical values. The formula is a formula that is closest to the actual situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and preset thresholds in the formula are set by technical personnel in this field according to actual conditions or obtained by simulating a large amount of data.

[0059] Working principle of the present invention:

[0060] The multi-clustering method for large-scale multi-source heterogeneous data described in the present invention is as follows: when working, in a selected time period, different business subsystems of the distribution network are continuously connected to collect target data sets online according to needs to form multi-source heterogeneous data sets; heterogeneous data from different sources are preprocessed through ETL tools, so that multiple formats of original data are converted into a unified target data format; the preprocessed multi-source heterogeneous data sets are classified according to voltage level, equipment type, and acquisition measurement type; a topology analysis engine is constructed for the classified multi-source heterogeneous data sets according to the mutual correlation between the distribution network and network elements; based on the topology analysis engine analysis, unsatisfactory data sets are eliminated to obtain the data sets to be fused; based on the HFCM clustering algorithm, the data sets to be fused are fused, and the data fusion results are output, so as to realize the cross-composite in-depth analysis of the historical data and quasi-real-time data of the distribution network, and provide guidance for the refined management of energy and user services, so as to realize the interaction between user and power grid supply and demand;

[0061] Among them, it also includes: analyzing the observation coefficient of the data set to be fused; obtaining the time section corresponding to the data set to be fused, and retrieving the research attraction value of the corresponding time section as YG; counting the data size of the data set to be fused as D1; ​​obtaining the distribution network corresponding to the data set to be fused; retrieving the scale value of the corresponding distribution network as GM, and retrieving the fault coefficient of the corresponding distribution network as GZ; using the formula GF=YG×g1+D1×g2+GM×g3+GZ×g4 to calculate the observation coefficient GF of the data set to be fused; determining the number of allocated processing terminals as L1 according to the observation coefficient GF, that is, allocating L1 processing terminals to fuse the data set to be fused, so as to maximize resource utilization and thereby improve data processing efficiency.

[0062] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0063] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A multi-clustering method for large-scale multi-source heterogeneous data, characterized in that: include: Step 1: During the selected time period, different business subsystems connected to the distribution network are continuously collected online according to demand to form a multi-source heterogeneous data set; Step 2: Preprocess heterogeneous data from different sources through ETL tools to convert various formats of original data into a unified target data format; preprocessing includes data screening and data repair; Step 3: Classify the preprocessed multi-source heterogeneous data sets according to voltage level, equipment type, and acquisition measurement type; A topology analysis engine is constructed for the classified multi-source heterogeneous data sets according to the interrelationships between the power distribution network and network elements; Step 4: Based on the analysis of the topology analysis engine, select the data sets from the multi-source heterogeneous data sets that meet the same time section, the same voltage, and the KCL law for current and power, and remove the data sets that do not meet the requirements to obtain the data sets to be fused; the data sets to be fused carry the time section; Step 5: Perform observation coefficient GF analysis on the data set to be fused, and allocate a corresponding number of processing terminals according to the observation coefficient GF to fuse the data set to be fused, wherein the fusion is based on the HFCM clustering algorithm; The observation coefficient GF analysis is performed on the fused data set. The specific analysis steps are as follows: Obtain the time section corresponding to the data set to be fused, and retrieve the research attraction value YG of the corresponding time section; The data size of the data set to be fused is counted as D1; ​​the distribution network corresponding to the data set to be fused is obtained, and the scale value GM and the fault coefficient GZ of the corresponding distribution network are retrieved; The observation coefficient GF of the data set to be fused is calculated using the formula GF=YG×g1+D1×g2+GM×g3+GZ×g4; where g1, g2, g3, and g4 are coefficient factors; According to the observation coefficient GF, a corresponding number of processing terminals are allocated to fuse the data set to be fused, including: The database stores a comparison table of observation coefficient ranges and allocation quantity thresholds; first, the observation coefficient range corresponding to the observation coefficient GF is determined, and then the allocation quantity threshold corresponding to the observation coefficient range is determined and marked as L1, that is, L1 processing terminals are allocated to fuse the data set to be fused; Among them, the data fusion results are monitored for access, and the research attraction value YG analysis is performed based on the access records. The specific analysis steps are as follows: Within a preset time, obtaining access records of data fusion results, wherein the access records include access start time and access end time; obtaining a time section corresponding to the data fusion result; For the same time section, the number of visits to the time section is counted as C1; the visit duration of each visit is accumulated to obtain the total visit duration ZT; the research attraction value YG of the time section is calculated using the formula YG=C1×a1+ZT×a2, where a1 and a2 are coefficient factors; Step 6: Output the data fusion results for distribution network staff to study and analyze, and provide guidance for refined energy management and user services; the data fusion results carry time sections.

2. A multi-clustering method for large-scale multi-source heterogeneous data according to claim 1, characterized in that: The scale value GM analysis of the distribution network is as follows: Obtain the power supply area of ​​the distribution network; count the length of the power supply line in the power supply area as DL, the number of households powered as HL, and the average power consumption per household as VL; use the formula GM=DL×a3+HL×a4+VL×a5 to calculate the scale value GM of the distribution network, where a3, a4, and a5 are coefficient factors.

3. The multi-clustering method for large-scale multi-source heterogeneous data according to claim 1, characterized in that: The distribution network is inspected and tracked, and the fault coefficient GZ of the distribution network is evaluated based on the inspection information; specifically: Obtain all maintenance information of the distribution network within a preset time period; the maintenance information includes faulty network components, maintenance duration, and maintenance level; The number of maintenance times of the distribution network is counted as G1; the number of faulty network components in each maintenance information is marked as GL, the maintenance time is marked as GT, and the maintenance level is marked as GD; The maintenance value JXi is calculated using the formula JXi=GL×b1+GT×b2+GD×b3, where b1, b2, and b3 are coefficient factors; Compare the maintenance value JXi with the maintenance threshold; count the number of times JXi is greater than the maintenance threshold as G2, and when JXi is greater than the maintenance threshold, obtain the difference between each time JXi is greater than the threshold and sum them up to get the over-inspection value CJ; use the formula CP=G2×b4+CJ×b5 to calculate the over-inspection coefficient CP, where b4 and b5 are coefficient factors; Using the formula The fault coefficient GZ is calculated, where ƒ1 and ƒ2 are coefficient factors.

4. The multi-clustering method for large-scale multi-source heterogeneous data according to claim 1, characterized in that: In step three, the voltage levels are divided into: 35kV, 20kV, 10kV; the equipment types are divided into: transformers, switch cabinets, and lines according to different business subsystems; the collected measurement types are divided into: state quantities and analog quantities, real-time data, and non-real-time data.

Images