Household change relationship identification method based on Spark framework and agglomerative hierarchical clustering algorithm

Abstract

The present application discloses a method for identifying household-to-electricity transformer relationships based on the Spark framework and the agglomerative hierarchical clustering algorithm, wherein the method includes: collecting time-series voltage data of transformers and users in the substation area; preprocessing the time-series voltage data using Spark SQL to obtain processed data; reducing the dimension of the processed data using the principal component analysis method and extracting the voltage time-series data features; clustering the voltage time-series data features based on the agglomerative hierarchical clustering algorithm to determine the classification of power users, and obtaining household-to-electricity transformer relationship identification results. This method, by introducing the Spark distributed computing platform and calling the Spark MLlib machine learning library, implements the principal component analysis method for dimensionality reduction and the agglomerative hierarchical clustering algorithm for clustering, and realizes the calculation of household-to-electricity transformer relationship identification, thereby solving the problem of low calculation efficiency of household-to-electricity transformer relationship identification after the amount of data increases.

Description

Technical Field

[0001] The present application relates to the technical field of power system analysis, and in particular to a method for identifying household-to-household relationship based on a Spark framework and an agglomerative hierarchical clustering algorithm. Background Art

[0002] In the power grid system, the user information of the distribution network is chaotic, lost or inaccurate, which seriously restricts the construction process of the smart grid: after the power company changes the wiring or the load distribution line is transformed, the information is not updated in time, resulting in user information not being consistent with the actual situation. In order to facilitate management, the power company implements sub-station management for low-voltage distribution network users, and the identification of the relationship between the station and the user is the basis for realizing lean power grid marketing, reducing power consumption and loss, and is also the prerequisite for power theft detection.

[0003] Application Contents

[0004] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0005] To this end, the first purpose of this application is to propose a household change relationship identification method based on the Spark framework and the agglomerative hierarchical clustering algorithm. By introducing the Spark distributed computing platform and calling the Spark MLlib machine learning library, the principal component analysis method for dimensionality reduction and the agglomerative hierarchical clustering algorithm for clustering are implemented to realize the calculation of household change relationship identification, thereby solving the problem of low calculation efficiency of household change relationship identification when the amount of data increases.

[0006] The second purpose of this application is to propose a household change relationship identification device based on the Spark framework and the agglomerative hierarchical clustering algorithm.

[0007] The third objective of the present application is to provide an electronic device.

[0008] A fourth objective of the present application is to provide a computer-readable storage medium.

[0009] To achieve the above-mentioned purpose, the first aspect of the present application provides a household change relationship identification method based on the Spark framework and the agglomerative hierarchical clustering algorithm, comprising the following steps:

[0010] Collect the time-series voltage data of transformers and users in the substation area;

[0011] Preprocessing the time series voltage data using Spark SQL (Structured Query Language) to obtain processed data;

[0012] Using principal component analysis to reduce the dimension of the processed data and extract voltage time series data features; and

[0013] Based on the agglomerative hierarchical clustering algorithm, cluster analysis is performed on the voltage time series data characteristics to determine the classification of power users and obtain the household-to-electricity transformer relationship identification result.

[0014] In addition, the household change relationship identification method based on the Spark framework and the agglomerative hierarchical clustering algorithm according to the above embodiment of the present application may also have the following additional technical features:

[0015] Optionally, the collecting of time-series voltage data of transformers and users in the substation area includes:

[0016] The time-series voltage data of the transformer and users in the substation area are fully pulled from the preset external system database through Sqoop.

[0017] Optionally, it also includes:

[0018] The time series voltage data of the transformer in the substation area and the users in the substation area are stored in HDFS (Hadoop Distributed File System) and associated with the Hive table.

[0019] Optionally, the preprocessing the time series voltage data by using Spark SQL includes:

[0020] Calculating missing values ​​of the time series voltage data, and performing missing value filling processing on the time series voltage data;

[0021] The characteristic vector of the time series voltage data is extracted and normalized to obtain the processed data.

[0022] To achieve the above-mentioned purpose, the second aspect of the present application provides a household change relationship identification device based on the Spark framework and the agglomerative hierarchical clustering algorithm, comprising:

[0023] The acquisition module is used to collect the time-series voltage data of transformers and users in the substation area;

[0024] A processing module, used for preprocessing the time series voltage data using Spark SQL to obtain processed data;

[0025] An extraction module, used to reduce the dimension of the processed data by using a principal component analysis method, and extract voltage time series data features; and

[0026] The identification module is used to perform cluster analysis on the voltage time series data characteristics based on an agglomerative hierarchical clustering algorithm to determine the classification of power users and obtain a result of identifying the relationship between users and transformers.

[0027] Optionally, the acquisition module includes:

[0028] The pulling unit is used to pull the time-series voltage data of the transformer and users in the substation area in full from a preset external system database through Sqoop.

[0029] Optionally, it also includes:

[0030] The storage module is used to store the time series voltage data of the transformer in the substation area and the users in the substation area in HDFS and associate it with the Hive table.

[0031] Optionally, the processing module includes:

[0032] A calculation unit, used for calculating missing values ​​of the time series voltage data, and performing missing value filling processing on the time series voltage data;

[0033] The extraction unit is used to extract the characteristic vector of the time series voltage data and perform normalization processing to obtain the processed data.

[0034] To achieve the above-mentioned objectives, the third aspect of the present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to execute the household change relationship identification method based on the Spark framework and the agglomerative hierarchical clustering algorithm as described in the above-mentioned embodiments.

[0035] To achieve the above-mentioned objectives, the fourth aspect of the present application provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the household change relationship identification method based on the Spark framework and the agglomerative hierarchical clustering algorithm as described in the above-mentioned embodiments.

[0036] Therefore, the time series voltage data of transformers and users in the substation area can be collected, and Spark SQL can be used to pre-process the time series voltage data to obtain the processed data, and the principal component analysis method can be used to reduce the dimension of the processed data, and the voltage time series data features can be extracted. Based on the agglomerative hierarchical clustering algorithm, the voltage time series data features are clustered and analyzed to determine the classification of power users, and the household-to-transformer relationship identification results can be obtained. Therefore, there is no need to consume manpower detection and additional equipment. It only needs to collect user voltage time series data according to a certain sampling rate, and the household-to-transformer relationship identification can be realized remotely. At the same time, the Spark distributed computing framework and memory-based parallel computing can greatly improve the computing efficiency, and solve the problem of high time complexity of the agglomerative hierarchical clustering algorithm.

[0037] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0039] Figure 1 A flowchart of a household change relationship identification method based on a Spark framework and an agglomerative hierarchical clustering algorithm provided according to an embodiment of the present application;

[0040] Figure 2 It is a flowchart of a household change relationship identification method based on Spark framework and agglomerative hierarchical clustering algorithm according to one embodiment of the present application;

[0041] Figure 3 This is an example diagram of a household change relationship identification device based on a Spark framework and an agglomerative hierarchical clustering algorithm according to an embodiment of the present application;

[0042] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0044] The following describes a household change relationship identification method based on the Spark framework and agglomerative hierarchical clustering algorithm proposed in accordance with an embodiment of the present application with reference to the accompanying drawings.

[0045] Specifically, Figure 1 A flowchart of a household change relationship identification method based on a Spark framework and an agglomerative hierarchical clustering algorithm provided in an embodiment of the present application.

[0046] It is understandable that in recent years, household transformer relationship recognition algorithms based on machine learning theory have been widely used, and there are many iterative computing scenarios in machine learning algorithms. Spark is a distributed computing framework that uses the most advanced DAG scheduler and can greatly improve computing efficiency through memory-based parallel computing. Spark MLlib is an extensible machine learning library in Spark, including machine learning algorithms such as classification, regression, clustering, and collaborative filtering. Various machine learning algorithms can be implemented through Spark MLlib. In the face of the continuous growth of the scale of existing power data in the substation area, using Spark to identify the household transformer relationship in the substation area can not only ensure the accuracy of the algorithm, but also greatly improve the computing efficiency. Under the Spark framework, the collected voltage time series data is processed and principal component analyzed to extract the static features of the time series data; then the agglomerative hierarchical clustering method is applied to cluster the voltage time series data to automatically divide the voltage time series data into corresponding categories to realize household transformer relationship recognition.

[0047] like Figure 1 As shown, the household change relationship identification method based on the Spark framework and the agglomerative hierarchical clustering algorithm includes the following steps:

[0048] In step S101, the time-series voltage data of the transformer and users in the substation are collected.

[0049] Optionally, in some embodiments, in combination Figure 1 and Figure 2 , collect the time series voltage data of transformers and users in the substation area, including: pulling the time series voltage data of transformers and users in the substation area from the preset external system database through Sqoop. Among them, Sqoop is a tool for transferring data between Hadoop and relational databases. It can import data from a relational database (such as MySQL, Oracle, Postgres, etc.) into Hadoop's HDFS, and can also import HDFS data into a relational database.

[0050] Specifically, the embodiment of the present application can obtain the original power data of the substation area from the external system, that is, the time-series voltage data of the users in the substation area, including voltage, current, power factor, etc. Among them, the preset external system database can be a Hive table.

[0051] Optionally, in some embodiments, in combination Figure 1 and Figure 2The above-mentioned household transformer relationship identification method based on Spark framework and agglomerative hierarchical clustering algorithm also includes: storing the time series voltage data of the transformer and the user in the substation area in HDFS and associating the Hive table. Among them, HDFS is a highly fault-tolerant distributed file system, and has low requirements for system server configuration. HDFS can not only ensure data fault tolerance, but also provide high-throughput data access, which is very suitable for application on large-scale data sets. Therefore, the problem of large data storage is solved based on the HDFS distributed storage system.

[0052] It is understandable that the embodiment of the present application can create a Hive table for storing the original data of the substation area by viewing the original data structure of the substation area, that is, the time-series voltage data of the users in the substation area. Among them, the original data structure of the substation area includes: collection point number, table number, substation area identification, use type, data type, date, phase sequence and the collection value at each moment (24 points a day). Based on the original data structure, a Hive external table is created to store the original data of the substation area.

[0053] Therefore, the embodiment of the present application can use Sqoop to pull all the data such as the current and voltage of the substation from the external system database and load the data into the Hive table. The parameter command includes: --target-dir (HDFS storage path), --num-mappers (number of mapjobs), --hive-database (database name of the Hive table), --hive-table (Hive table name).

[0054] In step S102, Spark SQL is used to pre-process the time series voltage data to obtain processed data.

[0055] Optionally, in some embodiments, in combination Figure 1 and Figure 2 , Spark SQL is used to preprocess the time series voltage data, including: calculating the missing values ​​of the time series voltage data and filling the missing values ​​of the time series voltage data; extracting the feature vector of the time series voltage data and normalizing it to obtain the processed data.

[0056] It is understandable that this application is private and can use SparkSQL to clean and preprocess the original data, for example, including missing value filling, feature vector selection, outlier processing, normalization, etc.

[0057] Specifically, the embodiment of the present application can first create a SparkSession, which provides a unified entry point for users to use various functions of Spark; then access the data in the Hive table through SparkSQL; remove the data with missing values ​​exceeding 20% ​​in the collected data, replace the outliers with the mean replacement principle, and finally normalize the data to obtain the processed data.

[0058] In step S103, the principal component analysis method is used to reduce the dimension of the processed data and extract the voltage time series data features.

[0059] Specifically, the embodiment of the present application can call the Spark MLlib package, first use principal component analysis to reduce the dimension of the processed data, and extract the voltage time series data features.

[0060] Therefore, Spark is based on memory computing, and the intermediate calculation results do not need to be stored on the disk, avoiding the time waste of disk reading and writing. It also models the relationship between RDDs through DAG (Directed acyclic graph), automatically schedules the execution order of jobs, reduces shuffle data, and greatly improves computing efficiency.

[0061] In step S104, cluster analysis is performed on the voltage time series data features based on an agglomerative hierarchical clustering algorithm to determine the classification of power users, and obtain a user-transformer relationship recognition result.

[0062] Specifically, the embodiment of the present application can also use an agglomerative hierarchical clustering algorithm to classify users according to transformers to identify the relationship between households and transformers.

[0063] According to the household-to-electricity transformer relationship identification method based on the Spark framework and the agglomerative hierarchical clustering algorithm proposed in the embodiment of the present application, the time-series voltage data of the transformer and the user in the substation area can be collected, and the time-series voltage data can be preprocessed using Spark SQL to obtain the processed data, and the principal component analysis method is used to reduce the dimension of the processed data, and the voltage time-series data features are extracted, and the voltage time-series data features are clustered based on the agglomerative hierarchical clustering algorithm to determine the classification of power users, and obtain the household-to-electricity transformer relationship identification result. Therefore, there is no need to consume manpower detection and additional equipment, and only the user voltage time-series data needs to be collected according to a certain sampling rate, and the household-to-electricity transformer relationship identification can be realized remotely. At the same time, the Spark distributed computing framework is used, and through memory-based parallel computing, the computing efficiency can be greatly improved, and the problem of high time complexity of the agglomerative hierarchical clustering algorithm is solved.

[0064] Next, a household change relationship identification device based on a Spark framework and an agglomerative hierarchical clustering algorithm proposed in accordance with an embodiment of the present application is described with reference to the accompanying drawings.

[0065] Figure 3 It is a block diagram of a household change relationship identification device based on a Spark framework and an agglomerative hierarchical clustering algorithm according to an embodiment of the present application.

[0066] like Figure 3 As shown, the household change relationship identification device 10 based on the Spark framework and the agglomerative hierarchical clustering algorithm includes: a collection module 100, a processing module 200, an extraction module 300 and an identification module 400.

[0067] The acquisition module 100 is used to collect the time-series voltage data of the transformer and the users in the area;

[0068] The processing module 200 is used to pre-process the time series voltage data using Spark SQL to obtain processed data;

[0069] The extraction module 300 is used to reduce the dimension of the processed data by using the principal component analysis method and extract the voltage time series data features; and

[0070] The identification module 400 is used to perform cluster analysis on the voltage time series data characteristics based on the agglomerative hierarchical clustering algorithm to determine the classification of power users and obtain the household-transformer relationship identification result.

[0071] Optionally, in some embodiments, the acquisition module 100 includes: a pulling unit.

[0072] Among them, the pulling unit is used to pull the time-series voltage data of the substation transformer and the substation users in the substation from the preset external system database through Sqoop.

[0073] Optionally, in some embodiments, the household change relationship identification device 10 based on the Spark framework and the agglomerative hierarchical clustering algorithm further includes:

[0074] The storage module is used to store the time series voltage data of the transformers and users in the substation area into HDFS and associate it with the Hive table.

[0075] Optionally, in some embodiments, the processing module 200 includes: a calculation unit and an extraction unit.

[0076] The calculation unit is used to calculate the missing values ​​of the time series voltage data and perform missing value filling processing on the time series voltage data;

[0077] The extraction unit is used to extract the characteristic vector of the time series voltage data and perform normalization processing to obtain processed data.

[0078] It should be noted that the aforementioned explanation of the embodiment of the household change relationship identification method based on the Spark framework and the agglomerative hierarchical clustering algorithm is also applicable to the household change relationship identification device based on the Spark framework and the agglomerative hierarchical clustering algorithm of this embodiment, and will not be repeated here.

[0079] According to the household-to-electricity transformer relationship identification device based on the Spark framework and the agglomerative hierarchical clustering algorithm proposed in the embodiment of the present application, the time series voltage data of the transformer and the user in the substation area can be collected, and the time series voltage data can be preprocessed using Spark SQL to obtain the processed data, and the principal component analysis method is used to reduce the dimension of the processed data, and the voltage time series data features are extracted, and the voltage time series data features are clustered based on the agglomerative hierarchical clustering algorithm to determine the classification of power users, and obtain the household-to-electricity transformer relationship identification result. Therefore, there is no need to consume manpower detection and additional equipment, and only the user voltage time series data needs to be collected according to a certain sampling rate, and the household-to-electricity transformer relationship identification can be realized remotely. At the same time, the Spark distributed computing framework is used, and through memory-based parallel computing, the computing efficiency can be greatly improved, and the problem of high time complexity of the agglomerative hierarchical clustering algorithm is solved.

[0080] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0081] A memory 1201 , a processor 1202 , and a computer program stored in the memory 1201 and executable on the processor 1202 .

[0082] When the processor 1202 executes the program, the household change relationship identification method based on the Spark framework and the agglomerative hierarchical clustering algorithm provided in the above embodiment is implemented.

[0083] Furthermore, the electronic device further comprises:

[0084] The communication interface 1203 is used for communication between the memory 1201 and the processor 1202 .

[0085] The memory 1201 is used to store computer programs that can be executed on the processor 1202 .

[0086] The memory 1201 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0087] If the memory 1201, the processor 1202 and the communication interface 1203 are implemented independently, the communication interface 1203, the memory 1201 and the processor 1202 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0088] Optionally, in a specific implementation, if the memory 1201, the processor 1202 and the communication interface 1203 are integrated on a chip, the memory 1201, the processor 1202 and the communication interface 1203 can communicate with each other through an internal interface.

[0089] The processor 1202 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0090] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the household change relationship identification method based on the Spark framework and the agglomerative hierarchical clustering algorithm is implemented as described above.

[0091] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0092] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0093] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0094] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0095] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0096] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0097] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0098] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A household change relationship identification method based on Spark framework and agglomerative hierarchical clustering algorithm, characterized in that: The following steps are involved: Collect the time-series voltage data of transformers and users in the substation area; Preprocessing the time series voltage data using Spark SQL to obtain processed data; Using principal component analysis to reduce the dimension of the processed data and extract voltage time series data features; as well as Based on the agglomerative hierarchical clustering algorithm, cluster analysis is performed on the voltage time series data characteristics to determine the classification of power users, and a household-to-transformer relationship recognition result is obtained; Wherein, the method further comprises: Based on the original data structure of the substation area, a Hive table is created, wherein the original data of the substation area is the time series voltage data of the users in the substation area, and the original data structure of the substation area includes the collection point number, table number, substation area identification, usage type, data type, date, phase sequence and the collection value at each moment; The collecting of the time series voltage data of the transformer and the users in the area includes: Pull all the time series voltage data of the transformer and users in the substation area from the preset external system database through Sqoop, and load the data into the Hive table; The method further comprises: The time series voltage data of the transformer and the users in the substation area are stored in HDFS and associated with the Hive table; The preprocessing of the time series voltage data by using Spark SQL includes: Calculating missing values ​​of the time series voltage data, and performing missing value filling processing on the time series voltage data; Extracting a characteristic vector of the time series voltage data and performing normalization processing to obtain the processed data; The preprocessing of the time series voltage data by using Spark SQL further includes: Create a SparkSession so that users can use Spark's functions through the SparkSession; Access data in Hive tables through SparkSQL; The principal component analysis method is used to reduce the dimension of the processed data and extract the voltage time series data features, including: The Spark MLlib package is called to use principal component analysis to reduce the dimension of the processed data and extract the characteristics of the voltage time series data.

2. A household change relationship identification device based on Spark framework and agglomerative hierarchical clustering algorithm, characterized in that: include: The acquisition module is used to collect the time-series voltage data of transformers and users in the substation area; A processing module, used for preprocessing the time series voltage data using Spark SQL to obtain processed data; An extraction module, used to reduce the dimension of the processed data by using a principal component analysis method, and extract voltage time series data features; as well as An identification module, used to perform cluster analysis on the voltage time series data characteristics based on an agglomerative hierarchical clustering algorithm to determine the classification of power users and obtain a result of identifying the relationship between users and transformers; Wherein, the device further comprises: A storage module is used to create a Hive table based on the original data structure of the substation area, wherein the original data of the substation area is the time series voltage data of the users in the substation area, and the original data structure of the substation area includes the collection point number, table number, substation area identification, use type, data type, date, phase sequence and the collection value at each moment; The acquisition module comprises: A pulling unit is used to pull the time series voltage data of the transformer and the user in the substation area from a preset external system database through Sqoop, and load the data into a Hive table; The storage module is also used to store the time series voltage data of the transformer and the user in the substation area in HDFS and associate it with the Hive table; The processing module comprises: A calculation unit, used for calculating missing values ​​of the time series voltage data, and performing missing value filling processing on the time series voltage data; An extraction unit, used for extracting a characteristic vector of the time series voltage data and performing normalization processing to obtain the processed data; The preprocessing of the time series voltage data by using Spark SQL further includes: Create a SparkSession so that users can use Spark's functions through the SparkSession; Access data in Hive tables through SparkSQL; The principal component analysis method is used to reduce the dimension of the processed data and extract the voltage time series data features, including: The Spark MLlib package is called to use principal component analysis to reduce the dimension of the processed data and extract the characteristics of the voltage time series data.

3. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the household change relationship identification method based on the Spark framework and the agglomerative hierarchical clustering algorithm as claimed in claim 1.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the household change relationship identification method based on the Spark framework and the agglomerative hierarchical clustering algorithm as claimed in claim 1.

Images