A power grid data frequent pattern mining method based on Spark and CS-FPtree
Patent Information
- Application Number
- CN202311603744.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-28
- Publication Date
- 2026-09-18
- Estimated Expiration
- 2043-11-28
AI Technical Summary
由于每条记录的事务项长度不唯一，传统的频繁模式树FPtree的子结点个数是不定的，为了方便搜索FPtree结点，必须额外定义头项表和线索指针记录存储位置，使结构定义复杂，进而使得操作和管理复杂、数据存储冗余，也将额外增加管理和应用的复杂度；此外，频繁模式树FPtree递归生成条件模式子树，基于子树再递归挖掘，聚合得到频繁模式基，如果数据量过大，其递归次数将受到内存容量限制，效率低、可扩展性差，影响挖掘结果的生成
[0011] The beneficial effects of this invention compared to existing technologies are as follows: This invention targets the mining strategy for big data of power grid users. Based on the Spark parallel computing cluster and PySpark API tool, it utilizes an improved CS-FPtree for frequent pattern mining. The improved CS-FPtree optimizes the definition of the frequent pattern tree from a structural perspective, simplifies the generation process, and reduces the number of iterations for frequent pattern mining. The use of the Spark distributed computing framework can effectively improve memory resource utilization and scalability.
Smart Images
Figure CN117688056B_ABST
Abstract
Description
Technical Field
[0001] This invention provides a method for mining frequent patterns in power grid data based on Spark and CS-FPtree, belonging to the technical field of frequent pattern mining in power grid data. Background Technology
[0002] As the number of power grid users increases year by year, the amount of electricity consumption data from users is also growing continuously. As a new factor of productivity, electricity consumption data will bring great value to power supply companies. Data mining technology introduced in recent years can discover hidden information in electricity consumption data, which can be used to improve power supply strategies and enhance business efficiency.
[0003] Currently used data mining techniques mainly rely on the frequent pattern tree (FP-tree) mining strategy. However, this data mining method has at least the following drawbacks in its application: Because the transaction item length of each record is not unique, the number of child nodes in a traditional frequent pattern tree (FPtree) is variable. To facilitate searching FPtree nodes, it is necessary to define additional header tables and thread pointer records, which complicates the structure definition, leading to complex operations and management, redundant data storage, and further increasing the complexity of management and application. In addition, the frequent pattern tree (FPtree) recursively generates conditional pattern subtrees, and then recursively mines based on the subtrees to aggregate and obtain the frequent pattern base. If the data volume is too large, the number of recursions will be limited by memory capacity, resulting in low efficiency, poor scalability, and affecting the generation of mining results. Summary of the Invention
[0004] In order to overcome the shortcomings of the prior art, the technical problem to be solved by this invention is to provide an improved method for mining frequent patterns in power grid data based on Spark and CS-FPtree.
[0005] To address the aforementioned technical problems, the present invention employs the following technical solution: a method for mining frequent patterns in power grid data based on Spark and CS-FPtree, comprising the following data mining steps: Step 1: Construct SparkSession and SprakConf objects, configure resource requirement parameters, load the power grid user dataset from HDFS, and create an instance of DataFrame (df data). Step 2: Create a processor to perform feature encoding and data binning on the instance df data to generate a binarized global transaction itemset; Step 3: Create a SparkContext object sc, call the parallelize method to generate transaction items, define a binary filter to fill in the names of items with a value of 1, set the minimum support to filter the item set, and generate the transaction item frequent basis and local transaction item set; Step 4: Define a CS-FPtree generator to generate a prefix pattern tree for valid transaction itemsets and generate a prefix path table based on frequent bases; Step 5: Define a subset generator to generate subsets of each path in the prefix path table, aggregate path weights, remove subsets that do not meet the minimum support, obtain the frequent pattern set, and mine the power grid data.
[0006] The specific method for creating instance data in step 1 is as follows: Step 1.1: Start PySpark, specify the masterURL, instantiate a SparkConf object to specify the cluster mode, configure resources, use the builder pattern to create a SparkSession object, and generate a sprak instance. Step 1.2: Read the data file from the specified location in HDFS and call spark.read to create an instance of DataFrame data.
[0007] The specific method for generating the binarized global transaction itemset in step 2 is as follows: Step 2.1: Convert character objects to discrete numerical objects: Define a converter to extract the category features of meter reading books, electricity price names, user categories, electricity consumption categories, power supply voltage, and importance levels from the df data into an RDD, and use the map operator to establish a mapping from feature values to feature value indexes for the fields; Step 2.2: Convert continuous value objects to discrete numerical types: Define a binning unit to convert the continuous numerical values of total electricity consumption and total electricity cost in the df data into discrete numerical values. Use the Weight of Evidence (WOE) to evaluate the binning performance and adjust the binning strategy accordingly. The expression for the WOE is: ; Step 2.3: Perform binarization on all features: Use the OneHotEncoderEstimator class in the SparkML library to implement one-hot encoding and generate a binarized global transaction set.
[0008] The specific method for generating the frequent basis of transaction items and the local transaction item set in step 3 is as follows: Step 3.1: Determine the support coefficient sp based on the minimum number of business rules and the confidence requirement, and calculate the minimum support min_support. The calculation formula is as follows: min_support=[Fmax×sp]; Step 3.2: Define a binary filter to retain valid transaction items with a feature value of 1, fill in the corresponding item names, delete items with a support lower than the minimum support, and obtain the valid transaction item RDD; Step 3.3: Calculate the frequency F of the transaction items, sort them from largest to smallest, and obtain the frequent basis of the transaction items.
[0009] The specific method for generating the prefix pattern tree of the valid transaction item set in step 4 is as follows: Step 4.1: Establish the initial root node, specifically an empty CS-FPtree with an initial frequency of 1; Step 4.2: Read the rows of valid transactions. Sort the transactions in the rows by frequency, extract the first item of the row, create a new node with a weight of 1, and insert it into the CS-Fptree according to the insertion rules: If the root node is a prefix, insert it into the left subtree; if it is not a prefix, insert it into the right subtree. Each inserted node writes its parent information. If the nodes are the same, no new node is created; instead, the accumulated weight is written directly. Step 4.3: Delete the inserted first item, retrieve the new first item, repeat step 4.2 until the current row is cleared, and then read the next record; Step 4.4: Repeat steps 4.2 and 4.3 until the records are empty, then the final CS-FPtree is generated, which is a binary prefix tree.
[0010] The specific method for obtaining the frequent pattern set in step 5 is as follows: Step 5.1: Read the transaction item node in the frequent basis, find the corresponding node in the CS-FPtree, visit it from its parent upwards until the root, and save the access path to the prefix path table. The prefix nodes on the path share the weight of the initial node. Step 5.2: Define a subset generator to calculate subsets for all paths in the prefix path table, obtain candidate item sets, and save them to the dictionary Freq_dict, where the dictionary Freq_dict = {frequent item subsets: first element weights}; Step 5.3: Define an aggregation filter, aggregate candidate subsets, filter subsets with support greater than the minimum support, and obtain the frequent pattern set of the effective transaction set.
[0011] The beneficial effects of this invention compared to existing technologies are as follows: This invention targets the mining strategy for big data of power grid users. Based on the Spark parallel computing cluster and PySpark API tool, it utilizes an improved CS-FPtree for frequent pattern mining. The improved CS-FPtree optimizes the definition of the frequent pattern tree from a structural perspective, simplifies the generation process, and reduces the number of iterations for frequent pattern mining. The use of the Spark distributed computing framework can effectively improve memory resource utilization and scalability. Attached Figure Description
[0012] The present invention will be further described below with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the framework of the present invention based on Spark distributed computing; Figure 2 This is a flowchart of the steps involved in the CS-FPtree frequent item mining method of this invention. Figure 3 This is a schematic diagram of the CS-FPtree structure used in this invention. Detailed Implementation
[0013] This invention specifically provides a method for mining frequent patterns in big data from power grid users based on the Spark distributed computing framework and CS-FPtree. It primarily utilizes the Spark distributed computing framework and PySpark API to implement frequent pattern mining, improving mining efficiency and memory utilization. By optimizing the frequent pattern tree into a binary tree, the structure definition and operations are simplified, improving the efficiency of writing power grid user transaction sets into the CS-FPtree. Furthermore, this invention generates a prefix table of transaction items by directly searching the CS-FPtree, eliminating the need for recursion and quickly calculating subsets of the prefix table, effectively improving both mining efficiency and accuracy.
[0014] like Figure 1 and Figure 2 As shown, the method for mining frequent patterns in big data of power grid users based on Spark and CS-FPtree provided by this invention includes the following specific steps: Step 1: Construct SparkSession and SparkConf objects, configure resource requirement parameters, load the power grid user dataset from HDFS, and create a DataFrame instance df; Step 2: Create a processor to perform feature encoding and data binning on the df data to generate a binarized global transaction itemset; Step 3: Create a SparkContext object sc, call the parallelize method to generate transaction item RDDs, define filters, set minimum support to filter itemsets, and generate transaction item frequent basis and local transaction item set RDDs; Step 4: Define a CS-FPtree generator to generate a prefix pattern tree for local transaction itemsets and generate a prefix path table based on frequent bases; Step 5: Define a subset generator to generate subsets of each path in the prefix path table, aggregate path weights, and filter by minimum support to obtain the frequent pattern set.
[0015] Step 1 specifically involves: Step 1.1: Start PySpark, specify the masterURL, instantiate a SparkConf object to specify the cluster mode, configure resources, use the builder pattern to create a SparkSession object, and generate a sprak instance. Step 1.2: Read the data file from the specified location in HDFS and call spark.read to create a DataFrame instance df.
[0016] Step 2 specifically involves: Step 2.1: Convert the character object to a discrete numerical object. Define a converter to extract the category features of the data file (df), such as meter reading, electricity price name, user category, electricity consumption category, power supply voltage, and importance level, into an RDD. Use the map operator to establish a mapping between feature values and feature value indices for the fields. Step 2.2: Convert continuous value objects to discrete numerical values. Define a binning unit to convert continuous numerical values such as total electricity consumption and total electricity cost in df into discrete numerical values. Use the Weight of Evidence (WOE) to evaluate the binning performance and adjust the binning strategy accordingly. Step 2.3: Binarize all features. Use the OneHotEncoderEstimator class from the SparkML library to perform one-hot encoding and generate a binarized global transaction set.
[0017] Step 3 specifically involves: Step 3.1: Determine the support coefficient sp based on the minimum number of business rules and the confidence requirement, and calculate the minimum support using the method min_support=[F max ×sp]; Step 3.2: Define a binary filter to retain valid transaction items with a feature value of 1, fill in the corresponding item names, delete items with a support lower than the minimum support, and obtain the valid transaction item RDD; Step 3.3: Calculate the frequency F of the transaction items, sort them from largest to smallest, and obtain the frequent basis of the transaction items.
[0018] Step 4 specifically involves: Step 4.1: Create the initial root node, which is an empty CS-FPtree with an initial frequency of 1; Step 4.2: Read rows of valid transactions. Sort the transactions in each row by frequency, extract the first item of the row, create a new node with a weight of 1, and insert it into the CS-FPtree. The basic insertion rule is: if the root node is a prefix, insert it into the left subtree; otherwise, insert it into the right subtree. Write parent information into each inserted node. If nodes are the same, do not create a new node; directly write the accumulated weight. Step 4.3: Delete the inserted first item, retrieve the new first item, repeat step 4.2 until the current row is cleared, and then read the next record; Step 4.4: Repeat steps 4.2 and 4.3 until the records are empty. Generate the final CS-FPtree, which is a binary prefix tree that implements compressed storage of the effective transaction set.
[0019] Step 5 specifically involves: Step 5.1: Read the transaction item nodes in the frequent base RDD, find the corresponding node in the CS-FPtree, visit upwards along its parent until the root, and save the access path to the prefix path table RDD. The prefix nodes on the path share the weight of the initial node. Step 5.2: Define a subset generator to generate subsets for all paths in the prefix path table, obtain candidate item sets, and save them to the dictionary Freq_dict, where Freq_dict = {frequent item subsets: first element weights}; Step 5.3: Define an aggregation filter to aggregate candidate subsets, filter subsets with support greater than the minimum support, and obtain the frequent pattern set of the effective transaction set, which is a dictionary.
[0020] In an embodiment of the present invention, data mining is performed using a slice of transaction set of power grid user data as an example: like Figure 1 As shown, SparkSession and SprakConf objects are constructed, resource requirement parameters are configured, the power grid user dataset is loaded from HDFS, and a DataFrame is created, with the schema including record IDs and transaction itemsets.
[0021] For ease of description, the transaction items after the binarization of the power grid user data features are named L1-L10 respectively. Items with a value of 1 are filtered to generate a slice of valid transaction set, as shown in the first column of Table 1 below.
[0022] Table 1: Abstract Representation of Sliced Transaction Sets of Power Grid User Data Based on the sliced data shown in Table 1, the maximum frequency of a transaction item is 7, the support coefficient is 0.2, and min_support=[7*0.2]=2. Filtering out the infrequent items in the transaction item set in the second column of Table 1 yields the frequent transaction set, which is then sorted and shown in the third column of Table 1. This set is saved as an RDD, with the frequent basis being the dictionary {L2:7, L1:6, L3:6, L4:2, L5:2}.
[0023] Read the RDD records line by line, insert each transaction node into the CS-Fptree, and the result is as follows. Figure 3 As shown.
[0024] Then, a prefix table is generated based on the frequent basis. L2 has no prefix, as shown in the second column of Table 2 below.
[0025] Table 2: Prefix table generates frequent transaction items Calculate the subset of the prefix table, as shown in the third column of Table 2; Aggregate the prefix subsets to obtain the frequent itemsets of the global transaction, as shown in the fourth column of Table 2; The above process is as follows Figure 2 As shown.
[0026] In addition, to verify the performance of the CS-FPtree method, a comparative experiment was conducted on the implementation examples using traditional methods for generating frequent itemsets based on FPtree. The test results are shown in Table 3 below.
[0027] Table 3: Test Results of Examples Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for mining frequent patterns in power grid data based on Spark and CS-FPtree, characterized in that: The data mining steps include the following: Step 1: Construct SparkSession and SprakConf objects, configure resource requirement parameters, load the power grid user dataset from HDFS, and create an instance of DataFrame (df data). Step 2: Create a processor to perform feature encoding and data binning on the instance data (df) to generate a binarized global transaction itemset. The specific method is as follows: Step 2.1: Convert character objects to discrete numerical objects: Define a converter to extract the category features of meter reading books, electricity price names, user categories, electricity consumption categories, power supply voltage, and importance levels from the df data into an RDD, and use the map operator to establish a mapping from feature values to feature value indexes for the fields; Step 2.2: Convert continuous value objects to discrete numerical types: Define a binning unit to convert the continuous numerical values of total electricity consumption and total electricity cost in the df data into discrete numerical values. Use the Weight of Evidence (WOE) to evaluate the binning performance and adjust the binning strategy accordingly. The expression for the WOE is: ； Step 2.3: Binarize all features: Use the OneHotEncoderEstimator class in the SparkML library to implement one-hot encoding and generate a binarized global transaction set; Step 3: Create a SparkContext object sc, call the parallelize method to generate transaction items, define a binary filter to fill in the names of items with a value of 1, set the minimum support to filter the item set, and generate the transaction item frequent basis and local transaction item set; Step 4: Define a CS-FPtree generator to generate a prefix pattern tree for valid transaction itemsets, and generate a prefix path table based on frequent bases. The specific method is as follows: Step 4.1: Establish the initial root node, specifically an empty CS-FPtree with an initial frequency of 1; Step 4.2: Read the rows of valid transactions. Sort the transactions in the rows by frequency, extract the first item of the row, create a new node with a weight of 1, and insert it into the CS-Fptree according to the insertion rules: If the root node is a prefix, insert it into the left subtree; if it is not a prefix, insert it into the right subtree. Each inserted node writes its parent information. If the nodes are the same, no new node is created; instead, the accumulated weight is written directly. Step 4.3: Delete the inserted first item, retrieve the new first item, repeat step 4.2 until the current row is cleared, and then read the next record; Step 4.4: Repeat steps 4.2 and 4.3 until the records are empty, then generate the final CS-FPtree, which is a binary prefix tree; Step 5: Define a subset generator to generate subsets of each path in the prefix path table, aggregate path weights, remove subsets that do not meet the minimum support, obtain the frequent pattern set, and mine the power grid data.
2. The method for mining frequent patterns in power grid data based on Spark and CS-FPtree according to claim 1, characterized in that: The specific method for creating instance data in step 1 is as follows: Step 1.1: Start PySpark, specify the masterURL, instantiate a SparkConf object to specify the cluster mode, configure resources, use the builder pattern to create a SparkSession object, and generate a sprak instance. Step 1.2: Read the data file from the specified location in HDFS and call spark.read to create an instance of DataFrame data.
3. The method for mining frequent patterns in power grid data based on Spark and CS-FPtree according to claim 2, characterized in that: The specific method for generating the frequent basis of transaction items and the local transaction item set in step 3 is as follows: Step 3.1: Determine the support coefficient sp based on the minimum number of business rules and the confidence requirement, and calculate the minimum support min_support. The calculation formula is as follows: min_support=[Fmax×sp]; Step 3.2: Define a binary filter to retain valid transaction items with a feature value of 1, fill in the corresponding item names, delete items with a support lower than the minimum support, and obtain the valid transaction item RDD; Step 3.3: Calculate the frequency F of the transaction items, sort them from largest to smallest, and obtain the frequent basis of the transaction items.
4. The method for mining frequent patterns in power grid data based on Spark and CS-FPtree according to claim 3, characterized in that: The specific method for obtaining the frequent pattern set in step 5 is as follows: Step 5.1: Read the transaction item node in the frequent basis, find the corresponding node in the CS-FPtree, visit it from its parent upwards until the root, and save the access path to the prefix path table. The prefix nodes on the path share the weight of the initial node. Step 5.2: Define a subset generator to calculate subsets for all paths in the prefix path table, obtain candidate item sets, and save them to the dictionary Freq_dict, where the dictionary Freq_dict = {frequent item subsets: first element weights}; Step 5.3: Define an aggregation filter, aggregate candidate subsets, filter subsets with support greater than the minimum support, and obtain the frequent pattern set of the effective transaction set.