Parallel frequent item set mining method based on Spark platform

Abstract

The invention provides a parallel frequent item set mining method based on a Spark platform, and belongs to the field of frequent pattern mining, and the method comprises the following steps: the Spark platform divides an original data set into a plurality of sub-data sets, stores the sub-data sets in a plurality of sub-nodes in a distributed manner, and converts the sub-data sets stored in the sub-nodes into corresponding Boolean matrixes; updating the Boolean matrix by combining repeated transactions; each sub-node carries out screening based on a minimum support degree threshold value on each Boolean matrix at the same time so as to respectively obtain a local frequent item set corresponding to each sub-node; the local frequent item sets are gathered to a central node to form a global candidate set, the global candidate set is sent to each sub-node through broadcast variables, each sub-node obtains a sub-global frequent item set, and the sub-global frequent item sets corresponding to the sub-nodes are combined to obtain the global frequent item set. According to the method, a larger-scale data set can be processed, and a frequent item set can be found more quickly.

Description

Technical Field

[0001] The present invention belongs to the field of frequent pattern mining, and in particular relates to a parallel frequent item set mining method based on a Spark platform. Background Art

[0002] In the field of big data analysis, association rule mining, as a powerful data mining technology, plays an important role in many scenarios such as market basket analysis, bioinformatics, and recommendation systems. Its purpose is to discover the association between item sets in massive data sets. Frequent item set mining can help us discover the co-occurrence relationship between items from a large amount of data and reveal the hidden patterns and trends in the data. This is crucial for understanding the inherent structure and laws of data. However, as the scale of information data continues to expand, the existing frequent item set mining algorithm (Apriori algorithm) has encountered the following problems:

[0003] 1. Large data set processing: When the data set is very large, the existing algorithms need to scan the entire database multiple times, making the mining process very time-consuming;

[0004] 2. Explosion of the number of frequent itemsets: In some data sets, especially those dense data sets with a large number of items, even for relatively high support thresholds, a large number of frequent itemsets may be generated;

[0005] 3. Memory limitation: A lot of memory is consumed when generating candidate sets, which may become a problem when memory resources are limited;

[0006] 4. Redundant rules: The generated association rules may include a lot of redundant information, which is not conducive to user understanding and decision-making.

[0007] In order to overcome these problems, researchers and practitioners have proposed a variety of optimization methods and algorithms, such as using more efficient data structures (such as the FP-Growth algorithm, which uses FP trees (frequent pattern trees) to compress the database and avoids the generation of candidate sets in the Apriori algorithm, greatly reducing the number of scans of the database). However, the FP-Growth algorithm also has some limitations and disadvantages: First, the FP-Growth algorithm relies on the FP tree (frequent pattern tree) data structure, in which all data must be loaded into memory, which will result in large memory consumption for large-scale data sets. Secondly, the FP-Growth algorithm is difficult to directly apply to distributed computing environments in its original form, because the entire FP tree needs to be constructed and stored in a single memory space. Although there is a distributed version of the FP-Growth implementation, it is relatively complex and requires additional computing and coordination overhead. Summary of the invention

[0008] The purpose of the present invention is to propose a parallel frequent itemset mining method based on the Spark platform, which can process larger data sets, find frequent itemsets faster, and does not require additional computing and coordination overhead.

[0009] The present invention is achieved through the following technical solutions:

[0010] A parallel frequent itemset mining method based on Spark platform includes the following steps:

[0011] Step S1, loading the original data set into the Spark platform, the Spark platform divides the original data set into multiple sub-data sets and stores them in multiple sub-nodes in a distributed manner, converts the sub-data sets stored in each sub-node into a corresponding transaction database, and converts each transaction database into a corresponding Boolean matrix, wherein each row of the Boolean matrix represents a transaction and each column represents an item;

[0012] Step S2, updating the Boolean matrix by merging repeated transactions so that each Boolean matrix only contains records that may become frequent item sets;

[0013] Step S3, each child node simultaneously screens each Boolean matrix based on the minimum support threshold to obtain local frequent item sets corresponding to each child node, respectively. The local frequent item sets include multi-order frequent item sets. In the process of obtaining each order of frequent item sets, the infrequent item sets of that order are generated from the items whose support is less than the minimum support threshold of that order, and the elements corresponding to the infrequent item sets of that order in the Boolean matrix are deleted to update the Boolean matrix;

[0014] Step S4: Aggregate all local frequent item sets to a central node and form a global candidate set. The global candidate set is sent to each child node through a broadcast variable. Each child node calculates the support of the global candidate set according to its stored sub-data set, and retains the item sets that meet the support conditions as sub-global frequent item sets. The central node merges the sub-global frequent item sets corresponding to each child node to obtain a global frequent item set.

[0015] Furthermore, in step S1, the original data set is loaded into the Spark platform through the textFile operation. The Spark platform preprocesses the original data set according to the goal of frequent item set mining to remove data or noise that does not meet the analysis requirements. The Spark platform automatically creates multiple partitions according to the size of the preprocessed original data set and available memory. These partitions are distributed on each child node, and the preprocessed original data set is divided into multiple sub-data sets, and each sub-data set is stored in a corresponding partition.

[0016] Furthermore, in step S1, each of the child nodes uses the fallMap operation to convert the format of the corresponding sub-dataset to convert each line of text into a transaction representation, thereby obtaining the corresponding transaction database, and traversing each transaction in the transaction database, using the mapPartition operation to convert each transaction into the form of 0 or 1 to generate a Boolean matrix. In the Boolean matrix, if an item appears in the corresponding transaction, the corresponding position in the Boolean matrix is ​​1, otherwise, it is 0.

[0017] Furthermore, the process of each sub-node acquiring a local frequent item set in step S3 includes the following steps:

[0018] Step S31, in the i-th iteration, generate an i-order candidate set according to the i-1-order frequent item set generated by the i-1-th iteration and the updated Boolean matrix, count the support of each item in the i-order candidate set, form an i-order frequent item set from items whose support is not less than the i-order minimum support threshold i and enter step S32, form an i-order infrequent item set from items whose support is less than the i-order minimum support threshold i, delete the transactions containing the i-order infrequent item set in the Boolean matrix to update the Boolean matrix, and enter step S32;

[0019] Step S32: determine whether the i-th order frequent item set is empty or has only one item. If so, the iteration ends and the frequent item sets of all orders are combined to form a local frequent item set. Otherwise, set i=i+1 and proceed to step S31 for the i+1th iteration.

[0020] Furthermore, in step S3, the process of calculating the support of the candidate set is: for each item in the candidate set, the support of the item can be obtained by performing a bit operation on the column corresponding to each row in the Boolean matrix, and the support set of each item is the support of the candidate set.

[0021] Furthermore, in step S4, the collect operation is used to aggregate the local frequent item sets to a central node, and form a global candidate set, which is the union of the local frequent item sets.

[0022] Furthermore, in step S4, after each child node receives the global candidate set, it calculates the support of each item in the global candidate item set according to the data of its child node to obtain the local support. The central node obtains the global support according to the local support, and retains the items whose support is not less than the global support threshold to form a sub-global frequent item set.

[0023] Furthermore, in step S4, each sub-global frequent item set is merged and deduplicated through a reduceByKey operation to obtain a global frequent item set, and the final support of each item in the global frequent item set is counted.

[0024] The present invention has the following beneficial effects:

[0025] 1. The present invention first loads the original data set to the Spark platform, and the Spark platform stores the original data set in a distributed manner in multiple child nodes, and converts each transaction database into a corresponding Boolean matrix, and then updates the Boolean matrix by merging repeated transactions. Then, each child node simultaneously filters each Boolean matrix based on the minimum support threshold to obtain the local frequent item sets corresponding to each child node, and finally obtains the global frequent item sets based on each local frequent item set, thereby realizing parallel processing of data. By introducing the Boolean matrix and filtering the Boolean matrix based on the minimum support threshold, the number of scans of the data set can be greatly reduced, thereby improving the efficiency of the entire method, finding frequent item sets faster, and processing larger data sets without the need for additional calculation and coordination overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present invention is further described in detail below with reference to the accompanying drawings.

[0027] Figure 1 It is a flow chart of the present invention.

[0028] Figure 2 It is a detailed flow chart of the present invention.

[0029] Figure 3 This is a schematic diagram of the RDD execution factor process on the Spark platform.

[0030] Figure 4 The flowchart of the present invention for obtaining local frequent itemsets.

[0031] Figure 5 This is an example of obtaining a local frequent set in the present invention. DETAILED DESCRIPTION

[0032] like Figures 1 to 3 As shown in FIG. 1 , a parallel frequent itemset mining method based on the Spark platform includes the following steps:

[0033] Step S1, loading the original data set into the Spark platform, the Spark platform divides the original data set into n sub-data sets and stores them in n sub-nodes in a distributed manner, converts the sub-data sets stored in each sub-node into a corresponding transaction database, and converts each transaction database into a corresponding Boolean matrix, wherein each row of the Boolean matrix represents a transaction and each column represents an item;

[0034] Specifically, you need to prepare the device configuration and compilation and running environment required for Spark distributed parallelism in advance, store the data set in the distributed file system HDFS, and use YARN to supervise the cluster devices. The read data will be created into a resilient distributed dataset (RDD), which represents an immutable, partitionable, and parallelizable data set.

[0035] The original data set is loaded into the Spark platform through the textFile operation. According to the goal of frequent item set mining, the Spark platform preprocesses the original data set to remove data or noise that does not meet the analysis requirements. The Spark platform automatically creates multiple partitions based on the size of the preprocessed original data set and available memory. These partitions are distributed on each child node and divide the preprocessed original data set into multiple sub-data sets, each of which is stored in the corresponding partition. After the original data set is loaded into the Spark platform, it needs to be cleaned, that is, operations such as missing value processing, outlier processing, and deduplication are performed to remove unnecessary information or noise data to ensure data quality and consistency.

[0036] Each child node uses the falltMap operation to convert the format of the corresponding sub-dataset to convert each line of text into a transaction representation, thereby obtaining the corresponding transaction database, and traversing each transaction in the transaction database, using the mapPartition operation to convert each transaction into the form of 0 or 1 to generate a Boolean matrix. In the Boolean matrix, if an item appears in the corresponding transaction, the corresponding position in the Boolean matrix is ​​1, otherwise it is 0.

[0037] For Boolean matrices, assume the following transaction database:

[0038] T1: {Apple, Banana, Mango}

[0039] T2: {Banana, Berry, Orange}

[0040] T3: {Apple, Orange}

[0041] T4: {Apple, Banana}

[0042] Then the Boolean matrix corresponding to the above transaction database is expressed as:

[0043]

[0044] After representing data through Boolean matrices, calculating the support of item sets can be achieved through simple bit operations, which is much faster than checking transaction data one by one. For example, to calculate the support of the item set {apple, banana}, you only need to perform AND bit operations: (1, 1, 1, 0, 0) AND (0, 1, 0, 1, 1) = (0, 1, 0, 0, 0). Boolean matrices also facilitate the use of Boolean algebra optimization, such as using the "counting rule" to quickly calculate the support of item sets, thereby avoiding row-by-row scanning.

[0045] Step S2, updating the Boolean matrix by merging repeated transactions so that each Boolean matrix only contains records that may become frequent item sets;

[0046] Step S3, each child node simultaneously screens each Boolean matrix based on the minimum support threshold to obtain local frequent item sets corresponding to each child node, respectively. The local frequent item sets include multi-order frequent item sets. In the process of obtaining each order of frequent item sets, the non-frequent item sets of that order are generated from the items whose support is less than the minimum support threshold of that order, and the elements corresponding to the non-frequent item sets of that order in the Boolean matrix are deleted to update the Boolean matrix;

[0047] like Figure 4 As shown in Figure 2, the process of each child node obtaining a local frequent item set includes the following steps:

[0048] Step S31, in the i-th iteration, generate an i-order candidate set according to the i-1-order frequent item set generated by the i-1-th iteration and the updated Boolean matrix, count the support of each item in the i-order candidate set, form an i-order frequent item set from items whose support is not less than the i-order minimum support threshold i and enter step S32, form an i-order infrequent item set from items whose support is less than the i-order minimum support threshold i, delete the transactions containing the i-order infrequent item set in the Boolean matrix to update the Boolean matrix, and enter step S32, where i is an integer;

[0049] Step S32: determine whether the i-th order frequent item set is empty or has only one item. If so, the iteration ends and the frequent item sets of all orders are combined to form a local frequent item set. Otherwise, set i=i+1 and proceed to step S31 for the i+1th iteration.

[0050] Before step S31, each row of the Boolean matrix is ​​filtered through a filter conversion operation to remove transactions containing known infrequent items, because transactions containing known infrequent items will not contribute to the mining results of frequent item sets and will not be used in subsequent calculations.

[0051] like Figure 5As shown in the figure, before the first iteration, the Boolean matrix is ​​initialized to count the frequency of each item's single occurrence, and a first-order candidate set is generated based on this. The first-order candidate set is a list of all single items. In the i-th iteration, the items in the i-1-order frequent item set are combined using the falltMap with the iterator to obtain the i-order candidate set.

[0052] In the above process, the process of calculating the support of the candidate set is: for each item in the candidate set, the support of the item can be obtained by performing a bitwise AND operation on the column corresponding to each row in the Boolean matrix. The support set of each item is the support of the candidate set.

[0053] Step S4: Aggregate all local frequent item sets to a central node and form a global candidate set. The global candidate set is sent to each child node through a broadcast variable. Each child node calculates the support of the global candidate set according to its stored sub-data set, and retains the item sets that meet the support conditions as sub-global frequent item sets. The central node merges the sub-global frequent item sets corresponding to each child node to obtain a global frequent item set.

[0054] Specifically, the collect operation is used to aggregate the local frequent item sets to a central node and form a global candidate set, which is the union of the local frequent item sets. After receiving the global candidate set, each child node calculates the support of each item in the global candidate set according to the data of its child node to obtain the local support. The central node obtains the global support according to the local support, and retains the items whose support is not less than the global support threshold to form a sub-global frequent item set.

[0055] The global support of global candidate item set i is supp i The calculation formula is Where D represents the total number of transactions corresponding to the original data set, supp i,l represents the local support of the global candidate item set i on the jth child node, D j Indicates the number of transactions corresponding to the data set of the jth child node, supp i，j ·D j represents the support count on the jth child node, and n is the number of child nodes.

[0056] Through the reduceByKey operation, the sub-global frequent itemsets are merged and deduplicated to obtain the global frequent itemset, and the final support of each item in the global frequent itemset is counted. The output results are usually sorted from high to low according to the order of the frequent itemsets. More specifically, the output gives the number of candidate sets and frequent itemsets at each stage, as well as the execution time. This method shows significant performance advantages when processing large data sets, especially when the data density is high and the transaction length is long. It has the advantages of high efficiency, flexibility, and scalability, and can cope with the frequent itemset mining tasks of large-scale data sets.

[0057] The above description is only a preferred embodiment of the present invention, and therefore cannot be used to limit the scope of implementation of the present invention. That is, equivalent changes and modifications made according to the scope of the patent application of the present invention and the contents of the specification should still fall within the scope covered by the patent of the present invention.

Claims

1. A parallel frequent itemset mining method based on Spark platform, characterized by: The steps include: Step S1, loading the original data set into the Spark platform, the Spark platform divides the original data set into multiple sub-data sets and stores them in multiple sub-nodes in a distributed manner, converts the sub-data sets stored in each sub-node into a corresponding transaction database, and converts each transaction database into a corresponding Boolean matrix, wherein each row of the Boolean matrix represents a transaction and each column represents an item; Step S2, updating the Boolean matrix by merging repeated transactions so that each Boolean matrix only contains records that may become frequent item sets; Step S3, each child node simultaneously screens each Boolean matrix based on the minimum support threshold to obtain local frequent item sets corresponding to each child node, respectively. The local frequent item sets include multi-order frequent item sets. In the process of obtaining each order of frequent item sets, the infrequent item sets of that order are generated from the items whose support is less than the minimum support threshold of that order, and the elements corresponding to the infrequent item sets of that order in the Boolean matrix are deleted to update the Boolean matrix; Step S4: Aggregate all local frequent item sets to a central node and form a global candidate set. The global candidate set is sent to each child node through a broadcast variable. Each child node calculates the support of the global candidate set according to its stored sub-data set, and retains the item sets that meet the support conditions as sub-global frequent item sets. The central node merges the sub-global frequent item sets corresponding to each child node to obtain a global frequent item set.

2. The method for parallelizing frequent itemsets mining based on Spark platform according to claim 1, characterized in that: In step S1, the original data set is loaded into the Spark platform through the textFile operation. The Spark platform preprocesses the original data set according to the goal of frequent item set mining to remove data or noise that does not meet the analysis requirements. The Spark platform automatically creates multiple partitions according to the size of the preprocessed original data set and the available memory. These partitions are distributed on each child node, and the preprocessed original data set is divided into multiple sub-data sets, and each sub-data set is stored in a corresponding partition.

3. The method for parallelizing frequent itemsets mining based on Spark platform according to claim 2, characterized in that: In step S1, each of the child nodes converts the format of the corresponding sub-dataset using the falltMap operation to convert each line of text into a transaction representation, thereby obtaining the corresponding transaction database, and traverses each transaction in the transaction database, and uses the mapPartition operation to convert each transaction into the form of 0 or 1 to generate a Boolean matrix. In the Boolean matrix, if an item appears in the corresponding transaction, the corresponding position in the Boolean matrix is ​​1, otherwise, it is 0.

4. The method for parallelizing frequent itemsets mining based on Spark platform according to claim 3, characterized in that: The process of each subnode acquiring a local frequent item set in step S3 includes the following steps: Step S31, in the i-th iteration, generate an i-order candidate set according to the i-1-order frequent item set generated by the i-1-th iteration and the updated Boolean matrix, count the support of each item in the i-order candidate set, form an i-order frequent item set from items whose support is not less than the i-order minimum support threshold i and enter step S32, form an i-order infrequent item set from items whose support is less than the i-order minimum support threshold i, delete the transactions containing the i-order infrequent item set in the Boolean matrix to update the Boolean matrix, and enter step S32; Step S32: determine whether the i-th order frequent item set is empty or has only one item. If so, the iteration ends and the frequent item sets of all orders are combined to form a local frequent item set. Otherwise, set i=i+1 and proceed to step S31 for the i+1th iteration.

5. The method for parallelizing frequent itemsets mining based on Spark platform according to claim 4, characterized in that: In step S3, the process of calculating the support of the candidate set is as follows: for each item in the candidate set, the support of the item can be obtained by performing a bit operation on the column corresponding to each row in the Boolean matrix. The support set of each item is the support of the candidate set.

6. The method for parallelizing frequent itemsets mining based on Spark platform according to claim 5, characterized in that: In step S4, the collect operation is used to aggregate the local frequent item sets to a central node, and form a global candidate set, which is the union of the local frequent item sets.

7. The method for parallelizing frequent itemsets mining based on Spark platform according to claim 6, characterized in that: In step S4, after receiving the global candidate set, each child node calculates the support of each item in the global candidate item set according to the data of its child node to obtain the local support. The central node obtains the global support according to the local support, and retains the items whose support is not less than the global support threshold to form a sub-global frequent item set.

8. The method for parallelizing frequent itemsets mining based on Spark platform according to claim 7, characterized in that: In step S4, the sub-global frequent itemsets are merged and deduplicated through the reduceByKey operation to obtain the global frequent itemset, and the final support of each item in the global frequent itemset is counted.

Images