A method for applying distributed frequent itemsets based on bit strings in product association

Abstract

The present invention relates to an application method of a distributed frequent item set based on a bit string in commodity association, and belongs to the field of data mining technology. The present invention improves the traditional FP-Growth association rule algorithm, converts the shopping list of commodities into the organizational form of a bit string, generates patterns through the BitwiseOperate method, and finally runs in the Spark big data processing framework to realize distributed computing. Compared with the traditional FP-Growth algorithm, the present invention mainly solves the problem that the memory is too large and the distribution is more complex when FPGrowth generates FPTree. The bit string form is used to save memory space, and the corresponding frequent item sets are quickly generated through bit operations. Finally, they are integrated in a distributed environment, which improves the mining efficiency of frequent item sets, and applies the algorithm to the analysis in the field of actual shopping commodities, providing a new research idea for frequent pattern mining.

Description

Technical Field

[0001] The invention relates to an application method of a distributed frequent item set based on bit strings in commodity association, belonging to the technical field of data mining. Background Art

[0002] In order to mine the potential commodity combination relationship in the commodity combination purchased by the user in the commodity shopping list, relevant data mining technology is needed, among which frequent item set mining in the field of association rules is a better method for dealing with such problems. Traditional frequent item sets include Apriori and FPGrowth, but Apriori has problems such as excessive data iteration and FPTree in FPGrowth may cause excessive memory usage. Therefore, the present invention converts each transaction into a related bit string by citing a data storage format similar to BitTable, and performs rapid pattern generation through the BitewiseOperate method. Secondly, it is improved in a distributed manner, by splitting it into multiple data structure trees similar to FPTree, distributing it to each partition to search for item sets, and obtaining all frequent item sets, which meets the requirements of rapid mining of commodity item sets. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a method for applying distributed frequent itemsets based on bit strings in commodity association, so as to solve the problem that the memory space occupied by FPTree in traditional FPGrowth is too large, and to provide a method for frequent itemset mining in partitioned traditional FPTree, which is finally applied to commodity association applications.

[0004] The technical solution of the present invention is: a method for applying distributed frequent item sets based on bit strings in commodity association, and the specific steps are:

[0005] Step 1: Read the shopping list of all items from the distributed database into memory and set the minimum support threshold.

[0006] Step 2: Get the absolute minimum support according to the minimum support threshold, aggregate the different items in the shopping list, remove the items with less than the minimum support to get all the frequently appearing 1-item sets, and sort them in descending order according to the support size, and then number the items from large to small.

[0007] Step 3: Traverse each shopping list in the product shopping list again and take the intersection with the full set of frequently appearing product 1 items in Step 2, and arrange the results of the intersection in the order of the product items in Step 2.

[0008] Step 4: Recursively generate n-1 commodity transaction items from the reserved commodity items in Step 3 from the front to the back, and perform Hash partitioning based on the tail item of the transaction, dividing each generated commodity prefix transaction item into the corresponding partition.

[0009] Step 5: Partition the commodity transaction items assigned to each partition according to different tail items, and generate multiple sub-FPTrees for each partition according to different transaction items.

[0010] Step 6: In each partition, traverse each branch of each sub-FPTree in turn, map according to the full set of products in each sub-FPTree, get the bit string to which each branch belongs, use BitwiseOperate operation on the bit string in a fast mode, and finally get all the product transaction sub-items. After performing the above operation on each sub-FPTree in each partition, the output is all the frequently appearing product items.

[0011] The Step 1 is specifically as follows:

[0012] Step 1.1: Create a SparkConf object, set the number of partitions in the object, set the memory size, the number of nodes, the master node, and other necessary parameters.

[0013] Step 1.2: Generate a SparkContext object based on the SparkConf object. This object is the interface for communication between the application and the Spark cluster.

[0014] Step 1.3: Use SparkContext to read the original shopping list from HDFS into memory and specify the number of partitions for parallel reading.

[0015] Step 1.4: Set the minimum support threshold min_sup, and satisfy 0<=|min_sup|<=1.

[0016] The Step 2 is specifically as follows:

[0017] Step 2.1: Read each product list transaction in turn, split the product items in each product transaction according to the separator, aggregate the same product items, and sort them in descending order from large to small according to the number of times the items appear.

[0018] Step 2.2: Calculate the absolute minimum support min_count by multiplying the total number db_count in the product list by the minimum support threshold min_sup.

[0019] Step 2.3: Filter the sorted items and remove the items with less than min_count to obtain the set of all frequent items.

[0020] Step 2.3: Number all frequent item sets.

[0021] The Step 3 is specifically as follows:

[0022] Step 3.1: Read each product shopping transaction in the original product shopping list transaction set, and take the intersection of each product shopping transaction and the sorted frequent 1 product transaction item.

[0023] Step 3.2: Rearrange the intersection results in the order of the frequent commodity transaction set.

[0024] The Step 4 is specifically as follows:

[0025] Step 4.1: Recursively generate a transaction set of n-1 commodity prefixes by taking the intersection and arranging the transactions from back to front.

[0026] Step 4.2: Scan the prefix commodity transaction set generated in each commodity transaction in turn, perform hash partitioning according to the item number in the tail item set of the commodity transaction set, and divide all prefix commodity transactions into each partition.

[0027] The Step 5 is specifically as follows:

[0028] Step 5.1: Further sort out the commodity prefix transactions in each partition and aggregate the items with the same tail.

[0029] Step 5.2: Convert all commodity transaction items with the same tail into tree branches and add them to each tree in turn.

[0030] Step 5.3: If the item encountered when adding to the branch exists, the support is added. If the item of the branch does not exist, it is added directly. The process stops when all the commodity transaction items with the same prefix of the tail items are extracted, and multiple sub-FPTrees are obtained.

[0031] The Step 6 is specifically as follows:

[0032] Step 6.1: Get the full item set list of each sub-FPTree, where each item is represented by <product item, support>.

[0033] Step 6.2: Eliminate items in the list of partial product sets whose support is less than the absolute minimum support.

[0034] Step 6.3: Extract each branch and take the intersection with the local product list. If it exists, it is 1, and if it does not exist, it is 0. Convert each branch into multiple bit strings.

[0035] Step 6.4: Perform BitwiseOperate operation on each bit string to obtain multiple substrings based on commodity transactions.

[0036] Step 6.5: Aggregate the substrings, add the support corresponding to the substrings, filter out the substrings with less than the minimum support, map the remaining substrings with the local product set, and obtain the full set in the current sub-FPTree.

[0037] Step 6.6: Operate all sub-FPTrees in sequence to obtain all frequent product item sets.

[0038] The beneficial effects of the present invention are as follows: the present invention improves the FPGrowth algorithm by adding the bit operation method to FPTree, and finally splits it. Compared with FP-Growth, the present invention mainly solves the problem that the storage of FPTree requires too much memory and is not suitable for partitioning in a distributed system. By introducing the bit string, the problem that the original types of different items need to be stored too large is solved, and the tree operation is performed in different partitions, which is suitable for parallelization, so that the combination relationship between commodities can be found more quickly, thereby finding the association between commodities. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of the steps of the present invention;

[0040] Figure 2 It is the Spark architecture diagram of the present invention;

[0041] Figure 3 It is the partition sub-FPTree graph of the present invention;

[0042] Figure 4 yes Figure 3 Sub-FPTree diagram of the Q5 partition. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical scheme and advantages of the present invention more specific, the present invention will be further described in detail below in conjunction with the accompanying drawings. Among them, the examples described below are only part of the embodiments of the present invention, rather than all the embodiments. Based on the examples in the present invention, they all belong to the scope of protection of the present invention.

[0044] Example 1: Figure 1-2 As shown in FIG. 1 , a method for applying a distributed frequent item set based on bit strings in commodity association is shown in FIG. 1 . The specific steps are as follows:

[0045] Step 1: Read the shopping list of all items from the distributed database into memory and set the minimum support threshold.

[0046] Step 2: Get the absolute minimum support based on the minimum support threshold, aggregate the different items in the shopping list, remove the items with less than the minimum support to get all the frequently appearing items, and sort them in descending order according to the support size, and then number the items from large to small.

[0047] Step 3: Traverse each shopping list in the product shopping list again and take the intersection with the full set of frequently appearing product 1 items in Step 2, and arrange the results of the intersection in the order of the product items in Step 2.

[0048] Step 4: Recursively generate n-1 commodity transaction items from the reserved commodity items in Step 3 from the front to the back, and perform Hash partitioning based on the tail item of the transaction, dividing each generated commodity prefix transaction item into the corresponding partition.

[0049] Step 5: Partition the commodity transaction items assigned to each partition according to different tail items, and generate multiple sub-FPTrees for each partition according to different transaction items.

[0050] Step 6: In each partition, traverse each branch of each sub-FPTree in turn, map according to the full set of products in each sub-FPTree, get the bit string to which each branch belongs, use BitwiseOperate operation on the bit string in a fast mode, and finally get all the product transaction sub-items. After performing the above operation on each sub-FPTree in each partition, the output is all the frequently appearing product items.

[0051] The Step 1 is specifically as follows:

[0052] Step 1.1: Create a SparkConf object, set the number of partitions in the object, set the memory size, the number of nodes, the master node, and other necessary parameters.

[0053] Step 1.2: Generate a SparkContext object based on the SparkConf object. This object is the interface for communication between the application and the Spark cluster.

[0054] Step 1.3: Use SparkContext to read the original shopping list from HDFS into memory and specify the number of partitions for parallel reading.

[0055] Step 1.4: Set the minimum support threshold min_sup, and satisfy 0<=|min_sup|<=1.

[0056] The Step 2 is specifically as follows:

[0057] Step 2.1: Read each product list transaction in turn, split the product items in each product transaction according to the separator, aggregate the same product items, and sort them in descending order from large to small according to the number of times the items appear.

[0058] Step 2.2: Calculate the absolute minimum support min_count by multiplying the total number db_count in the product list by the minimum support threshold min_sup.

[0059] Step 2.3: Filter the sorted items and remove the items with less than min_count to obtain the set of all frequent items.

[0060] Step 2.3: Number all frequent item sets.

[0061] The Step 3 is specifically as follows:

[0062] Step 3.1: Read each product shopping transaction in the original product shopping list transaction set, and take the intersection of each product shopping transaction and the sorted frequent 1 product transaction item.

[0063] Step 3.2: Rearrange the intersection results in the order of the frequent commodity transaction set.

[0064] The Step 4 is specifically as follows:

[0065] Step 4.1: Recursively generate a transaction set of n-1 commodity prefixes by taking the intersection and arranging the transactions from back to front.

[0066] Step 4.2: Scan the prefix commodity transaction set generated in each commodity transaction in turn, perform hash partitioning according to the item number in the tail item set of the commodity transaction set, and divide all prefix commodity transactions into each partition.

[0067] The Step 5 is specifically as follows:

[0068] Step 5.1: Further sort out the commodity prefix transactions in each partition and aggregate the items with the same tail.

[0069] Step 5.2: Convert all commodity transaction items with the same tail into tree branches and add them to each tree in turn.

[0070] Step 5.3: If the item encountered when adding to the branch exists, the support is added. If the item of the branch does not exist, it is added directly. The process stops when all the commodity transaction items with the same prefix of the tail items are extracted, and multiple sub-FPTrees are obtained.

[0071] The Step 6 is specifically as follows:

[0072] Step 6.1: Get the full item set list of each sub-FPTree, where each item is represented by <product item, support>.

[0073] Step 6.2: Eliminate items whose support is less than the absolute minimum support in the list of the local product set.

[0074] Step 6.3: Extract each branch and take the intersection with the local product list. If it exists, it is 1, and if it does not exist, it is 0. Convert each branch into multiple bit strings.

[0075] Step 6.4: Perform BitwiseOperate operation on each bit string to obtain multiple substrings based on commodity transactions.

[0076] Step 6.5: Aggregate the substrings, add the support corresponding to the substrings, filter out the substrings with less than the minimum support, map the remaining substrings with the local product set, and obtain the full set in the current sub-FPTree.

[0077] Step 6.6: Operate all sub-FPTrees in sequence to obtain all frequent product item sets.

[0078] The following is a further example based on the above technical solution:

[0079] Take an intermittent shopping list as an example, as shown in Table 1.

[0080]

[0081]

[0082] Table 1: Product purchase list

[0083] In order to better represent it in the following, each commodity item is represented by a letter, and the corresponding relationship is shown in Table 2 below, specifically {I1=>bread, I2=>milk, I3=>eggs, I4=>cheese, I5=>juice, I6=>cola, I7=>cheese, I8=>potatoes}.

[0084]

[0085] Table 2: Original transaction and sorting table

[0086] The original commodity transaction set is the left part of Table 2. It can be seen from the table that there are 9 transactions in total. Assuming that the current minimum support threshold min_sup is 0.2, the absolute minimum support is 2, that is, the minimum support of the frequent item set needs to be greater than 2.

[0087] Items are extracted for each commodity transaction, and the commodity items with the same items are aggregated and added. Different commodity items are arranged in descending order of support, and commodity items with support less than 2 are removed from the total table. The results are shown in Table 3.

[0088] <![CDATA[I2:7,I1:6,I4:2,I5:2,I6:2,I7:1,I8:1]]> <![CDATA[I2:7,I1:6,I4:2,I5:2,I6:2]]>

[0089] Table 3: Aggregation sorting table

[0090] From Table 3, we can get that the full set of frequent 1-item product sets that need to be retained is S1 = {I2, I1, I3, I4, I5}. The retained full product item set S1 is intersected with the item set in the original product transaction set and arranged in the order of the frequent 1-item product set S1. The final result is shown on the right side of Table 2.

[0091] Secondly, for each filtered and sorted commodity transaction item, generate its prefix item: recursively generate the first n-1 items from the commodity transaction each time, for example, {I2, I1, I5} generates {I2, I1, I5}, {I2, I1} and {I2} in turn based on these three items; the suffix items they represent are I5, I1 and I2 respectively.

[0092] Assume that 5 partitions are given, and the current commodity transaction item is composed of 5 different commodity items. In this way, each commodity item can be assigned to a different partition and each partition has only one spanning tree. The numbers representing the commodity items are hash partitioned. According to their support, they are numbered, and I2 is numbered 1, I1 is numbered 2, and so on. Now, Hash the modulus of 1 in I2 to get 1, that is, I2 can be assigned to the Q1 partition, I1 to the Q2 partition, etc. The specific transaction item partitions are shown in Table 4.

[0093] Sort Item Set Q1 Q2 Q3 Q4 Q5 I2, I1, I5 I2 I2, I1 I2, I1, I5 I2, I4 I2 I2, I4 I2, I3 I2 I2, I3 I2, I1, I4 I2 I2, I1 I2, I1, I4 I1, I3 I1 I1, I3 I2, I3 I2 I2, I3 I1, I3 I1 I1, I3 I2, I1, I3, I5 I2 I2, I1 <![CDATA[I2,I 1 ,I3]]> I2, I1, I3, I5 I2, I1, I3 I2 I2, I1 I2, I1, I5

[0094] Table 4: Transaction item partition table

[0095] According to the prefix items assigned to each partition, the sub-FPTree is constructed for the items in the item set in turn, and multiple sub-FPTrees are obtained in each partition. In the current case, each item is assigned to a different partition and there is only one sub-FPTree. The multiple sub-FPTrees obtained in each partition and the partition situation are as follows: Figure 3shown.

[0096] After obtaining multiple sub-FPTrees in each partition, frequent item sets can be mined. For each sub-FPTree tree, the full set of this local tree needs to be obtained, such as Figure 4 As shown, the full set obtained by Q5 is {I2:3, I1:3, I3:1, I5:3}, where the support of I3 is less than 2 and can be represented as 0 in the subsequent bit string representation.

[0097] By taking out the full item set {I2, I1, I3, I5} and two branches {I2, I1, I5} and {I2, I1, I3, I5}, we get two strings {1101:1} and {1101:2}, where I3 support is less than 2 and is represented as 0 in bit string representation.

[0098] By aggregating "1101" we get {1101:3}, and then we can do BitwiseOperate on "1101" to get Si={1001, 0101, 1101, 0001}, that is, we get the set of all substrings. We can do bit string reduction on the substrings to get {I2, I5}, {I1, I5}, {I2, I1, I5}, {I5} are all sub-item sets of 3, that is, {milk, juice}, {bread, juice}, {milk, bread, juice}

[0099] Corresponding operations can be performed on other subtrees in other partitions to obtain all frequent product item sets.

[0100] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A method for applying distributed frequent itemsets based on bit strings in commodity association, characterized by: Step 1: Read the shopping list of all items from the distributed database into memory and set the minimum support threshold; Step 2: Obtain the absolute minimum support according to the minimum support threshold, aggregate different items in the shopping list, remove items with less than the minimum support to obtain all frequently occurring 1-item sets, and sort them in descending order according to the support, and then number the items from large to small; Step 3: Traverse each shopping list in the product shopping list again and take the intersection with the set of frequently appearing product 1 items in Step 2, and arrange the intersection results in the order of the product items in Step 2; Step 4: Recursively generate n-1 commodity transaction items from the reserved commodity items in Step 3 from the front to the back, and perform Hash partitioning based on the tail item of the transaction, and divide each generated commodity prefix transaction item into the corresponding partition; Step 5: Partition the commodity transaction items assigned to each partition according to different tail items, and generate multiple sub-FPTrees for each partition according to different transaction items; Step 6: In each partition, traverse each branch of each sub-FPTree in turn, map according to the full set of products in each sub-FPTree, get the bit string to which each branch belongs, use BitwiseOperate operation on the bit string in a fast mode, and finally get all the product transaction sub-items. After performing the above operation on each sub-FPTree in each partition, the output is all the frequently appearing product items.

2. The method for applying the bit string-based distributed frequent item set in commodity association according to claim 1 is characterized in that: The Step 1 is specifically as follows: Step 1.1: Create a SparkConf object, set the number of partitions in the object, set the memory size, the number of nodes to run, and the master node to run; Step 1.2: Generate a SparkContext object based on the SparkConf object. This object is the interface for communication between the application and the Spark cluster. Step 1.3: Use SparkContext to read the original shopping list from HDFS into memory and specify the number of partitions for parallel reading. Step 1.4: Set the minimum support threshold min_sup, and satisfy 0<=|min_sup|<=1.

3. The method for applying the bit string-based distributed frequent item set in commodity association according to claim 1 is characterized in that: The Step 2 is specifically as follows: Step 2.1: Read each product list transaction in turn, split the product items in each product transaction according to the separator, aggregate the same product items, and sort them in descending order according to the number of times the items appear; Step 2.2: Calculate the absolute minimum support min_count by multiplying the total number db_count in the product list by the minimum support threshold min_sup; Step 2.3: Filter the sorted items and remove the items with less than min_count to obtain the set of all frequent items. Step 2.3: Number all frequent item sets.

4. The method for applying the bit string-based distributed frequent item set in commodity association according to claim 1, characterized in that: The Step 3 is specifically as follows: Step 3.1: Read each shopping transaction in the original shopping list transaction set, and take the intersection of each shopping transaction and the sorted frequent 1 item transaction item; Step 3.2: Rearrange the intersection results in the order of the frequent commodity transaction set.

5. The method for applying the bit string-based distributed frequent item set in commodity association according to claim 1, characterized in that: The Step 4 is specifically as follows: Step 4.1: Recursively generate a transaction set of n-1 commodity prefixes by taking the intersection and arranging the transactions from back to front; Step 4.2: Scan the prefix commodity transaction set generated in each commodity transaction in turn, perform hash partitioning according to the item number in the tail item set of the commodity transaction set, and divide all prefix commodity transactions into each partition.

6. The method for applying the bit string-based distributed frequent item set in commodity association according to claim 1, characterized in that: The Step 5 is specifically as follows: Step 5.1: Further sort out the commodity prefix transactions in each partition and aggregate the items with the same tail; Step 5.2: Convert all commodity transaction items with the same tail into tree branches and add them to each tree in turn; Step 5.3: If the item encountered when adding to the branch exists, the support is added. If the item of the branch does not exist, it is added directly. The process stops when all the commodity transaction items with the same prefix of the tail items are extracted, and multiple sub-FPTrees are obtained.

7. The method for applying the bit string-based distributed frequent item set in commodity association according to claim 1, characterized in that: The Step 6 is specifically as follows: Step 6.1: Get the full item set list of each sub-FPTree, where each item is represented by <item, support>; Step 6.2: Eliminate items whose support is less than the absolute minimum support in the list of the local product set; Step 6.3: Extract each branch and take the intersection with the local product list. If it exists, it is 1, and if it does not exist, it is 0. Convert each branch into multiple bit strings. Step 6.4: Perform BitwiseOperate operation on each bit string to obtain multiple substrings based on commodity transactions; Step 6.5: Aggregate the substrings, add the support corresponding to the substrings, filter out the substrings with less than the minimum support, map the remaining substrings with the local product set, and obtain the full set in the current sub-FPTree; Step 6.6: Operate all sub-FPTrees in sequence to obtain all frequent product item sets.

Images