An algorithm for mining high-utility and high-occupancy itemsets in massive data based on suffix partitioning
By using a suffix-based efficient high-occupancy-ratio itemset mining algorithm, the problem of ignoring itemset weights in frequent itemset mining is solved, achieving efficient mining on large-scale datasets and improving algorithm performance and the accuracy of finding high-occupancy itemsets.
Patent Information
- Application Number
- CN202210060831.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-19
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2042-01-19
AI Technical Summary
Existing frequent itemset mining methods ignore the weight features of itemsets, resulting in the neglect of high-utility itemsets. Furthermore, the computational performance of existing algorithms degrades in massive datasets, making it difficult to effectively mine high-utility itemsets with high occupancy ratios on large-scale datasets.
An efficient high-utility-occupancy-ratio itemset mining algorithm based on suffix partitioning is adopted. By partitioning and vertically storing data in the preprocessing stage, combined with a bidirectional stack structure and pruning strategy, the itemset mining process is optimized. This includes downward closure of support attribute, downward closure of utility-occupancy-ratio attribute, 3-itemset full pruning strategy, and itemset link optimization strategy to achieve efficient mining.
This algorithm effectively computes high-utilization-ratio itemsets in large-scale datasets, reducing space and time overhead and improving algorithm efficiency. It enables efficient mining on large-scale datasets to discover practically meaningful high-utilization-ratio itemsets.
Smart Images

Figure CN114528332B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of massive data mining and processing, and relates to a method for mining high-utility and high-occupancy ratio item sets in retailer and online e-commerce transaction data. Background Art
[0002] Pattern (item set) mining on massive data is one of the current research hotspots in the field of data mining. Nowadays, data mining has been widely used in fields such as shopping basket analysis, marketing, financial transactions and biological analysis, and has increasingly become a research hotspot that has attracted widespread attention in various fields at home and abroad.
[0003] Frequent itemset mining (FIM) returns a large number of frequently occurring itemsets in a data set, from which the associations and frequent patterns between itemsets are discovered. However, frequent pattern mining only considers whether an item set appears in a transaction, ignoring the number of itemsets in the transaction and their weights. Therefore, frequent pattern mining may obtain very common but low-utility itemsets, miss rare but high-profit itemsets, and often have no practical significance. For example, in a retailer transaction data set, milk and bread are frequent itemsets that frequently appear in the shopping basket, but their utility values (unit profits) are often relatively low; while caviar and champagne are rare infrequent itemsets in the shopping basket, but their utility values are often high. Therefore, FIM often ignores infrequent itemsets that can bring huge profit utilities, but these ignored high-utility itemsets can often bring huge profit utilities to retailers.
[0004] High Utility Itemset Mining (HUIM) is an extension of frequent itemset mining that considers itemset weights. Based on FIM, weights are assigned to items in the transaction table, and each item is defined to have its internal utility (such as the number of goods sold) and external utility (such as profit, importance, etc.). The utility value of an item is the product of internal utility and external utility, representing the total profit or contribution value of the pattern. HUIM can discover the association and correlation between patterns in large transaction or relational databases, and can help many business decisions such as classification design, cross-marketing, and customer shopping habits analysis. However, HUIM ignores the quantitative characteristics of the item set, and the high-utility item set mined may be a special case that occurs by chance, so it is inappropriate to use it as a sales decision.
[0005] Nowadays, most studies use either frequency (such as frequent itemset mining) or utility (such as high-utility itemset mining) as the main measurement criteria. On the one hand, using these two measurement methods alone has its own limitations. Itemsets with high frequency may have low utility values, while itemsets with high utility values often have low frequency and may be accidental exceptions without practical value. If these itemsets are recommended to users, they have no practical significance and may even be misleading. On the other hand, existing algorithms can usually only calculate small and medium-sized data sets of 100,000 transactions, and as the amount of data increases, the performance of the algorithm will be greatly reduced. In addition, considering the utility value alone cannot reflect the overall relationship between the itemset and the transaction table. It is necessary to comprehensively consider the occupancy or importance of the itemset in supporting transactions from the aspects of frequency and utility. Summary of the invention
[0006] In order to overcome the above problems existing in the prior art, the present invention proposes a high-utility and high-occupancy ratio itemset mining algorithm based on suffix partitioning in massive data.
[0007] The technical solution adopted by the present invention to solve the technical problem is: a high-utility and high-occupancy ratio itemset mining algorithm based on suffix partitioning in massive data, which specifically includes two stages: preprocessing and itemset mining. The preprocessing stage specifically partitions the data set according to the suffix and stores it as a vertical structure; the specific process of the itemset mining stage is:
[0008] For itemsets whose length is not greater than 2, their utility occupancy ratio is directly calculated to determine whether they are high-utility occupancy ratio itemsets. Itemsets with low-utility occupancy ratios are directly pruned.
[0009] For item sets with a length greater than 2, a linked list-based bidirectional stack structure is constructed to store the extended items in the partition. The items in the bidirectional stack are used to build a set enumeration tree. The item sets are traversed and mined in the set enumeration tree according to the depth-first search order. The item sets are pruned according to the support downward closure property, the utility occupancy ratio downward closure property, and the 3-item set complete pruning strategy. The item set link optimization strategy and the remaining TID cross count optimization strategy are used for mining.
[0010] The above-mentioned high-utility and high-occupancy-ratio itemset mining algorithm based on suffix partitioning in massive data, the partitioning process is specifically: performing sequence scanning on a given data set T, partitioning all subsequences of the data set T, and putting itemsets with the same suffix into the same partition.
[0011] The above-mentioned high-utility and high-occupancy-ratio item set mining algorithm based on suffix partitioning in massive data, the vertical storage process is specifically as follows: in each partition, a vertical format utility list is constructed for each item, and the utility list is arranged in descending order according to the support corresponding to the item, and the items whose support is not less than a preset support threshold and their corresponding utility lists are read into the memory in sequence.
[0012] In the above-mentioned high-utility and high-occupancy ratio itemset mining algorithm based on suffix partitioning in massive data, the support downward closure property is specifically as follows: when the upper bound of the support of a partition is less than a preset minimum support threshold, the partition has no high-utility-occupancy ratio itemset that meets the support threshold, and the partition is pruned; when the support of an itemset is less than a preset minimum support threshold, the itemset and its superset do not meet the support threshold, and the itemset and its superset are pruned.
[0013] In the above-mentioned high-utility and high-occupancy ratio item set mining algorithm based on suffix partitioning in massive data, the utility occupancy ratio downward closure property is specifically: the utility occupancy ratio upper bound The calculation formula is:
[0014]
[0015] Where t is a transaction containing the itemset X, is the transaction utility value of item set X in the partition, is the utility value of transaction t where X is located, is the pre-set minimum support threshold, n is the number of transactions in the dataset, is the utility occupancy ratio of item set X in transaction t, is the residual utility occupancy ratio of item set X in transaction t;
[0016]
[0017] in, represents the jth item in the item set X, represents the utility value of the jth item in item set X in transaction t, is the subscript corresponding to the last term in X, is the utility value of transaction t where X is located;
[0018]
[0019] in, Representing a transaction The jth item in Representing a transaction The utility value of the jth item in , , is the subscript of the next term after the last term of X, is the utility value of transaction t where X is located;
[0020] When the upper bound of the utility occupancy ratio is less than the preset utility occupancy ratio threshold, X and its superset do not meet the utility occupancy ratio threshold, and the item set X and its superset are pruned; when the upper bound of the utility occupancy ratio of the first item in the partition is less than the preset utility occupancy ratio threshold, the partition has no high utility occupancy ratio item set that meets the utility occupancy ratio threshold, and the partition is pruned.
[0021] In the above-mentioned high-utility and high-occupancy-ratio itemset mining algorithm based on suffix partitioning in massive data, the bidirectional stack structure based on linked lists includes a base sequence and an operand sequence.
[0022] The high-utility and high-occupancy-ratio itemset mining algorithm based on suffix partitioning in the above-mentioned massive data, the bidirectional stack structure storage process based on the linked list includes:
[0023] A. The items in the partition are sorted in descending order of support, and every item loaded into memory is frequent;
[0024] B. The first item in the partition is stored in the cardinality sequence, and subsequent items are stored in the operand sequence in order;
[0025] C. First, put the items in the operand sequence into the bidirectional stack structure in descending order from bottom to top, and then put the items in the radix sequence into the bidirectional stack structure;
[0026] D. Extract items from the bidirectional stack structure from top to bottom, and the sequence output from the bidirectional stack structure is used in the subsequent item set mining process and pruning process.
[0027] The high-utility and high-occupancy ratio itemset mining algorithm based on suffix partitioning in the above-mentioned massive data, the item set link optimization strategy is specifically as follows: when mining an item set with a length greater than 3, link the item set to its ancestor item set in the previous partition and the parent item set in the current partition, and obtain the support information of the ancestor item set and the parent item set. If the ancestor item set is a frequent item set, then the support information of both the ancestor item set and the parent item set is obtained. The list is cross-counted to obtain the support of the item set and determine whether to generate and calculate the item set; if the ancestor item set or parent item set is not frequent, the item set and its superset can be pruned in the set enumeration tree.
[0028] The above-mentioned high-utility and high-occupancy ratio itemset mining algorithm based on suffix partitioning in the massive data, and the specific processing process based on the remaining TID cross count optimization strategy include:
[0029] Step 2.1: Initialization and Point to and The first element of is an index pointing to the position of an item in the itemset;
[0030] Step 2.2: For the remaining Upper Bound Initialize, , where n is the number of transactions in the dataset, is the minimum support threshold, Representing Item Sets The length of the corresponding tid list, Representing Item Sets The length of the corresponding tid list, and ;
[0031] Step 2.3: Judgement The size of , then there is no need to perform cross counting operations, and directly and its superset for pruning; , then perform the cross counting operation;
[0032] Step 2.4: During the cross counting process, compare and The tid pointed to, express The tid pointed to by the index, express The tid pointed to by the index: 1) , then the tid pointed to by the index is and An element in the intersection of the two, so the public tid pointed to by the index is stored in middle, and Move one position backward respectively; 2) , then ind1 moves back one position; 3) ,but ;
[0033] Step 2.5: Repeat step 2.4 until <0 or and One of the two points to At the end of the set, the cross counting process ends; then according to The size of a set determines the itemset Whether the support of meets the threshold, if If the usage is frequent, continue to calculate its utility occupancy ratio, otherwise directly and its superset for pruning.
[0034] The beneficial effects of the present invention are:
[0035] (1) This patent proposes a new high-utility high-occupancy itemset mining algorithm - Suffix-based High-utility Occupancy Itemset Mining Algorithm (SHO algorithm), which can perform effective calculations on large-scale massive data;
[0036] (2) A suffix-based partitioning strategy is proposed to divide the transaction table into several suffix-based partitions and store them vertically. These partitions do not overlap with each other, and the union of all partitions is the entire data set. Each partition is small enough for the allocated memory, so that the partition can be read into the memory in its entirety. The support and utility occupancy ratio of the item set can be calculated in a certain partition without traversing the entire data set, which greatly saves the space and time overhead of the algorithm.
[0037] (3) An item set link optimization strategy and a residual TID cross count optimization strategy are proposed to prune item sets as early as possible.
[0038] (4) The pruning strategy and the upper bound of the utility occupancy ratio are designed so that the utility occupancy ratio satisfies the downward closure property, thereby effectively pruning away a large number of hopeless item sets and reducing the computational overhead. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] The present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0040] Figure 1 It is the algorithm flow chart of the present invention;
[0041] Figure 2 The preprocessing stage and partition reading flow chart of the algorithm of the present invention;
[0042] Figure 3 The transaction table database T in the embodiment of the present invention;
[0043] Figure 4 is the external utility value corresponding to each item in the embodiment of the present invention;
[0044] Figure 5 This is a schematic diagram of the result of dividing a data set according to suffixes according to an embodiment of the present invention;
[0045] Figure 6 It is a vertical storage structure according to an embodiment of the present invention;
[0046] Figure 7 A linked list-based bidirectional stack structure built for partition P6;
[0047] Figure 8 The depth-first growth enumeration tree structure of partition P5;
[0048] Fig. 9 This is the experimental result diagram of the impact of support changes on algorithm performance;
[0049] Fig.10 This is the experimental result diagram of the impact of changes in utility occupancy ratio on algorithm performance. DETAILED DESCRIPTION
[0050] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0051] This embodiment discloses an itemset mining algorithm with high utility and high occupancy ratio based on suffix partitioning in massive data, and the algorithm includes the following steps:
[0052] Step 1: Preprocess the dataset;
[0053] Step 2: Prune and mine each partition to obtain the final high-utility and high-occupancy ratio item set;
[0054] The preprocessing is to partition the data set according to the suffix and store it in a vertical structure. The specific process is as follows Figure 2 The specific process of partition processing is: perform a sequence scan on a given data set, assuming that transaction t contains different items, namely ,in , is the jth item in t, Yes Internal utility in transaction t. Subsequence will be divided into When the processing of t is completed, w subsequences will be placed in the corresponding partition. When the sequence scanning on the data set T is completed, m suffix-based partitions will be obtained. These partitions do not overlap with each other, and their union is the data set T. Therefore, the item set support and utility value can be calculated in a certain suffix partition, which is convenient for item set mining.
[0055] The specific process of vertical storage is: in the partition, a vertical utility list is constructed for each item, that is, a list of all tids where the item appears and its corresponding utility value. The utility list will be sorted in descending order according to its size, and its size is equal to the support count corresponding to the item. The first item in the partition has the largest support count, and the item at the last position in the partition has the smallest support. The items and their utility lists are read into the memory in sequence until the support count of an item is less than the support threshold specified by the user. Since the items are stored in descending order of support, the support of the items after the current item is less than the given support threshold at this time. The items with support less than the threshold are not high utility occupancy ratio item sets, so these item sets will be pruned.
[0056] In this embodiment, a Figure 3 The dataset shown in the figure is used in the following steps. Figure 3 The data set shown is used as an example to describe Figure 4 is the external utility value of each item in the data set. According to the above partitioning process, Figure 3 The data set is divided into seven partitions P1-P7, such as Figure 5 As shown, the partitioned data is stored in a vertical structure, such as Figure 6 As shown, in Figure 6 In (a), each partition stores items and their corresponding utility list information in a vertical format. In partition P1, the first record is i1={(t1,4), (t2,5), (t3,2), (t4,2),(t6,2), (t9,1), (t 10 ,3)}, which means that item i1 appears in 7 transactions in the data set, where (t1,4) means that the internal utility value of item i1 in transaction t1 is 4, that is, iu(i1, t1)=4; Figure 6 As shown in (b), the utility lists in each partition are sorted in descending order by support count. For example, in partition P3, the support counts of items i1, i2, and i3 are 4, 3, and 5, respectively. After sorting by support, the first record is utility-list(i3)={(t3,1),(t4,3), (t5,5), (t6,3), (t9,2)}, followed by item i1, and item i2 is at the end.
[0057] The specific steps of the itemset mining phase are as follows: Figure 1 As shown, it should be noted that Figure 1 X1, X2, and X3 refer to the itemsets currently being traversed. In the item set mining phase of step 2, each partition is read from disk into memory in turn. First, the 1-item set ( Figure 1 The corresponding X1) and 2-item set ( Figure 1 Calculate the corresponding X2 in the table to determine whether it is a high-utility and high-occupancy ratio item set; for item sets with a length greater than 2 ( Figure 1 Corresponding to X3 in the figure, a bidirectional stack structure based on a linked list is constructed to store the extended items in the partition. The items in the bidirectional stack are used to build a set enumeration tree. The item sets are traversed and mined in the set enumeration tree according to the depth-first search order. The item sets are pruned according to the support downward closure attribute, the utility occupancy ratio downward closure attribute, and the 3-item set complete pruning strategy. The item set link optimization strategy and the remaining TID cross count optimization strategy are used for mining.
[0058] Bidirectional stack structure (such as Figure 7 As shown in the figure, it consists of two parts: the cardinality sequence and the operand sequence. First, the items in the partition are arranged in descending order of support, and each item loaded into the memory is frequent. The first item in the partition will be stored in the cardinality sequence on the left, and the subsequent 1-item sets will be placed in the operand sequence on the right in order. In the operand sequence, the item with the lowest support is stored at the tail, and the item with the highest support is stored at the head; then, the items in the operand sequence are placed in the bidirectional stack structure first, and then the items in the cardinality sequence are placed in the bidirectional stack structure; finally, the items in the bidirectional stack structure are extracted from the top to the bottom, and the sequence output from the bidirectional stack structure can be used to generate the set enumeration tree and the subsequent item set mining process.
[0059] In this embodiment, Figure 5 The P6 partition in the structure is based on the linked list bidirectional stack structure. Figure 7 As shown, There are five different items in , middle The support of is less than the preset support threshold, so will not be read into memory. Won't appear Since the items are sorted in descending order according to their support, ,item The support count of is the largest, so it is placed in the cardinality sequence on the left. , , , will be placed into the operand sequence on the right in sequence; then the items in the operand sequence will be output in sequence; Located at the head of the operand sequence, first Output to the bidirectional stack structure, and then output , and . Then output the cardinality sequence And put it on the top of the bidirectional stack structure, thus obtaining the bidirectional stack structure of the partition. When constructing the set enumeration tree, the items in the bidirectional stack structure will be The bidirectional stack structure item list will be used to generate item sets with the same suffix in this partition. The bidirectional stack structure can logically organize the items contained in the partition according to the order of use. By using the bidirectional stack structure, the search space can be effectively reduced and the algorithm efficiency can be improved.
[0060] A detailed explanation of the specific pruning strategy and optimization strategy:
[0061] 1. Downward closure property of support
[0062] According to the results of data set partitioning and vertical storage, the first item in the partition is the suffix item. The upper bound of the maximum support count in the partition can be directly obtained through the suffix item. If the upper bound of the maximum support is less than the preset minimum support threshold , it is impossible to generate high-utility and high-occupancy ratio itemsets in this partition, so the partition can be skipped directly; when the support of the itemset is less than the preset minimum support threshold, the itemset and its superset do not meet the support threshold, and the itemset and its superset are pruned.
[0063] 2. Downward Closure Property of Utility Occupancy Ratio
[0064] Utility Occupancy Ratio Upper Bound The calculation formula is:
[0065]
[0066] Where t is a transaction containing the itemset X, is the transaction utility value of item set X in the partition, is the utility value of transaction t, is the pre-set minimum support threshold, n is the number of transactions in the dataset, is the utility occupancy ratio of item set X in transaction t, is the residual utility occupancy ratio of item set X in t;
[0067]
[0068] in, represents the jth item in the item set X, represents the utility value of the jth item in item set X in transaction t, is the subscript of the last term of X, is the utility value of transaction t where X is located;
[0069]
[0070] in, Representing a transaction The jth item in Representing a transaction The utility value of the jth item in , , is the subscript of the term after the last term of X, is the utility value of transaction t where X is located;
[0071] When the upper bound of the utility occupancy ratio is less than the preset utility occupancy ratio threshold, X and its superset do not meet the utility occupancy ratio threshold, and the item set X and its superset are pruned; when the upper bound of the utility occupancy ratio of the first item in the partition is less than the preset utility occupancy ratio threshold, the partition has no high utility occupancy ratio item set that meets the utility occupancy ratio threshold, and the partition is pruned.
[0072] 3. 3-itemset complete pruning strategy
[0073] In the partition P with item i as the suffix item i middle, is an item list containing m items. First, all 1-item sets and 2-item sets are mined through simple sequence scanning. Since both 1-item sets and 2-item sets are frequent, their utility occupancy ratio can be directly calculated and whether they are high utility occupancy ratio item sets can be determined; next, a 3-item set is generated by combining two 2-item sets, but before generating the 3-item set, it can be determined whether all its 2-subsets are frequent. If not, the candidate set can be skipped directly without cross counting and calculating its support and utility occupancy ratio; finally, the depth-first set enumeration tree is combined to mine item sets of length 3 or more in the running instance. Assume that the support threshold is 0 when constructing the set enumeration tree, that is, all 1-item sets in the partition are frequent and 1-item sets can be read into memory. The depth-first growth enumeration tree structure is as follows Figure 8 shown. Figure 8 contains information about five partitions from P1 to P5. 2, i3} and {i 2, i4} performs cross counting to generate {i 2, i 3, i4}, where the item set {i 2, i4} and {i 3, i4} has been calculated in the current partition P4, and the item set {i 2, i3} has been calculated in partition P3, so the property of support being closed downward can be used to calculate the 3-item set {i 2, i 3, i4} is completely pruned. If any of the 2-subsets is a non-frequent item set, there is no need to generate {i 2, i 3, i4}, avoiding waste of resources.
[0074] By using the set enumeration tree to prune the 3-candidate itemsets, the computational overhead of the cross-count can be greatly reduced, and many candidate itemsets can be skipped, thereby shortening the time overhead of itemset mining. The complete pruning strategy of 3-itemsets can not only reduce the search space, but also improve the efficiency of the algorithm.
[0075] 4. Itemset Link Optimization Strategy
[0076] Specifically, when the length of an itemset is greater than 2, the itemsets are mined sequentially by depth-first search (DFS). Although the designed 3-itemset complete pruning strategy can effectively prune 3-itemsets, when the number of layers of the set enumeration tree increases, its search space is still very large. Therefore, an itemset link pruning optimization strategy (LI strategy) is proposed to prune itemsets with a length greater than 3.
[0077] Ancestor itemset: Assume X is an itemset with a length greater than 3. , is the last item in item set X. The ancestor item set of item set X .
[0078] Parent itemset: Assume X is an itemset stored in the set enumeration tree, the itemset stored at the parent node of X is called the parent itemset of X.
[0079] In the process of depth-first search on the set enumeration tree, the item set X is generated by the parent item set. When determining whether to generate a k-item set X (k>3), the parent item set of X can be easily obtained. If the parent item set of item set X is not frequent, X and its superset will be directly pruned. On the contrary, if item set X is frequent, the utility list of the parent item set of item set X can be obtained, and the utility of the parent item set and ancestor item set of item set X can be calculated. The set performs cross counting, and then calculates and obtains the utility list of item set X to determine whether item set X is a high utility occupancy ratio item set. In the running example, the item set Is the parent set Generated, when mining itemsets, link X to , and the item set has been calculated in P3, according to Whether it is frequent enough to predict whether itemsets need to be generated ,like Frequent, then and of The set is crossed and counted to obtain The support of the item set is determined by the .
[0080] The main idea of this optimization strategy is to link the itemset with its two subsets. By deleting the last item, the itemset is logically connected to its ancestor itemset. When mining an itemset with a length greater than 3, the itemset is linked to its ancestor itemset in the previous partition and the parent itemset in the current partition to obtain the support information of the ancestor itemset and the parent itemset. If the ancestor itemset is a frequent itemset, the support information of both the ancestor itemset and the parent itemset is calculated. The list is cross-counted to obtain the support of the item set and determine whether to generate and calculate the item set; if the ancestor item set or parent item set is not frequent, the item set and its superset can be pruned in the set enumeration tree.
[0081] 5. Remaining Upper bound crossing counting strategy
[0082] The specific processing process based on the remaining TID cross count optimization strategy includes:
[0083] Step 2.1: Initialization and Point to and The first element of is an index pointing to the position of an item in the itemset;
[0084] Step 2.2: For the remaining Upper Bound Initialize, , where n is the number of transactions in the dataset, is the minimum support threshold, Representing Item Sets The length of the corresponding tid list, Representing Item Sets The length of the corresponding tid list, and ;
[0085] Step 2.3: Judgement The size of , then there is no need to perform cross counting operations, and directly and its superset for pruning; , then perform the cross counting operation;
[0086] Step 2.4: During the cross counting process, compare and The tid pointed to, express The tid pointed to by the index, express The tid pointed to by the index: 1) , then the tid pointed to by the index is and An element in the intersection of the two, so the public tid pointed to by the index is stored in middle, and Move one position backward respectively; 2) , then ind1 moves back one position; 3) ,but ;
[0087] Step 2.5: Repeat step 2.4 until <0 or and One of the two points to At the end of the set, the cross counting process ends; then according to The size of a set determines the itemset Whether the support of meets the threshold, if If the usage is frequent, continue to calculate its utility occupancy ratio, otherwise directly and its superset for pruning.
[0088] In this embodiment, given an item set X, by adding two different items a and b to X, the 1-extended item set of X can be obtained. and Similarly, by adding a and b to X, we get the 2-expanded itemset of X Θ(X) is the remainder of the itemset X In this embodiment, it is defined and They are and Corresponding tid list, | | and | |Respectively and The length of the corresponding tid list, and | | | |, is an index pointing to the position of an item in an itemset. In a partition, if itemset X is a 1-itemset, then and is a 2-item set, currently generated is a 3-item set. The collection can be directly read by the smaller support items Set as 2-itemset For example, when accessing partition P3, the itemset of Collections can be accessed directly through The corresponding Get .
[0089] When the length of the newly generated itemset is greater than 2, Optimize the cross counting operation. Assuming the minimum support count threshold is 4, , , then the remaining The optimization strategy process is as follows:
[0090] In this embodiment, during the cross counting process, first initialize and Point to and The first element of , Next, for the remaining Upper Bound Initialize, , A cross counting operation is required. In the first round of comparison, , , ,at this time , and Move back one position. In the second round of comparison, tid( , then Deposit Item Set of gather , and and Move one position backward respectively. In the third round of comparison, , , ,at this time , and then Move back one position. In the fourth round of comparison, , , ,Will Move back one position. In the fifth round of comparison, , , ,at this time .because Therefore, the cross counting process is stopped.
[0091] In summary, the introduction of the remaining Upper Bound The result of the intersection and union of two sets can be evaluated, and useless item sets can be stopped and discarded as early as possible. The larger the pre-set support threshold, the more calculation steps the optimization strategy skips, and the better the optimization effect.
[0092] Parameter Used value Support Threshold in Percentage(%)(syn) 0.3 ~ 0.9 Support Threshold in Percentage(%)(rea) 0.1~ 0.22 Utility-Occupancy Threshold in Percentage(%)(syn) 5 ~ 60 Utility-Occupancy Threshold in Percentage(%)(rea) 10 ~ 80 <![CDATA[Transaction Number(10 6 ) (his)]]> 0.1 ~100 Number of Items(syn) 500 ~ 4000 Average transaction width... 5 ~ 20
[0093] In the experiment, the performance of the high-utility and high-occupancy ratio itemset mining algorithm (SHO algorithm) based on suffix partitioning of the present invention is tested from two aspects: a preset support threshold (min_sup) and a preset utility occupancy ratio threshold (min_uo), and compared with the most advanced HUOPM algorithm and BA algorithm.
[0094] In order to study the impact of support threshold on the algorithm, this experiment uses three different types of data sets, including a real data set (Retail) and two synthetic data sets (T100KI1KD5 and T100KI1KD10). The performance of HUOPM, BA and SHO algorithms are compared for different support thresholds (min_sup). The experimental results are shown in Figure 2. Fig. 9 shown.
[0095] Depend on Fig. 9 It can be seen that with the increase of the support threshold, the running time of the three groups of experiments shows an overall downward trend, among which the running time of the BA algorithm and the HUOPM algorithm is relatively close. Fig. 9 In (a), given a data set n=100K, d=1000, w=10, where n represents the number of transactions in the data set, d represents the number of items in the data set, and w represents the average transaction length, the utility occupancy ratio threshold min_uo=0.3 is set. When the support threshold min_sup is 0.3%, the running time of the HUOPM algorithm is 64.9 times that of the SHO algorithm, and the running time of the BA algorithm is 56.8 times that of the SHO algorithm; as the support threshold min_sup increases, when the support threshold min_sup is 0.6%, the running time of the HUOPM algorithm is 328.2 times that of the SHO algorithm, and the running time of the BA algorithm is 189 times that of the SHO algorithm. This fully demonstrates that the performance of the SHO algorithm is significantly better than that of HUOPM and BA on the real data set and the above two synthetic data sets.
[0096] exist Fig. 9 In (b), given a dataset of n=100K, d=1000, w=5, and setting the utility occupancy ratio threshold min_uo=0.2, the SHO algorithm is compared with the HUOPM algorithm and the BA algorithm respectively. The SHO algorithm has the most obvious advantage when the support threshold min_sup=0.9%, which is 15.7 times that of the HUOPM algorithm, and when the support threshold min_sup=0.3%, it is 6.1 times that of the BA algorithm.
[0097] exist Fig. 9 In the real data set retail in (c), given the utility occupancy ratio threshold min_uo=0.1, the performance of the SHO algorithm is more stable, and its running time will not be greatly affected when the support is very small.
[0098] In order to study the impact of utility occupancy ratio on threshold algorithm, this experiment uses real data set (Retail) and two synthetic data sets (T100KI1KD5 and T100KI1KD10) for experiments. Given the support threshold min_sup=0.5%, the performance of HUOPM, BA and SHO algorithms are compared for different utility occupancy ratios (min_uo). The experimental results are as follows: Fig.10 shown.
[0099] exist Fig.10 In (a), given a data set n=100K, d=1000, w=10, where n represents the number of transactions in the data set, d represents the number of items in the data set, and w represents the average transaction length, the support threshold min_sup=0.5% is set. In the best case among each set of thresholds, the SHO algorithm is about 540.3 times faster than the HUOPM algorithm, and the SHO algorithm is about 240.2 times faster than the BA algorithm.
[0100] exist Fig.10 In (b), given a data set n=100K, d=1000, w=5, where n represents the number of transactions in the data set, d represents the number of items in the data set, and w represents the average transaction length, the support threshold min_sup=0.5% is set. The SHO algorithm is about 129.5 times faster than the HUOPM algorithm on average, and the SHO algorithm is about 81.5 times faster than the BA algorithm on average.
[0101] exist Fig.10 In (c), a real retailer dataset is used for testing. Given a support threshold min_sup=0.16%, the runtimes of the three algorithms show an overall downward trend as the utility occupancy ratio threshold min_uo increases, and SHO performs better when the utility occupancy ratio threshold min_uo is larger.
[0102] On the one hand, as the utility occupancy ratio threshold min_uo increases, the mining time fluctuates less; on the other hand, the impact of the utility occupancy ratio threshold min_uo on the mining time is much smaller than the impact of the support threshold min_sup on the mining time. The reason is that the utility occupancy ratio is a more comprehensive transaction feature attribute that can objectively reflect the occupancy of the item set in the entire data set. Therefore, the high utility occupancy ratio pattern calculated in different data sets can better reflect the overall characteristics of the data set than the frequent pattern.
[0103] The above embodiments are only exemplary embodiments of the present invention and are not intended to limit the present invention. The protection scope of the present invention is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present invention within the essence and protection scope of the present invention, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present invention.
Claims
1. An algorithm for mining high-utility and high-occupancy itemsets based on suffix partitioning in massive data, characterized by: Specifically, it includes two stages: preprocessing and item set mining. The preprocessing stage is to partition the data set according to the suffix and store it in a vertical structure. The specific process of the item set mining stage is as follows: For itemsets whose length is not greater than 2, their utility occupancy ratio is directly calculated to determine whether they are high-utility occupancy ratio itemsets. Itemsets with low-utility occupancy ratios are directly pruned. For item sets with a length greater than 2, a linked list-based bidirectional stack structure is constructed to store the extended items in the partition. The items in the bidirectional stack are used to build a set enumeration tree. The item sets are traversed and mined in the set enumeration tree in a depth-first search order. The item sets are pruned according to the support downward closure property, the utility occupancy ratio downward closure property, and the 3-item set complete pruning strategy. The item set link optimization strategy and the remaining TID cross count optimization strategy are used for mining. The support downward closure property is specifically: when the upper bound of the support of a partition is less than a preset minimum support threshold, the partition has no high utility occupancy ratio itemset that meets the support threshold, and the partition is pruned; when the support of an itemset is less than a preset minimum support threshold, the itemset and its superset do not meet the support threshold, and the itemset and its superset are pruned; The utility occupancy ratio downward closure property is specifically: the utility occupancy ratio upper bound The calculation formula is: Where t is a transaction containing the itemset X, Γ X is the supporting transaction set of item set X, pat_tu(X) is the transaction utility value of item set X in the partition, tu(t) is the utility value of transaction t where X is located, min_sup is the preset minimum support threshold, n is the number of transactions in the data set, uo(X,t) is the utility occupancy ratio of item set X in transaction t, and ruo(X,t) is the residual utility occupancy ratio of item set X in transaction t; Where x[j] represents the jth item in the item set X, u(x[j],t) represents the utility value of the jth item in the item set X in transaction t, los(X,t) is the subscript corresponding to the last item in X, and tu(t) is the utility value of transaction t in which X is located; Where t[j] represents the jth item in transaction t, u(t[j],t) represents the utility value of the jth item in transaction t, los(X,t)+1≤j≤w, los(X,t)+1 is the subscript of the next item after the last item of X, and tu(t) is the utility value of transaction t in which X is located; When the upper bound of the utility occupancy ratio is less than the preset utility occupancy ratio threshold, X and its superset do not meet the utility occupancy ratio threshold, and the item set X and its superset are pruned; when the upper bound of the utility occupancy ratio of the first item in a partition is less than the preset utility occupancy ratio threshold, the partition has no high utility occupancy ratio item set that meets the utility occupancy ratio threshold, and the partition is pruned; The bidirectional stack structure based on the linked list includes a base sequence and an operand sequence; The linked list-based bidirectional stack structure storage process includes: A. The items in the partition are sorted in descending order of support, and every item loaded into memory is frequent; B. The first item in the partition is stored in the cardinality sequence, and the subsequent items are stored in the operand sequence in order; C. First, put the items in the operand sequence into the bidirectional stack structure in descending order from bottom to top, and then put the items in the radix sequence into the bidirectional stack structure; D. Extract items from the bidirectional stack structure from top to bottom, and use the sequence output from the bidirectional stack structure for subsequent item set mining and pruning processes; The item set link optimization strategy is specifically as follows: when mining an item set with a length greater than 3, link the item set to its ancestor item set in the previous partition and the parent item set in the current partition to obtain the support information of the ancestor item set and the parent item set. If the ancestor item set is a frequent item set, cross-count the tid lists of the ancestor item set and the parent item set to obtain the support of the item set and determine whether to generate and calculate the item set; if the ancestor item set or the parent item set is not frequent, the item set and its superset can be pruned in the set enumeration tree; Ancestor itemset: If X is an itemset of length greater than 3, X = {X[1], X[2], ..., X[l]}, X[l] is the last item in itemset X, and the ancestor itemset of itemset X = {X[1], X[2], ..., X[l-1]} = XX[l], which contains the first (l-1) items of itemset X; Parent item set: If X is an item set stored in the set enumeration tree, the item set stored at the parent node of X is called the parent item set of X; The specific processing process of the optimization strategy based on the remaining TID cross count includes: Step 2.1: Initialize ind1 and ind2 to point to tid_list(X a ) and tid_list(X b ), where ind is an index pointing to the position of an item in the itemset; Step 2.2: Upper bound Θ(X) on the remaining tid ab ) to initialize, Θ(X ab )=min{|tid_list(X a )|,|tid_list(X b )|}-min_sup×n, where n is the number of transactions in the dataset, min_sup is the minimum support threshold, |tid_list(X a )| represents the item set X a The length of the corresponding tid list, |tid_list(X b )| represents the item set X b The length of the corresponding tid list, and |tid_list(X a )|≥tid_list(X b )|; Step 2.3: Determine Θ(X ab ), Θ(X ab )<0, then there is no need to perform the cross counting operation, and directly ab and its superset for pruning; Θ(X ab )≥0, then perform the cross counting operation; Step 2.4: During the cross-counting process, compare the tids pointed to by ind1 and ind2. tid(ind1) represents the tid pointed to by the ind1 index, and tid(ind2) represents the tid pointed to by the ind2 index: 1) If tid(ind1) = tid(ind2), it means that the tid pointed to by the index is an element in the intersection of tid_list(X a ) and tid_list(X b ). Then, store the common tid pointed to by the index into tid_list(X ab ), and move ind1 and ind2 one position backward respectively; 2) If tid(ind1) < tid(ind2), move ind1 one position backward; 3) If tid(ind1) > tid(ind2), then Θ(X ab ) - 1; Step 2.5: Repeat step 2.4 until Θ(X ab )<0 or one of ind1 and ind2 points to the end of the tid_list set, the cross counting process ends; then according to tid_list(X ab ) The size of the set determines the item set X ab Whether the support of meets the threshold, if X ab If it is frequent, continue to calculate its utility occupancy ratio, otherwise directly calculate X ab and its superset for pruning; The 3-itemset complete pruning strategy is as follows: in the partition P with item i as the suffix item i In the bidirectional stack structure list (P i )={a,b,c,…,m} is an item list containing m items. All 1-item sets and 2-item sets are mined through simple sequence scanning. Since both 1-item sets and 2-item sets are frequent, their utility occupancy ratio can be directly calculated and whether they are high utility occupancy ratio item sets can be judged. A 3-item set is generated by combining two 2-item sets, but before generating the 3-item set, it is judged whether all its 2-subsets are frequent. If not, the candidate set can be directly skipped. The depth-first set enumeration tree is combined to mine the item sets of length 3 or more in the running instance.
2. The high-utility and high-occupancy ratio itemset mining algorithm based on suffix partitioning in massive data according to claim 1 is characterized in that: The partition processing process is specifically as follows: a given data set T is sequence scanned, all subsequences of the data set T are divided, and item sets with the same suffix are placed in the same partition.
3. The high-utility and high-occupancy ratio itemset mining algorithm based on suffix partitioning in massive data according to claim 1 is characterized in that: The vertical storage process is specifically as follows: in each partition, a vertical format utility list is constructed for each item, the utility list is arranged in descending order according to the support corresponding to the item, and the items whose support is not less than a preset support threshold and their corresponding utility lists are read into the memory in sequence.
Citation Information
Patent Citations
Method and device for item-sets mining
CN106033447A
High utility itemset mining algorithm with data stream reducing candidate itemsets
CN106777182A
High-utility item set mining method and device and data processing equipment
CN107870956A
estDec+ method for finding frequent itemsets over data streams using compressed-Prefix trees
KR1020070055675A
