Efficient mining of non-dominated frequent-utility itemsets in large datasets

Abstract

The application provides a high-efficiency mining method of non-dominated frequent-utility item sets in large-scale data, comprising: dividing and grid storing a data set according to a prefix; initializing a multi-layer index linked list structure according to attribute values of items stored in the grid, and reading the grid in an anti-diagonal sequence; for each grid, taking the maximum utility value stored in the multi-layer index linked list as a conditional constraint, and sequentially reading the prefix shards corresponding to the items in the grid which satisfy the constraint; in each shard, adding the items which satisfy the conditional constraint to an expansion set; sequentially expanding the items in the expansion set, obtaining item sets which have a dominating ability in the two attributes of support and utility value, and adding the item sets to a result set; and the application proposes a prefix division strategy, a grid storage mode and three sets of pruning strategies, can perform high-efficiency calculation on large-scale massive data, effectively prunes a large number of hopeless item sets, and greatly saves space overhead and time overhead of the method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data mining and processing, and particularly relates to an efficient mining method of non-dominated frequent-utility item sets in large-scale data. BACKGROUND

[0002] Data mining is a core step of the knowledge discovery process in a database system, and is widely applied in e-commerce retail, financial services, biological medicine and other fields. Nowadays, most of the researches use either the frequency (such as frequent item set mining FIM) or the utility value (such as high utility item set mining HUIM) as the main measurement standard. Using these two measurement methods alone has their own limitations, FIM only considers whether the item set appears or not, ignoring its different weights, while HUIM considers that the item set has different importance and weights, but ignores its quantity characteristics. Therefore, considering both the frequency and the utility value, the returned result has higher value.

[0003] For users, both FIM and HUIM need to set the support threshold and the utility threshold in advance, and it is often difficult to find the appropriate threshold at one time. If the threshold is too high, the number of returned item sets is too small, which cannot provide sufficient information for decision-making. On the contrary, if the threshold is too low, the number of returned item sets is too large, which produces a large amount of redundant information, causing space waste and interference with the final decision. Top-k high utility item set mining returns the top-k result item sets that meet the conditions according to the threshold k selected by the user, which can help users control the number of result sets and make decisions within a certain range to a certain extent. However, the value of k also needs to be determined by experts with certain professional knowledge. For ordinary users, they prefer to avoid setting the threshold and directly return a set of decision schemes according to the selected preferences. Each individual on the result set is the optimal scheme under different weights, so as to enhance the user experience and provide the most valuable decision results.

[0004] Although there are a few methods that can handle frequent-high utility item set mining, there are great challenges in executing the mining task in large-scale data. The classic method is usually based on the assumption that the data volume is small enough, and the transaction data set can be completely read into the memory, but this operation cannot be performed on massive data, and the traditional method cannot be calculated. SUMMARY

[0005] The present application aims to provide an efficient mining method of non-dominated frequent-utility item sets in large-scale data, which aims to solve the above problems in the prior art.

[0006] The embodiment of the application provides a fast method for mining a skyline frequent-high utility item set in mass data, wherein the non-dominated frequent-high utility item set is the skyline frequent-high utility item set, abbreviated as SFUI.

[0007] The application has the following advantages:

[0008] (1) The application provides a new skyline frequent-high utility item set mining method, i.e., a skyline frequent-high utility item set mining method based on prefix partitioning, which can effectively calculate large-scale mass data.

[0009] (2) The application provides a strategy based on prefix partitioning, which uses local calculation in a slice to replace overall calculation on a data set. The huge data set is divided into a plurality of small slices based on prefixes, each of which can be completely read into the memory for calculation, so that the data set exceeding the computer memory can also be effectively calculated. The prefix slices are stored in a two-dimensional grid file, and the item sets in the same slice all contain a specific prefix item, so that an item set containing a specific prefix item can be calculated in a slice without traversing the entire data set, greatly saving the space and time overheads of the method.

[0010] (3) The application designs a new data structure M-list for storing the support-utility value pairs with the dominating ability in the current state of the data set, solves the problem of storing a large number of continuous repeated values in the traditional array structure, and effectively reduces the space overhead.

[0011] (4) The application designs a new grid storage method, which uses a two-dimensional grid to record and store the information of the slice and the item set. The grid file takes the support as the horizontal axis and the overestimated utility value as the vertical axis, and the coordinates of the item record the information of the support and the maximum utility value of the item.

[0012] (5) The application designs three groups of pruning strategies, which are 1) grid pruning, 2) PTWU pruning, and 3) RU pruning. Through the pruning strategies, the upper bound of the utility value of the skyline frequent-high utility item set is found, so that the support and the utility value satisfy the downward closed property and the dominating relationship, thereby effectively pruning a large number of hopeless item sets and reducing the calculation overhead. DETAILED DESCRIPTION

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0014] Figure 1 The overall method flowchart of the present application.

[0015] Figure 2 The result diagram of prefix partitioning according to the example data set.

[0016] Figures 3(a)-3(b) The chain table structure based on multi-layer index.

[0017] Figures 4(a)-4(b) The inter-slice grid and intra-slice grid.

[0018] Figures 5(a)-5(b) The anti-diagonal sequence of the grid.

[0019] Figure 6 The structure diagram of the set enumeration tree.

[0020] Figure 7 The concave-convex point diagram. DETAILED DESCRIPTION

[0021] In order to make the technical problems to be solved by the present application, technical solutions and beneficial effects clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0022] It should be noted that the terms "first", "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0023] Please refer to Figure 1 , for the convenience of description, only part related to the present embodiment is shown, which is described as follows:

[0024] A method for mining skyline frequent-high utility item sets in massive data, comprising:

[0025] Stage one: partitioning and storing the data set.

[0026] Each transaction in the dataset is divided into several sub-transactions, the first item of each sub-transaction (single item set or multi-item set) is taken as its prefix, and then the multiple sub-transactions are put into the corresponding prefix partition according to the prefix. In particular, if the length of the transaction is 1, it is the prefix itself. Then, through the same division rule, each transaction in the dataset is continuously divided and stored in the corresponding partition. The same operation is repeated until all transaction items in the dataset are traversed.

[0027] After the final division, the originally large dataset that cannot be read into memory for calculation is divided into several smaller partitions that can adapt to memory. These partitions do not overlap with each other, and the result of combining all the partitions is the entire transaction record database. Logically, the present application divides the dataset into several shards, and then physically stores the shard information on the disk using a grid storage form. The two-dimensional grid file takes the item as the primary key, and indexes the attribute value and item set information, which is beneficial to data collection and storage, thereby improving the efficiency of data reading and mining.

[0028] First, the specific process of the division processing is: given a transaction t = {t[1], t[2],..., t[w]}, assuming that the sub-transaction t' is a subset of the transaction t, that is,

[0029] Step 1.1: At the beginning, t' = t, the first item in t' is considered as the prefix item, that is, the prefix item prefix(t') = t[1], t[1] = i, and t' is put into the shard Pi according to the prefix item division.

[0030] Step 1.2: Next, t' = t / i, similarly, t' will be continuously put into the corresponding shard until its length is 1. Sub-transactions with the same prefix will be stored in the same shard, and transactions in the same prefix shard have the same prefix.

[0031] Step 1.3: Repeat step 1.2 to divide all transactions in the dataset. Finally, several prefix-based partitions will be obtained, which do not overlap with each other, and their union is the entire dataset.

[0032] In this embodiment, a dataset as shown in Table 1 is proposed, and the dataset shown in Table 1 is described in subsequent steps. Table 2 is the external utility value eu of each single item set in the dataset. TID in Table 1 represents the unique identification of the transaction, and tu represents the transaction utility value, whose calculation formula is where t is a certain transaction, i is a certain item in the transaction, and u(i, t) is the utility value of item i in t. According to the above partitioning process, the method scans the dataset D in Table 1, and the first transaction t1 = {(i2, 3), (i5, 1), (i7, 1)} is partitioned into three sub-transactions: {(i2, 3), (i5, 1), (i7, 1)}, {(i5, 1), (i7, 1)}, and {(i7, 1)}. Since the prefix item of {(i2, 3), (i5, 1), (i7, 1)} is i2, {(i2, 3), (i5, 1), (i7, 1)} will be partitioned into the P2 shard, and similarly, {(i5, 1), (i7, 1)} and {(i7, 1)} will be placed into shards P5 and P7, respectively. The same partitioning operation is then performed on the remaining transactions t2-t 10 , thereby partitioning the dataset D in Table 1 into seven partitions P1-P7, as shown in Figure 2 .

[0033] Example dataset D in Table 1

[0034]

[0035]

[0036] External utility value eu corresponding to each single item set Item in Table 2

[0037]

[0038] Step 2: Record and store the information of shards and their item sets using a two-dimensional plane grid. The grid file takes support (support, sup) as the X-axis and the overestimated utility value (ptwu) as the Y-axis, and stores the item in the corresponding position in the grid according to the support and maximum utility value of the item, as shown in Figures 4(a)-4(b) .

[0039] Phase Two: Process the prefix-based shards obtained in Phase One in turn, and calculate the item sets in the shards to obtain the final skyline frequent-high-utility item set.

[0040] The application applies a multi-layer index-based linked list structure M-list. The original linked list layer of the M-list stores compressed data, and each utility value value is stored only once, avoiding repeated storage of the maximum utility value, thereby greatly saving memory space. Above the original linked list layer, a multi-layer index index is constructed according to the utility value, as shown in FIG. 3(a) and FIG. 3(b), and the number of index nodes of each layer is half of the number of index nodes of the lower layer. That is, the number of nodes of the first layer index is half of the number of nodes of the original linked list layer, the number of nodes of the second layer index is half of the number of nodes of the first layer index, and so on. When the number of nodes is large, more index layers can be extracted until the number of nodes of the highest layer index is not greater than 2.

[0041] Unlike the array structure in the conventional scheme, the M-list does not need to traverse all the data and modify them according to the array subscript value. When the M-list needs to be modified, it is traversed from the upper index to the bottom layer, and if the node to be modified is an index node, the index also needs to be modified correspondingly, and finally the corresponding node in the original linked list layer is found to be modified. Therefore, the M-list can effectively avoid the problem that the array-based structure in the existing scheme needs to traverse the entire array, and does not need to modify the utility value one by one, thereby improving the modification efficiency.

[0042] In this stage, a new grid-based structure is applied, which does not need to perform a complete scan on the items and their utility lists of the data set, and can effectively mine all skyline frequent-high utility item sets in the data set.

[0043] Grid preprocessing: In the process of grid division, if the point appears on the grid line, it is not convenient for data reading. In this scheme, the position of the point is taken as the boundary, and the region is directly divided by using the point coordinates. In this technical scheme, in order to avoid the case that the point appears on the grid boundary, the mean division method is adopted. In the mean division, the mean A of the initial point and the last point in each grid is first calculated, and then the mean of each mean point A1, A2…An is calculated in turn to serve as the division boundary. Such division has more overall characteristics.

[0044] GAP (Grid-Adjacent-Partition): GAP is a grid file with support sup as the horizontal axis and the utility value ptwu as the vertical axis, which is used to store the 1-item set (i.e. the item set with length 1) in the data set. GAP stores the item and its information into the corresponding grid, and traverses the item in the grid according to the anti-diagonal sequence when processing, thereby processing the partition. By using the anti-diagonal sequence, the item with greater advantages in the two dimensions of support and utility value is read in and calculated earlier, so that the value of the M-list improves faster, and the pruning effect is more significant.

[0045] Grid-in-pieces (GIP): Similar to GAP, GIP is also a two-dimensional grid with sup as the horizontal axis and ptwu as the vertical axis, which is used to store the items in a certain piece, and these items are divided into different grids according to their sup and ptwu. In GIP, the M-list divides the two-dimensional plane into two regions, the dominated region and the non-dominated region. The region in the lower left corner is the region dominated by the M-list, and the items in the dominated region do not have advantages in both support and utility value, so neither the item in the partition nor its superset can be a skyline frequent-high utility item set, and this item does not need to be read into the memory for calculation. Only the items in the non-dominated region need to be read into the memory, so that many hopeless items can be pruned in advance.

[0046] According to the above theory, when reading the grid file of a certain partition, according to the inter-piece grid reading scheme, the grid-in-pieces is traversed according to the anti-diagonal order, and only the region not dominated by the M-list needs to be read by comparing the (sup, ptwu) attribute values of the single item set in the partition and the attribute limits in the M-list. At the same time, a set enumeration tree is constructed for the items in the partition according to the reading order, as shown in Figure 6 The earlier the item is read, the stronger the domination ability of the item set generated by the item, which is beneficial to quickly improve the M-list, thereby reducing the search space and improving the pruning efficiency.

[0047] Anti-diagonal order: After the grid is divided, the grid file GAP and GIP are read according to the anti-diagonal order. The anti-diagonal order designed by the present application starts from the upper right corner and ends at the lower left corner. In each small grid in the grid file, the pieces or items are read in order. The earlier the item is read, the greater the sup and ptwu, so it is more likely to dominate other item sets. Similarly, the prefix-based item set generated in the piece is more likely to be a skyline frequent-high utility item set. Conversely, the items or pieces closer to the lower left corner are read later because their sup and ptwu are relatively low. In summary, through the anti-diagonal order, the item set with more advantages in support and utility value will be mined earlier, and therefore the values in the M-list will also be improved faster, thereby making the pruning effect more obvious.

[0048] Specifically, step 1: initialize the M-list according to the GAP grid obtained in stage one. First, take the element in the upper right corner of the GAP grid, initialize the M-list according to the support corresponding to the utility value, and then use the elements in the uppermost row and the rightmost column to update the M-list for one round, and further improve the values of the M-list. In particular, if none of the grids in the upper right corner, the uppermost row and the rightmost column contains any item, the next layer (the uppermost row + the rightmost column) of the grid will be traversed until an item is included, so that the M-list is assigned an initial value.

[0049] Step 2: Read the GAP grid in anti-diagonal order.

[0050] The anti-diagonal order starts from the upper right corner and ends at the lower left corner. The order is shown in Fig. 5(a) and Fig. 5(b). It is assumed that the GAP contains 9 grids in total, and the traversal order starts from grid 1 and ends at grid 9. First, compare the grid coordinates G[x][y] with the WP point. If the grid coordinates and sum(G[x][y]) ≥ sum(WP), it means that the diagonal line where the grid is located will not be dominated by the M-list, and the grid and the items or fragments contained in the grid on the diagonal line need to be read and interpreted. Otherwise, if sum(G[x][y]) < sum(WP), the pruning ends the reading.

[0051] Supplementary explanation:

[0052] As shown in Fig. 1, in a grid file, C-vex is defined as the set of convex points in the grid. Specifically, C-vex = (sup[i], u[i]), (0 < i ≤ m), Figure 7 C-cave is defined as the set of concave points. Specifically,

[0053]

[0054]

[0055] where sup[i] is the support of the i-th point in the M-list, u[j] is the utility value of the j-th point in the M-list, m is the number of nodes in the M-list, m = |M-list| = |C-vex|, |C-cave| = |C-vex|.

[0056] G[x][y] is the grid in the x-th row and y-th column in the grid file, and its coordinate sum can be represented as sum(G[x][y]) = x + y. The point with the largest coordinate sum in C-vex is called the strongest point (Strongest Point, SP), i.e.

[0057] sum(SP) = max{sumG[x][y] | G[x][y] ∈ C-vex}

[0058] The point with the smallest coordinate sum in C-cave is called the weakest point (Weakest Point, WP), i.e. sum(WP) = min{sumG[x][y] | G[x][y] ∈ C-vex}

[0059] Step 3: Read the grids on each anti-diagonal line in the GAP in turn. For each grid, read its contained fragments in turn. If ptwu(i) ≥ mutil(sup(i)), the i-th fragment Pi needs to be read.​

[0060] Step 4: Read each item in Pi in turn, similarly, if the grid coordinates of the current read item j satisfy sum(G[x][y])≥sum(WP), and ptwu(j)≥mutil(sup(j)), then the item j in the slice Pi will be read into memory and participate in the calculation; otherwise, if ptwu(j)<mutil(sup(j)), then the item j has been dominated, and j and its superset cannot be a skyline frequent-high utility item set, so the item j is directly skipped; if sum(G[x][y])<sumWP, then the slice reading is completed, and the next slice calculation is entered.

[0061] Step 5: In the calculation of the slice Pi, first, judge the prefix item i, if u(i)≥mutil(sup(i)), then the item i is the current item set with the dominating ability, update the M-list and put the item i into the result set.

[0062] Step 6: Then add the item i to the set E.

[0063] Step 7: Traverse the item sets in E in turn, and expand them in the breadth-first search (BFS) order. When expanding the item set, add the items in the residual item set R(X) of the item set X to X in turn, that is, Xa=X∪a. Then judge the item set Xa. If uXa+ru(Xa)≥mutil(sup(Xa)), then add the item set Xa to the set Eset, otherwise, if uXa+ruXa<mutilsupXa, then Xa and its superset will be pruned. Further, if u(Xa)≥mutil(sup(Xa)), then Xa is the current item set with the dominating ability in the support and utility value attributes, so the M-list will be updated according to the support and utility value corresponding to Xa, and Xa will be put into the result set. After each item set expansion, an Eset set is returned, and the elements in the Eset set will be added to the NE set.

[0064] Supplementary explanation:

[0065] uXa is the actual utility value of the item set Xa, ruXa is the residual utility value of the item set Xa, mutil(sup(Xa)) represents the maximum utility value corresponding to the support of Xa, and the corresponding value is stored in the M-list structure.

[0066] Step 8: When the elements in E are traversed, it means that the item sets of this length have been expanded. Assign E to NE, continue to expand the item sets of the dependent level, and repeat step 7 until E is empty.

[0067] The above-described embodiments are only used to illustrate the technical solutions of the present application, but not limit them; although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. An efficient mining method of non-dominated frequent-utility itemsets in large scale data, characterized by, The method comprises the following steps: Stage one, dividing and storing the data set; Divide each transaction of the data set into several sub-transactions, and take the first item of each sub-transaction as its prefix, and then put the sub-transactions into the corresponding prefix partition according to the prefix; Record and store the information of the shards and the item set by using a two-dimensional plane grid; the grid file takes support as the X axis and high-estimated utility value as the Y axis, and the items are stored in the corresponding position in the grid according to the support and the maximum utility value of the items; Stage two, based on the grid obtained in stage one, initialize the chain table structure M-list of the multi-layer index according to the attribute values of the items stored in the grid; first, take the element in the upper right corner of the grid, and initialize the M-list according to the corresponding utility value of the support; update the M-list by using the elements in the uppermost row and the rightmost column of the grid; start from the upper right corner of the grid and end at the lower left corner, read the grid in the anti-diagonal sequence; for each grid, take the maximum utility value stored in the multi-layer index chain table as the conditional constraint, and read the prefix shards corresponding to the items meeting the constraint in the grid in sequence; in each shard, add the items meeting the maximum utility value conditional constraint to the expansion set; Expand the items in the expansion set in sequence, obtain the item set having the dominance ability in the support and the utility value, and add the item set to the result item set; the original chain table layer of the M-list stores the data in a compressed manner, and each utility value is stored only once, so that the repeated storage of the maximum utility value is avoided, and the memory space is greatly saved; above the original chain table layer, a multi-layer index is constructed according to the utility value, and the number of index nodes of each layer is half of that of the lower layer; when the number of nodes is large, more index levels can be extracted, until the number of nodes of the highest layer index is not greater than 2; When the M-list needs to be modified, traverse from the upper index to the bottom layer, if the node to be modified is an index node, the index also needs to be modified, and finally the corresponding node in the original chain table layer is modified.

2. The method as claimed in claim 1, wherein, In the process of grid division, the position of the point is taken as the boundary, and the point coordinates are directly used for dividing the region; in order to avoid the point appearing at the boundary of the grid, the mean value division method is adopted, and the mean value of two is taken as the division boundary, which has better overall performance.

3. The method as claimed in claim 1, wherein the non-dominated frequent-utility itemsets are mined efficiently in large scale data. The grid comprises an inter-shard grid and an intra-shard grid; The inter-shard grid is a grid file taking support as the horizontal axis and high-estimated utility value as the vertical axis, and is used for storing 1-item sets in the data set, i.e. item sets with a length of 1; the inter-shard grid stores the items and their information in the corresponding grid, and the items in the grid are processed in the anti-diagonal sequence during processing, so that the shards are processed; The intra-shard grid is also a grid taking support as the horizontal axis and high-estimated utility value as the vertical axis, and is used for storing items in a shard; the items are divided into different grids according to their support and maximum utility value; in the intra-shard grid, the M-list divides the two-dimensional plane into two regions, the dominated region and the non-dominated region; Only the items in the non-dominated region need to be read into the memory, so that many hopeless items are pruned in advance.

4. The method as claimed in claim 3, wherein the non-dominated frequent-utility itemsets are mined efficiently in large scale data. When reading in a partition of the grid file, according to the inter-slice grid reading scheme, the intra-slice grid is traversed according to the anti-diagonal order, and only the region not dominated by the M-list needs to be read by comparing the attribute values of the single-item set in the partition with the attribute limits in the M-list.

5. The method as claimed in claim 1, wherein the non-dominated frequent-utility itemsets are mined efficiently in large scale data. During itemset expansion, the itemset will be... The remaining itemset After the items in the set are added to X in sequence, the itemset is... Make a judgment: If Then the itemset Add to the collection Eset, or vice versa. Its supersets will be pruned; among them, For itemsets The actual utility value, Itemset The residual utility value, express The maximum utility value corresponding to the support level is stored in the M-list structure.

6. The method as claimed in claim 5, wherein the non-dominated frequent-utility itemsets are mined efficiently in large scale data. If , then item set is the current item set with dominance power on the support and utility value attributes, so the M-list will be updated according to the corresponding support and utility values, and will be put into the result set.

Images