Parallel frequent itemset mining method based on Spark

Abstract

The present invention proposes a Spark-based parallel frequent item set mining method, comprising the following steps: S1, obtaining a support count matrix through matrix storage and providing a support count query for creating a conditional FP-tree; then, based on the obtained support count, eliminating items that do not meet the minimum support to obtain a frequent 1-item set matrix; then, decomposing the matrix storing the frequent 1-item sets using a non-negative matrix factorization strategy (SNMF); S2, obtaining a frequent 1-item set F-list from the frequent 1-item set matrix of S1, and uniformly grouping the frequent 1-item sets obtained by S1 using a genetic algorithm grouping strategy (GS-GA); S3, merging conditional pattern bases using an efficient tree structure reduction strategy (ERTS), deleting paths in the FP-tree that do not meet the maximum frequent item set; then, mining the grouped FP-trees in parallel to obtain frequent item sets, and outputting the results to a distributed file system (HDFS). The present invention solves the problems of low spatiotemporal efficiency, high inter-node communication overhead, and redundant search in creating a conditional FP-tree, and significantly improves both spatiotemporal efficiency and recommendation non-empty rate.

Description

Technical Field

[0001] The present invention relates to the field of big data mining, and in particular to a Spark-based parallel frequent itemset mining method. Background Art

[0002] In recent decades, with the continuous development of the Internet, information and communications technology, and sensor technology, more and more companies and institutions have acquired large amounts of data, which contains a wealth of potential value. How to quickly and accurately mine valuable information from this massive amount of data has become one of the most pressing issues in society.

[0003] Association rules are an important means of mining potential value from massive amounts of data. Their goal is to identify interesting associations or correlations between datasets. The association rule mining process is often divided into two stages: first, all frequent itemsets are found in the database; second, strong association rules are generated from these frequent itemsets. Mining frequent itemsets is a crucial step in association rule mining. Frequent itemset mining methods include: multiple candidate generation, such as Apriori, partitioning, and sampling; pattern growth, such as FP-Growth, HMmine, FPMax, and Close+; and vertical format, such as Eclat and CHARM. Multiple candidate generation methods iteratively generate candidate sets and compare minimum support to obtain frequent itemsets. A typical algorithm is the Apriori algorithm proposed by Agrawal et al.; pattern growth methods construct a tree structure from the frequent items generated by mining, and traverse the tree to generate frequent itemsets. A typical algorithm is the FP-Growth algorithm proposed by J. Han et al.; and vertical format methods transform horizontal datasets into vertical datasets and obtain frequent itemsets through intersection operations. A typical algorithm is the Eclat algorithm. These algorithms are effective when the data size is small, but when the data size reaches gigabytes or even higher, they become extremely inefficient due to the upper limits of storage and computing power, making them unsuitable for mining massive data. Therefore, the concept of parallel computing is particularly important. Improving frequent itemset mining algorithms and combining them with distributed computing models has become a major research direction.

[0004] With the widespread application of frequent itemset mining algorithms in distributed computing models, distributed computing frameworks, such as Spark, have attracted widespread attention due to their fast computational speed, ease of use, versatility, and support for multiple operating modes. A growing number of researchers have proposed Spark-based parallel FP-Growth algorithms, and these have been successfully applied to big data mining. Among them, Chunduri et al. proposed a scalable algorithm that integrates machine learning and Apache Spark, addressing the lack of support values ​​for the FP-growth algorithm and enabling the porting of the improved FP-growth algorithm to the Spark computing model for distributed computing. Despite successful application in Spark, this algorithm still suffers from three drawbacks: low spatiotemporal efficiency in creating conditional FP-trees, high inter-node communication overhead, and redundant searches. Summary of the Invention

[0005] The present invention aims to at least solve the technical problems existing in the prior art, and particularly innovatively proposes a Spark-based parallel frequent itemset mining method.

[0006] In order to achieve the above-mentioned object of the present invention, the present invention provides a Spark-based parallel frequent itemset mining method, comprising the following steps:

[0007] S1, obtains the support count matrix through matrix storage and provides support count query for creating conditional FP-tree; then, based on the obtained support count, removes items that do not meet the minimum support to obtain a frequent 1-item set matrix; then, decomposes the matrix storing frequent 1-item sets using the non-negative matrix factorization strategy SNMF, solving the problem of low time and space efficiency in creating conditional FP-tree;

[0008] S2, the frequent 1-itemset F-list can be obtained from the frequent 1-itemset matrix of S1. The genetic algorithm grouping strategy GS-GA is used to evenly group the frequent 1-itemset obtained from S1, reducing the FP-tree structure generated by each group and solving the problem of high communication overhead between nodes;

[0009] S3 adopts the efficient tree structure reduction strategy ERTS to merge the conditional pattern base and delete the paths in the FP-tree that do not meet the maximum frequent item set, thus solving the problem of redundant search. Then, the grouped FP-trees are mined in parallel to obtain frequent item sets and the results are output to the distributed file system HDFS.

[0010] The frequent item sets include frequent 1 item sets, and the frequent item sets also include frequent 2, 3, 4, 5...n item sets;

[0011] The parallel mining is a parallel operation, where multiple groups are recursively mined simultaneously. Therefore, recursive mining is omitted and we directly refer to parallel mining. However, the mining process in FP-tree naturally includes recursive mining.

[0012] Furthermore, the S1 includes the following steps:

[0013] S1-1, support count matrix acquisition: First, obtain database information from the distributed file system HDFS and convert it into the initial abstract elastic distributed dataset RDD. Use the RDD's fallMap() function to generate all items from the dataset. Then, apply the Map() function to each item in the dataset to generate (item,1) key-value pairs. Then, store the support counts of each pairwise combination of items in each transaction in a matrix. Finally, use reduceByKey to perform support statistics and obtain the support count matrix.

[0014] S1-2, frequent 1-itemset matrix acquisition: According to the obtained support count, eliminate the transactions that do not meet the minimum support, obtain the frequent 1-itemset, and update the storage information of the matrix to obtain the frequent 1-itemset matrix;

[0015] S1-3, matrix decomposition: The non-negative matrix factorization function FNMF is used to decompose the frequent 1-item set matrix, which solves the problem of large space occupation when creating conditional FP-tree.

[0016] Furthermore, the matrix decomposition in S1-3 includes:

[0017] The non-negative matrix factorization function FNMF is:

[0018]

[0019] Where V, W, H are non-negative matrices; Indicates that V is a non-negative matrix of dimension a×b, Indicates that W is a non-negative matrix of dimension a×c, Indicates that H is a non-negative matrix of dimension c×b; a, b, c represent the matrix dimensions;

[0020] min means taking the minimum value;

[0021] For penalty items;

[0022] ||·|| F Indicates the F paradigm;

[0023] γ is the sparsity parameter;

[0024] T represents the transpose of a matrix.

[0025] Furthermore, the genetic algorithm grouping strategy GS-GA in S2 includes the following steps:

[0026] S2-1, optimal solution acquisition: After obtaining the frequent 1-item set F-list, first use the "high-weight binary" encoding HWB to convert the F-list to obtain the initial population; then use the individual fitness calculation formula FCIF to screen the initial population and obtain the next generation population; finally, based on the obtained new population and genetic variation population, the optimal solution is obtained;

[0027] S2-2, frequent 1-itemset grouping: After obtaining the optimal solution, group the frequent 1-itemsets according to the optimal solution.

[0028] Furthermore, the "high-weight binary" code HWB in S2-1 includes:

[0029] When the frequent 1-item set is {X1,X2,...,X n}, X k When the support count of is encoded in binary, the probability of its normal encoding individual is P = ((m+1) / 2 m ) k , where X k is the kth normal coding individual, k∈{1,2,...,n}, m is the number of items in each item set, and n means that there are n items in the frequent 1-item set;

[0030] And the weights of each binary integer satisfy in is any symbol, N * represents a positive integer, For the existence symbol.

[0031] Furthermore, the body fitness calculation formula FCIF in S2-1 includes:

[0032] Given a set {x0,x1,...,x n}, the existence values ​​are {y0,y1,...,y n}, the population capacity is C, then the individual fitness is:

[0033]

[0034] where y i Represents x i The value of existence;

[0035] x i represents the i-th item set;

[0036] represents the correlation coefficient,

[0037] Furthermore, the S2-2 includes:

[0038] After obtaining the optimal solution, the frequent one-item set F-list is grouped according to the encoding information of the optimal solution. When n-1 groupings are completed, the remaining individuals automatically become a group. After obtaining the group list Group-list, the obtained group list Group-list is stored in HDFS so that any node in the cluster can access it. Where n is the total number of items in the frequent one-item set.

[0039] Furthermore, the step S3 includes the following steps:

[0040] S3-1, Conditional Pattern Base Merging: Parallel recursive FP-trees are used to generate conditional pattern bases. The conditional pattern base merging method MMCPB is used to merge the conditional pattern bases, reducing the FP-tree structure and generating a relatively small FP-tree, thus solving the problem of redundant search.

[0041] S3-2, non-maximum frequent item set deletion: using the maximum frequent item set pruning method MPMFI, the paths of non-maximum frequent item sets in the FP-tree generated in the previous stage are deleted in parallel to form an FP-tree with a small tree structure, which solves the problem of redundant search;

[0042] S3-3, frequent item set mining: After reducing the FP-tree structure, the FP-trees of each group are mined in parallel to obtain frequent item sets. The mining results of each group are merged to obtain the frequent item sets of the entire data set. At the same time, the results calculated in each partition are merged into the same partition and the results are output to HDFS.

[0043] Furthermore, the conditional mode base merging method MMCPB in S3-1 includes:

[0044] Let the suffix pattern be a, the storage method of the current item β = {a}, the conditional pattern base is {S1:u,S2:v}, if S1 is a proper superset of S2, then delete the elements in S1 that are not in S2 and merge them with a to generate a new path and add the support count; where u is the support of S1, v is the support of S2, and S1 and S2 represent the item sets of the conditional pattern base.

[0045] Furthermore, the maximum frequent itemset pruning method MPMFI in S3-2 includes:

[0046] Given an item set X, for any full path L X ,if Make And sup(L' X )≥min_sup, then, and X∪Y is not the maximum frequent item set; delete the non-maximum frequent item set path and update the support count; where L' X represents an arbitrary full path, sup(·) represents the support count, min_sup is the minimum support, and Y represents an arbitrary itemset.

[0047] In summary, by employing the above-mentioned technical solution, the present invention solves the problems of low spatiotemporal efficiency, high inter-node communication overhead, and redundant searches in creating conditional FP-trees. This method significantly improves both spatiotemporal efficiency and the rate of non-empty recommendations. Furthermore, the knowledge mined by this method can provide significant benefits in biology, medicine, astronomy, and geography.

[0048] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0050] Figure 1 Schematic diagram of the FP-tree with header table of the present invention.

[0051] Figure 2 Schematic diagram of the FP-tree pruning process of the present invention = {E}.

[0052] Figure 3 It is a schematic diagram of the MMCPB pruning process of the present invention.

[0053] Figure 4 It is a schematic diagram of the FP-tree of the present invention.

[0054] Figure 5 It is a schematic diagram of the MPMFI pruning process of the present invention.

[0055] Figure 6 This is a comparison chart of the acceleration ratios of the FIMAFP-Spark algorithm of the present invention on various data sets.

[0056] Figure 7 It is a comparison of the running time of different algorithms of the present invention on different data sets.

[0057] Figure 7 (a) is the recommendation non-empty rate of each algorithm on the com-amazon dataset.

[0058] Figure 7 (b) is the recommendation non-empty rate of each algorithm on the Kosarak dataset.

[0059] Figure 7 (c) is the non-empty recommendation rate of each algorithm on the Wikipedia dataset.

[0060] Figure 7 (d) is the recommendation non-empty rate of each algorithm on the LiveJournal dataset.

[0061] Figure 7 (e) is the non-empty recommendation rate of each algorithm on the webdocs dataset. DETAILED DESCRIPTION

[0062] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0063] The present invention can be applied to the fields of biology, medicine, astronomy and geography. For example, the method can be used to process medical imaging data to quickly identify abnormal parts. The method can also be used to process geographical landslide prediction, proteomic data analysis research, and pathogenic factor association analysis.

[0064] The specific plan is as follows:

[0065] 1. Frequent 1-itemset acquisition

[0066] Currently, Spark-based FP-Growth algorithms in big data environments often traverse the conditional pattern base multiple times when creating a conditional FP-tree, which can lead to low spatial and temporal efficiency. To address this issue, a SNMF strategy is proposed to decompose the matrix storing support counts during the process of obtaining frequent single-item sets, thus resolving the low spatial and temporal efficiency of conditional FP-tree creation.

[0067] The strategy consists of three main steps:

[0068] 1) Support count matrix acquisition: This method calculates the support of each item, i.e., the frequency of each item's occurrence. A matrix is ​​proposed to store the frequency of each item in each transaction, and to provide support count queries for creating conditional FP-trees, thus solving the time-consuming problem of creating conditional FP-trees.

[0069] 2) Obtaining the frequent 1-itemset matrix: Based on the obtained support count, remove transactions that do not meet the minimum support to obtain the frequent 1-itemset, and update the storage information of the matrix to obtain the frequent 1-itemset matrix;

[0070] 3) Matrix Decomposition: A non-negative matrix factorization function (FNMF) is proposed to decompose the frequent 1-item set matrix, solving the problem of large space usage when creating a conditional FP-tree.

[0071] 1.1 Obtaining the Support Count Matrix

[0072] In the process of obtaining support counts, we first obtain database information from the distributed file system HDFS (Hadoop Distributed FileSystem) and convert it into the initial abstract resilient distributed dataset RDD (Resilient Distributed Datasets). The falltMap() function of RDD is used to generate all items from the dataset. Then, on each item in the dataset, the Map() function is applied to generate (item,1) key-value pairs. Then, the support counts of each pairwise combination of items in each transaction are stored in a matrix. Finally, reduceByKey is used to perform support statistics to obtain the support count matrix.

[0073] 1.2 Frequent 1-itemset matrix acquisition

[0074] After obtaining the support count matrix, first delete the transactions with less than the minimum support according to the support count; then sort the transaction set in descending order according to the support count to generate the frequent one-item set F-list; finally, prune the support matrix according to the F-list to obtain the frequent one-item set matrix.

[0075] 1.3 Matrix Decomposition

[0076] After obtaining the frequent 1-itemset matrix, the matrix storage will take up a lot of space in a big data environment. In order to reduce the space occupation of the matrix, the FNMF function is proposed to decompose the frequent 1-itemset matrix and reduce the space occupation.

[0077] Theorem 1 (Non-negative Matrix Factorization Function FNMF): If the objective function of non-negative matrix factorization NMF (Non-negative Matrix Factorization) is Then the non-negative matrix factorization function FNMF is

[0078]

[0079] Where V, W, H are non-negative matrices; Indicates that V is a non-negative matrix of dimension a×b, Indicates that W is a non-negative matrix of dimension a×c, Indicates that H is a non-negative matrix of dimension c×b; a, b, c represent the matrix dimensions;

[0080] is the penalty term, ||·|| F represents the F-normal form, γ is the sparsity parameter, and ·T represents the transpose of the matrix.

[0081] Proof: Assuming W is the independent variable, we can get:

[0082]

[0083] Among them, tr() represents the trace function of the matrix, and ∧ represents the sparse parameter.

[0084] Find the partial derivative of W, let θF(W) / θW ij =0, and its piecewise derivative is:

[0085]

[0086]

[0087] again The sparse parameter ∧=[-VH T +WHH T +2W T -4γH T W] ij , the non-negative constraint makes the KKT (Karush-Kuhn-Tucker) condition ∧⊙∨=0 hold, where · ij represents the element in the i-th row and j-th column of the matrix, and ∨ represents the complement of the sparsity parameter ∧.

[0088] Assume H is the independent variable, find the partial derivative of H, and let θF(H) / θH ij =0, and its piecewise derivative is:

[0089]

[0090] again but The sparse parameter α can be obtained as [-W T V+W T WH+2γ 2 H-2γWW T ] ij , similarly, the non-negativity constraint makes the KKT condition hold. Proof complete.

[0091] The specific process of decomposing the matrix according to the FNMF function is:

[0092] Transform the objective function: According to the calculation method of matrix multiplication, the objective function is transformed into row and column elements. The element of the i-th row and j-th column of the matrix WH can be expressed as:

[0093]

[0094] Where e∈(0,c], d∈(e-1,c];

[0095] So the objective function becomes:

[0096]

[0097] Obtaining an unconstrained optimization problem: To avoid negative numbers in the calculation process, the FNMF function is used based on the Lagrange multiplier method to transform the standard constrained optimization problem of the original objective function into an unconstrained optimization problem. The objective function is converted to:

[0098]

[0099] Among them, if W ie and H ej They are all greater than or equal to 0 and need to be multiplied by -1 to become less than or equal to 0.

[0100] Decomposition matrix: To decompose the frequent 1-item set matrix, obtain the matrix W ie and H ej , first find the partial derivative of the transformed objective function and obtain θL / θW ie and θL / θH ej ; Then set the partial derivative to zero, remove the constant term in the equation according to the complementary slackness of the KKT condition, and calculate the decomposed matrix W ie and H ej .W ie and H ej The values ​​are as follows:

[0101]

[0102]

[0103] 2. Frequent 1-itemset grouping

[0104] In a big data environment, the size of the frequent single-item set F-list obtained is very large. The Spark-based FP-Growth algorithm does not consider load balancing when grouping, resulting in high communication overhead between nodes. Therefore, to address the high communication overhead between nodes, a genetic algorithm grouping strategy GS-GA is proposed.

[0105] The implementation process is as follows: 1) Obtaining the optimal solution: Based on the frequent single-item set F-list obtained in the previous stage, we first propose a "high weight binary" encoding HWB (High Weight Binary) to convert the F-list to obtain the initial population; then we propose a formula of calculating individual fitness (FCIF) to screen the initial population and obtain the next generation population; finally, based on the obtained new population and the genetic variant population, we obtain the optimal solution;

[0106] 2) Frequent 1-itemset grouping: After obtaining the optimal solution, the frequent 1-itemsets are grouped according to the optimal solution.

[0107] 2.1 Obtaining the optimal solution

[0108] After obtaining the frequent single-item set F-list, genetic algorithms cannot perform genetic mutation on the decimal F-list because they calculate individual fitness and select suitable individuals for genetic mutation by exchanging code segments to generate the optimal solution. Therefore, to select suitable individuals for genetic mutation, we propose HWB coding to convert the F-list and the FCIF formula to calculate individual fitness. Based on individual fitness, suitable individuals are selected for genetic mutation by exchanging code segments to obtain the optimal solution.

[0109] Theorem 2 (“High Weight Binary” Encoding HWB): When the frequent 1-item set is {X1,X2,...,X n}, X k When the support count of is encoded in binary, the probability of its normal encoding individual is P = ((m+1) / 2 m ) k , where X k is the kth normal coding individual, k∈{1,2,...,n}, m is the number of items in each item set, n represents the number of items in the frequent 1-item set. And the weight of each binary integer satisfies in is any symbol, N * represents a positive integer, For the existence symbol.

[0110] Proof: Let X1 be converted into binary, take the number of sets as m, and store the results in the array from low to high position, and get X1 = [x0, x1, ..., x m-1 ], where x i ∈{0,1},i∈(0,m-1). Assume that the set of components of item i is {x mi ,x mi+1 ,...,x mi+m-1}, it is easy to know that the set has a total of 2 valuesm There are m cases where the values ​​of different components are 1, and they are mutually exclusive. So there are only m cases where the codes corresponding to the values ​​of these components are normal codes. Considering the case where all components are not set, there are m+1 cases in total, and the probability is (m+1) / 2 m , and there are n items in the item set, which is the number of items in the frequent 1-item set, then X1 is the probability of a normal coded individual ((m+1) / 2 m ) n . Also prove: the weight relationship of each bit of a binary integer. According to mathematical induction, if p = 1, then q = 0, there are 2 p >2 q , the equation holds; take p=k+1, then q=k, we have The equation holds.

[0111] Theorem 3 (individual fitness calculation formula FCIF): Given the total set {x0,x1,...,x n}, the existence values ​​are {y0,y1,...,y n}, the population capacity is C, then the individual fitness is:

[0112]

[0113] where y i Represents x i The existence value of x i represents the i-th item set, the correlation coefficient

[0114] Proof: Assume is x, is y, n means the number of items in the frequent 1-item set is n. When x≤C, Fitness=y, it is easy to know Then Fitness = y is a continuous function. Similarly, x is continuous. When x>C, take any F0(x0,y0), and the Fitness function Fitness(x,y) is continuous with respect to y at F0, then |f(x0,y)-f(x0,y0)|<ε / 2,ε is an arbitrary constant. And since f(x,y0±δ1) is continuous with respect to x, then There are |f(x,y0-δ1)-f(x0,y0-δ1)|<ε / 2, |f(x0,y0+δ1)-f(x0,y0)|<ε / 2. Now take δ=min(δ1,δ2)(δ1→x0,δ2→y0), then when |x-x0|<δ,|y-y0|<δ, from the monotonicity of Fitness(x,y) with respect to y and the inequality obtained by the above deduction, we can obtain f(x,y)≤f(x,y0+δ1)<f(x0,y0+δ1)+ε / 2<f(x0,y0)+ε / 2+ε / 2=f(x0+y0)+ε and f(x,y)≥f(x,y0-δ1)>f(x0,y0-δ1)-ε / 2>f(x0,y0-δ1)-ε / 2>f(x0,y0)-ε / 2-ε / 2=f(x0+y0)-ε, and the two are combined as |f(x,y)-f(x0,y0)|<ε. Then Fitness(x,y), or f(x,y), is a continuous function. Since all functions are single-valued and non-negative, this formula can be used to calculate fitness. Proof complete.

[0115] The specific process of obtaining the optimal solution based on HWB coding and FCIF formula is as follows:

[0116] 1) Generate the initial population: Use HWB encoding to convert the values ​​in the frequent 1-item set F-list. If the converted code length is greater than the number of F-list items, the lower-order value is retained. Otherwise, the upper-order value is padded with 1. This is how the initial population is generated.

[0117] 2) Calculate population fitness: After obtaining the initial population, use the FCIF formula to calculate the fitness of individuals in the population.

[0118] 3) Confirm the next generation population: First, add up the fitness of all individuals in the previous population to get the total; second, calculate the probability of each individual in the population being selected and having its own genes used to pass on to future generations; then group the various probabilities into intervals; finally, generate a random number γ∈(0,1), and determine in which probability interval the random number appears, indicating that the individual was selected, thus obtaining the next generation population.

[0119] 4) Crossover Individuals: Based on the population obtained in the previous step, randomly pair the population and set the crossover point position of the individual chromosome pairing; then exchange some genes between the paired individual chromosomes to obtain new individuals. Assume that the population is {f:2,g:3,h:4,i:5}, the corresponding individual survival value is {3,4,5,6}, and the environmental capacity is 8. The genes for the chromosomes of two individuals are f = [1, 0, 1, 0] and g = [1, 1, 1, 1]. Using FCIF, the individual fitnesses are Fitness(f) = 8 and Fitness(g) = 6. A crossover reaction occurs after the third position, generating f' = [1, 0, 1, 1] and g' = [1, 1, 1, 0], respectively, and their fitnesses become Fitness(f) = 8 and Fitness(g) = 10.

[0120] 5) Mutation: A new population is formed from the new individuals obtained in the previous step. The location of the gene mutation for each individual in the new population is determined. Then, based on a certain probability, the original gene at the mutation location is inverted to obtain the mutant individual. The fitness of the mutant individual is then calculated, and the corresponding solution is obtained based on the fitness of the mutant individual. Finally, the corresponding solution is compared with the maximum environmental capacity. If the solution meets the maximum environmental capacity, mutation is stopped and the corresponding solution is the optimal solution. Otherwise, iterate until the condition is met. Suppose the genes of an individual's chromosome are [0,0,1,0], and a mutation occurs at the second position, resulting in [0,1,1,0]. The new fitness value obtained based on the gene mutation is the optimal solution. If the optimal solution meets the maximum environmental capacity, mutation is stopped. Otherwise, iterate until the condition is met.

[0121] 2.2 Frequent 1-itemset grouping

[0122] After obtaining the optimal solution, the frequent single-item set F-list is grouped according to the encoding information of the optimal solution. After completing n-1 groupings, the remaining individuals are automatically grouped together. After obtaining the group list Group-list, the obtained group list is stored in HDFS, making it accessible to any node in the cluster.

[0123] 3. Parallel mining of frequent itemsets

[0124] In a big data environment, the Spark-based FP-Growth algorithm suffers from redundant searches during frequent itemset mining. Therefore, to address this redundant search issue, we propose an ERTSS strategy for pruning the FP-tree. This strategy consists of three phases: 1) Conditional pattern base merging: Conditional pattern bases are generated through a parallel recursive FP-tree. MMCPB (a method of merging conditional pattern bases) is proposed to merge conditional pattern bases, reduce the FP-tree structure, and generate a smaller FP-tree, thus addressing the redundant search issue. 2) Non-maximum frequent itemset removal: MPMFI (a method of pruning maximumfrequent itemsets) is proposed to remove non-maximum frequent itemset paths from the FP-tree generated in the previous phase, resulting in a smaller FP-tree structure and addressing the redundant search issue. 3) Frequent itemset mining: After reducing the FP-tree structure, the FP-tree is recursively mined to generate frequent itemsets.

[0125] 3.1 Conditional Pattern Base Merging

[0126] For the merging of conditional pattern bases, first, a set of support counts of each item is generated based on the element item; second, according to the item sorting of the item header table, the FP-tree is recursively mined to generate the conditional pattern base; finally, the MMCPB method is proposed to merge the conditional pattern bases, reduce the FP-tree structure, and generate an FP-tree with a relatively small tree structure.

[0127] Theorem 4 (Conditional Pattern Base Merging Method MMCPB): If the suffix pattern is a, the current item is stored as β = {a}, and the conditional pattern base is {S1:u,S2:v}, where u and v are supports. If S1 is a proper superset of S2, remove the elements in S1 that are not in S2 and merge them with a to generate a new path, and add the support counts. S1 and S2 represent the item sets of the conditional pattern base.

[0128] Proof: Known sup(S1)=u, sup(S2)=v Assume that the minimum support is η, where sup(·) represents the support count; the combination of S1 and S2 has the following cases:

[0129] u+v<η, the support count of the merged set does not meet the minimum support, and S1 is a non-frequent itemset.

[0130] u+v≥η, where u<η, v<η, the support count of the merged set meets the minimum support and is a frequent item set, and u<η, indicating that S1 is a non-frequent item set.

[0131] u+v≥η, where u<η, v>η, the support count of the merged set meets the minimum support and is a frequent item set, u<η, indicating that S1 is a non-frequent item set.

[0132] u+v≥η, where u>η, v>η, the support count of the merged set meets the minimum support and is a frequent item set. u>η indicates that S1 is a frequent item set.

[0133] If S1 is an infrequent itemset, deleting elements from S1 that are not in S2 will not affect the set of frequent itemsets. This is negligible if S1 is a frequent itemset, and performing such pruning greatly improves computational efficiency. Proof complete.

[0134] The following example uses the transaction dataset T2 as shown in Table 1. The example process of pruning the FP-tree according to the MMCPB method is as follows:

[0135] Table 1 Transaction dataset T2

[0136] TID Items 1 A,B,E 2 B,D 3 B,C 4 A,B,D 5 A,C 6 B,C 7 A,C 8 A,B,C,E 9 A, B, C

[0137] Construct an FP-tree with a header table using the transaction data set T2, such as Figure 1 As shown in the figure, traverse the item header table in reverse order, use the current item header table item as the condition, and search the FP-tree to find the conditional pattern base. Taking α = {E} as an example, the generated conditional pattern base is {{NULL, B, A}:1, {NULL, B, A, C}:1}.

[0138] After obtaining the conditional pattern base of α = {E}, construct the FP-tree with its conditional pattern base as follows Figure 2 As shown in (a). In the conditional pattern base of α={E}, {NULL,B,A} and {NULL,B,A,C} both contain {NULL,B,A}, but {NULL,B,A} has no other elements except B and A, so {NULL,B,A} is not a true superset, and {NULL,B,A,C} is a true superset. According to the MMCPB method, the conditional pattern base is merged to obtain the path {NULL,B,A,E}:2. The path after merging the conditional pattern base is deleted and the support count is updated. In this way, the FP-tree of α={E} is pruned to obtain the pruned FP-tree as shown in Figure 2 (b) shown.

[0139] Similarly, the FP-tree constructed with the transaction data set T2 is as follows Figure 3As shown in (a), when α = {D}, {C}, {A}, {B}, the MMCPB method is used to merge the conditional pattern bases, delete the relevant paths, and update the support count. In this way, the FP-tree is pruned to obtain the pruned FP-tree as shown in Figure 3 (b) shown.

[0140] After pruning using the MMCPB method, Figure 3 Compared with the FP-tree before pruning, the FP-tree structure after pruning is relatively smaller, which avoids redundant search to a certain extent.

[0141] 3.2 Deletion of non-maximum frequent itemsets

[0142] Although the tree structure is reduced after merging the conditional pattern bases, it is still very large in a big data environment. To further reduce the tree structure, the MPMFI method is proposed to delete the paths in the tree that do not meet the maximum frequent itemset and update the support counts of the relevant items to generate a smaller FP-tree.

[0143] Theorem 5 (Maximum Frequent Itemset Pruning Method MPMFI): Given an item set X, for any full path L X ,if Make And sup(L' X )≥min_sup, then, and X∪Y is not the maximum frequent item set. Delete the non-maximum frequent item set path and update the support count. Where min_sup is the minimum support, is any symbol, Y represents any item set, True is contained in the symbol.

[0144] Proof: Given sup(L' X )≥min_sup, then X∪L' X is a frequent item set, that is, sup(X∪L' X )≥min_sup. And because but again and Then, according to Property 1, X∪Y is not a maximum frequent itemset. Proof completed.

[0145] Let's take a given FP-tree as an example. Figure 4 The example process of pruning FP-tree according to MPMFI is as follows:

[0146] Assuming the minimum support min_sup=2, given the item set X={A}, search the FP-tree to find the full path of A, and get {{NULL,B,A}:4,{NULL,B,A,E}:1,{NULL,B,A,C}:2,{NULL,B,A,C,E,}:2,{NULL,A}:2,{NULL,A,C}:2}. And sup({NULL,B,A,C,E})=2=min_sup. According to Theorem 5, {NULL,B,A} is not a full path. Delete this path and update the support count. Similarly, use MPMFI to delete the paths that do not meet the maximum frequent itemset in the FP-tree. In this way, prune the FP-tree and obtain the pruned FP-tree as follows Figure 5 (b) shown.

[0147] After pruning using the MPMFI method, Figure 5 Compared with the FP-tree before pruning, the FP-tree structure after pruning is relatively smaller, which avoids redundant search to a certain extent.

[0148] 3.3 Frequent Itemset Mining

[0149] After reducing the FP-tree structure, the FP-trees of each group are mined in parallel to obtain frequent item sets. The mining results of each group are merged to obtain the frequent item sets of the entire data set. At the same time, the results calculated in each partition are merged into the same partition and the results are output to HDFS.

[0150] 4. Effectiveness of the Spark-based Parallel Frequent Itemset Mining Algorithm (PAFMFI-Spark)

[0151] To verify the feasibility of the PAFMFI-Spark algorithm for parallel mining of frequent itemsets in a big data environment, we applied the PAFMFI-Spark algorithm to five datasets: Wikipedia, com-amazon, LiveJournal, webdocs, and kosarak. The specific information is shown in Table 2. The PAFMFI-Spark algorithm, IFPS algorithm, ECFAR algorithm, and HBPFP-DC algorithm are compared in terms of recommendation non-emptiness rate.

[0152] Table 2 Experimental dataset

[0153] Dataset Wikipedia LiveJournal com-amazon webdocs kosarak Number of samples 2394385 4847571 334863 1692082 990002 Number of itemsets 2 2 2 5267656 41270 Size (MB) 63.3 1000 12 1481.9 32.1

[0154] 5.1 Parallel Performance Analysis of the PAFMFI-Spark Algorithm

[0155] To verify the feasibility of the PAFMFI-Spark algorithm for parallel mining of frequent itemsets in a big data environment, we compared the performance of the Wikipedia, com-amazon, LiveJournal, webdocs, and Kosarak datasets, using speedup ratio as a metric. To ensure accuracy, each algorithm was run 10 times on each dataset, and the average run time was taken as the final experimental result. The speedup ratio was then calculated. This analysis of the algorithm speedup ratios confirms the feasibility of the PAFMFI-Spark algorithm for parallel mining of frequent itemsets in a big data environment. The experimental results are shown below:

[0156] from Figure 6As can be seen from the figure, when processing different datasets, the speedup of the PAFMFI-Spark algorithm on multiple nodes is consistently greater than that on a single node, and this speedup increases with the size of the dataset. In particular, when processing relatively large datasets such as LiveJournal, webdocs, and Wikipedia, the speedup of the PAFMFI-Spark algorithm shows a continuous upward trend with increasing node counts. The speedup growth trend is more pronounced when the number of nodes is 2 for the LiveJournal and webdocs datasets than for the Wikipedia dataset. On the relatively smaller datasets such as com-amazon and kosarak, the speedup of the PAFMFI-Spark algorithm reaches its maximum when the number of nodes is 2 and 3, respectively, increasing by 0.21 and 0.06 compared to the previous node count. Subsequently, the speedup decreases with increasing node counts. This result is due to the fact that when processing small datasets, parallel processing distributes samples across nodes. However, as the number of nodes increases, the communication overhead between nodes also increases, resulting in a decrease in the algorithm's speedup. When processing relatively large datasets such as LiveJournal, webdocs, and Wikipedia, the PAFMFI-Spark algorithm's speedup increases with the number of nodes. Specifically, when processing LiveJournal and webdocs, the PAFMFI-Spark algorithm's speedup increases by 0.77 and 0.83, respectively, compared to the single-node state. This is because the larger the number of samples in a dataset, the longer it takes for a single node to process them. However, as the number of nodes increases, the PAFMFI-Spark algorithm's GS-GA strategy evenly distributes the dataset's samples to each node, reducing inter-node communication overhead and indirectly improving the algorithm's efficiency in parallel mining frequent itemsets, leading to an increase in the algorithm's speedup. This demonstrates that the PAFMFI-Spark algorithm is suitable for parallel processing of relatively large datasets, and that the use of the GS-GA strategy in the PAFMFI-Spark algorithm can effectively improve its operational efficiency.

[0157] 5.2 Analysis of the Recommendation Non-emptiness Rate of the PAFMFI-Spark Algorithm

[0158] To analyze the performance of the PAFMFI-Spark algorithm in mining frequent itemsets in a big data environment, we compared the recommendation non-empty rate with the IFPS algorithm, the ECFAR algorithm, and the HBPFP-DC algorithm on the Wikipedia, com-amazon, LiveJournal, webdocs, and Kosarak datasets. The experimental results are shown below:

[0159] from Figure 7As can be seen from the table, the PAFMFI-Spark algorithm always has the highest recommendation non-empty rate compared to other algorithms on various datasets. Moreover, as the dataset size increases and the minimum support decreases, the gap between the PAFMFI-Spark algorithm's recommendation non-empty rate and other algorithms becomes more obvious. This gap is particularly evident on the com-amazon and webdocs datasets. Figure 7 As shown in (a), when processing the com-amazon dataset with a minimum support of 0.05, the PAFMFI-Spark algorithm's recommendation non-emptiness rate increases by 21.05%, 15.01%, and 27.78% compared to the IFPS, HBPFP-DC, and ECFAR algorithms, respectively. As shown in 7(e), when processing the webdocs dataset with a minimum support of 0.05, the PAFMFI-Spark algorithm's recommendation non-emptiness rate increases by 37.21%, 9.26%, and 47.50% compared to the IFPS, HBPFP-DC, and ECFAR algorithms, respectively. This result is due to the PAFMFI-Spark algorithm's use of the ERTSS strategy, which merges conditional pattern bases and removes paths in the FP-tree that do not satisfy the maximum frequent itemset requirement. This solves the problem of redundant search, avoids mining excessively useless frequent itemsets when the minimum support is low, and improves the quality of the final frequent itemsets. The IFPS and ECFAR algorithms, however, do not employ any optimization measures to address redundant search issues, resulting in excessive generation of useless frequent itemsets when the minimum support is low. While the HBPFP-DC algorithm employs transaction pruning and suffix length information extraction strategies to avoid redundant searches, the pruned FP-tree structure is not sufficiently small, and compared to the PAFMFI-Spark algorithm, it is less effective at eliminating frequent item redundancy. Consequently, the frequent itemsets mined by the PAFMFI-Spark algorithm consistently achieve a higher recommendation non-empty rate compared to other algorithms across all datasets.

[0160] In summary, the Spark-based parallel frequent itemset mining algorithm (PAFMFI-Spark) we proposed has significantly improved the recommendation non-empty rate and parallel efficiency.

[0161] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A Spark-based parallel frequent itemset mining method, characterized in that: The following steps are involved: S1, obtains the support count matrix through matrix storage and provides a support count query to create a conditional FP-tree; then, based on the obtained support count, removes items that do not meet the minimum support to obtain a frequent 1-item set matrix; then, decomposes the matrix storing frequent 1-item sets using the non-negative matrix factorization strategy SNMF; S1-1, support count matrix acquisition: First, obtain database information from the distributed file system HDFS and convert it into the initial abstract elastic distributed dataset RDD. Use the RDD's fallMap() function to generate all items from the dataset. Then, apply the Map() function to each item in the dataset to generate (item,1) key-value pairs. Then, store the support counts of each pairwise combination of items in each transaction in a matrix. Finally, use reduceByKey to perform support statistics and obtain the support count matrix. S1-2, frequent 1-itemset matrix acquisition: According to the obtained support count, eliminate the transactions that do not meet the minimum support, obtain the frequent 1-itemset, and update the storage information of the matrix to obtain the frequent 1-itemset matrix; S1-3, matrix decomposition: use the non-negative matrix factorization function FNMF to decompose the frequent 1-item set matrix; The non-negative matrix factorization function FNMF is: Where V, W, H are non-negative matrices; Indicates that V is a non-negative matrix of dimension a×b, Indicates that W is a non-negative matrix of dimension a×c, Indicates that H is a non-negative matrix of dimension c×b; a, b, c represent the matrix dimensions; min means taking the minimum value; For penalty items; ||·|| F Indicates the F paradigm; γ is the sparsity parameter; T represents the transpose of the matrix; S2, the frequent 1-itemset F-list can be obtained from the frequent 1-itemset matrix of S1, and the frequent 1-itemset obtained from S1 is evenly grouped using the genetic algorithm grouping strategy GS-GA; S2-1, optimal solution acquisition: After obtaining the frequent single-item set F-list, first use "high-weight binary" encoding HWB to convert the F-list to obtain the initial population; then use the individual fitness calculation formula FCIF to screen the initial population and obtain the next generation population; finally, based on the obtained new population and genetic variation population, the optimal solution is obtained; The "high-weight binary" code HWB includes: When the frequent 1-item set is {X1,X2,...,X n }, X k When the support count of is encoded in binary, the probability of its normal encoding individual is P = ((m+1) / 2 m ) k , where X k is the kth normal coding individual, k∈{1,2,...,n}, m is the number of items in each item set, and n means that there are n items in the frequent 1-item set; And the weights of each binary integer satisfy in is any symbol, N * represents a positive integer, For the symbol of existence; The body fitness calculation formula FCIF includes: Given a set {x0,x1,...,x n }, the existence values ​​are {y0,y1,...,y n }, the population capacity is C, then the individual fitness is: where y i Represents x i The value of existence; x i represents the i-th item set; represents the correlation coefficient, S2-2, frequent 1-itemset grouping: After obtaining the optimal solution, group the frequent 1-itemset according to the optimal solution; After obtaining the optimal solution, the frequent 1-item set F-list is grouped according to the encoding information of the optimal solution. After completing n-1 groupings, the remaining individuals automatically become a group. After obtaining the group list Group-list, the obtained group list Group-list is stored in HDFS so that any node in the cluster can access it. Where n is the total number of items in the frequent 1-item set. S3 uses the efficient tree structure reduction strategy ERTS to merge the conditional pattern base and delete the paths in the FP-tree that do not meet the maximum frequent item set; then the grouped FP-trees are mined in parallel to obtain frequent item sets and the results are output to the distributed file system HDFS.

2. A Spark-based parallel frequent itemset mining method according to claim 1, characterized in that: The S3 includes the following steps: S3-1, Conditional Pattern Base Merging: Parallel recursive FP-trees are used to generate conditional pattern bases. The conditional pattern base merging method MMCPB is used to merge the conditional pattern bases, reducing the FP-tree structure and generating a relatively small FP-tree, thus solving the problem of redundant search. S3-2, non-maximum frequent item set deletion: using the maximum frequent item set pruning method MPMFI, the paths of non-maximum frequent item sets in the FP-tree generated in the previous stage are deleted in parallel to form an FP-tree with a small tree structure, which solves the problem of redundant search; S3-3, frequent item set mining: After reducing the FP-tree structure, the FP-trees of each group are mined in parallel to obtain frequent item sets. The mining results of each group are merged to obtain the frequent item sets of the entire data set. At the same time, the results calculated in each partition are merged into the same partition and the results are output to HDFS.

3. The Spark-based parallel frequent itemset mining method according to claim 2, characterized in that: The conditional mode base merging method MMCPB in S3-1 includes: Let the suffix pattern be a, the storage method of the current item β = {a}, the conditional pattern base is {S1:u,S2:v}, if S1 is a proper superset of S2, then delete the elements in S1 that are not in S2 and merge them with a to generate a new path and add the support count; where u is the support of S1, v is the support of S2, and S1 and S2 represent the item sets of the conditional pattern base.

4. The Spark-based parallel frequent itemset mining method according to claim 2, characterized in that: The maximum frequent itemset pruning method MPMFI in S3-2 includes: Given an item set X, for any full path L X ,if Make And sup(L' X )≥min_sup, then, and X∪Y is not the maximum frequent item set; delete the non-maximum frequent item set path and update the support count; where L' X represents an arbitrary full path, sup(·) represents the support count, min_sup is the minimum support, and Y represents an arbitrary itemset.

Images