A stream data accurate frequent item set mining method

Abstract

This invention discloses a method for accurate frequent itemset mining of streaming data, relating to the field of data mining technology, including the following steps: S1, accessing and managing structured streaming data through a distributed streaming processing engine; S2, converting streaming transaction data into a vertical format of item and transaction ID sets, and maintaining the vertical database state using state management functions; S3, filtering infrequent items based on a minimum support threshold, performing equivalence class partitioning, and partitioning the conditional database to each computing node based on the equivalence classes; This invention, by deeply integrating an incremental mining architecture with a vertical data format with a distributed streaming processing engine, ensures the complete accuracy of frequent itemset mining results while achieving a high throughput of tens of thousands of transactions per second, thus successfully breaking through the technical bottleneck of "difficulty in balancing accuracy and efficiency" in existing technologies, and achieving significant technical progress.

Description

Technical Field

[0001] This invention relates to the field of data mining technology, and in particular to a method for mining precise frequent itemsets in streaming data. Background Technology

[0002] Frequent itemset mining is a core technology for applications such as association rule analysis, real-time recommendation systems, and network intrusion detection. In fields such as the Internet of Things, e-commerce, and financial risk control, data is generated in a high-speed, continuous, and unbounded streaming format, creating an urgent need for frequent itemset mining technology capable of real-time and accurate analysis of the latest data streams. However, traditional designs for static data can no longer meet the stringent requirements of streaming scenarios, prompting the emergence and development of frequent itemset mining technology for streaming data.

[0003] Existing techniques for frequent itemset mining of streaming data mainly revolve around the sliding window model and distributed computing, but they are diverging in their technical approaches: 1. Precise data mining algorithms based on prefix tree structures: Represented by DS-Tree and CPS-Tree, these algorithms store data within a window using a tree structure and employ methods similar to FP-Growth for precise data mining. To adapt to distributed environments, implementations based on the Hadoop platform, such as CanTree-GTree, have emerged. 2. Fast mining algorithms based on approximation strategies: Represented by FIMoTS, this type of algorithm actively discards potential infrequent itemsets by setting upper and lower bounds, sacrificing the completeness of the results in exchange for improved processing speed, and belongs to approximation mining; 3. Algorithms based on vertical data structures: such as EclatDS, which attempts to introduce vertical data formats (such as tidset) suitable for static data into streaming mining, but its design is mainly geared towards single-machine environments and has scalability bottlenecks when dealing with massive streaming data.

[0004] While the aforementioned technologies provide a preliminary solution for streaming mining, they all reveal serious shortcomings when dealing with high-throughput, low-latency real-time big data streams: 1. The dilemma of balancing "accuracy" and "efficiency": Existing technologies are caught in a dilemma. Precision algorithms based on prefix trees (such as CanTree-GTree) suffer from high latency in dynamic updates and computation due to their complex structure and the batch processing characteristics of the Hadoop framework; while approximate algorithms (such as FIMoTS) actively discard itemsets in pursuit of efficiency, resulting in inaccurate mining results and poor reliability. 2. Inefficient dynamic updates become a performance bottleneck: Algorithms based on prefix tree structures (such as DS-Tree, CPS-Tree) require traversing and adjusting the tree structure when inserting or deleting a single transaction during window sliding, resulting in a time complexity of O(logn). This causes significant computational overhead in high-throughput streaming data scenarios (hundreds of thousands of records per second or more), becoming a major performance bottleneck. 3. Poor distributed scalability and load balancing: Existing distributed algorithms (such as Hadoop-based implementations) require intermediate results to be persisted to disk, which cannot meet the low latency requirements of stream processing. Furthermore, they generally employ simple data sharding strategies and fail to optimize for frequent itemset mining computational features (such as equivalence classes), resulting in uneven computational load across cluster nodes, triggering a "long tail effect," and underutilizing computing power.

[0005] The root cause of the aforementioned technical difficulties lies in the mismatch between its core design and the essential needs of streaming data: 1. Mismatch between data structure and the dynamic nature of streaming data: Prefix tree structures are good at expressing data rather than handling frequent additions and deletions. Using them for continuously changing sliding windows inevitably leads to high update costs. The potential of more efficient vertical data structures in distributed stream processing has not been fully explored. 2. Mismatch between distributed architecture and stream processing characteristics: Using computing frameworks designed for batch processing, such as Hadoop, to process streaming data results in high latency due to their inherent disk I / O and task scheduling. Meanwhile, the advantages of emerging in-memory computing frameworks have not been effectively applied to this field. 3. Mismatch between parallel strategies and mining task characteristics: Simple data partitioning cannot cope with the imbalance in itemset support calculations during frequent itemset mining. The lack of a parallel strategy that can intelligently partition based on computational load (rather than simply data volume) is the root cause of load imbalance.

[0006] In summary, the field of frequent itemset mining of streaming data urgently needs a new distributed solution that can simultaneously guarantee result accuracy, processing efficiency, and cluster load balancing. Summary of the Invention

[0007] The purpose of this invention is to solve the problems in the prior art by proposing a method for accurate and frequent itemset mining of streaming data.

[0008] A method for mining exact frequent itemsets in streaming data includes the following steps: S1. Structured streaming data access and window management are achieved through a distributed stream processing engine; S2. Convert the streaming transaction data into a vertical format of project and transaction ID sets, and use the state management function to maintain the vertical database state. S3. Filter infrequent items based on the minimum support threshold and perform equivalence class partitioning. Then, partition the conditional database to each computing node based on the equivalence classes. S4. Execute the StrStream-Eclat algorithm in parallel on each computing node to calculate the intersection of transaction ID sets through distributed aggregation operators; S5. Update the state using an incremental state update mechanism and output the mining results.

[0009] Preferably, in step S1, the structured stream data access and window management are performed through a distributed stream processing engine, specifically including: (1) Create a data stream through the stream processing API data source; (2) Use window functions to define the processing window and set the window duration and sliding interval parameters; (3) Configure watermark parameters to process delayed data; (4) Group the data by window through the grouping operation.

[0010] Preferably, in step S2, the streaming transaction data is converted into a vertical format of project and transaction ID sets, and the vertical database state is maintained using state management functions, specifically including: (1) Expand the transaction data into <project, transaction ID> key-value pairs; (2) Generate a vertical data format of <project, transaction ID set> by key aggregation; (3) Maintain the state of the vertical database using the state management function of the stream processing engine; (4) Set the status timeout parameter to automatically clean up expired data.

[0011] Preferably, in step S3, filtering infrequent items based on the minimum support threshold and performing equivalence class partitioning, and partitioning the conditional database to each computing node based on the equivalence classes, specifically includes: (1) Filter infrequent items based on the minimum support threshold parameter; (2) Use breadth-first or depth-first strategy parameters for equivalence class partitioning; (3) Use a distributed dataset to automatically partition the conditional database across computing nodes; (4) Set the number of partitions to control the parallelism.

[0012] Preferably, in step S4, the StrStream-Eclat algorithm is executed in parallel on each computing node to calculate the intersection of the transaction ID sets through a distributed aggregation operator, specifically including: (1) Run the core logic of the StrStream-Eclat algorithm in parallel on each executor; (2) The intersection of transaction sets is calculated through key reduction operations; (3) Use accumulator parameters to collect global statistics; (4) Configure execution memory parameters to optimize computing performance.

[0013] Preferably, in step S5, an incremental state update mechanism is used to update the state and output the mining results, specifically including: (1) Use stateful mapping grouping operations to implement incremental state updates; (2) Output the complete results through batch processing functions or output mode parameters; (3) Set checkpoint location parameters to ensure fault recovery; (4) Configure the trigger interval parameter to control the processing rhythm.

[0014] Preferably, the data source supports Kafka, file system, or Socket access.

[0015] A system for mining exact frequent itemsets in streaming data includes: The streaming data access module is used to access streaming data and manage windows. The vertical format conversion module is used to convert data to a vertical format and manage its state; The equivalence class partitioning module is used to partition tasks based on equivalence classes and distribute them to computing nodes. The distributed mining module is used to perform precise frequent itemset mining in parallel across nodes; The state management module is used to incrementally update the state and output the results.

[0016] Compared with existing technologies, the advantages of this invention are: 1. This invention achieves a high throughput of tens of thousands of transactions per second by deeply integrating an incremental mining architecture with a vertical data format and a distributed stream processing engine, while ensuring the complete accuracy of frequent itemset mining results. This successfully breaks through the technical bottleneck of "difficulty in balancing accuracy and efficiency" in the prior art and achieves significant technical progress.

[0017] 2. This invention uses a vertical data format to replace the traditional prefix tree structure, which reduces the complexity of data updates when the window slides and effectively improves the update efficiency. At the same time, based on the intelligent task partitioning strategy of equivalence class and combined with the adaptive parallel scheduling mechanism, the load difference between each computing node is controlled within 5%, effectively eliminating the "long tail effect" and greatly improving the utilization rate of computing power. Attached Figure Description

[0018] Figure 1 This is an overall flowchart of the frequent itemset mining method of the present invention.

[0019] Figure 2 This is a schematic diagram of vertical format conversion in this invention.

[0020] Figure 3 This is an example diagram of a sliding window in this invention.

[0021] Figure 4 This is the overall system architecture of the method of the present invention in a preferred embodiment (taking a Spark cluster as an example).

[0022] Figure 5 This is a comparison chart of the throughput of this invention with other algorithms.

[0023] Figure 6 This is a comparison chart of the memory consumption of this invention with other algorithms.

[0024] Figure 7 This is a comparison diagram of the system scalability experiments of this invention and other algorithms. Detailed Implementation

[0025] To facilitate understanding of this application and to make the aforementioned objectives, features, and advantages of this application more apparent, a detailed description of specific embodiments of this application is provided below in conjunction with the accompanying drawings. Numerous specific details are set forth in the following description to provide a thorough understanding of this application, and preferred embodiments are shown in the accompanying drawings. However, this application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of this application. This application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified. In the description of this application, "several" means at least one, such as one, two, etc., unless otherwise explicitly specified. It should be noted that when an element is referred to as being "fixed to" another element, it can be directly attached to the other element or there may be an intervening element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or there may be an intervening element. The terms "vertical," "horizontal," "left," "right," and similar expressions used herein are for illustrative purposes only and do not represent the only possible implementations. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is only for describing particular implementations and is not intended to limit the scope of this application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0026] Reference Figure 1 As shown, a method for mining precise frequent itemsets in streaming data employs the StrStream-Eclat algorithm and includes a five-step execution flow: "data access and window management → vertical format conversion and state management → equivalence class partitioning → distributed precise mining → incremental update and output". In the data access and window management steps: a sliding window is defined using the window function and watermark parameter of the stream processing engine (e.g., the window function and watermark parameter of SparkStructured Streaming), and the discretization of streaming data is achieved by combining grouping operations (e.g., groupBy operation); In the vertical format conversion and state management step: convert the stream transaction data into the vertical data format of <item, set of transactionIds>, and use the incremental state update operator of the stream processing framework (such as the mapGroupsWithState operator in Spark Structured Streaming) to achieve incremental state maintenance with O(1) time complexity, as Figure 2 shown; In the equivalence class partitioning step: perform distributed task partitioning based on frequent item equivalence classes, and achieve load balancing through a query optimizer and an adaptive query execution mechanism (such as the Spark Catalyst optimizer and the AQE mechanism), so that the load difference between each computing node is controlled within 5%; In the distributed exact mining execution step: parallelize the execution of the core logic of the StrStream-Eclat algorithm on each computing node, and use a distributed key-based reduction operator (such as the reduceByKey operator in Spark) to achieve efficient intersection calculation of the set of transactionIds, ensuring 100% accuracy of the mining results; In the incremental update and output step: set checkpoint location parameters to ensure fault recovery. Incremental state management mechanism: use the incremental state update function of the stream processing engine to only perform incremental processing on the changed data, avoiding duplicate calculation of the full amount of data; A stream data exact frequent itemset mining system, including: A stream data access module, used to access stream data and perform window management; A vertical format conversion module, used to convert data into a vertical format and manage the state; An equivalence class partitioning module, used to partition tasks based on equivalence classes and distribute them to computing nodes; A distributed mining module, used to parallelize the execution of exact frequent itemset mining on each node; A state management module, used to incrementally update the state and output the results.

[0027] The present invention uses a distributed stream processing engine (in a preferred embodiment, it is Apache Spark Structured Streaming), as Figure 4 shown, combines an optimized vertical data format and a window management mechanism, and proposes the StrStream-Eclat algorithm, which specifically includes the following steps: Step S1: Structured stream data access and window management Use a distributed stream processing engine (such as an Apache Spark cluster and its Structured Streaming component) to implement the following steps: Steps and parameters: (1) Create a data stream (DataStream) from data sources such as Kafka and the file system through a stream processing API (e.g., Spark Structured Streaming API); (2) Define a processing window using window functions (e.g., Spark's window function), and set the window duration parameter (windowDuration) and the sliding interval parameter (slideDuration); (3) Configure the watermark parameter (watermark) to handle late data; (4) Group by window through a grouping operation (e.g., Spark's groupBy operation) to form discretized data processing units.

[0028] Step S2: Vertical Data Format Conversion and State Management Use the SQL processing and state management capabilities of a distributed computing engine (e.g., Spark SQL engine and its state store) to implement the following steps: Steps and Parameters: (1) Use a flattening map operation (e.g., Spark's flatMap operation) to expand transaction data into the <item, transactionId> format; (2) Aggregate through a key grouping operation (e.g., Spark's groupByKey operation) to generate the vertical data format <item, set of transactionIds>; (3) Utilize the state management function of the stream processing engine (e.g., Structured Streaming's state management) to maintain the vertical database state; (4) Set the state timeout parameter (timeoutConf) to automatically clean up expired data.

[0029] Step S3: Equivalence Class Conditional Database Partitioning Use the query optimization and execution components of a distributed computing engine (e.g., Spark Catalyst optimizer and Tungsten execution engine) to implement the following steps: Steps and Parameters: (1) Filter infrequent items according to the minimum support threshold parameter (minSupport); (2) Adopt a breadth-first or depth-first strategy parameter (traversalStrategy) for equivalence class partitioning; (3) Use a distributed dataset (e.g., Spark's distributed dataset RDD or Dataset) to automatically partition the conditional database to each computing node; (4) Set the number of partitions (numPartitions) parameter to control the degree of parallelism.

[0030] Step S4: Distributed Precision Mining Execution Implement the following steps using the execution components of a distributed computing engine (such as Spark Executors and task schedulers): Steps and parameters: (1) Run the core logic of the StrStream-Eclat algorithm in parallel on each executor; (2) Achieve efficient intersection calculation of transaction sets through key reduction operations (such as Spark's reduceByKey operation); (3) Use accumulator parameters (e.g., Spark's Accumulator) to collect global statistics; (4) Configure execution memory parameters (executorMemory) to optimize computing performance.

[0031] Step S5: Incremental state update and result output Implement the following steps using the output and state storage components of a stream processing engine (such as the output receiver and state storage of Structured Streaming): Steps and parameters: (1) Use stateful mapping grouping operations (such as the mapGroupsWithState operation of Structured Streaming) to implement incremental state updates; (2) Output the complete results through batch processing functions or output mode parameters (such as foreachBatch or outputMode); (3) Set checkpoint location parameters to ensure fault recovery; (4) Configure the trigger interval parameter (triggerInterval) to control the processing rhythm.

[0032] The StrStream-Eclat algorithm proposed in this patent can effectively solve the three major technical problems mentioned in the background. Its core principle lies in the synergistic innovation of a new generation of stream processing framework, optimized vertical data structure and intelligent task partitioning strategy, which achieves a fundamental breakthrough in technical architecture. The core principle is elaborated as follows: 1. The principle for resolving the "precision versus efficiency contradiction" The root of the technical problem lies in the fact that existing technologies are forced to make trade-offs between accuracy and efficiency due to architectural limitations. Approximate algorithms sacrifice accuracy for speed, while precise algorithms cannot meet real-time requirements due to inefficient computational frameworks.

[0033] This patented solution: The core of the technical solution is to fundamentally unify the accuracy and efficiency of processing by integrating a high-performance distributed stream processing framework with a precise mining algorithm kernel and leveraging the advantages of in-memory computing.

[0034] A high-performance distributed stream processing framework is employed: leveraging its declarative API and unified batch-stream processing model, the query optimizer automatically generates the optimal execution plan, maximizing execution efficiency while ensuring processing correctness. (In a preferred embodiment, the Spark Structured Streaming framework is used, with its Catalyst query optimizer achieving the above effect).

[0035] Retaining the core of the Eclat precise algorithm: Based on the vertical data format, the Eclat algorithm can completely find all frequent itemsets that meet the minimum support by performing intersection operations on the transaction ID sets, ensuring 100% accuracy of the results.

[0036] Leveraging the advantages of in-memory computing: Employing a memory-based computing model avoids frequent disk I / O, enabling precise calculations to achieve processing speeds comparable to approximate algorithms. (In a preferred embodiment, this is achieved by replacing Hadoop HDFS with Spark's memory-based computing model).

[0037] 2. The principle behind solving the problem of "low efficiency of dynamic updates" The root of the technical problem is that the prefix tree structure requires frequent adjustments to the tree topology when adding or deleting data during transactions, and its O(log n) time complexity is not suitable for high-throughput streaming data scenarios.

[0038] This patented solution: The core of the technical solution is to adopt a vertical data format suitable for adding and deleting data, and combine it with an incremental state management mechanism and optimized memory storage to achieve constant-level update efficiency.

[0039] Constant-time update for vertical data format: Using the storage format of <project, transaction ID set>, the data update when the window slides is simplified to the union (insertion) and difference (deletion) operations of the set, and the time complexity is stable at O(1).

[0040] Incremental state management mechanism: Utilizing the incremental state update function of the stream processing engine, incremental processing is performed only on changed data, avoiding redundant calculations of all data. (In a preferred embodiment, this mechanism is implemented through the mapGroupsWithState operator of Structured Streaming).

[0041] Columnar memory storage optimization: By employing advanced columnar memory management techniques, the layout of data in memory is optimized, further improving the efficiency of collection operations. (In a preferred embodiment, this optimization is achieved through columnar memory management of the Spark Tungsten execution engine).

[0042] 3. The principle for solving the problems of "distributed scalability and uneven load distribution" The root cause of the technical problem is that simple data sharding cannot cope with the inherent uneven computational load in frequent itemset mining, resulting in low cluster resource utilization.

[0043] This patented solution: The core of the technical solution is to perform intelligent task partitioning based on the natural partitioning characteristics of frequent itemsets—equivalence classes—and combine adaptive parallel scheduling and distributed collaborative computing to achieve ultimate load balancing and horizontal scaling.

[0044] Intelligent task partitioning based on equivalence classes: Tasks are distributed according to equivalence classes, the natural partitioning unit of frequent itemsets, to ensure that the computational load handled by each computing node is relatively balanced.

[0045] Adaptive parallel scheduling: Combining dynamic resource allocation and adaptive query execution mechanisms, it automatically optimizes partitioning strategies and execution plans based on data characteristics. (In a preferred embodiment, this effect is achieved by combining Spark's dynamic resource allocation and adaptive query execution (AQE) mechanisms.)

[0046] Distributed collaborative mining: Parallel intersection calculation of transaction sets is achieved through efficient distributed aggregation operators, making full use of cluster computing resources. (In a preferred embodiment, efficient intersection calculation is achieved through Spark's reduceByKey operators, etc.) To verify the actual effect of the StrStream-Eclat algorithm in this invention, the DS-Tree, FIMoTS, and CanTree-GTree algorithms were compared experimentally. The experimental results are as follows: Figure 5-7 As shown, from Figure 5 It can be seen that the algorithm in this application is 3-4 times faster than DS-Tree and FIMoTS in terms of throughput (relative performance %); from Figure 6 It can be seen that the memory consumption of the algorithm in this application is reduced by more than 40% compared with the prefix tree algorithm, and the deployment cost is significantly reduced; from Figure 7 As can be seen, the algorithm in this application has far superior system scalability compared to the CanTree-GTree algorithm, achieving a near-linear speedup on a 16-node cluster.

[0047] As is known from common technical knowledge, this invention can be implemented through other embodiments that do not depart from its spirit or essential characteristics. Therefore, the disclosed embodiments described above are merely illustrative in all respects and are not the only ones. All modifications within the scope of this invention or its equivalents are included in this invention.

Claims

1. A method for mining precise frequent itemsets in streaming data, characterized in that: Includes the following steps: S1. Structured streaming data access and window management are achieved through a distributed stream processing engine; S2. Convert the streaming transaction data into a vertical format of project and transaction ID sets, and use the state management function to maintain the vertical database state. S3. Filter infrequent items based on the minimum support threshold and perform equivalence class partitioning. Then, partition the conditional database to each computing node based on the equivalence classes. S4. Execute the StrStream-Eclat algorithm in parallel on each computing node to calculate the intersection of transaction ID sets through distributed aggregation operators; S5. Update the state using an incremental state update mechanism and output the mining results.

2. The method for mining precise frequent itemsets in streaming data according to claim 1, characterized in that: In step S1, structured stream data access and window management are performed through a distributed stream processing engine, specifically including: (1) Create a data stream through the stream processing API data source; (2) Use window functions to define the processing window and set the window duration and sliding interval parameters; (3) Configure watermark parameters to process delayed data; (4) Group the data by window through the grouping operation.

3. The method for mining precise frequent itemsets in streaming data according to claim 1, characterized in that: In step S2, the streaming transaction data is converted into a vertical format of project and transaction ID sets, and the vertical database state is maintained using the state management function. Specifically, this includes: (1) Expand the transaction data into <project, transaction ID> key-value pairs; (2) Generate a vertical data format of <project, transaction ID set> by key aggregation; (3) Maintain the state of the vertical database using the state management function of the stream processing engine; (4) Set the status timeout parameter to automatically clean up expired data.

4. The method for mining precise frequent itemsets in streaming data according to claim 1, characterized in that: In step S3, infrequent items are filtered based on the minimum support threshold, and equivalence classes are partitioned. The conditional database is then partitioned into different computational nodes based on these equivalence classes. Specifically, this includes: (1) Filter infrequent items based on the minimum support threshold parameter; (2) Use breadth-first or depth-first strategy parameters for equivalence class partitioning; (3) Use a distributed dataset to automatically partition the conditional database across computing nodes; (4) Set the number of partitions to control the parallelism.

5. The method for mining precise frequent itemsets in streaming data according to claim 1, characterized in that: In step S4, the StrStream-Eclat algorithm is executed in parallel on each computing node. The intersection calculation of the transaction ID sets is achieved through distributed aggregation operators, specifically including: (1) Run the core logic of the StrStream-Eclat algorithm in parallel on each executor; (2) The intersection of transaction sets is calculated through key reduction operations; (3) Use accumulator parameters to collect global statistics; (4) Configure execution memory parameters to optimize computing performance.

6. The method for mining precise frequent itemsets in streaming data according to claim 1, characterized in that: In step S5, the incremental state update mechanism is used to update the state and output the mining results, specifically including: (1) Use stateful mapping grouping operations to implement incremental state updates; (2) Output the complete results through batch processing functions or output mode parameters; (3) Set checkpoint location parameters to ensure fault recovery; (4) Configure the trigger interval parameter to control the processing rhythm.

7. The method for mining precise frequent itemsets in streaming data according to claim 1, characterized in that: The data source supports Kafka, file system, or Socket access.

8. A system for mining exact frequent itemsets in streaming data, used to implement the method described in any one of claims 1-7, characterized in that, include: The streaming data access module is used to access streaming data and manage windows. The vertical format conversion module is used to convert data to a vertical format and manage its state; The equivalence class partitioning module is used to partition tasks based on equivalence classes and distribute them to computing nodes. The distributed mining module is used to perform precise frequent itemset mining in parallel across nodes; The state management module is used to incrementally update the state and output the results.

Images