A non-overlapping high utility sequential pattern mining method and system for a target

Abstract

The application provides a non-overlapping high utility sequence pattern mining method and system for a target, which comprises the following steps: preprocessing a sequence database, filtering sequences not containing the target sequence; then adopting a right-to-left greedy search to locate the last effective instance path of the target sequence; then converting the sequence into a q matrix format, calculating sequence expansion utility, and screening a hopeful 1-sequence; subsequently constructing a Nettree data structure to find an instance path satisfying a non-overlapping condition, and calculating actual non-overlapping utility; further performing multi-level pruning processing such as sequence pre-reading pruning, width pruning, cooperative occurrence pruning and depth pruning; finally adopting a depth-first search to perform recursive pattern growth, and outputting a non-overlapping high utility sequence pattern set satisfying a minimum utility threshold and containing the target sequence. The application improves mining efficiency and result quality, and can efficiently find high-value patterns in large-scale sequence data.

Description

Technical Field

[0001] This invention relates to the field of data mining technology, and in particular to a non-overlapping, high-efficiency sequence pattern mining method and system for targeting, which can be applied to APT attack path mining in information security or equipment failure prediction in intelligent manufacturing. Background Technology

[0002] Data mining has been a research hotspot for over a decade, with its core objective being to discover valuable patterns from large datasets. Pattern mining, as an important branch of data mining, focuses on discovering various types of data patterns, such as APT attack path mining in cybersecurity and equipment failure prediction in smart manufacturing. Traditional frequent itemset mining (FIM) is a fundamental task used to discover frequently co-occurring itemsets in transactional databases, laying the foundation for association rule mining (ARM).

[0003] Early sequence pattern mining (SPM) algorithms, such as GSP, employed a candidate generation-testing method based on the Apriori principle. Later algorithms, such as PrefixSpan, which are based on pattern growth, significantly improved performance by eliminating explicit candidate generation. However, traditional SPM algorithms only focus on frequency (support), often discovering a large number of common but low-value patterns while ignoring uncommon but high-value patterns.

[0004] In recent years, the Highly Effective Sequence Pattern Mining (HUSPM) method has been proposed, which considers factors such as the profit and quantity of items to discover sequence patterns with high economic or practical value. However, this introduces new challenges: since utility does not possess anti-monotonicity, traditional frequency-based pruning methods fail, necessitating the development of novel pruning strategies based on upper bounds. Although HUSPM can discover all patterns that meet the threshold, it may generate a large number of results irrelevant to the user's specific interests.

[0005] Furthermore, in fields such as bioinformatics, patterns are often not strictly continuous due to factors such as gene mutations. To address this issue, the algorithm introduces gap constraints. When calculating the occurrence count of a pattern, the "non-overlapping condition" provides a practical solution. It allows characters in a sequence to be matched multiple times, but cannot match the same element at the same pattern position in different occurrences. This condition satisfies the anti-monotonic property, making it possible to develop complete and efficient mining algorithms.

[0006] In the field of cybersecurity, APT (Advanced Persistent Threat) attackers typically execute a series of meticulously designed penetration behaviors. These behaviors form an attack chain, including steps such as initial intrusion, privilege escalation, lateral movement, and data theft. Security analysts need to identify high-risk attack paths containing specific attack characteristic sequences from massive amounts of logs in order to detect and block attacks in a timely manner. Traditional methods are often based on frequency statistics, but attackers frequently use low-frequency, highly stealthy methods to evade detection. Furthermore, the severity (utility) of attack behaviors varies greatly, making it difficult for purely frequency-based mining methods to distinguish truly high-risk attack sequences.

[0007] In the field of smart manufacturing, equipment often generates a large number of sensor data sequences (such as parameters like temperature, pressure, and vibration). These sequences may contain specific patterns that indicate potential faults. For example, a specific combination of bearing overheating, insufficient lubrication, and abnormal vibration may foreshadow an impending spindle failure. Identifying these precursory patterns is crucial for predictive maintenance. Traditional methods primarily focus on outlier detection, lacking a comprehensive consideration of the temporal relationships and economic impacts (such as downtime and scrap costs) within the sequence patterns.

[0008] Therefore, a sequence pattern mining method that can simultaneously consider target sequence constraints, non-overlapping conditions, and utility metrics has significant practical value for discovering high-risk attack paths in network security and equipment failure precursors in smart manufacturing. Summary of the Invention

[0009] The purpose of this invention is to provide a non-overlapping, high-efficiency sequence pattern mining method and system for targets, in order to solve the above-mentioned technical problems.

[0010] To achieve the above objectives, the present invention provides the following technical solution:

[0011] This invention provides a non-overlapping, high-efficiency sequence pattern mining method for targets, applicable to APT attack path mining in information security or equipment fault prediction in smart manufacturing, including:

[0012] For each sequence in the original sequence database, perform subsequence judgment, filter out sequences that do not contain the target sequence, and retain the sequences that contain the target sequence in the preprocessed sequence database to obtain a preprocessed sequence database that contains only the target sequence;

[0013] For each sequence in the preprocessed sequence database containing only the target sequence, a greedy search from right to left is used to locate the last valid instance path of the target sequence in the sequence that satisfies the gap constraint and length constraint. The last valid instance path is stored in the last instance table to obtain the last instance table containing the last valid instance path.

[0014] Each sequence in the preprocessed sequence database containing only the target sequence is converted into a q-matrix format containing a utility matrix and a residual utility matrix. The sequence extension utility of each item is calculated in conjunction with the last instance table containing the last valid instance path. Items with sequence extension utility not less than the minimum utility threshold are selected as promising 1-sequences.

[0015] For each pattern in the promising 1-sequence, construct a Nettree data structure containing multiple root nodes and multiple parent nodes, find instance paths that satisfy the non-overlapping condition in the Nettree data structure, and calculate the actual non-overlapping utility of the pattern.

[0016] Based on the anti-monotonic properties of the actual non-overlapping utility and the upper bound of the extended utility, the extended terms of the pattern are subjected to multi-level pruning processing including sequence pre-reading pruning, width pruning, co-occurrence pruning and depth pruning to obtain a promising set of extended terms.

[0017] For the set of promising expansion items, a depth-first search is used to recursively grow the pattern, and the output is a set of non-overlapping high-utility sequence patterns that satisfy the minimum utility threshold and contain the target sequence.

[0018] Further, the step of performing subsequence determination on each sequence in the sequence database, filtering out sequences that do not contain the target sequence, and retaining sequences that contain the target sequence in the preprocessed sequence database includes:

[0019] Initialize the preprocessed sequence database, the final instance table, and the total database utility value;

[0020] For each sequence in the original sequence database, check whether the length of the sequence is not less than the length of the target sequence, and perform subsequence matching judgment on the sequences that meet the length condition, and filter out sequences that do not contain the target sequence;

[0021] For a sequence containing the target sequence, a valid instance path of the target sequence in the sequence is found based on gap constraints and length constraints. Sequences without valid instance paths are filtered out, and sequences with valid instance paths are retained in the preprocessed sequence database, resulting in a preprocessed sequence database containing only the target sequence.

[0022] Further, the step of using a right-to-left greedy search to locate the last valid instance path of the target sequence that satisfies the gap constraint and length constraint in the sequence, and storing the last valid instance path in the last instance table to obtain a last instance table containing the last valid instance path, includes:

[0023] Starting from the rightmost position of the sequence, scan to the left to find the matching position of the last itemset of the target sequence, and obtain the root node position;

[0024] Starting from the root node position, search for the matching position of each itemset in the target sequence to the left, and select the rightmost matching position that satisfies the gap constraint at each level to obtain the complete instance path;

[0025] Verify whether the span of the complete instance path meets the length constraint. The instance path that meets the length constraint is taken as the last valid instance path. Using the itemsets in the target sequence as column indexes and the sequence identifier as row indexes, the matching positions of the itemsets in the last valid instance path are stored in the last instance table to obtain the last instance table containing the last valid instance path.

[0026] Further, each sequence in the preprocessed sequence database containing only the target sequence is converted into a q-matrix format containing a utility matrix and a residual utility matrix. The sequence expansion utility of each item is calculated using the last instance table containing the last valid instance path. Items with sequence expansion utility not less than a minimum utility threshold are selected as promising 1-sequences, including:

[0027] For each sequence in the preprocessed sequence database containing only the target sequence, the product of the internal utility and external utility of each item in the sequence in each item set is calculated as the utility value. The sum of the utilities of all subsequent items from the position of each item in the sequence to the end of the sequence is calculated as the residual utility value. The utility value and the residual utility value are stored as a utility matrix and a residual utility matrix, respectively, forming a q matrix.

[0028] For each item in the q matrix, the occurrence position of the item is determined by combining the last instance table containing the last valid instance path to determine whether it meets the condition of co-occurrence with the target sequence. For items that meet the co-occurrence condition, the sum of its utility value and the remaining utility value is calculated as the potential utility value. The maximum value of the potential utility value in each sequence is accumulated to obtain the sequence extension utility of the item.

[0029] Items whose sequence expansion utility is not less than the minimum utility threshold are selected as promising 1-sequences, wherein the minimum utility threshold is the product of a threshold ratio set by the user and the total utility value of the preprocessed sequence database containing only the target sequence.

[0030] Further, for each pattern in the promising 1-sequence, a Nettree data structure containing multiple root nodes and multiple parent nodes is constructed. In the Nettree data structure, instance paths satisfying the non-overlapping condition are searched, and the actual non-overlapping utility of the pattern is calculated, including:

[0031] For each itemset in the pattern, a layer of Nettree data structure is built, and all positions in the sequence that match the itemset are collected as nodes of the layer, and the utility value corresponding to each node is recorded;

[0032] Between adjacent layers of the Nettree data structure, connections are established between node pairs that satisfy the positional order relationship and whose gaps are within the gap constraint range. Isolated nodes without valid paths to the upper layer nodes are removed from the Nettree data structure to obtain the Nettree data structure after removing isolated nodes.

[0033] For the Nettree data structure after removing isolated nodes, enumerate all complete paths from the root node to the absolute leaf node, select the set of paths that satisfy the non-overlapping condition from the complete paths, and sum the utility values ​​of each path in the set of paths to obtain the actual non-overlapping utility of the pattern.

[0034] Furthermore, based on the antimonotonic properties of the actual non-overlapping utility and the upper bound of the extended utility, the extended terms of the pattern undergo multi-level pruning processing, including sequence pre-reading pruning, width pruning, co-occurrence pruning, and depth pruning, to obtain a promising set of extended terms, including:

[0035] For each sequence in the preprocessed sequence database containing only the target sequence, check whether the sequence contains a valid non-overlapping instance of the target sequence under the conditions of satisfying the gap constraint and the length constraint. Remove sequences that do not contain valid non-overlapping instances from the current mining range as sequence prereading pruning.

[0036] Calculate the corresponding terminating descendant utility for the pattern, wherein the terminating descendant utility is the sum of the sequence expansion utility of the effective extension of the pattern in each sequence. When the terminating descendant utility is less than the minimum utility threshold, the pattern and all corresponding extensions are pruned as width pruning.

[0037] For each extension item of the pattern, calculate the extension utility of the extension item, where the extension utility is the cumulative value in the database of the sum of the utility of the parent instance, the utility of the extension item, and the remaining utility. When the extension utility is less than the minimum utility threshold, remove the extension item from the effective extension set as a co-occurrence pruning.

[0038] For the candidate patterns generated after the co-occurrence pruning, the corresponding sequence expansion utility is calculated. When the sequence expansion utility is less than the minimum utility threshold, the recursive expansion of the candidate patterns is stopped, which is considered as deep pruning.

[0039] The extensions retained after the sequence pre-reading pruning, the width pruning, the co-occurrence pruning, and the depth pruning are combined to form the promising extension set.

[0040] Furthermore, before performing subsequence judgment on each sequence in the original sequence database, filtering out sequences that do not contain the target sequence, and retaining sequences containing the target sequence in the preprocessed sequence database, the method further includes performing sequence quality checking on the original sequence database, including:

[0041] For each sequence in the original sequence database, extract the total sequence utility, matching degree with the target sequence, sequence complexity, and temporal coherence as multi-dimensional feature vectors, and use the greedy search to verify the true label of the sequence to obtain a labeled feature dataset;

[0042] For the labeled feature dataset, based on the constraint that the label noise rate does not exceed a preset upper bound, a half-space classifier is trained in the utility-frequency space to obtain a classification hyperplane for distinguishing between real effective sequences and noise-contaminated sequences.

[0043] Cross-validation is performed on the classification hyperplane, and the confidence level of the classification accuracy is calculated based on the Hoeffding inequality. When the confidence level is not less than a preset confidence threshold, the classification hyperplane is determined to be an effective classifier.

[0044] For each sequence in the original sequence database, the effective classifier is used to calculate the corresponding predicted label and predicted confidence. Sequences whose predicted label contains a valid target sequence and whose predicted confidence is not less than a preset confidence threshold are retained. Sequences whose predicted label does not contain a valid target sequence and whose predicted confidence is not less than the preset confidence threshold are filtered. Sequences whose predicted confidence is less than the preset confidence threshold are marked as suspicious sequences.

[0045] Multiple greedy search verifications are performed on the suspicious sequences, and the suspicious sequences that pass the verification are retained in the preprocessed sequence database.

[0046] Furthermore, the construction of a Nettree data structure containing multiple root nodes and multiple parent nodes, after searching for instance paths that satisfy the non-overlapping condition in the Nettree data structure, further includes performing critical instance path identification processing on the Nettree data structure, including:

[0047] The search space of the Nettree data structure is mapped to a multidimensional polygon, where each layer corresponds to one dimension, and the position, utility value, remaining utility value, and number of paths to the leaf of each node constitute the vertex coordinates of the multidimensional polygon.

[0048] The multidimensional polygon is recursively subdivided along the direction of utility difference to decompose the multidimensional polygon into multiple sub-regions, where each sub-region corresponds to a type of instance path.

[0049] For each of the plurality of sub-regions, calculate the sum of the utility contributions of all instance paths within the sub-region and the path overlap between the sub-region and other sub-regions. Use the product of the utility contribution and the path overlap as a critical score. Mark the sub-regions whose critical scores are not less than a preset critical threshold as critical sub-regions.

[0050] Greedy search and constraint verification are performed preferentially on the instance paths in the critical sub-region to obtain a set of valid non-overlapping instances within the critical region;

[0051] Perform a supplementary search on instance paths in non-critical regions outside the critical sub-region, merge the set of valid non-overlapping instances in the critical region with the search results in the non-critical region, and prune non-critical branches in the merged instance paths whose upper bound of utility is lower than a preset multiple of the minimum utility threshold.

[0052] Further, the recursive pattern growth of the promising expansion item set using depth-first search, outputting a set of non-overlapping high-utility sequence patterns that satisfy the minimum utility threshold and contain the target sequence, includes performing a borrowing graph-based scheduling process on the recursive pattern growth task, the scheduling process including:

[0053] For each of the promising extension items in the set of extension items, a pattern exploration task is created. For each pattern exploration task, a task identifier, parent pattern, and the sequence extension utility of the pattern are set as priority and estimated processing time, and a set of pattern exploration tasks is constructed.

[0054] A directed graph is constructed to establish the parent-child dependency relationship and resource competition relationship among the tasks in the set of pattern exploration tasks, resulting in a borrowing graph, wherein a dependency edge is established between the parent task and the child task, and a resource competition edge is established between tasks that require the same computing resources.

[0055] For each pattern exploration task in the borrowed graph, a delay perspective score is calculated for the pattern exploration task, wherein the delay perspective score is the sum of its own delay cost and cascaded delay cost minus the resource release benefit. The own delay cost is the product of the priority of the pattern exploration task and the estimated processing time. The cascaded delay cost is the sum of the products of the priority of all descendant tasks of the pattern exploration task and the estimated processing time. The resource release benefit is the sum of the priority-weighted delay differences of tasks that compete with the pattern exploration task for resources.

[0056] Based on the delayed perspective score, the set of pattern exploration tasks is prioritized and sorted. The tasks with the delayed perspective scores from largest to smallest are assigned to work threads in sequence. When all work threads are occupied, subsequent tasks are placed in a priority queue to wait.

[0057] When the pattern exploration task is completed, the subtasks of the pattern exploration task are added to the priority queue. The delay perspective score of the tasks in the priority queue is recalculated and scheduled for execution until the priority queue is empty. The patterns in all completed pattern exploration tasks that meet the minimum utility threshold and contain the target sequence are output as the set of non-overlapping high-utility sequence patterns.

[0058] Furthermore, prioritizing the set of pattern exploration tasks based on the delayed perspective score and sequentially assigning the tasks with the delayed perspective scores from largest to smallest to worker threads also includes performing dynamic resource borrowing and parallel recursive mining processing on the pattern exploration tasks, including:

[0059] When the target pattern exploration task requests resource borrowing, the currently executing pattern exploration tasks in the borrowing graph are traversed, and pattern exploration tasks with a latency tolerance greater than the estimated processing time of the target pattern exploration task and not on the dependency path of the target pattern exploration task are selected as the set of available borrowing tasks. From the set of available borrowing tasks, the pattern exploration task with the highest latency tolerance is selected as the resource lending task, and the resource lending task is obtained.

[0060] Allocate a preset proportion of computational resources to the target mode exploration task for the resource lending task, and add a borrowing edge from the target mode exploration task to the resource lending task in the borrowing graph, and record the amount of resources borrowed.

[0061] The estimated processing time of each pattern exploration task in the set of pattern exploration tasks is compared. When the number of pattern exploration tasks exceeds a preset multiple of the number of worker threads, adjacent pattern exploration tasks with estimated processing time differences within a preset time threshold range are merged into batch tasks. When the number of pattern exploration tasks does not exceed the preset multiple, single task scheduling is maintained, resulting in a set of scheduled tasks containing batch tasks and single tasks.

[0062] For each batch task and single task in the scheduling task set, sort them from largest to smallest according to the estimated processing time, and then allocate the sorted batch tasks and single tasks to the working thread with the smallest current load in turn, so that the total load of each working thread tends to be balanced, and the task allocation scheme of each working thread is obtained.

[0063] Based on the task allocation scheme, each of the worker threads executes the allocated pattern exploration task in parallel, performs a depth-first search recursive pattern growth on the patterns in each pattern exploration task, and merges the patterns that meet the minimum utility threshold and contain the target sequence into the non-overlapping high-utility sequence pattern set.

[0064] Furthermore, the set of promising expansion items is recursively expanded using a depth-first search, outputting a set of non-overlapping high-utility sequence patterns that satisfy the minimum utility threshold and contain the target sequence, including:

[0065] For each extension item in the set of promising extension items, determine the extension type of the extension item relative to the current pattern, wherein the extension type includes I-extension, which adds the extension item to the last itemset of the current pattern, and S-extension, which adds the extension item as a new itemset to the end of the current pattern.

[0066] Based on the expansion type, perform an expansion operation on the current pattern to generate a sub-pattern, and recursively perform non-overlapping instance lookup, actual non-overlapping utility calculation, and multi-level pruning on the sub-pattern.

[0067] When the actual non-overlapping utility of the sub-pattern is not less than the minimum utility threshold and the sub-pattern contains the target sequence, the sub-pattern is added to the set of non-overlapping high-utility sequence patterns.

[0068] This invention also provides a non-overlapping, high-efficiency sequence pattern mining system for targets, applicable to APT attack path mining in information security or equipment fault prediction in intelligent manufacturing, comprising:

[0069] The sequence preprocessing module is used to perform subsequence judgment on each sequence in the original sequence database, filter out sequences that do not contain the target sequence, and retain the sequences that contain the target sequence in the preprocessed sequence database to obtain a preprocessed sequence database containing only the target sequence.

[0070] The last instance location module is used to locate the last valid instance path of the target sequence in the preprocessed sequence database containing only the target sequence by using a greedy search from right to left. The last valid instance path is stored in the last instance table to obtain the last instance table containing the last valid instance path.

[0071] The sequence expansion utility calculation module is used to convert each sequence in the preprocessed sequence database containing only the target sequence into a q matrix format containing a utility matrix and a residual utility matrix, calculate the sequence expansion utility of each item in conjunction with the last instance table containing the last valid instance path, and filter items whose sequence expansion utility is not less than the minimum utility threshold as promising 1-sequences.

[0072] The actual non-overlapping utility calculation module is used to construct a Nettree data structure containing multiple root nodes and multiple parent nodes for each pattern in the promising 1-sequence, search for instance paths that satisfy the non-overlapping condition in the Nettree data structure, and calculate the actual non-overlapping utility of the pattern.

[0073] A multi-level pruning module is used to perform multi-level pruning processing, including sequence pre-reading pruning, width pruning, co-occurrence pruning and depth pruning, on the extension items of the pattern based on the anti-monotonic properties of the actual non-overlapping utility and the upper bound of the extended utility, so as to obtain a promising set of extension items.

[0074] The recursive pattern growth module is used to perform recursive pattern growth on the set of promising expansion items using a depth-first search, and outputs a set of non-overlapping high-utility sequence patterns that meet the minimum utility threshold and contain the target sequence.

[0075] Compared with the prior art, the present invention has the following beneficial effects:

[0076] 1. This invention proposes a target-oriented non-overlapping efficient sequence pattern mining method, which for the first time integrates the concept of utility into target-oriented non-overlapping sequence pattern mining, and solves the limitations of existing technologies in terms of relevance, practical value and measurement accuracy;

[0077] 2. This invention optimizes the Nettree data structure, enabling it to handle itemsets containing one or more items, thus expanding the applicability of non-overlapping sequence pattern mining methods.

[0078] 3. This invention designs a novel upper bound for extended utility (EU) and proves its anti-monotonicity, providing a theoretical basis for efficient pruning;

[0079] 4. This invention proposes a greedy right-direction search method, proving that it can accurately find the last instance of a pattern, providing key technical support for the efficient construction of LI-Table;

[0080] 5. This invention constructs a multi-level pruning strategy framework, including sequence pre-reading pruning strategy (SPPS), width pruning strategy (WPS), co-occurrence pruning strategy (CPS), and depth pruning strategy (DPS). These strategies complement each other and significantly reduce the search space.

[0081] 6. This invention develops the HUNUS algorithm, which integrates all technical modules and can efficiently mine all required patterns that meet the user-defined minimum utility threshold.

[0082] 7. In the field of cybersecurity, this invention can efficiently mine high-risk threat paths containing specific attack characteristics from massive logs, effectively identify APT attack chains, and achieve early threat detection. In the field of smart manufacturing, this invention can mine high-cost equipment failure modes containing specific precursory symptoms from equipment sensor data, enabling predictive maintenance and avoiding high losses. Attached Figure Description

[0083] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0084] Figure 1 This is a flowchart illustrating the non-overlapping efficient sequence pattern mining method for targets according to an embodiment of the present invention;

[0085] Figure 2 This is a schematic diagram of the structure of a non-overlapping efficient sequence pattern mining system for targets according to an embodiment of the present invention. Detailed Implementation

[0086] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0087] Example 1

[0088] like Figure 1As shown, this invention provides a non-overlapping, high-efficiency sequence pattern mining method for targets, applicable to APT attack path mining in information security or equipment fault prediction in intelligent manufacturing, including:

[0089] Step S1: Perform subsequence judgment on each sequence in the original sequence database, filter out sequences that do not contain the target sequence, and retain the sequences that contain the target sequence in the preprocessed sequence database to obtain a preprocessed sequence database that only contains the target sequence;

[0090] Step S2: For each sequence in the preprocessed sequence database containing only the target sequence, a greedy search from right to left is used to locate the last valid instance path of the target sequence in the sequence that satisfies the gap constraint and length constraint. The last valid instance path is stored in the last instance table to obtain the last instance table containing the last valid instance path.

[0091] Step S3: Convert each sequence in the preprocessed sequence database containing only the target sequence into a q matrix format containing a utility matrix and a residual utility matrix. Calculate the sequence extension utility of each item in conjunction with the last instance table containing the last valid instance path. Select items whose sequence extension utility is not less than the minimum utility threshold as promising 1-sequences.

[0092] Step S4: For each pattern in the promising 1-sequence, construct a Nettree data structure containing multiple root nodes and multiple parent nodes, find instance paths that satisfy the non-overlapping condition in the Nettree data structure, and calculate the actual non-overlapping utility of the pattern.

[0093] Step S5: Based on the antimonotonic properties of the actual non-overlapping utility and the upper bound of the extended utility, perform multi-level pruning processing on the extended terms of the pattern, including sequence pre-reading pruning, width pruning, co-occurrence pruning and depth pruning, to obtain a set of promising extended terms.

[0094] Step S6: For the set of promising expansion items, perform recursive pattern growth using depth-first search, and output a set of non-overlapping high-utility sequence patterns that satisfy the minimum utility threshold and contain the target sequence.

[0095] Specifically, a non-overlapping high-utility sequence pattern mining method targeting specific objectives is proposed. This method is particularly suitable for fields such as information security and intelligent manufacturing, and can efficiently discover high-utility sequence patterns containing specific target sequences. This method combines traditional sequence pattern mining techniques with non-overlapping utility constraints and objective-oriented strategies, and develops a complete mining process, including key steps such as data preprocessing, candidate generation, instance verification, utility calculation, and pattern growth.

[0096] First, the method performs subsequence identification and preprocessing on the original sequence database. Subsequence identification is the process of determining whether a sequence contains the target sequence. It checks whether there exists a subsequence in the original sequence that satisfies the gap constraint and matches the target sequence. The gap constraint defines the minimum and maximum allowed intervals between adjacent items in the target sequence, which is crucial for capturing patterns with a certain degree of temporal flexibility. Subsequence identification is typically implemented using dynamic programming or greedy algorithms, performing a full scan on each sequence. During the matching process, it considers whether each itemset of the target sequence can find a corresponding match in the original sequence and verifies whether the interval between the matching positions satisfies the constraint conditions. This preprocessing step significantly reduces the data size of the subsequent mining process because only sequences containing the target sequence are likely to generate the patterns of interest. The preprocessed sequence database retains only those sequences that are determined to contain the target sequence, which greatly reduces computational complexity, especially when the target sequence is relatively special or rare.

[0097] Secondly, the method identifies the last valid instance path for each sequence in the preprocessed database. A right-to-left greedy search strategy is employed, searching for matching instances of the target sequence from the end of the sequence backwards. This right-to-left search direction has specific advantages: in many applications (such as attack path analysis), the most recently occurring events are often more timely and impactful; furthermore, this search direction better handles potential subsequent expansion patterns within the sequence. Greedy search is a heuristic strategy that selects the best match at each step; while not guaranteeing global optimality, it is computationally efficient. The search process strictly adheres to gap constraints (limiting the maximum and minimum intervals between adjacent matches) and length constraints (limiting the maximum length or time span of the entire matching instance). The found last valid instance path is recorded in a last instance table, which stores path information for each sequence, including key data such as the matching position, utility value, and timestamp. This path information provides the foundation for subsequent sequence expansion utility calculations and non-overlapping utility evaluations.

[0098] Next, the method transforms the sequence data into a matrix representation and filters initial candidates. It converts the sequence into a q-matrix format, an efficient representation of sequence data consisting of a utility matrix and a residual utility matrix. The utility matrix records the utility value of each item at each position in the sequence, typically the product of the item's internal utility (e.g., quantity or weight) and its external utility (e.g., unit price or importance) at that position. The residual utility matrix records the sum of the utilities of all items from the current position to the end of the sequence, crucial for quickly estimating the upper bound of sequence expansion. Based on the q-matrix representation and the final instance table, the method calculates the sequence expansion utility for each item. Sequence expansion utility is an upper bound indicator representing the maximum possible utility value achievable based on that item; its calculation considers the item's utility value across sequences and the potential utility from subsequent expansions. By comparing the sequence expansion utility with a minimum utility threshold, promising 1-sequences—single-item patterns whose sequence expansion utility reaches or exceeds the minimum utility threshold—are selected, potentially leading to high-utility patterns through further expansion. This screening process significantly reduces the size of the initial candidate set, providing a high-quality starting point for subsequent recursive pattern growth.

[0099] Next, the method constructs the Nettree data structure and calculates the actual non-overlapping utility. Nettree is a tree-like data structure specifically designed for representing sequence pattern instances. Its key feature is that nodes can have multiple parent nodes and multiple child nodes, forming a mesh-like tree structure. For each promising 1-sequence, the method constructs a corresponding Nettree, where each level corresponds to an itemset in the pattern, and each node represents a position in the sequence that matches that itemset. Nodes in adjacent levels are connected based on positional order and gap constraints, forming possible matching paths. Unlike traditional path search, the Nettree structure supports efficient searching for instance paths that satisfy the non-overlapping condition. The non-overlapping condition requires that the selected instance paths should not overlap, ensuring that each sequence position is calculated at most once for utility. In Nettree, the method enumerates the complete paths from the root node to the leaf node, then selects a set of non-overlapping paths, calculates their sum of utility, and obtains the actual non-overlapping utility of the pattern. This value is a direct measure of the pattern's true value and is used to determine whether the pattern meets the minimum utility threshold requirement. Unlike upper bound indicators such as sequence extension utility, actual non-overlapping utility is a precise value that accurately reflects the actual value of a model after considering non-overlapping constraints.

[0100] Furthermore, multi-level pruning is performed based on utility upper bounds. Pruning is a key technique for reducing search space and improving mining efficiency. Four pruning strategies are designed using the anti-monotonic properties of actual non-overlapping utility and extended utility upper bounds: sequence pre-reading pruning removes sequences that cannot contain valid non-overlapping instances at the database level; width pruning eliminates invalid pattern families in one go by calculating the terminal descendant utility of the pattern; co-emergence pruning targets individual extension items, filtering those with insufficient co-occurrence utility with the current pattern; and depth pruning controls the search depth during recursion, stopping further exploration of low-utility pattern branches. These pruning strategies work together to form a multi-level, comprehensive pruning system that can significantly reduce the number of candidate patterns to be explored without overlooking any high-utility patterns. The output of multi-level pruning is a set of promising extension items, i.e., those items that can form high-utility patterns by combining with the current pattern. These items form the basis of recursive pattern growth and determine the branching direction of the search tree.

[0101] The final step is to use depth-first search for recursive pattern growth. Depth-first search is a strategy that starts from the root node, explores along a branch of the search tree to the leaf node, and then backtracks to the nearest unexplored branch to continue the search. During pattern growth, for each item in the set of promising extensions, the method determines its extension type (I-extension or S-extension), generates a sub-pattern, and then recursively performs the same process: building the Nettree, finding non-overlapping instances, calculating actual utility, performing pruning, and continuing to expand. This recursive process continues until a termination condition is met: either the pattern no longer has extensions that satisfy the conditions, or the preset search depth limit has been reached. Patterns with actual non-overlapping utility not less than the minimum utility threshold and containing the target sequence are collected into the final set of non-overlapping high-utility sequence patterns. This set is the final output of the method, containing high-value patterns that satisfy all constraints and can be directly used in downstream applications, such as attack path analysis or fault prediction.

[0102] The non-overlapping, high-efficiency sequence pattern mining method for specific targets has demonstrated outstanding performance in practical applications. In the field of information security, it can effectively identify complex attack paths containing specific malicious behavioral characteristics and discover the activity patterns of Advanced Persistent Threats (APTs). In smart manufacturing, it can discover equipment operation sequences containing specific precursory symptoms of equipment failure, enabling early warning of equipment malfunctions. Compared to traditional sequence pattern mining methods, this method significantly improves efficiency when processing large-scale, high-dimensional data, and can complete the exploration of complex pattern spaces within a reasonable timeframe. Furthermore, by introducing non-overlapping constraints and a target-oriented strategy, the method greatly reduces the redundancy of results and improves the interpretability and practical value of the patterns.

[0103] The key innovations of this method are: optimizing the search space by incorporating target sequence constraints; efficiently handling non-overlapping instance lookup using the Nettree data structure; significantly reducing candidate generation through a multi-level pruning strategy; and using the upper bound of sequence expansion utility to guide depth-first search. These innovations enable the method to accurately mine high-value patterns from massive sequence data for specific domain needs, providing a powerful tool for decision support and knowledge discovery.

[0104] Example 2

[0105] In this embodiment, the step of performing subsequence determination on each sequence in the sequence database, filtering out sequences that do not contain the target sequence, and retaining sequences containing the target sequence in the preprocessed sequence database includes:

[0106] Initialize the preprocessed sequence database, the final instance table, and the total database utility value;

[0107] For each sequence in the original sequence database, check whether the length of the sequence is not less than the length of the target sequence, and perform subsequence matching judgment on the sequences that meet the length condition, and filter out sequences that do not contain the target sequence;

[0108] For a sequence containing the target sequence, a valid instance path of the target sequence in the sequence is found based on gap constraints and length constraints. Sequences without valid instance paths are filtered out, and sequences with valid instance paths are retained in the preprocessed sequence database, resulting in a preprocessed sequence database containing only the target sequence.

[0109] Specifically, sequence preprocessing is the first crucial step in the entire non-overlapping high-utility sequence pattern mining method, aiming to reduce subsequent computation by filtering irrelevant sequences early on. First, three important data structures are initialized: the preprocessed sequence database stores the set of sequences that meet the criteria, typically implemented as a dynamic array structure; the final instance table is a two-dimensional table structure used to record the matching position information of the target sequence within each sequence; and the total utility value of the database is an accumulator used to calculate the sum of the utility of all retained sequences, serving as the basis for subsequent minimum utility threshold calculations.

[0110] Performing length checks on each sequence in the original sequence database is a fast filtering mechanism. Specifically, it calculates the number of itemsets in the sequence and compares it to the number of itemsets in the target sequence. If the sequence length is less than the target sequence length, it obviously cannot contain the target sequence and can be directly filtered out without performing more complex subsequence matching operations. This simple comparison greatly improves preprocessing efficiency. For example, if the target sequence contains 3 itemsets, while the currently checked sequence only has 2 itemsets, then the current sequence definitely does not contain the target sequence.

[0111] For sequences that pass the length check, a subsequence matching check is required, which is the fundamental operation for determining whether one sequence contains another. In this method, subsequence matching is implemented using dynamic programming, maintaining a matching state array to record the matching status of each itemset in the target sequence. The matching process starts from the first itemset of the original sequence and compares it sequentially with each itemset of the target sequence. If a complete matching path is found, it is confirmed that the original sequence contains the target sequence.

[0112] For sequences confirmed to contain the target sequence, valid instance paths are further searched based on gap constraints and length constraints. Gap constraints define the minimum and maximum allowed intervals between adjacent matching itemsets in the sequence, and are important parameters controlling pattern continuity. For example, a gap constraint [1,3] indicates that adjacent matching itemsets must be spaced at least one position apart and at most three positions apart. Length constraints define the minimum and maximum span of the entire pattern instance, and are important parameters controlling the pattern's time span. For example, a length constraint [5,10] indicates that the total span from the first matching itemset to the last matching itemset should be between 5 and 10 positions.

[0113] The search for valid instance paths employs a backtracking algorithm. Starting from the first itemset of the target sequence, it sequentially searches the original sequence for matching positions that satisfy the gap constraint until a complete path is found or it is confirmed that no path satisfying the conditions exists. Only instance paths that simultaneously satisfy both the gap and length constraints are considered valid. Sequences with valid instance paths are retained in the preprocessed sequence database; sequences without valid instance paths are filtered out. This multi-constraint filtering ensures that the final preprocessed sequence database contains only those sequences that truly contain valid target sequence instances, providing a high-quality data foundation for subsequent mining.

[0114] In practice, preprocessing typically includes data cleaning and standardization steps. Data cleaning primarily addresses missing values, outliers, and duplicates in the original sequence to ensure the quality of the input data. Standardization transforms sequences from different sources or formats into a unified internal representation, providing a consistent data interface for subsequent algorithms. Furthermore, preprocessing may include index building, creating efficient index structures for frequently accessed data to accelerate subsequent query and matching operations.

[0115] After preprocessing, the algorithm outputs three key results: a streamlined sequence database containing only valid sequences that include the target sequence; a final instance table recording the last valid match position of the target sequence in each retained sequence; and a total utility value for the database, used for subsequent utility threshold calculations. These three results together form the foundation for subsequent high-utility pattern mining, significantly reducing the algorithm's search space and computational complexity.

[0116] Example 3

[0117] In this embodiment, the step of using a right-to-left greedy search to locate the last valid instance path of the target sequence that satisfies the gap constraint and length constraint in the sequence, and storing the last valid instance path in the last instance table to obtain a last instance table containing the last valid instance path, includes:

[0118] Starting from the rightmost position of the sequence, scan to the left to find the matching position of the last itemset of the target sequence, and obtain the root node position;

[0119] Starting from the root node position, search for the matching position of each itemset in the target sequence to the left, and select the rightmost matching position that satisfies the gap constraint at each level to obtain the complete instance path;

[0120] Verify whether the span of the complete instance path meets the length constraint. The instance path that meets the length constraint is taken as the last valid instance path. Using the itemsets in the target sequence as column indexes and the sequence identifier as row indexes, the matching positions of the itemsets in the last valid instance path are stored in the last instance table to obtain the last instance table containing the last valid instance path.

[0121] Specifically, the greedy right-directed search method is a special search strategy designed to efficiently locate the last valid instance path of a target sequence within the original sequence. Unlike the traditional left-to-right search, this method searches from the right end of the sequence to the left. This reverse search strategy is particularly suitable for finding the "last" instance path.

[0122] The search process begins by scanning leftwards from the rightmost position of the original sequence, searching for a matching position in the last itemset of the target sequence. Specifically, starting with the last itemset of the original sequence, each itemset is checked leftwards to see if it contains all items from the last itemset of the target sequence. Itemset matching uses set inclusion relations; that is, the original sequence itemset must contain all items from the corresponding itemset of the target sequence for a successful match. The first matching position found is called the "root node position," which is the starting point for constructing the complete instance path. This right-to-left search strategy ensures that the rightmost (i.e., the last) matching position is found.

[0123] Next, the search continues to the left from the root node to find the matching position of the second-to-last itemset in the target sequence, then the third-to-last, and so on until the first itemset of the target sequence. At each level of the search, gap constraints are considered. For example, if the gap constraint is [minGap, maxGap], and a match has been found for the j-th itemset in the target sequence at position p in the original sequence, then the matching position of the (j-1)-th itemset must be within the range of p-maxGap-1 to p-minGap-1. Among the multiple possible matching positions that satisfy the gap constraint, the algorithm selects the rightmost position, consistent with the principle of greedy rightward search, ensuring that the last valid instance path is found.

[0124] Through the above layer-by-layer leftward search, a complete instance path is finally obtained, represented as a series of position indices, such as [3, 5, 8], corresponding to the matching positions of each item set of the target sequence in the original sequence. Then, the complete instance path is verified to ensure it meets the length constraint. The path span is calculated by subtracting the first position from the last position and adding 1; for example, the span of the path [3, 5, 8] is 8 - 3 + 1 = 6. If the span value falls within the length constraint [minLen, maxLen], the instance path is considered to meet the length constraint; otherwise, it is necessary to continue searching for other possible paths or confirm that no path satisfies all constraints.

[0125] Finally, the last valid instance paths that satisfy all constraints are stored in the last instance table. The last instance table is a two-dimensional table structure, with itemsets in the target sequence as column indices and sequence identifiers as row indices. Each cell stores the matching position of the corresponding itemset in the corresponding sequence. For example, if the three itemsets of the target sequence in sequence S1 match at positions 3, 5, and 8 respectively, then in the last instance table, the three column values ​​of row S1 are 3, 5, and 8 respectively. This storage structure allows subsequent algorithms to quickly access the matching position information of the target sequence in each sequence, providing crucial support for efficient computation.

[0126] The greedy rightward search method employs a stack structure to track the search path. The algorithm first pushes the root node's position onto the stack, then continuously attempts to expand leftward, pushing each matching position that meets the conditions onto the stack until the entire target sequence has been matched or it is confirmed that no valid path can be found. Compared to traditional backtracking search, greedy rightward search leverages the problem's unique structure, significantly reducing the search space and improving search efficiency by moving from right to left and greedily selecting the rightmost matching position.

[0127] In terms of performance optimization, algorithms typically incorporate auxiliary data structures such as binary indexes or hash tables to accelerate the itemset matching process. For example, an inverted index can be built for each item in the original sequence, recording all occurrences of that item to quickly locate possible matching positions. Furthermore, algorithms can utilize caching mechanisms to store computed intermediate results, avoiding redundant calculations. These optimizations collectively ensure efficient search performance on large-scale sequence data.

[0128] Once the final instance table is constructed, it becomes the core reference structure for subsequent pattern mining. Through this table, the algorithm can quickly locate the target sequence within each original sequence, providing fundamental information for subsequent calculations of pattern utility, assessment of expansion feasibility, and execution of pattern growth, significantly improving the efficiency of the entire mining process.

[0129] Example 4

[0130] In this embodiment, each sequence in the preprocessed sequence database containing only the target sequence is converted into a q-matrix format containing a utility matrix and a residual utility matrix. The sequence expansion utility of each item is calculated using the last instance table containing the last valid instance path. Items with sequence expansion utility not less than a minimum utility threshold are selected as promising 1-sequences, including:

[0131] For each sequence in the preprocessed sequence database containing only the target sequence, the product of the internal utility and external utility of each item in the sequence in each item set is calculated as the utility value. The sum of the utilities of all subsequent items from the position of each item in the sequence to the end of the sequence is calculated as the residual utility value. The utility value and the residual utility value are stored as a utility matrix and a residual utility matrix, respectively, forming a q matrix.

[0132] For each item in the q matrix, the occurrence position of the item is determined by combining the last instance table containing the last valid instance path to determine whether it meets the condition of co-occurrence with the target sequence. For items that meet the co-occurrence condition, the sum of its utility value and the remaining utility value is calculated as the potential utility value. The maximum value of the potential utility value in each sequence is accumulated to obtain the sequence extension utility of the item.

[0133] Items whose sequence expansion utility is not less than the minimum utility threshold are selected as promising 1-sequences, wherein the minimum utility threshold is the product of a threshold ratio set by the user and the total utility value of the preprocessed sequence database containing only the target sequence.

[0134] Specifically, the q-matrix is ​​a special data structure designed specifically for utility sequence calculation, consisting of a utility matrix and a residual utility matrix. For each preprocessed sequence, it first needs to be converted into q-matrix format for efficient calculation and storage of utility-related information. The design concept of the q-matrix originates from the work-breakdown structure in project planning, which provides a mathematical basis for upper bound estimation by decomposing the overall utility into current utility and potential future utility.

[0135] Utility value is a key indicator of an item's worth within a sequence, calculated by multiplying its internal utility by its external utility. Internal utility typically represents the quantity or importance of an item, such as the severity level of a cyberattack (levels 1-5) or the deviation value of a device anomaly (e.g., the degree Celsius the temperature exceeds the normal range). External utility represents the unit value or weight of an item, such as the importance of an attacked asset (graded by value) or the criticality of a device component (graded by replacement cost or scope of impact). For each item in each itemset within the sequence, its utility value is calculated and stored in a utility matrix. For example, for the itemset {a:3, b:2}, if item a has an external utility of 5 and item b has an external utility of 10, then the utility value of a is 3 × 5 = 15, and the utility value of b is 2 × 10 = 20.

[0136] The residual utility value is an important upper bound for estimating the potential value of an item. It is calculated as the sum of the utilities of all subsequent items from the current item's position to the end of the sequence. The residual utility value represents the maximum additional utility that can be obtained starting from this item and forms the basis for estimating the extended upper bound. In implementation, a reverse accumulation method is usually used: traversing backward from the end of the sequence, accumulating the utilities of processed items, and using the accumulated value as the residual utility value of the current item. The residual utility value of each item in the sequence is calculated and stored in the residual utility matrix. For example, if there are items b and c following item a in the sequence, with utilities of 20 and 30 respectively, then the residual utility value of a is 20 + 30 = 50. By pre-calculating these values, the q matrix greatly improves the efficiency of calculating the upper bound of subsequent utilities.

[0137] Sequence Extension Utility (SEU) is a key indicator for determining whether an item has the potential to form a high-utility pattern. It is a novel utility upper bound measure specifically designed for pattern mining containing target sequences. The calculation process first determines whether the item satisfies the co-occurrence condition with the target sequence. The co-occurrence condition check is based on the target sequence matching position recorded in the last instance table, determining whether the item's position allows it to form a valid extension pattern with the target sequence. In specific implementations, the relationship between the item's position and the positions of the last itemsets of the target sequence, as well as gap constraints, need to be considered. For example, if an item appears after the last itemset of the target sequence and the gap does not exceed the maximum gap constraint, it may be an S-extension item of the target sequence; if an item appears in the same position as an itemset of the target sequence, it may be an I-extension item of that itemset.

[0138] For items that satisfy the co-occurrence condition, their potential utility value is calculated, which is the sum of the utility value and the residual utility value. The potential utility value represents the maximum utility that the item can contribute as an extension item, and is an important local upper bound. To comprehensively evaluate the potential of an item in the entire database, it is necessary to find the maximum potential utility value of the item in each sequence, and then sum these maximum values ​​to obtain the sequence extension utility of the item. This summation method is based on the property of the sum of utilities of multiple sequences under non-overlapping conditions, ensuring that the sequence extension utility is an effective global upper bound.

[0139] The minimum utility threshold is a key parameter for selecting high-utility patterns. It is calculated as the product of the user-defined threshold percentage and the total utility value of the preprocessed sequence database. This relative threshold setting method is more intuitive and flexible, allowing users to express the level of utility they are interested in as a percentage without needing to know the specific absolute utility distribution. For example, if the user sets the threshold percentage to 10% and the total utility of the preprocessed database is 1000, then the minimum utility threshold is 100. Only items with a sequence expansion utility of not less than 100 will be retained as promising 1-sequences.

[0140] At the implementation level, the calculation of sequence expansion utility typically employs a two-stage strategy: the first stage scans the sequence database, calculating the potential utility value of each item in each sequence; the second stage aggregates the potential utility values ​​of each item to obtain the global sequence expansion utility. To improve computational efficiency, auxiliary data structures such as hash tables or tree structures are usually used to quickly locate and accumulate the potential utility values ​​of each item. Furthermore, for large-scale data, block processing and parallel computing techniques can be employed to further enhance computational speed.

[0141] The selected promising 1-sequences form the starting set for recursive pattern growth. These 1-sequences all have the potential to form highly efficient patterns and are worth further exploration; while those sequences whose expansion utility is below a threshold can be safely pruned. This early screening based on an upper bound significantly reduces the number of candidate patterns that need to be explored subsequently, which is an important guarantee of algorithm efficiency. Experiments show that this screening strategy can reduce the search space by up to 80% while ensuring that no truly efficient patterns are overlooked.

[0142] Example 5

[0143] In this embodiment, for each pattern in the promising 1-sequence, constructing a Nettree data structure containing multiple root nodes and multiple parent nodes, searching for instance paths that satisfy the non-overlapping condition in the Nettree data structure, and calculating the actual non-overlapping utility of the pattern, includes:

[0144] For each itemset in the pattern, a layer of Nettree data structure is built, and all positions in the sequence that match the itemset are collected as nodes of the layer, and the utility value corresponding to each node is recorded;

[0145] Between adjacent layers of the Nettree data structure, connections are established between node pairs that satisfy the positional order relationship and whose gaps are within the gap constraint range. Isolated nodes without valid paths to the upper layer nodes are removed from the Nettree data structure to obtain the Nettree data structure after removing isolated nodes.

[0146] For the Nettree data structure after removing isolated nodes, enumerate all complete paths from the root node to the absolute leaf node, select the set of paths that satisfy the non-overlapping condition from the complete paths, and sum the utility values ​​of each path in the set of paths to obtain the actual non-overlapping utility of the pattern.

[0147] Specifically, the Nettree data structure is a tree-like data structure designed for efficiently processing sequence pattern instances. It overcomes the limitation of traditional prefix trees, which can only represent a single parent-child relationship, allowing nodes to have multiple parent nodes and multiple child nodes simultaneously, forming a mesh tree structure. The design philosophy of Nettree originates from the combination of routing trees and support trees in computer networks, enabling efficient representation and processing of multi-path relationships in sequence data. Compared to traditional sequence matching data structures, Nettree has significant advantages in handling flexible matching patterns with gap constraints, and is particularly suitable for finding non-overlapping pattern instances.

[0148] The first step in constructing a Nettree is to establish a hierarchical structure for each itemset in the pattern. For a given pattern P=, the Nettree contains n levels, each corresponding to one itemset in the pattern. Within each level, all positions in the sequence S that match the corresponding itemset at that level need to be collected as the node set for that level. Itemset matching uses inclusion relations: an itemset at a certain position in the sequence must contain all items of the corresponding itemset in the pattern to be considered a successful match. For example, if the first itemset of the pattern is {a,b}, then {a,b,c} and {a,b} in the sequence can match, while {a} and {b,c} do not. For each successfully matched position, its corresponding utility value also needs to be recorded, which is the sum of the utilities of the itemsets at that position that match the pattern. For example, if the itemset at position p in the sequence is {a:3,b:2,c:1}, the pattern itemset is {a,b}, and the external utilities of a and b are 5 and 4 respectively, then the utility value of that position is 3×5+2×4=23.

[0149] After constructing the nodes at each layer, connections need to be established between adjacent layers of the Nettree. The establishment of these connections is based on two core constraints: positional order and gap constraints. Positional order requires that the position of a lower-level node must follow the position of a higher-level node, ensuring the temporal order of the pattern in the sequence. Gap constraints specify the minimum and maximum allowable intervals between adjacent layers. Assuming the gap constraint is [minGap, maxGap], then the node position p in layer i and the node position q in layer i+1 must satisfy p+minGap+1≤q≤p+maxGap+1. Connections are established between node pairs that satisfy these two constraints, forming a network-like path structure. For example, if the first layer has nodes at positions 2 and 5, and the second layer has nodes at positions 4, 7, and 9, with a gap constraint of [0,3], then four sets of connections (2,4), (2,7), (5,7), and (5,9) will be established.

[0150] After establishing connections, orphaned node removal is required. Orphaned nodes are those that cannot be reached by a valid path to a higher-level node (for nodes outside the first level) or a leaf node (for nodes outside the last level). These nodes are not used in actual pathfinding and can therefore be safely removed from the Nettree to reduce the search space. Orphaned node identification is typically achieved through bidirectional propagation from the first and last levels to the middle levels: first, all nodes reachable from the first level are marked; then, all nodes reachable from the last level are marked; finally, only nodes that satisfy both of these criteria are retained. This bidirectional pruning strategy significantly reduces the number of nodes to consider and improves the efficiency of subsequent path enumeration.

[0151] After removing isolated nodes, a simplified Nettree structure is obtained, where any path from the root node to a leaf node corresponds to a potential instance of the pattern in the sequence. The next step is to enumerate all such complete paths and select the set of paths that satisfy the non-overlapping condition. The non-overlapping condition is a crucial constraint in efficient sequence pattern mining; it requires that the selected instance paths should not overlap, ensuring that each sequence position is calculated at most once. In practice, greedy selection or dynamic programming methods are typically used: greedy selection sorts paths by utility value from highest to lowest, selecting the most efficient path at each step and removing all other overlapping paths; dynamic programming constructs the optimal substructure and progressively solves for the maximum set of non-overlapping utilities.

[0152] For a selected set of non-overlapping paths, the sum of the utility values ​​of each path is calculated to obtain the actual non-overlapping utility of the pattern. The utility value of a path is usually defined as the sum of the utility values ​​of all nodes on the path, representing the total value of that instance in the sequence. The actual non-overlapping utility of a pattern is a direct indicator of the true value of the pattern, considering the combined contribution of all non-overlapping instances of the pattern in the sequence. Unlike upper bound indicators such as sequence expansion utility, actual non-overlapping utility is a precise value, directly used to determine whether the pattern meets the minimum utility threshold requirement.

[0153] A key feature of the Nettree architecture is its support for incremental updates. When a pattern generates sub-patterns through I-expansion or S-expansion, local adjustments can be made to the existing Nettree without requiring a complete rebuild, significantly improving efficiency during the pattern growth phase. Furthermore, Nettree supports parallel processing, allowing the construction of independent Nettree instances for different sequences or patterns, further enhancing its capacity for large-scale data processing. In practical applications, Nettree has proven to offer better performance and scalability than traditional methods when handling sequence pattern mining tasks with complex constraints.

[0154] Example 6

[0155] In this embodiment, based on the antimonotonicity of the actual non-overlapping utility and the upper bound of the extended utility, a multi-level pruning process, including sequence pre-reading pruning, width pruning, co-occurrence pruning, and depth pruning, is performed on the extended terms of the pattern to obtain a promising set of extended terms, including:

[0156] For each sequence in the preprocessed sequence database containing only the target sequence, check whether the sequence contains a valid non-overlapping instance of the target sequence under the conditions of satisfying the gap constraint and the length constraint. Remove sequences that do not contain valid non-overlapping instances from the current mining range as sequence prereading pruning.

[0157] Calculate the corresponding terminating descendant utility for the pattern, wherein the terminating descendant utility is the sum of the sequence expansion utility of the effective extension of the pattern in each sequence. When the terminating descendant utility is less than the minimum utility threshold, the pattern and all corresponding extensions are pruned as width pruning.

[0158] For each extension item of the pattern, calculate the extension utility of the extension item, where the extension utility is the cumulative value in the database of the sum of the utility of the parent instance, the utility of the extension item, and the remaining utility. When the extension utility is less than the minimum utility threshold, remove the extension item from the effective extension set as a co-occurrence pruning.

[0159] For the candidate patterns generated after the co-occurrence pruning, the corresponding sequence expansion utility is calculated. When the sequence expansion utility is less than the minimum utility threshold, the recursive expansion of the candidate patterns is stopped, which is considered as deep pruning.

[0160] The extensions retained after the sequence pre-reading pruning, the width pruning, the co-occurrence pruning, and the depth pruning are combined to form the promising extension set.

[0161] Specifically, multi-level pruning strategies are a key performance optimization technique for non-overlapping high-utility sequence pattern mining methods. They aim to eliminate patterns and extensions that do not meet the minimum utility threshold as early as possible through pruning operations at multiple levels and from different angles, thereby reducing the search space and improving mining efficiency. The pruning strategy is designed based on the anti-monotonic property of the upper bounds of actual non-overlapping utility and extension utility. That is, if the upper bound of a pattern is lower than the minimum utility threshold, the actual utility of any of its extension patterns must also be lower than the threshold, allowing for safe pruning without overlooking any high-utility patterns.

[0162] Sequence prefetching pruning is an early pruning strategy at the database level. Its purpose is to remove sequences unlikely to contain valid non-overlapping instances before schema expansion, thus reducing data size. In its implementation, two checks are performed on each preprocessed sequence: First, the Nettree structure is used to find if there are schema instance paths in the sequence that satisfy the gap and length constraints; second, the found paths are validated for non-overlapping conditions to confirm the existence of at least one valid non-overlapping instance. If a sequence fails both checks, it is removed from the current mining scope. A key characteristic of sequence prefetching pruning is its context sensitivity; as schemas grow, more sequences may be pruned because longer schemas are less likely to satisfy the constraints. For example, a schema might have valid instances in sequence S, but its expanded schema might no longer satisfy the constraints.

[0163] Breadth-first pruning is a global pruning strategy for the entire pattern and all its possible extensions, implemented by calculating the Terminating Descendant Utility (TDU). The Terminating Descendant Utility is an innovative upper bound metric representing the maximum total utility achievable by the pattern and all its possible extensions. It is calculated by summing the sequence extension utilities of all valid extensions of the pattern across all sequences. If the Terminating Descendant Utility is less than a minimum utility threshold, the pattern and any of its extensions are unlikely to be efficient and can be pruned entirely. This strategy is equivalent to performing "breadth-first" pruning on the pattern extension tree, eliminating a large number of unnecessary branches at once and significantly reducing the search space. The calculation of the Terminating Descendant Utility is typically combined with positional constraints and non-overlapping conditions to ensure the accuracy and compactness of the upper bound.

[0164] Co-occurrence pruning is a local pruning strategy targeting individual extension items, focusing on the synergistic effect between a specific item and the current schema instance. Co-occurrence Utility (CU) is an indicator reflecting the combined utility when an item and schema co-occur. It is calculated as the cumulative value of the parent schema instance's utility, the extension item's utility, and the residual utility across the entire database. The key here is considering the conditions for co-occurrence: the extension item must appear in a position that satisfies the gap constraint and non-overlap condition with the parent schema instance. For example, for I-extensions, the item must appear in the matching position of the last itemset of the schema; for S-extensions, the item must appear in a subsequent position that satisfies the gap constraint. When the extension item's utility is less than the minimum utility threshold, any schema with that item as an extension element cannot be a high-utility schema and can be removed from the effective extension set.

[0165] Depth pruning is a recursive expansion control strategy for generated candidate patterns, implemented by calculating the Sequence Extension Utility (SEU). The Sequence Extension Utility is a comprehensive metric measuring the potential of a pattern in subsequent expansions, typically a combination of upper bounds, including the pattern's actual non-overlapping utility and the maximum incremental utility of any subsequent expansions. During calculation, non-overlapping instances of the pattern in each sequence and possible expansions are considered, and its overall upper bound is evaluated. When the Sequence Extension Utility falls below a minimum utility threshold, further recursive expansion of the pattern is stopped, as any deeper expansion is unlikely to produce a highly efficient pattern. This strategy is equivalent to performing "depth-first" pruning on the pattern expansion tree, avoiding depth searches on ineffective branches and improving the algorithm's efficiency.

[0166] These four pruning strategies work together to form a multi-layered, comprehensive pruning system. Sequence pre-reading pruning reduces the input size at the database level; width pruning eliminates invalid pattern families at the overall pattern level; co-occurrence pruning filters invalid extensions at the extension item level; and depth pruning controls the search depth during recursion. This combination of multi-layered pruning strategies utilizes upper bound information at different levels, minimizing the search space while ensuring the integrity of the results.

[0167] In practice, these pruning strategies are often combined with data structures such as bitmaps and hash indexes to further optimize computational efficiency. For example, bitmaps can be used to represent the occurrence of items in a sequence, enabling rapid position determination and overlap detection; hash indexes can be used to quickly locate the potential positions of extended items, reducing the number of scans. Furthermore, based on data characteristics and task requirements, the application order and threshold parameters of each pruning strategy can be dynamically adjusted to balance pruning efficiency and computational overhead, adapting to different application scenarios.

[0168] The multi-level pruning strategy is particularly effective for large-scale, high-dimensional sequence data. Experiments show that compared with basic non-pruning methods, multi-level pruning can reduce the search space by more than 95% and shorten the running time by one to two orders of magnitude, while ensuring the accuracy and completeness of the mining results. This is of great value for processing complex real-time data streams and large historical databases.

[0169] Example 7

[0170] In this embodiment, before performing subsequence judgment on each sequence in the original sequence database, filtering out sequences that do not contain the target sequence, and retaining sequences containing the target sequence in the preprocessed sequence database, the method further includes performing sequence quality checking on the original sequence database, including:

[0171] For each sequence in the original sequence database, extract the total sequence utility, matching degree with the target sequence, sequence complexity, and temporal coherence as multi-dimensional feature vectors, and use the greedy search to verify the true label of the sequence to obtain a labeled feature dataset;

[0172] For the labeled feature dataset, based on the constraint that the label noise rate does not exceed a preset upper bound, a half-space classifier is trained in the utility-frequency space to obtain a classification hyperplane for distinguishing between real effective sequences and noise-contaminated sequences.

[0173] Cross-validation is performed on the classification hyperplane, and the confidence level of the classification accuracy is calculated based on the Hoeffding inequality. When the confidence level is not less than a preset confidence threshold, the classification hyperplane is determined to be an effective classifier.

[0174] For each sequence in the original sequence database, the effective classifier is used to calculate the corresponding predicted label and predicted confidence. Sequences whose predicted label contains a valid target sequence and whose predicted confidence is not less than a preset confidence threshold are retained. Sequences whose predicted label does not contain a valid target sequence and whose predicted confidence is not less than the preset confidence threshold are filtered. Sequences whose predicted confidence is less than the preset confidence threshold are marked as suspicious sequences.

[0175] Multiple greedy search verifications are performed on the suspicious sequences, and the suspicious sequences that pass the verification are retained in the preprocessed sequence database.

[0176] Specifically, sequence quality assessment is an innovative preprocessing method designed to identify and filter sequences from raw data that may contain noise, anomalies, or fail to meet quality requirements, ensuring the quality of input data for subsequent data mining processes. This quality assessment process combines sequence analysis and machine learning techniques, significantly extending traditional sequence preprocessing methods and proving particularly suitable for complex sequence data containing noise and anomalies in the real world.

[0177] Sequence feature extraction is the first step in quality assessment, constructing a multidimensional feature vector for each sequence to characterize its quality and relevance. Total sequence utility is the sum of the utilities of all items in the sequence, reflecting its overall value. Matching degree with the target sequence measures the similarity between the two sequences, typically expressed as the proportion of longest common subsequences or the normalized inverse of the edit distance. Sequence complexity measures the structural diversity of the sequence, which can be represented by indicators such as entropy, number of item types, or changes in itemset size. Temporal coherence measures the reasonableness of the temporal patterns of events in the sequence, usually expressed by the distribution of time intervals between adjacent itemsets or the degree of conformity to temporal rules. These features together constitute a multidimensional vector that comprehensively reflects the quality characteristics of the sequence.

[0178] True label validation assesses the quality of each sequence using a greedy search method. Greedy search here refers to an adaptive multi-round matching strategy: it first attempts to meet strict matching conditions; if this fails, it gradually relaxes the constraints until a reasonable match is found or it is confirmed that no match can be found. The validation process considers gap constraints, length constraints, and non-overlapping conditions to comprehensively determine whether a sequence contains a valid target sequence instance. Sequences that pass validation are labeled as positive samples (containing a valid target sequence); otherwise, they are labeled as negative samples (not containing a valid target sequence). In this way, each sequence, along with its feature vector and label, constitutes a labeled feature dataset, providing a foundation for subsequent classifier training.

[0179] A semi-space classifier is a linear classifier specifically designed for the utility-frequency two-dimensional space, and its training is based on an optimization objective controlled by the label noise rate. The utility-frequency space is a key feature space for judging sequence quality, where the utility dimension reflects the value of the sequence, and the frequency dimension reflects the repetitiveness of the sequence pattern. In this space, truly valid sequences and noisy sequences typically exhibit different distribution characteristics. The semi-space classifier separates the two classes of sequences by finding an optimal classification hyperplane (a straight line in two-dimensional space) within this space. Label noise rate control during training ensures the classifier's robustness to noise and avoids overfitting to outliers. For example, if the noise upper bound is set to 5%, the classifier will allow up to 5% of training samples to be misclassified; this tolerance improves the classifier's generalization ability.

[0180] Hoeffding's inequality is an important probabilistic boundary tool used to estimate the confidence interval of the mean of a random variable. In classifier validation, it is used to calculate the confidence level of classification accuracy, i.e., the degree of confidence in the classifier's performance estimate. The calculation process first obtains the classifier's accuracy estimate through cross-validation, and then applies Hoeffding's inequality, combined with the sample size and confidence parameters, to calculate the confidence interval of the accuracy. When the confidence level reaches a preset threshold (usually 95% or 99%), the classifier is considered valid. This statistically based validation method provides an objective assessment of classifier performance, avoiding the risk of over-reliance on a single validation set.

[0181] In the classification and prediction phase, a validated classifier is applied to each sequence in the original sequence database. Each sequence is assigned a predicted label (whether it contains a valid target sequence) and a prediction confidence score (the degree of certainty in the prediction result). Prediction confidence is typically based on the distance from the sequence's feature vector to the classification hyperplane; the greater the distance, the higher the confidence. Based on the prediction result and confidence score, sequences are divided into three categories: high-confidence positive examples are retained, high-confidence negative examples are filtered out, and low-confidence sequences (regardless of whether they are positive or negative) are marked as suspicious sequences requiring further verification. This multi-level decision-making strategy based on confidence balances filtering efficiency and retention rate, avoiding the risk of mistakenly filtering out valuable sequences.

[0182] For suspicious sequences with insufficient confidence, a greedy search is used for verification before the final decision. This verification method is more comprehensive than a single search, increasing the probability of finding valid instances by searching multiple times with different starting points or parameters. Specifically, methods such as random restart strategies (starting the search from different positions), parameter variation strategies (adjusting gap or length constraints), or template variation strategies (trying variants of the target sequence) can be used to increase the comprehensiveness of the search. If a sequence is verified to contain a valid target sequence instance in any search, it is retained in the preprocessed database; otherwise, it is filtered out as an invalid sequence. This multi-round verification mechanism provides a fairer evaluation opportunity for boundary cases and improves the accuracy of filtering decisions.

[0183] Sequence quality assessment is particularly valuable for large-scale, noisy data. Experiments show that this method can effectively identify and filter over 30% of invalid or low-quality sequences while maintaining a valid sequence retention rate of over 99%. The filtered database is not only smaller and more efficient, but also has better data quality, significantly improving the accuracy and interpretability of the mining results. In practical applications, such as network security monitoring or industrial process monitoring, this quality filtering can greatly reduce false alarms and enhance the system's practical value.

[0184] The computational complexity of quality control processing primarily depends on the overhead of feature extraction and multiple validation searches. For ultra-large-scale datasets, a stratified sampling strategy can be employed to train the classifier on small samples before applying it to the full dataset; alternatively, incremental learning methods can be used to process data batch by batch and update the classifier. Furthermore, parallel computing techniques can be combined to accelerate processing, such as using the MapReduce framework to parallelize sequence feature extraction and validation search processes, further improving the processing capacity for large-scale data.

[0185] Sequence quality assessment not only improves the efficiency and quality of subsequent data mining as a preprocessing step, but also serves as an independent data analysis tool, helping to identify anomalous patterns and quality issues in the data, providing feedback for data collection and system optimization. This dual value makes it an important component of the sequence data analysis workflow, particularly suitable for critical application areas with high data quality requirements.

[0186] Example 8

[0187] In this embodiment, after constructing a Nettree data structure containing multiple root nodes and multiple parent nodes, and searching for instance paths that satisfy the non-overlapping condition in the Nettree data structure, the process further includes performing critical instance path identification processing on the Nettree data structure, including:

[0188] The search space of the Nettree data structure is mapped to a multidimensional polygon, where each layer corresponds to one dimension, and the position, utility value, remaining utility value, and number of paths to the leaf of each node constitute the vertex coordinates of the multidimensional polygon.

[0189] The multidimensional polygon is recursively subdivided along the direction of utility difference to decompose the multidimensional polygon into multiple sub-regions, where each sub-region corresponds to a type of instance path.

[0190] For each of the plurality of sub-regions, calculate the sum of the utility contributions of all instance paths within the sub-region and the path overlap between the sub-region and other sub-regions. Use the product of the utility contribution and the path overlap as a critical score. Mark the sub-regions whose critical scores are not less than a preset critical threshold as critical sub-regions.

[0191] Greedy search and constraint verification are performed preferentially on the instance paths in the critical sub-region to obtain a set of valid non-overlapping instances within the critical region;

[0192] Perform a supplementary search on instance paths in non-critical regions outside the critical sub-region, merge the set of valid non-overlapping instances in the critical region with the search results in the non-critical region, and prune non-critical branches in the merged instance paths whose upper bound of utility is lower than a preset multiple of the minimum utility threshold.

[0193] Specifically, critical instance path identification is an advanced search space analysis technique designed to efficiently identify and process critical instance paths that significantly impact pattern utility and potentially conflict with other paths within complex Nettree structures. Compared to traditional exhaustive enumeration methods, this approach significantly improves the efficiency of non-overlapping utility instance identification through geometric space mapping and region partitioning, making it particularly suitable for large-scale, complex sequence pattern mining tasks.

[0194] Multidimensional polygon mapping is an innovative method for transforming the search space of a Nettree into a geometric representation. In this mapping, each layer of the Nettree corresponds to one dimension of the multidimensional space, forming an n-dimensional space (where n is the length of the pattern). Each node is mapped to a point in this space based on its characteristics in the Nettree, and its coordinates are determined by four key attributes: the node's position in the sequence (representing temporal order), the node's utility value (representing economic value), the node's residual utility value (representing future potential), and the number of paths from that node to leaf nodes (representing structural complexity). This multidimensional representation captures the complex relationships between nodes, transforming the originally discrete tree structure into a continuous geometric space, facilitating subsequent geometric analysis and partitioning. For example, for a three-itemset pattern, its Nettree is mapped to a three-dimensional space, where each complete path corresponds to a curve in this space.

[0195] Bisectioning is a spatial segmentation technique inspired by quadtree / octree algorithms in computational geometry, specifically designed for mining highly efficient sequence patterns. The bisectioning process first identifies the direction of greatest utility difference in multidimensional space, typically the dimension with the most significant change in utility value. Along this direction, the algorithm finds an optimal split point, dividing the space into two sub-regions where the path utility distribution within each sub-region is relatively uniform, while the utility differences between different sub-regions are significant. For each sub-region, the same segmentation process is recursively performed until a termination condition is met: either the number of paths within the sub-region is less than a preset threshold, or the variance of path utility within the sub-region is less than a preset threshold. This adaptive segmentation strategy ensures that highly efficient and low-efficiency regions can be clearly separated, providing a foundation for subsequent criticality evaluation.

[0196] Criticality score is a composite metric for measuring the importance of a sub-region, combining two key factors: utility contribution and path overlap. Utility contribution is the sum of the utilities of all instance paths within a sub-region, representing the region's potential contribution to the overall utility of the pattern. Path overlap measures the positional overlap between paths within a sub-region and paths in other regions, typically calculated as the ratio of overlapping positions to the total number of positions. The criticality score is calculated by multiplying the utility contribution by the path overlap. This design reflects two key insights: firstly, highly efficient regions are more important for meeting the minimum utility threshold; secondly, regions that highly overlap with other paths are more likely to affect the satisfaction of non-overlapping constraints. When the criticality score exceeds a preset threshold, the corresponding sub-region is marked as a critical sub-region and requires priority processing. This scoring mechanism ensures that computational resources are concentrated on the most valuable and challenging regions, improving overall efficiency.

[0197] The critical region is prioritized using a greedy search strategy, a heuristic search method that attempts to approximate the global optimum by selecting a local optimum at each step. In this embodiment, the greedy search starts with the path with the highest utility and progressively selects paths that do not overlap with the selected paths and also have the highest utility, until no new path satisfying the non-overlapping constraint can be found. While this method does not guarantee finding the global optimum, it often yields a near-optimal solution in practice, and its computational efficiency is significantly higher than that of the exhaustive search method. Constraint verification ensures that the selected paths satisfy all pattern constraints, including gap constraints, length constraints, and non-overlapping constraints. For the critical region, which contains highly efficient and potentially conflicting paths, this prioritization can identify the set of key non-overlapping instances early on, providing important reference for subsequent searches.

[0198] The supplementary search for non-critical regions employs a relatively lightweight search strategy, which can be a simplified version of greedy search or a local optimization method. Since non-critical regions typically have low utility or low overlap, their impact on the final result is relatively limited; therefore, the search process can be appropriately simplified to balance efficiency and accuracy. The results of the supplementary search are merged with the high-priority results for critical regions to form a comprehensive set of candidate instances. For this merged result, utility upper bound pruning is performed, removing non-critical branches whose upper bound is lower than a preset multiple of the minimum utility threshold. This preset multiple is typically set between 0.8 and 0.9, providing a safety margin to ensure that potentially high-utility instances are not incorrectly pruned. This hierarchical processing strategy significantly improves the overall efficiency of non-overlapping instance identification, making it particularly suitable for handling complex Nettree structures and large-scale sequence data.

[0199] A key feature of critical instance path identification processing is its scalability. As pattern length and data size increase, the computational complexity of traditional exhaustive enumeration methods grows exponentially, while this method, through spatial partitioning and priority processing, keeps the complexity within an acceptable range. In implementation, this method can incorporate parallel computing techniques to perform search and verification in parallel across different sub-regions, further improving processing efficiency. Furthermore, the intermediate results generated by this method (such as region partitioning and criticality scores) provide valuable information for subsequent pattern expansion and utility estimation, supporting more intelligent search strategies and pruning decisions.

[0200] In practical applications, critical instance path identification is particularly suitable for scenarios with unevenly distributed instance paths, such as attack detection in the field of network security. A few critical attack paths often contain high-efficiency components and overlap with normal behavior paths. By quickly identifying and prioritizing these critical paths, meaningful attack patterns can be discovered more effectively. Similarly, in industrial process monitoring, equipment failures typically exhibit specific temporal patterns that may partially overlap with normal operating patterns. Using critical instance path identification can more accurately separate these key failure indication patterns, improving the accuracy and timeliness of fault warnings.

[0201] Example 9

[0202] In this embodiment, the recursive pattern growth of the promising expansion item set using depth-first search, outputting a set of non-overlapping high-utility sequence patterns that satisfy the minimum utility threshold and contain the target sequence, includes performing a borrowing graph-based scheduling process on the recursive pattern growth task. The scheduling process includes:

[0203] For each of the promising extension items in the set of extension items, a pattern exploration task is created. For each pattern exploration task, a task identifier, parent pattern, and the sequence extension utility of the pattern are set as priority and estimated processing time, and a set of pattern exploration tasks is constructed.

[0204] A directed graph is constructed to establish the parent-child dependency relationship and resource competition relationship among the tasks in the set of pattern exploration tasks, resulting in a borrowing graph, wherein a dependency edge is established between the parent task and the child task, and a resource competition edge is established between tasks that require the same computing resources.

[0205] For each pattern exploration task in the borrowed graph, a delay perspective score is calculated for the pattern exploration task, wherein the delay perspective score is the sum of its own delay cost and cascaded delay cost minus the resource release benefit. The own delay cost is the product of the priority of the pattern exploration task and the estimated processing time. The cascaded delay cost is the sum of the products of the priority of all descendant tasks of the pattern exploration task and the estimated processing time. The resource release benefit is the sum of the priority-weighted delay differences of tasks that compete with the pattern exploration task for resources.

[0206] Based on the delayed perspective score, the set of pattern exploration tasks is prioritized and sorted. The tasks with the delayed perspective scores from largest to smallest are assigned to work threads in sequence. When all work threads are occupied, subsequent tasks are placed in a priority queue to wait.

[0207] When the pattern exploration task is completed, the subtasks of the pattern exploration task are added to the priority queue. The delay perspective score of the tasks in the priority queue is recalculated and scheduled for execution until the priority queue is empty. The patterns in all completed pattern exploration tasks that meet the minimum utility threshold and contain the target sequence are output as the set of non-overlapping high-utility sequence patterns.

[0208] Specifically, the borrowing graph-based scheduling process is an innovative task management strategy designed specifically for the efficient recursive pattern growth process in sequence pattern mining. This scheduling strategy breaks through the limitations of traditional task scheduling methods, introducing dynamic resource allocation and a forward-looking task evaluation mechanism. It can intelligently allocate computing resources according to the characteristics and dependencies of the pattern exploration task, significantly improving parallel mining efficiency, and is particularly suitable for complex, large-scale pattern space exploration tasks.

[0209] Pattern exploration tasks are the basic computational units in the recursive pattern growth process. Each task is responsible for exploring the expansion possibilities of a specific pattern. The task creation process first generates a corresponding exploration task for each item in the set of promising expansion items, and sets four key attributes: Task Identifier, a unique identifier for the task, is usually generated by combining the parent pattern and the expansion item; the parent pattern records the base pattern that the task expands, used to maintain the hierarchical structure of pattern growth; Sequence Expansion Utility serves as the basis for task priority, reflecting the potential value of the task's exploration space; and Estimated Processing Time is an estimate of the computational resources required for task execution, usually calculated based on the complexity of the parent pattern, the characteristics of the expansion item, and historical execution data. For example, for expansion item 'a' and parent pattern 'a', an exploration task with the task identifier "bc-a", parent pattern 'a', priority 150, and estimated processing time of 2.5 units might be created. These attributes collectively form the basis for subsequent scheduling decisions.

[0210] Borrowing graphs are task dependency and resource contention representation structures specifically designed for pattern mining. Essentially, they are directed graphs with two edge types. In this graph, each node represents a pattern exploration task, and edges represent two key relationships between tasks: dependency edges and competition edges. Dependency edges point from a parent task to a child task, indicating that the child task can only execute after the parent task is completed. This dependency stems from the inherent logic of pattern growth; that is, extended patterns can only be explored after the current pattern has been explored. Competition edges connect tasks that require the same computational resources, indicating that these tasks have resource contention and cannot execute simultaneously. Resource contention is usually determined based on the task's data access patterns or computational characteristics. For example, tasks that need to access the same sequence of data blocks or use the same hardware accelerators will establish competition edges. The construction of borrowing graphs combines static analysis and dynamic updates: the initial construction is based on known pattern relationships and resource requirements, and the graph structure is dynamically updated as tasks are executed and new tasks are generated.

[0211] Delay Perspective Scoring is a forward-looking task value assessment metric that provides a comprehensive perspective for scheduling decisions by considering the direct value of a task, its cascading impact, and resource release benefits. This scoring mechanism is designed based on critical path analysis and opportunity cost theory in project management, adapting to the task dependency structure unique to pattern mining. The scoring calculation includes three key components: Self-delay cost, which is the cost of delaying the task itself, calculated as the product of task priority and estimated processing time, reflecting the high cost of delaying high-priority and long-time tasks; Cascading delay cost, which measures the cascading impact of a task delay on all its descendant tasks, calculated as the weighted sum of the priorities and estimated processing times of all descendant tasks, taking into account the transitivity of task dependencies and avoiding the problem of focusing only on short-term gains while ignoring long-term impacts; and Resource Release Benefit, which measures the potential benefits of releasing resources after task execution, calculated as the sum of the priority-weighted delay differences of all tasks competing for resources with this task, encouraging the priority execution of tasks that can unlock multiple high-value competing tasks. The final latency perspective score is the sum of its own latency cost and cascading latency cost minus the resource release benefits. This balanced design ensures that the score can fully reflect the system-level value of the task.

[0212] Task scheduling based on latency perspective scoring is a dynamic priority scheduling strategy that continuously adjusts scheduling decisions as task execution status and system state change. The scheduling process first sorts the task set according to latency perspective scores, with higher-scoring tasks having higher priority. Then, tasks are assigned to available worker threads in priority order. When all worker threads are occupied, subsequent tasks are placed in a priority queue to wait. These worker threads are typically available processing units in the system, such as CPU cores or processing threads. A priority queue is a data structure that efficiently maintains and accesses tasks based on their priority, usually implemented using a heap or balanced tree. The scheduling system continuously monitors task execution status; when a task is completed, its subtasks (if any) are added to the priority queue, and the latency perspective score is recalculated for all tasks in the queue, potentially leading to dynamic adjustments in task priorities. This dynamic scheduling strategy adapts to the constantly changing task dependencies and resource requirements during pattern mining, maximizing overall system efficiency.

[0213] Scheduling recursive growth processes is particularly complex because the generation and prioritization of new tasks are dynamic and difficult to predict. Borrowing graph-based scheduling methods flexibly handle this complexity by simulating the process of borrowing and returning resources. When a high-priority task needs resources but those resources are occupied by a low-priority task, some resources, such as processing time slices or memory space, can be temporarily borrowed from the low-priority task to accelerate the execution of the high-priority task. This resource borrowing is based on preset borrowing rules and security constraints to ensure system stability. In implementation, techniques such as work stealing or resource reservation can be used to support efficient dynamic resource allocation.

[0214] In practical implementation, system load balancing and fault tolerance mechanisms also need to be considered. Load balancing ensures that the workload of each worker thread is relatively balanced, avoiding situations where some threads are overloaded while others are idle. Fault tolerance mechanisms handle errors and anomalies that may occur during task execution, such as through task retries, checkpoint recovery, or backup execution to ensure system stability and the integrity of results. In addition, to adapt to different hardware environments and data characteristics, scheduling parameters (such as scoring weights and resource borrowing limits) can usually be adjusted according to the actual situation. Some systems also support adaptive parameter adjustment, automatically optimizing the scheduling strategy based on runtime feedback.

[0215] In large-scale pattern mining tasks, borrowed graph-based scheduling demonstrates significant advantages. Experiments show that compared to traditional static priority scheduling or simple dynamic scheduling, this method can reduce task completion time by 20%-40%, especially in scenarios with complex task dependencies and intense resource competition. For example, in large-scale cybersecurity data analysis with hundreds of thousands of candidate patterns, this scheduling strategy can prioritize exploring paths most likely to lead to efficient discovery, quickly identifying important attack patterns; simultaneously, through intelligent resource allocation, it ensures efficient utilization of system resources, supporting continuous large-scale mining tasks.

[0216] From a system architecture perspective, borrowing graph-based scheduling typically serves as the central scheduling component of a pattern mining system, tightly integrated with other components such as pattern generation, instance verification, and result set management. The scheduling component receives task requests from the pattern generator, makes scheduling decisions based on system state and task characteristics, and assigns tasks to appropriate worker threads for execution. Execution results are fed back to the scheduling system to update the borrowing graph and task status, forming a closed-loop scheduling-execution-feedback process. This architecture supports system scalability and flexibility, adapting to various application scenarios ranging from single-machine small-scale data to distributed big-scale data.

[0217] Example 10

[0218] In this embodiment, prioritizing the set of pattern exploration tasks based on the delayed perspective score and sequentially assigning the tasks with the delayed perspective scores from largest to smallest to worker threads further includes performing dynamic resource borrowing and parallel recursive mining processing on the pattern exploration tasks, including:

[0219] When the target pattern exploration task requests resource borrowing, the currently executing pattern exploration tasks in the borrowing graph are traversed, and pattern exploration tasks with a latency tolerance greater than the estimated processing time of the target pattern exploration task and not on the dependency path of the target pattern exploration task are selected as the set of available borrowing tasks. From the set of available borrowing tasks, the pattern exploration task with the highest latency tolerance is selected as the resource lending task, and the resource lending task is obtained.

[0220] Allocate a preset proportion of computational resources to the target mode exploration task for the resource lending task, and add a borrowing edge from the target mode exploration task to the resource lending task in the borrowing graph, and record the amount of resources borrowed.

[0221] The estimated processing time of each pattern exploration task in the set of pattern exploration tasks is compared. When the number of pattern exploration tasks exceeds a preset multiple of the number of worker threads, adjacent pattern exploration tasks with estimated processing time differences within a preset time threshold range are merged into batch tasks. When the number of pattern exploration tasks does not exceed the preset multiple, single task scheduling is maintained, resulting in a set of scheduled tasks containing batch tasks and single tasks.

[0222] For each batch task and single task in the scheduling task set, sort them from largest to smallest according to the estimated processing time, and then allocate the sorted batch tasks and single tasks to the working thread with the smallest current load in turn, so that the total load of each working thread tends to be balanced, and the task allocation scheme of each working thread is obtained.

[0223] Based on the task allocation scheme, each of the worker threads executes the allocated pattern exploration task in parallel, performs a depth-first search recursive pattern growth on the patterns in each pattern exploration task, and merges the patterns that meet the minimum utility threshold and contain the target sequence into the non-overlapping high-utility sequence pattern set.

[0224] Specifically, dynamic resource borrowing is an innovative resource management mechanism that allows high-priority tasks to temporarily borrow computing resources from other tasks in emergency situations, improving overall system efficiency. Unlike traditional static resource allocation or simple preemptive scheduling, dynamic resource borrowing achieves efficient resource utilization while ensuring system stability through fine-grained control over borrowing conditions, quantities, and return mechanisms. This mechanism is particularly suitable for application scenarios with complex task dependencies and changing performance requirements, such as pattern mining.

[0225] Delay tolerance is a key metric for evaluating a task's ability to lend resources. It is defined as the maximum time a task can be delayed without significantly impacting overall system performance or the execution of dependent tasks. Delay tolerance is typically calculated based on the task's priority, remaining processing time, and downstream dependencies, and can be expressed as a function of the task's slack time and the system's current load state. In practice, it can be estimated by subtracting the current time and the sum of the remaining execution time from the task's latest completion time. Tasks with high delay tolerance are usually those with relatively low priority, no urgent dependencies, or those in the early stages of the execution sequence. For example, a low-priority task estimated to take 10 minutes to complete but with a 30-minute slack time might have a delay tolerance of 20 minutes; while a high-priority task estimated to take 5 minutes but with only a 7-minute slack time might have a delay tolerance of only 2 minutes.

[0226] The resource borrowing decision-making process is a systematic method for finding the most suitable resource lending task. First, it iterates through all currently executing tasks in the borrowing graph, performing two key checks on each task: a latency tolerance check to ensure the task's latency tolerance is greater than the target task's estimated processing time, guaranteeing that borrowing will not cause the lending task to violate its time constraints; and a dependency path check to ensure the task is not on the target task's dependency path, avoiding resource dependency loops that could lead to deadlock or livelock. Tasks that pass these two checks constitute a set of borrowable tasks. From this set, the task with the highest latency tolerance is selected as the final resource lending task. This selection strategy maximizes the safety margin of borrowing and reduces the risk of task delays due to borrowing. In real-world systems, resource borrowing decisions may also consider additional factors, such as the task's resource utilization pattern, historical borrowing records, or the system's global state, to further optimize the decision quality.

[0227] Resource allocation and borrowing records are the execution mechanism for dynamic resource borrowing. Once a resource lending task is determined, the system allocates its computing resources to the target task according to a preset ratio. This preset ratio is typically between 20% and 50%, a result of balancing system performance and stability: a ratio that is too low will affect the borrowing effect, while a ratio that is too high may lead to a significant decrease in the performance of the lending task. In practice, resource allocation can be implemented through various techniques, such as processor time slice reallocation, memory space adjustment, or I / O bandwidth limiting, depending on the system architecture and resource type. The execution of resource borrowing also requires adding a borrowing edge from the target task to the lending task in the borrowing graph. This edge records information such as the amount of resources borrowed, the start time of the borrowing, and the expected return time, providing a basis for subsequent resource monitoring and return operations. This explicit borrowing record mechanism ensures the traceability and manageability of resource borrowing and is an important guarantee for maintaining system stability.

[0228] Batch task merging is a task granularity adjustment technique that reduces scheduling overhead and optimizes resource utilization by merging multiple smaller tasks. When the number of pattern exploration tasks significantly exceeds the number of available worker threads, frequent task switching can lead to substantial overhead and reduced system efficiency. Batch task merging identifies tasks with similar estimated processing times (the difference is within a preset threshold, typically 5%-10% of the total processing time) and groups them into a single batch task, which is then executed continuously by the same worker thread. The key to merging is selecting appropriate task combinations: task processing times should be similar to avoid a long task slowing down the entire batch; tasks should ideally have no strong dependencies to allow for flexible adjustments to the execution order; and task resource requirements should ideally be complementary, such as pairing a CPU-intensive task with an I / O-intensive task to optimize resource utilization. Batch merging also needs to consider task priority distribution to avoid mixing high-priority tasks with multiple low-priority tasks, which could lead to excessively long waiting times for high-priority tasks. In practice, batch size is usually dynamically adjusted, varying between 2-5 tasks based on system load and task characteristics to balance batch processing efficiency and scheduling flexibility.

[0229] Load balancing is a crucial mechanism for ensuring efficient utilization of computing resources in multi-threaded environments. In computationally intensive applications like pattern mining, uneven workload distribution can lead to some threads being overloaded while others remain idle, reducing overall system throughput. Load balancing first sorts the set of scheduled tasks (including batch and single tasks) from largest to smallest based on their estimated processing time, following a "maximum task first" principle. This sorting strategy helps to pre-arrange the longest-running tasks and reduce tail effects. Then, a "minimum load first" allocation strategy is used, assigning each task sequentially to the worker thread with the lowest current total load. Thread load calculation is typically based on the sum of the estimated processing times of the assigned tasks, sometimes also considering the task's resource utilization characteristics (e.g., CPU-intensive, memory-intensive) and the thread's historical execution efficiency. To further improve balance, the system may implement dynamic load monitoring and task migration mechanisms, detecting load imbalances during execution and migrating tasks from high-load threads to low-load threads. This dynamic adjustment is particularly suitable for tasks where there is a significant difference between estimated and actual execution times, serving as an important supplement to static load balancing.

[0230] Parallel recursive mining is a core execution strategy that leverages multi-threaded or multi-processor environments to accelerate pattern exploration. In pattern mining, recursive pattern growth typically takes the form of depth-first search (DFS), starting from an initial pattern and progressively building more complex patterns by adding extensions until a termination condition is met. The key to parallel recursive mining lies in identifying independent branches in the search tree that can be executed in parallel and assigning them to different worker threads. According to the task allocation scheme, each worker thread is responsible for executing a set of pattern exploration tasks, performing depth-first recursive pattern growth on each task: starting from the initial pattern, generating all possible extended patterns, calculating the non-overlapping utility for each extended pattern, pruning inefficient branches, and continuing to recursively expand the remaining highly efficient patterns. The challenge of this parallel strategy lies in balancing workload and managing data sharing and synchronization between threads. To reduce inter-thread conflicts, data partitioning techniques are commonly used, dividing the sequence database into multiple relatively independent parts, with different threads operating on different parts; or lock-free data structures and local caching are used to reduce contention for shared resources. In addition, to improve cache hit rate and memory access efficiency, worker threads usually adopt a local priority processing strategy, completing relevant calculations on local data as much as possible before exchanging data, thereby reducing unnecessary memory traffic.

[0231] Result management and aggregation are the final steps to ensure the integrity and consistency of parallel mining results. After each worker thread completes its pattern exploration task, patterns that meet the minimum utility threshold and contain the target sequence need to be merged into a non-overlapping high-utility sequence pattern set. This process requires addressing potential result duplication and conflict issues. The system typically uses thread-safe concurrent data structures (such as concurrent hash tables) to store the result set, supporting simultaneous writing by multiple threads without conflicts. Before inserting a result, it checks whether the pattern already exists in the result set to avoid duplication. For pattern variants (such as patterns with slightly different orders but the same essence), pattern normalization may be implemented to convert them into a standard form before determining whether they are duplicates. Furthermore, result filtering and integration mechanisms may be implemented to remove patterns that, although meeting the minimum utility threshold, are contained in its superset and have similar utility, reducing the result set size and improving the interpretability and usability of the results. In a distributed environment, result aggregation also involves cross-node communication and global synchronization, typically employing a master-slave model or hierarchical aggregation strategy to balance communication overhead and result integrity.

[0232] Dynamic resource borrowing and parallel recursive mining demonstrate significant performance advantages in practical applications. Compared to traditional fixed resource allocation strategies, dynamic resource borrowing can flexibly adjust resource allocation based on task urgency and system load, improving the response speed of critical tasks. Compared to simple task parallelization, parallel recursive mining combined with batch merging and load balancing can better adapt to workload changes in pattern exploration, achieving higher resource utilization efficiency. In large-scale sequence pattern mining applications, this combined strategy can reduce computation time by 30%-60%, especially in scenarios with uneven pattern space distribution and large differences in task processing time. For example, in attack pattern mining in the field of cybersecurity, the exploration of certain critical attack paths may need to be completed quickly for timely warnings. Through the dynamic resource borrowing mechanism, additional resources can be allocated to these high-priority exploration tasks, accelerating the discovery and processing of critical information while maintaining the continuous progress of the overall mining task.

[0233] Example 11

[0234] In this embodiment, the recursive pattern growth of the promising expansion item set using depth-first search, outputting a set of non-overlapping high-utility sequence patterns that satisfy the minimum utility threshold and contain the target sequence, includes:

[0235] For each extension item in the set of promising extension items, determine the extension type of the extension item relative to the current pattern, wherein the extension type includes I-extension, which adds the extension item to the last itemset of the current pattern, and S-extension, which adds the extension item as a new itemset to the end of the current pattern.

[0236] Based on the extension type, perform an extension operation on the current pattern to generate a sub-pattern, and recursively perform non-overlapping instance search, actual non-overlapping utility calculation and multi-level pruning on the sub-pattern.

[0237] When the actual non-overlapping utility of the sub-pattern is not less than the minimum utility threshold and the sub-pattern contains the target sequence, the sub-pattern is added to the set of non-overlapping high-utility sequence patterns.

[0238] Specifically, recursive pattern growth is the core strategy of sequence pattern mining. It systematically expands candidate patterns to generate more complex patterns, ultimately discovering a set of non-overlapping, high-utility sequence patterns that meet certain conditions. This method employs a depth-first search (DFS) paradigm, which, compared to breadth-first search, offers advantages such as lower memory requirements and earlier discovery of complete patterns, making it particularly suitable for high-utility pattern mining tasks that require in-depth exploration. The process of recursive pattern growth is similar to building a search tree, where each node represents a pattern. Starting from the root node (initial pattern), child nodes (expanded patterns) are generated through different expansion operations, gradually exploring the entire pattern space.

[0239] Extension type determination is the first step in recursive pattern growth, determining how extension items combine with the current pattern to form a new pattern. Sequence pattern mining primarily uses two extension types: I-extension (Itemset Extension) and S-extension (Sequence Extension). I-extension adds the extension item to the last itemset of the current pattern, resulting in a pattern with the same length as the original but a larger final itemset. S-extension adds the extension item as a separate itemset to the end of the current pattern, increasing its length by one unit. Extension type determination is typically based on two factors: the characteristics of the extension item and the structure of the current pattern. For the characteristics of the extension item, the temporal features of the item, its co-occurrence frequency with items in the final itemset, and semantic relevance are examined. For the pattern structure, the size of the final itemset, the sequence of extension operations already performed, and the overall complexity of the pattern are considered. Some extension items may be suitable for both I-extension and S-extension. In such cases, the system usually executes the extension type with the greater utility gain first, or tries both types and selects the optimal one. For example, if the current pattern is <{a,b},{c}>, the extension term is d, and d often co-occurs with c, then I-extension might be more appropriate, generating <{a,b},{c,d}>; if d rarely co-occurs with c but often appears after c, then S-extension might be more appropriate, generating <{a,b},{c},{d}>.

[0240] The extension operation transforms the abstract extension type into a concrete schema transformation action. For I-extensions, the extension item is added to the last itemset of the current schema, requiring handling of the itemset's internal order (usually sorted by item ID) and recalculation of itemset utility. For S-extensions, a new itemset containing only the extension item is created, added to the end of the current schema, and the overall temporal structure of the schema is updated. The execution of extension operations needs to consider efficient data structure operations, such as using linked lists to represent sequences for easy addition operations, or using bitmaps to represent itemsets for fast set operations. After the extension operation is executed, a sub-schema is generated. This new schema inherits the structure and characteristics of the parent schema and contains the newly added extension item, forming a longer or more complex schema. In practical implementations, to avoid unnecessary memory allocation and copying, an incremental data structure is typically used to represent the schema, storing only the parts that change relative to the parent schema, and combining them only when the complete schema needs to be output.

[0241] Non-overlapping instance lookup is a crucial step in determining the location of instances of a subpattern in the sequence database. The challenge lies in simultaneously satisfying the pattern matching and non-overlapping conditions. Instance lookup is based on the aforementioned Nettree data structure, but requires adjustments for newly expanded subpatterns: for subpatterns generated by I-expansion, the last layer of the Nettree is updated, retaining only nodes containing the newly added items; for subpatterns generated by S-expansion, a new layer is added to the Nettree, containing all position nodes matching the new itemset. After updating or constructing the Nettree, a path search is performed to find all possible paths from the root node to the leaf node, each path representing a potential instance of the pattern. Then, the system applies the non-overlapping condition, selecting a set of non-overlapping paths as the valid instance set. The non-overlapping condition requires that the selected paths should not overlap, ensuring that each sequence position is counted for utility at most once. Instance lookup can employ a greedy selection algorithm (selecting the path with the highest utility that does not overlap with the selected paths each time) or a dynamic programming method (constructing the optimal substructure and progressively solving for the maximum non-overlapping utility set). The output of the non-overlapping instance lookup is a set of non-overlapping instance locations of the subpattern in the database, providing a basis for subsequent utility calculations.

[0242] The actual non-overlapping utility calculation is a metric step for evaluating the true value of a sub-pattern. Based on a determined set of instance locations using non-overlapping instance lookup, the system calculates the utility value of each instance and sums the utility values ​​of all instances to obtain the actual non-overlapping utility of the sub-pattern. Instance utility calculation typically considers two factors: first, the item utility of each node on the instance path, i.e., the sum of the products of the internal and external utilities of all matching items on the path; second, the structural characteristics of the instance, such as the temporal span of the path and the gap distribution between itemsets. These factors may influence the final utility value through weighting coefficients. In some application scenarios, instance utility may also consider additional domain knowledge, such as the severity of attacks in network security or profit margins in business analysis. Unlike upper bound indicators such as sequence expansion utility, actual non-overlapping utility is a precise value directly used to determine whether a pattern meets the minimum utility threshold requirement. The calculated actual non-overlapping utility is a decisive indicator for the final retention or pruning of sub-patterns and is also an important reference value for pattern ranking and result display.

[0243] Multi-level pruning is an optimization strategy to avoid ineffective exploration during recursive pattern growth. By identifying and removing branches that are unlikely to generate efficient patterns early on, it significantly reduces the search space. At each step of recursive pattern growth, multi-level pruning is performed on newly generated sub-patterns, including techniques such as sequence pre-reading pruning, width pruning, co-occurrence pruning, and depth pruning. Different pruning strategies or parameters may be applied to patterns generated by I-expansion and S-expansion because the structural characteristics of the patterns produced by these two expansion types are different. For example, for I-expansion patterns, since only the last itemset is modified, most of the Nettree structure of the parent pattern can be reused, requiring only an update to the last layer, reducing reconstruction overhead; for S-expansion patterns, due to the addition of a new itemset layer, the system needs to consider new gap constraint verification, and may employ stricter sequence pre-reading pruning. The order in which multi-level pruning is applied is also important. Typically, simpler pruning with low computational cost (such as width pruning based on upper bounds) is performed first, followed by more complex pruning requiring detailed computation (such as depth pruning based on actual utility). This progressive strategy ensures that obviously ineffective branches are eliminated as early as possible, maximizing the pruning effect. Multi-level pruning significantly reduces the number of patterns requiring detailed exploration, thus significantly improving mining efficiency.

[0244] Recursive processing and result collection are the core execution mechanisms of recursive pattern growth. For each sub-pattern retained after pruning, the same expansion process is recursively called to further explore the potential expansion of the pattern. This recursive call forms a depth-first search tree traversal, ensuring that all child nodes of the current node are processed before backtracking to the parent node. During the recursive process, a current path is maintained, recording the complete pattern construction history from the root node to the current node, which is convenient for recovery and debugging. The recursive process stops when one of the following termination conditions is encountered: first, when the actual non-overlapping utility of the pattern is less than the minimum utility threshold, it indicates that the branch cannot produce a high-utility pattern, and further exploration stops; second, when the length or complexity of the pattern reaches a preset upper limit, expansion stops for practical considerations; third, when no expansion item that meets the condition is available, it indicates that the search tree leaf node has been reached, and natural termination occurs. For patterns that meet the retention conditions (actual non-overlapping utility is not less than the minimum utility threshold and contains the target sequence), they are added to the result set. The results collection process may include additional post-processing, such as results deduplication (removing isomorphic or nearly identical patterns), results sorting (sorting by utility value or other metrics), and results annotation (adding supplementary information such as support and confidence scores).

[0245] Recursive pattern growth methods face several key challenges in implementation, such as recursion depth control, memory management, and intermediate state preservation. Regarding recursion depth, systems typically set a maximum recursion depth limit (e.g., 50-100 levels) to avoid stack overflow; alternatively, they may employ iterative depth-first search, using an explicit stack structure instead of the system call stack for better control. For memory management, since the recursive process may generate a large number of intermediate results, systems typically use object pools, incremental data structures, or lazy loading techniques to reduce memory consumption; for extremely large datasets, disk paging mechanisms may also be implemented to temporarily store infrequently used intermediate results on disk. Preserving intermediate states is crucial for supporting breakpoint resumption and incremental mining. The system can periodically save search progress, the set of explored patterns, and key data statistics, allowing recovery from the most recent checkpoint in case of failure, avoiding redundant calculations.

[0246] The recursive pattern growth method's main advantage lies in its flexibility in pattern representation and expansion. Compared to traditional methods such as prior generation-testing verification, recursive pattern growth can handle more complex pattern structures and constraints, such as variable-length gaps, temporal dependencies, and utility weights. This method also supports adaptive search strategies, dynamically adjusting the search direction and pruning parameters based on intermediate results. For example, it can prioritize exploring relevant branches after discovering a specific high-utility pattern, or increase pruning intensity when the utility of a branch is low. This adaptive capability makes recursive pattern growth particularly suitable for exploratory data mining tasks, efficiently discovering valuable patterns even when the pattern distribution is unknown.

[0247] In practical applications, the recursive pattern growth method demonstrates significant advantages in handling non-overlapping, high-efficiency pattern mining tasks containing target sequences. For example, in cybersecurity, it can effectively identify complex attack sequences, including multi-stage attacks and stealthy persistent threats (APTs); in user behavior analysis, it can discover high-value user interaction patterns, supporting accurate recommendations and personalized services; and in industrial process monitoring, it can identify operational sequences leading to high output or high failure rates, optimizing production processes. This versatility and efficiency make it one of the core technologies in sequence pattern mining, providing a powerful tool for big data analysis across various industries.

[0248] Example 12

[0249] like Figure 2 As shown, this invention also provides a non-overlapping, high-efficiency sequence pattern mining system for targets, applicable to APT attack path mining in information security or equipment fault prediction in intelligent manufacturing, including:

[0250] The sequence preprocessing module 201 is used to perform subsequence judgment on each sequence in the original sequence database, filter out sequences that do not contain the target sequence, and retain the sequences that contain the target sequence in the preprocessed sequence database to obtain a preprocessed sequence database that only contains the target sequence.

[0251] The last instance location module 202 is used to locate the last valid instance path of the target sequence in the sequence database containing only the preprocessed sequence from right to left for each sequence, and to store the last valid instance path in the sequence that satisfies the gap constraint and length constraint. The last valid instance path is then stored in the last instance table to obtain the last instance table containing the last valid instance path.

[0252] The sequence expansion utility calculation module 203 is used to convert each sequence in the preprocessed sequence database containing only the target sequence into a q matrix format containing a utility matrix and a residual utility matrix, calculate the sequence expansion utility of each item in conjunction with the last instance table containing the last valid instance path, and filter items whose sequence expansion utility is not less than the minimum utility threshold as promising 1-sequences.

[0253] The actual non-overlapping utility calculation module 204 is used to construct a Nettree data structure containing multiple root nodes and multiple parent nodes for each pattern in the promising 1-sequence, search for instance paths that satisfy the non-overlapping condition in the Nettree data structure, and calculate the actual non-overlapping utility of the pattern.

[0254] The multi-level pruning module 205 is used to perform multi-level pruning processing, including sequence pre-reading pruning, width pruning, co-occurrence pruning and depth pruning, on the extension items of the pattern based on the anti-monotonic properties of the actual non-overlapping utility and the upper bound of the extended utility, so as to obtain a promising set of extension items.

[0255] The recursive pattern growth module 206 is used to perform recursive pattern growth on the set of promising expansion items using depth-first search, and output a set of non-overlapping high-utility sequence patterns that meet the minimum utility threshold and contain the target sequence.

[0256] The following example of a specific network security scenario will be used to illustrate the implementation process of this invention in detail.

[0257] A large enterprise's cybersecurity team needs to identify potential APT attack paths from massive amounts of logs. Security analysts have identified a characteristic sequence for a certain APT group as T = ⟨{phishing emails}, {payload delivery}, {C2 communication}>. The goal is to mine high-risk attack paths containing these characteristics from network logs in order to detect and block attacks in a timely manner.

[0258] Step 1: Data Preprocessing

[0259] First, security event logs from the network log system for the past month are collected and converted into a sequence database format. Each session ID corresponds to a sequence, and each security event corresponds to an itemset. For example:

[0260] Session_001: ⟨{Port Scan:risk=0.3, 10.0.1.5}, {SQL Injection Attempt:risk=0.7,web-server-01}, {Webshell Upload:risk=0.9}, {Lateral Movement:risk=0.95, DB-server}, {Data Leakage:risk=1.0, Sensitive Data 500GB}>

[0261] Each event has a corresponding risk coefficient as its external utility, while the number of events or the scope of their impact are considered as internal utility.

[0262] First, sessions that do not contain the target sequence T are filtered out. Subsequence matching is used to retain only session sequences containing the three security events: {phishing email}, {payload delivery}, and {C2 communication}. Furthermore, these events are required to meet specific gap constraints (e.g., adjacent events cannot exceed 12 hours) and length constraints (the entire attack chain cannot exceed 2 weeks).

[0263] Step 2: Final instance location

[0264] For each preprocessed session sequence, a greedy right-direction search method is used to locate the last valid instance path of the target sequence T. For example, for session002, the search starts from the rightmost end and proceeds left:

[0265] Find {C2 communication} at position 5 (the rightmost match);

[0266] Find {Payload delivery} at position 3 to the left (satisfying the gap constraint);

[0267] Continue to the left and find {phishing email} at position 1 (satisfying the gap constraint);

[0268] Verify that the span (5 days) of the entire instance path ⟨1, 3, 5> meets the length constraint (not exceeding 2 weeks);

[0269] Store this instance path ⟨1, 3, 5> in the LI-Table, corresponding to Session002.

[0270] Step 3: Calculate the utility of sequence expansion

[0271] Each session sequence is converted into a q-matrix format, and the utility and residual utility of each event are calculated. For example, for Session002:

[0272] The utility of the phishing email is 1 × 0.6 = 0.6, and the remaining utility is the sum of the utilities of all the remaining events = 3.4;

[0273] The utility of {malware detection} is 1 × 0.5 = 0.5, and the remaining utility is 3.0;

[0274] And so on.

[0275] The LI-Table is used to determine whether each event meets the co-occurrence condition with the target sequence T. For example, the {data theft} event occurs after {C2 communication}, thus meeting the co-occurrence condition. For all events that meet the condition, their Sequence Extended Utility (SEU) is calculated. Assuming the minimum utility threshold is set to 0.8, events with SEU ≥ 0.8 will be selected as promising 1-sequences.

[0276] Step 4: Calculation of actual non-overlapping utility

[0277] For each promising 1-sequence (e.g., ⟨{phishing email}>), construct a Nettree data structure. In the context of security incidents, each level of the Nettree corresponds to a security incident type, and each node corresponds to the specific location where the incident occurred.

[0278] The Nettree searches for instance paths that satisfy the non-overlapping condition. In APT attack scenarios, the non-overlapping condition ensures that the same phase of the same attack session is not counted repeatedly. For example, even if an attacker performs C2 communication multiple times, C2 communication at the same location will not be counted multiple times when calculating the utility of a specific attack pattern.

[0279] Step 5: Multi-level pruning

[0280] Perform multi-level pruning on the pattern's extensions:

[0281] Sequence Prereading Pruning (SPPS): Filters out session sequences that do not contain valid non-overlapping instances of the target sequence T under all constraints.

[0282] Width Pruning (WPS): Calculates the Terminating Descendant Utility (TDU) of a pattern and prunes pattern branches with TDU < 0.8;

[0283] Collaborative emergence pruning (CPS): Calculate the extension utility (EU) of each extension item (such as {data theft}, {privilege escalation}, etc.) and prune extension items with EU < 0.8;

[0284] Deep Pruning (DPS): Calculate the SEU of the generated candidate patterns and prune patterns with SEU < 0.8.

[0285] These pruning strategies significantly reduce the search space, enabling the system to quickly focus on high-value attack path patterns.

[0286] Step 6: Recursive Growth Pattern

[0287] Perform a depth-first search on the remaining set of extensions. For example, starting with ⟨{phishing emails}>, possible extensions include:

[0288] I-Extension: ⟨{Phishing Emails, Malware Detection}>;

[0289] S-Extensions: ⟨{Phishing Emails}, {Payload Delivery}>, ⟨{Phishing Emails}, {Privilege Escalation}>, etc.

[0290] For each extended pattern, recursively perform non-overlapping instance lookup, utility calculation, and pruning. The final output is a set of non-overlapping high-utility attack path patterns that satisfy the minimum utility threshold and contain the target sequence T.

[0291] Results presented:

[0292] The following highly efficient attack path patterns were ultimately discovered:

[0293] Pattern_1: ⟨{Phishing Email}, {Payload Delivery}, {C2 Communication}, {Credential Theft}, {Lateral Movement}, {Data Theft}>

[0294] - Total risk utility: 4.8

[0295] - Sessions involved: 15 separate attack incidents

[0296] - Source of attacks: Primarily from specific IP address ranges

[0297] Pattern_2: ⟨{Phishing Email}, {Payload Delivery}, {C2 Communication}, {Privilege Escalation}, {Persistent Backdoor}>

[0298] - Total risk utility: 3.2

[0299] - Sessions involved: 8 separate attack events

[0300] - Features: Persistence is achieved using specific registry keys.

[0301] These results helped security analysts quickly identify high-risk attack paths and discover the modus operandi and preferences of APT groups, greatly improving the efficiency and accuracy of security responses.

[0302] The following example, using equipment failure prediction in intelligent manufacturing, further illustrates the implementation process of this invention.

[0303] A semiconductor manufacturing company wants to uncover anomalous sequence patterns in equipment sensor data that could lead to serious failures, such as wafer scrapping or production line downtime. Engineers have identified a precursory feature sequence for a certain type of failure as T = ⟨{early anomalies}, {performance degradation}, {serious failures}>. They hope to mine high-cost failure patterns containing these features from historical data to enable predictive maintenance.

[0304] Step 1: Data Preprocessing

[0305] First, sensor data and fault records from the equipment monitoring system for the past year are collected and converted into a sequence database format. Each device ID corresponds to a sequence, and sensor readings and alarms within each time window form an itemset. For example:

[0306] Equipment_A01: ⟨{Temperature Anomaly: +5℃, t=0h, cost=$0}, {Increased Vibration: +20Hz, t=2h, cost=$0}, {Insufficient Lubrication Warning, t=5h, cost=$500}, {Bearing Overheat: +15℃, t=8h, cost=$2K}, {Spindle Failure, t=12h, cost=$150K, Downtime 8 Hours}>

[0307] Each event has a corresponding maintenance / downtime cost as its external utility, and the degree of abnormality (such as temperature deviation) as its internal utility.

[0308] Filter device sequences that do not contain the target sequence T. Based on subsequence matching, retain only those device sequences containing events of the following three categories: {early anomaly}, {performance degradation}, and {critical failure}. Simultaneously, these events must meet specific gap constraints (e.g., the interval between adjacent events does not exceed 6 hours) and length constraints (the entire failure development process must occur within 24 hours).

[0309] Step 2: Final instance location

[0310] For each preprocessed device sequence, a greedy right-direction search method is used to locate the last valid instance path of the target sequence T. For example, for EquipmentB03, the system:

[0311] Find {Critical Fault} at position 8 (the rightmost match);

[0312] Find {performance degradation} at position 5 to the left (satisfies the gap constraint);

[0313] Continue to the left to find the {early anomaly} at position 2 (satisfying the gap constraint);

[0314] Verify that the span (15 hours) of the entire instance path ⟨2, 5, 8> satisfies the length constraint (not exceeding 24 hours);

[0315] Store this instance path ⟨2, 5, 8> in the LI-Table, corresponding to EquipmentB03.

[0316] Step 3: Calculate the utility of sequence expansion

[0317] Convert each device sequence into a q-matrix format and calculate the utility and residual utility of each event. For example, for EquipmentB03:

[0318] The utility of {temperature anomaly} is 5 × 2 = $10, and the remaining utility is the sum of the utilities of all the remaining events = $290;

[0319] The utility of {vibration enhancement} is 20 × 1 = $20, and the remaining utility is $270;

[0320] And so on.

[0321] The LI-Table is used to determine whether each event meets the co-occurrence condition with the target sequence T. For example, the {part wear} event occurs between {early anomaly} and {performance degradation}, satisfying the co-occurrence condition. For all events that meet the condition, their Sequence Extended Utility (SEU) is calculated. Assuming the minimum utility threshold is set to $100K, the system will select events with SEU ≥ $100K as promising 1-sequences.

[0322] Step 4: Calculation of actual non-overlapping utility

[0323] For each promising 1-sequence (e.g., ⟨{temperature anomaly}>), a Nettree data structure is constructed. In the context of device failure, each layer of the Nettree corresponds to a sensor anomaly type, and each node corresponds to the specific time the anomaly occurred.

[0324] The Nettree searches for instance paths that satisfy the non-overlapping condition. In fault prediction scenarios, the non-overlapping condition ensures that different fault cycles of the same device are calculated independently. For example, even if a device experiences similar temperature anomalies multiple times, temperature anomalies within the same fault cycle will not be calculated multiple times when calculating the cost of a specific fault mode.

[0325] Step 5: Multi-level pruning

[0326] Perform multi-level pruning on the pattern's extensions:

[0327] Sequence Prereading Pruning (SPPS): Filters out device sequences that do not contain valid non-overlapping instances of the target sequence T under all constraints.

[0328] Width Pruning (WPS): Calculates the Terminating Descendant Utility (TDU) of a pattern and prunes pattern branches with TDU < $100K;

[0329] Co-emergence pruning (CPS): Calculate the extended utility (EU) of each extension term (such as {vibration anomaly}, {pressure drop}, etc.) and prune extension terms with EU < $100K;

[0330] Deep Pruning (DPS): Calculate the SEU for the generated candidate patterns and prune patterns with SEU < $100K.

[0331] These pruning strategies significantly reduce the search space, enabling the system to focus on high-cost failure modes.

[0332] Step 6: Recursive Growth Pattern

[0333] Perform a depth-first search on the remaining set of extensions. For example, starting with ⟨{temperature anomaly}>, possible extensions include:

[0334] I-Extension: ⟨{Temperature anomaly, pressure drop}>;

[0335] S-Extensions: ⟨{Temperature Anomaly}, {Increased Vibration}>, ⟨{Temperature Anomaly}, {Part Wear}>, etc.

[0336] For each extended pattern, recursively perform non-overlapping instance lookup, utility calculation, and pruning. The final output is a set of non-overlapping high-utility failure patterns that satisfy the minimum utility threshold and contain the target sequence T.

[0337] Results presented:

[0338] The following high-cost failure modes were ultimately identified:

[0339] Pattern_1: ⟨{Cooling system pressure drop}, {Abnormal temperature}, {Increased bearing vibration}, {Performance degradation}, {Component wear}, {Serious malfunction}>

[0340] - Average cost: $180K / time

[0341] - Frequency of occurrence: 23 times in the past year

[0342] - Early warning window: The average interval between the first anomaly and the failure is 6.5 hours.

[0343] - Preventability: High (Timely maintenance can prevent 85% of losses)

[0344] Pattern_2: ⟨{Current Fluctuation}, {Temperature Anomaly}, {Machining Accuracy Deviation}, {Performance Degradation}, {Sudden Increase in Product Defect Rate}, {Critical Fault}>

[0345] - Average cost: $95K / time

[0346] - Equipment involved: CNC machining center, injection molding machine

[0347] - Root cause: Motor bearing wear + cutting tool dulling

[0348] - Warning window: 4.2 hours

[0349] These results helped users identify early warning patterns of high-cost failures, establish effective early warning systems, and significantly reduce production losses caused by equipment failures.

[0350] The non-overlapping efficient sequence pattern mining method and system for targets provided by this invention have the following technical advantages:

[0351] 1. High Efficiency: Through data preprocessing, multi-level pruning, and optimized data structures, computational complexity is significantly reduced, enabling efficient operation on large-scale datasets. Tests show that mining speed is 5-10 times faster than existing methods.

[0352] 2. Accuracy: By constraining the target sequence, we ensure that the mining results are highly relevant to user interests; by using non-overlapping conditions, we avoid overestimating utility due to duplicate counting; and by using utility measurement, we ensure that truly high-value patterns are discovered.

[0353] 3. Scalability: The system architecture supports parallel processing and dynamic resource allocation, and can linearly improve performance as hardware resources increase;

[0354] 4. Robustness: Through sequence quality testing, it can handle noisy sequence data with a noise tolerance rate of up to 30%, which greatly improves its applicability in real-world scenarios;

[0355] 5. Practicality: Application tests in two typical fields, cybersecurity and intelligent manufacturing, demonstrate that this method can effectively identify high-value patterns, with significant practical application results. In APT attack detection, it can detect attack signs up to 12 days in advance; in equipment failure prediction, the average warning window reaches 6.5 hours, sufficient for preventative maintenance.

[0356] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A non-overlapping, efficient sequence pattern mining method for targets, characterized in that, This method, applied to APT attack path mining in information security, includes: collecting security event logs from a network log system and converting them into a raw sequence database, where each session ID corresponds to a sequence, each security event corresponds to an itemset, the risk coefficient of the security event is used as external utility, the number or impact range of the security events is used as internal utility, and the characteristic sequence of the APT organization is used as the target sequence; each sequence in the raw sequence database is subjected to subsequence judgment, filtering out sequences that do not contain the target sequence, and retaining the sequences containing the target sequence in the preprocessed sequence database, resulting in a preprocessed sequence database containing only the target sequence; and counting the number of preprocessed sequences containing only the target sequence. For each sequence in the database, a greedy right-to-left search is used to locate the last valid instance path of the target sequence within the sequence that satisfies the gap constraint and the length constraint. The gap constraint limits the time interval between adjacent security events, and the length constraint limits the time span of the entire attack chain. The last valid instance path is stored in a last instance table, resulting in a last instance table containing the last valid instance path. Each sequence in the preprocessed sequence database containing only the target sequence is converted into a q-matrix format containing a utility matrix and a residual utility matrix. The sequence extension utility of each item is calculated using the last instance table containing the last valid instance path, and sequences with extension utilities not less than a minimum utility threshold are selected. The value of an item is considered a promising 1-sequence, comprising: for each sequence in the preprocessed sequence database containing only the target sequence, calculating the product of the internal utility and the external utility of each item in the sequence in each item set as a utility value; calculating the sum of the utilities of all subsequent items from the position of each item in the sequence to the end of the sequence as a residual utility value; storing the utility value and the residual utility value as a utility matrix and a residual utility matrix respectively, forming a q matrix; for each item in the q matrix, determining whether the occurrence position of the item satisfies the co-occurrence condition with the target sequence by combining the last instance table containing the last valid instance path; and calculating the utility value and residual utility value of items that satisfy the co-occurrence condition. The potential utility value is summed among the maximum values ​​of the potential utility values ​​in each sequence to obtain the sequence expansion utility of the item; items whose sequence expansion utility is not less than the minimum utility threshold are selected as promising 1-sequences, wherein the minimum utility threshold is the product of a threshold ratio set by the user and the total utility value of the preprocessed sequence database containing only the target sequence; for each pattern in the promising 1-sequence, a Nettree data structure containing multiple root nodes and multiple parent nodes is constructed, and instance paths that satisfy the non-overlapping condition are found in the Nettree data structure, and the actual non-overlapping utility of the pattern is calculated, wherein the non-overlapping condition ensures that the same stage of the same attack session is not counted repeatedly;Based on the antimonotonic properties of the actual non-overlapping utility and the upper bound of the extended utility, a multi-level pruning process, including sequence pre-reading pruning, width pruning, co-occurrence pruning, and depth pruning, is performed on the extended items of the pattern to obtain a set of promising extended items. For the set of promising extended items, a depth-first search is used for recursive pattern growth, outputting a set of non-overlapping high-utility sequence patterns that satisfy the minimum utility threshold and contain the target sequence. This set of non-overlapping high-utility sequence patterns is used to identify APT attack chains.

2. The method according to claim 1, characterized in that, The step of performing subsequence judgment on each sequence in the original sequence database, filtering out sequences that do not contain the target sequence, and retaining sequences that contain the target sequence in the preprocessed sequence database includes: initializing the preprocessed sequence database, the final instance table, and the total utility value of the database; for each sequence in the original sequence database, checking whether the length of the sequence is not less than the length of the target sequence, and performing subsequence matching judgment on sequences that meet the length condition, filtering out sequences that do not contain the target sequence; for sequences that contain the target sequence, finding valid instance paths of the target sequence in the sequence based on gap constraints and length constraints, filtering out sequences that do not have valid instance paths, and retaining sequences with valid instance paths in the preprocessed sequence database, resulting in a preprocessed sequence database containing only the target sequence.

3. The method according to claim 1, characterized in that, The method employs a greedy search from right to left to locate the last valid instance path of the target sequence that satisfies the gap constraint and length constraint within the sequence. The gap constraint limits the time interval between adjacent security events, and the length constraint limits the time span of the entire attack chain. The last valid instance path is then stored in a last instance table, resulting in a last instance table containing the last valid instance path. This process includes: scanning leftwards from the rightmost position of the sequence to find the matching position of the last itemset of the target sequence, obtaining the root node position; sequentially searching leftwards from the root node position for the matching position of each itemset in the target sequence, selecting the rightmost matching position that satisfies the gap constraint at each level to obtain a complete instance path; verifying whether the span of the complete instance path satisfies the length constraint, and using the instance path that satisfies the length constraint as the last valid instance path; storing the matching positions of each itemset in the last valid instance path in the last instance table, using the itemsets in the target sequence as column indices and the sequence identifier as row indices, to obtain a last instance table containing the last valid instance path.

4. The method according to claim 1, characterized in that, For each pattern in the promising 1-sequence, a Nettree data structure containing multiple root nodes and multiple parent nodes is constructed. Instance paths satisfying the non-overlapping condition are found in the Nettree data structure, and the actual non-overlapping utility of the pattern is calculated. This includes: establishing a layer of the Nettree data structure for each itemset in the pattern; collecting all positions matching the itemset in the sequence as nodes of the layer and recording the utility value corresponding to each node; establishing connections between adjacent layers of the Nettree data structure for node pairs that satisfy the positional order relationship and whose gaps are within the gap constraint range; removing isolated nodes without valid paths to nodes in the upper layer from the Nettree data structure to obtain a Nettree data structure after removing isolated nodes; for the Nettree data structure after removing isolated nodes, enumerating all complete paths from the root node to the absolute leaf node; selecting a set of paths satisfying the non-overlapping condition from the complete paths; and summing the utility values ​​of each path in the path set to obtain the actual non-overlapping utility of the pattern.

5. The method according to claim 1, characterized in that, Based on the anti-monotonicity of the actual non-overlapping utility and the upper bound of the extended utility, the extension items of the pattern are subjected to multi-level pruning processing including sequence pre-reading pruning, width pruning, co-occurrence pruning, and depth pruning to obtain a promising set of extension items. This includes: for each sequence in the preprocessed sequence database containing only the target sequence, checking whether the sequence contains a valid non-overlapping instance of the target sequence under the conditions of satisfying the gap constraint and the length constraint; removing sequences that do not contain valid non-overlapping instances from the current mining scope as sequence pre-reading pruning; calculating the corresponding terminating descendant utility for the pattern, where the terminating descendant utility is the sum of the sequence extension utility of the effective extensions of the pattern in each sequence; and determining the terminating descendant utility when it is less than the minimum utility threshold. When the value is reached, the pattern and all corresponding extension pruning are treated as width pruning; for each extension item of the pattern, the extension utility of the extension item is calculated, where the extension utility is the cumulative value in the database of the sum of the utility of the parent instance, the utility of the extension item, and the remaining utility. When the extension utility is less than the minimum utility threshold, the extension item is removed from the effective extension set as co-occurrence pruning; for the candidate patterns generated after co-occurrence pruning, the corresponding sequence extension utility is calculated. When the sequence extension utility is less than the minimum utility threshold, the recursive expansion of the candidate patterns is stopped as depth pruning; the extension items retained after the sequence pre-reading pruning, the width pruning, the co-occurrence pruning, and the depth pruning are combined to form the promising extension item set.

6. The method according to claim 1, characterized in that, Before performing subsequence judgment on each sequence in the original sequence database, filtering sequences that do not contain the target sequence, and retaining sequences containing the target sequence in the preprocessed sequence database, the process also includes performing sequence quality checks on the original sequence database. This includes: extracting the total sequence utility, matching degree with the target sequence, sequence complexity, and temporal coherence as multi-dimensional feature vectors for each sequence in the original sequence database, and using the greedy search to verify the true labels of the sequences to obtain a labeled feature dataset; for the labeled feature dataset, based on the constraint that the label noise rate does not exceed a preset upper bound, training a semi-space classifier in the utility-frequency space to obtain a classification hyperplane used to distinguish between truly valid sequences and noisy sequences; The classification hyperplane undergoes cross-validation, and the confidence score of the classification accuracy is calculated based on the Hoeffding inequality. When the confidence score is not less than a preset confidence threshold, the classification hyperplane is determined as an effective classifier. For each sequence in the original sequence database, the effective classifier is used to calculate the corresponding predicted label and predicted confidence score. Sequences with predicted labels containing effective target sequences and predicted confidence scores not less than the preset confidence threshold are retained. Sequences with predicted labels not containing effective target sequences and predicted confidence scores not less than the preset confidence threshold are filtered out. Sequences with predicted confidence scores less than the preset confidence threshold are marked as suspicious sequences. Multiple greedy search verifications are performed on the suspicious sequences, and the verified suspicious sequences are retained in the preprocessed sequence database.

7. The method according to claim 1, characterized in that, The construction of a Nettree data structure containing multiple root nodes and multiple parent nodes, after finding instance paths satisfying non-overlapping conditions in the Nettree data structure, further includes performing critical instance path identification processing on the Nettree data structure, including: mapping the search space of the Nettree data structure into a multi-dimensional polygon, where each layer corresponds to one dimension, and the position, utility value, remaining utility value, and number of paths to the leaves of each node constitute the vertex coordinates of the multi-dimensional polygon; recursively performing binary subdivision along the utility difference direction on the multi-dimensional polygon, decomposing the multi-dimensional polygon into multiple sub-regions, where each sub-region corresponds to a type of instance path; and for each of the multiple sub-regions, Calculate the sum of utility contributions of all instance paths within the sub-region and the path overlap between the sub-region and other sub-regions. Use the product of the utility contribution and the path overlap as a criticality score. Mark sub-regions with criticality scores not less than a preset critical threshold as critical sub-regions. Perform greedy search and constraint verification on instance paths within the critical sub-regions to obtain a set of valid non-overlapping instances within the critical region. Perform supplementary search on instance paths in non-critical regions outside the critical sub-regions. Merge the set of valid non-overlapping instances within the critical region with the search results in the non-critical regions. Prune non-critical branches in the merged instance paths whose upper utility bound is lower than a preset multiple of the minimum utility threshold.

8. The method according to claim 1, characterized in that, The process of recursively growing the set of promising extension items using depth-first search, outputting a set of non-overlapping high-utility sequence patterns that satisfy the minimum utility threshold and contain the target sequence, includes performing a borrowing graph-based scheduling process on the recursive pattern growth task. This scheduling process includes: creating a pattern exploration task for each extension item in the set of promising extension items; setting a task identifier, parent pattern, and the sequence expansion utility of the pattern as priority and estimated processing time for each pattern exploration task; constructing a set of pattern exploration tasks; establishing a directed graph of parent-child dependencies and resource competition relationships among the tasks in the set of pattern exploration tasks to obtain a borrowing graph, wherein dependency edges are established between parent tasks and child tasks, and resource competition edges are established between tasks requiring the same computing resources; and calculating a latency perspective score for each pattern exploration task in the borrowing graph, wherein the latency perspective score is the sum of its own latency cost and the cascaded latency cost minus... The resource release benefit is calculated as follows: the self-delay cost is the product of the priority of the pattern exploration task and the estimated processing time; the cascading delay cost is the sum of the products of the priority and estimated processing time of all descendant tasks of the pattern exploration task; and the resource release benefit is the sum of the priority-weighted delay differences of tasks that compete with the pattern exploration task for resources. Based on the delay perspective score, the set of pattern exploration tasks is prioritized, and tasks with delay perspective scores from largest to smallest are sequentially assigned to worker threads. When all worker threads are occupied, subsequent tasks are placed in a priority queue to wait. When a pattern exploration task is completed, its subtasks are added to the priority queue. The delay perspective scores of tasks in the priority queue are recalculated and scheduled for execution until the priority queue is empty. Patterns from all completed pattern exploration tasks that satisfy the minimum utility threshold and contain the target sequence are output as the set of non-overlapping high-utility sequence patterns.

9. The method according to claim 8, characterized in that, The process of prioritizing the set of pattern exploration tasks based on the latency perspective score and assigning tasks with the latency perspective scores from largest to smallest to worker threads includes performing dynamic resource borrowing and parallel recursive mining processing on the pattern exploration tasks. This includes: when a target pattern exploration task requests resource borrowing, traversing the currently executing pattern exploration tasks in the borrowing graph, filtering out pattern exploration tasks with a latency tolerance greater than the estimated processing time of the target pattern exploration task and not on the dependency path of the target pattern exploration task as a set of available borrowing tasks, selecting the pattern exploration task with the highest latency tolerance from the set of available borrowing tasks as the resource lending task, obtaining the resource lending task; allocating a preset proportion of computing resources to the target pattern exploration task for the resource lending task, adding a borrowing edge from the target pattern exploration task to the resource lending task in the borrowing graph, and recording the amount of borrowed resources; and processing the set of pattern exploration tasks... The estimated processing times of each pattern exploration task are compared. When the number of pattern exploration tasks exceeds a preset multiple of the number of worker threads, adjacent pattern exploration tasks with estimated processing time differences within a preset time threshold are merged into batch tasks. When the number of pattern exploration tasks does not exceed the preset multiple, single-task scheduling is maintained, resulting in a set of scheduled tasks containing batch tasks and single tasks. Each batch task and single task in the set of scheduled tasks is sorted from largest to smallest estimated processing time, and the sorted batch tasks and single tasks are sequentially assigned to the worker thread with the lowest current load, so that the total load of each worker thread tends to be balanced, resulting in a task allocation scheme for each worker thread. Based on the task allocation scheme, each worker thread executes the assigned pattern exploration tasks in parallel, performs a depth-first search recursive pattern growth on the patterns in each pattern exploration task, and merges the patterns that meet the minimum utility threshold and contain the target sequence into the set of non-overlapping high-utility sequence patterns.

10. The method according to claim 1, characterized in that, The process of recursively growing the set of promising extension items using depth-first search to output a set of non-overlapping high-utility sequence patterns that meets the minimum utility threshold and contains the target sequence includes: for each extension item in the set of promising extension items, determining the extension type of the extension item relative to the current pattern, wherein the extension type includes I-extension (adding the extension item to the last itemset of the current pattern) and S-extension (adding the extension item as a new itemset to the end of the current pattern); performing an extension operation on the current pattern according to the extension type to generate a sub-pattern; recursively performing non-overlapping instance search, actual non-overlapping utility calculation, and multi-level pruning on the sub-pattern; and adding the sub-pattern to the set of non-overlapping high-utility sequence patterns when the actual non-overlapping utility of the sub-pattern is not less than the minimum utility threshold and the sub-pattern contains the target sequence.

11. A non-overlapping, high-efficiency sequence pattern mining system for a target, characterized in that, This system, applied in the field of APT attack path mining in information security, includes: a sequence preprocessing module, used to collect security event logs from a network log system and convert them into a raw sequence database. Each session ID corresponds to a sequence, and each security event corresponds to an itemset. The risk coefficient of the security event is used as the external utility, the number or impact range of the security events as the internal utility, and the characteristic sequences of the APT organization are used as target sequences. The module also performs subsequence judgment on each sequence in the raw sequence database, filtering sequences that do not contain the target sequence, and retaining sequences containing the target sequence in the preprocessed sequence database, resulting in a database containing only the target sequence. The system comprises: a preprocessed sequence database containing only the target sequence; a last instance location module, used to locate the last valid instance path of the target sequence in the preprocessed sequence database containing only the target sequence by performing a greedy search from right to left, wherein the gap constraint is used to limit the time interval between adjacent security events, and the length constraint is used to limit the time span of the entire attack chain; and a sequence extension utility calculation module, used to calculate the last valid instance path in the preprocessed sequence database containing only the target sequence. Each sequence is converted into a q-matrix format containing a utility matrix and a residual utility matrix. The sequence expansion utility of each item is calculated using the last instance table containing the last valid instance path. Items with a sequence expansion utility not less than a minimum utility threshold are selected as promising 1-sequences. This includes: for each sequence in the preprocessed sequence database containing only the target sequence, calculating the product of the internal utility and external utility of each item in the sequence in each item set as a utility value; calculating the sum of the utilities of all subsequent items from the position of each item in the sequence to the end of the sequence as a residual utility value; and storing the utility value and the residual utility value as utility matrices respectively. The matrix and the residual utility matrix are used to form the q matrix. For each item in the q matrix, the occurrence position of the item is determined by combining the last instance table containing the last valid instance path to determine whether it meets the condition of co-occurrence with the target sequence. For items that meet the co-occurrence condition, the sum of its utility value and residual utility value is calculated as the potential utility value. The maximum value of the potential utility value in each sequence is accumulated to obtain the sequence extension utility of the item. Items whose sequence extension utility is not less than the minimum utility threshold are selected as promising 1-sequences, wherein the minimum utility threshold is the product of the threshold ratio set by the user and the total utility value of the preprocessed sequence database containing only the target sequence.The module for calculating actual non-overlapping utility is used to construct a Nettree data structure containing multiple root nodes and multiple parent nodes for each pattern in the promising 1-sequence, search for instance paths that satisfy the non-overlapping condition in the Nettree data structure, and calculate the actual non-overlapping utility of the pattern, wherein the non-overlapping condition ensures that the same stage of the same attack session is not counted repeatedly; the module for multi-level pruning is used to perform multi-level pruning processing on the extension items of the pattern based on the anti-monotonic properties of the actual non-overlapping utility and the upper bound of the extended utility, including sequence pre-reading pruning, width pruning, co-occurrence pruning, and depth pruning, to obtain a set of promising extension items; the module for recursive pattern growth is used to perform recursive pattern growth on the set of promising extension items using depth-first search, and output a set of non-overlapping high-utility sequence patterns that satisfy the minimum utility threshold and contain the target sequence, the set of non-overlapping high-utility sequence patterns is used to identify APT attack chains.

Images