Complex mode online detection method for mobile data stream

Abstract

The invention discloses a complex mode online detection method for a mobile data stream, which can ensure that candidate instances between modes can be quickly and efficiently aggregated while keeping high efficiency so as to meet the requirement of online detection. According to the method, a potential hierarchical clustering structure between complex modes is utilized, each complex mode is associated with a similar mode through a content-aware structure entropy minimization algorithm to construct an association graph, and closely connected clusters are formed; according to the method, candidate instances capable of being aggregated are recognized by preferentially checking common sub-modes of complex modes in clusters, so that construction of an aggregation plan is accelerated, and an exponential-level search space does not need to be traversed; for'residual 'instances not aggregated in clusters, a lightweight differential compensation mechanism is designed to further find aggregable instances between them across clusters. Based on a tree-shaped multi-level aggregation plan generated by the mechanism, candidate instances are created and aggregated step by step among complex modes, and efficient online detection is realized.

Description

Technical Field

[0001] This invention relates to the field of complex pattern detection in mobile data streams. Specifically, it presents a method for constructing a fast and scalable aggregation plan of common candidate instances among complex patterns while maintaining high efficiency, thereby effectively meeting the online detection requirements of complex patterns in mobile data streams. Background Technology

[0002] Many mobile applications (such as mobile payments and navigation) generate massive amounts of streaming data at high rates and require real-time processing and analysis. Traditionally, for relatively simple applications (such as transaction logs and device location), data items in the data stream are processed independently, typically resulting in linear time and space complexity. In recent years, however, the development of complex or intelligent mobile applications based on the Internet of Things (IoT), such as supply chains, smart transportation, and network management, has created a need to detect complex patterns in data streams. These patterns are often combinations of user-defined data items. In practical applications, complex pattern detection is crucial for numerous mobile applications, including supply chains, mobile financial services, and application maintenance.

[0003] Complex pattern detection is typically performed by a complex event processing system deployed on a backend server, edge device, or by a server-agnostic function. Based on the incoming moving data stream, the complex event processing system continuously creates combinations of input data items as candidate instances that partially match one or more complex patterns. Simultaneously, it checks whether these instances meet other requirements of the complex pattern, such as the order of data items, time intervals, and attribute requirements, and then stores the qualifying candidate instances in memory.

[0004] In early complex event processing systems, instances of each complex pattern were created and stored independently. However, as pattern length (i.e., the number of basic events required by a complex pattern) grows rapidly, the size of candidate instances also increases dramatically, leading to a significant waste of computational and storage resources. Since early complex event processing systems only processed a small number of complex patterns, this problem did not significantly impact their performance. To enable complex event processing systems to scale to an increasing number of complex patterns, a natural approach is to aggregate common candidate instances from multiple complex patterns. Existing research has also devised various methods to minimize the overall instance size of a given complex pattern through instance aggregation, the key being the construction of an aggregation plan that specifies which instances are aggregated across which complex patterns. To find the optimal aggregation plan, existing methods identify aggregable instances by traversing a subset of the basic event types required by the complex pattern, while utilizing combinatorial optimization or heuristics to construct the aggregation plan. However, when complex patterns require… 1 basic event (i.e., its length is 1) When ), there will be The solution space for the aggregation plan expands rapidly with the length and number of patterns, being a subset of the basic event set. Initially, this exponentially large solution space does not lead to excessive system load, as traditional methods typically only need to handle a few dozen patterns with a maximum length of [missing information]. Complex patterns, both left and right. However, with the rapid development and popularization of mobile applications, the length (over 20) and number (up to thousands) of patterns are constantly increasing. Especially when generating complex patterns using generative large language models, building instance aggregation plans has become a new scalability bottleneck for complex event processing systems. The high time cost necessitates offline construction of aggregation plans, and because data streams need to be processed in real time, this results in a considerable downtime for complex event processing systems. Furthermore, due to the high construction cost, the quality of offline-built aggregation plans still involves a trade-off between construction complexity, thus having a long-term impact on the performance of complex event processing systems. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to solve the problems of inefficient instance aggregation plan construction and inability to execute online in complex pattern detection of mobile data streams. By combining content-aware structural entropy minimization and differential compensation mechanisms, the present invention accelerates candidate instance aggregation for complex patterns, thereby supporting online detection while reducing instance size and improving detection efficiency.

[0006] This invention provides an online method for detecting complex patterns in mobile data streams, comprising:

[0007] Step 1: Obtain the set of complex patterns to be detected, and classify each complex pattern... Defined as a quadruple Each complex pattern matching instance is composed of data items, i.e., basic events, from the mobile data stream; where It matches. The set of basic event types required in the instance It is a match The maximum permissible time difference between any pair of basic events in an instance. This indicates the requirements for basic event properties; These refer to logical operators, which specify the logical relationships required between basic events;

[0008] The logical operators include SEQ, AND, OR, and NOT; where the SEQ logical operator requires that all required basic event types appear in an instance of the complex pattern, and the basic events of each type appear in the required chronological order; the complex pattern P requires the basic events of types A, B, and C to appear, and A.t < B.t < C.t, where A.t, B.t, and C.t represent the occurrence times of the basic events of types A, B, and C respectively. Therefore, the requirements for logical relationships and event types in the complex pattern P can be represented by SEQ{A, B, C}. At the same time, for the three basic events that make up an instance that matches the complex pattern P, it should also satisfy C.t - A.t ; The AND logical operator requires that all required basic event types appear within a time interval of , but there is no requirement for their order. AND{A, B, C} represents the requirement for abnormal vehicle speeds at positions A, B, and C within a certain time, but does not care about their order; The OR logical operator represents a set of specified alternative basic event types; OR{A, B, C} means that when a basic event of any type in {A, B, C} appears and its attributes meet the requirements in , then such a basic event is a matching instance of OR{A, B, C}; The NOT logical operator requires that a specific type of event does not exist at a certain position in the complex pattern; The!E is used to represent the NOT operator on the event type E. SEQ{A, B,!C, D} means that within the time of , basic events of types A, B, and D appear, but there is no C-type basic event that satisfies A.t < B.t < C.t < D.t; After extracting the characteristics such as the event types of the extraction pattern, the similarity is quickly calculated by counting the original event types commonly required between complex patterns; A pattern association graph is constructed by connecting each complex pattern with its similar patterns. The patterns in the pattern association graph are used as graph nodes, and the patterns that share common sub-patterns in the association graph will form a tightly connected cluster;

[0009] Step 2: Based on the pattern association graph constructed in Step 1, perform structural entropy minimization analysis. By calculating the structural entropy change of merging a pair of nodes into a cluster or merging a pair of clusters into a single cluster, select the optimal merging pair to obtain the optimal partitioning scheme that makes the internal structure of the cluster most compact and the inter-cluster structure most sparse. The structural entropy minimization process uses the merging operation to automatically cluster similar complex patterns into clusters. Further, it compares the required basic event sequences and constraints of all patterns within a cluster to identify common pattern fragments, i.e., sub-patterns, that can share candidate instances among multiple patterns within a cluster, and marks the cluster with these sub-patterns. This clustering process is used to construct an instance aggregation plan among complex patterns, i.e., to determine which common sub-pattern instances are aggregated among which complex patterns. For each cluster aggregated by complex patterns, the instances of the marked sub-patterns are aggregated among the complex patterns contained in the cluster.

[0010] Step 3: For the remaining candidate instances that cannot be covered by the intra-cluster shared structure in Step 2, a lightweight differential compensation mechanism is adopted to identify cross-cluster aggregation opportunities through cluster-level search; cluster pairs with inter-cluster edges are added to the cluster-level candidate list, where inter-cluster edges refer to edges where the two endpoint nodes belong to two different clusters respectively; the similarity between cluster pairs is calculated, and the cluster pairs with the highest similarity are checked first to identify sub-patterns that can generate more aggregateable instances within the constrained construction time; when checking each pair of clusters, if there is a common sub-pattern among the remaining candidate instances of their complex patterns, the differential compensation mechanism will create a new parent point for the pair of clusters and mark it with the identified common sub-pattern to update the aggregation plan;

[0011] Step 4: Based on the intra-cluster instance aggregation plan constructed in Step 2 and the cross-cluster compensation aggregation plan generated in Step 3, implement instance aggregation during real-time detection; first, input the moving event stream, inserting the basic events of each type into a shareable buffer so that the detection process of each pattern can call the basic events in the buffer to create candidate instances; then, in the multi-level aggregation plan, candidate instance creation of a node will not be performed until matching instances of all its parent nodes appear, thereby avoiding the repeated creation and storage of instances of common sub-patterns at all levels; where:

[0012] Step 2 specifically includes:

[0013] 2-1, Structural Entropy Minimization and Optimal Cluster Partitioning: Graph nodes are encoded by dividing them into multiple layers of clusters. The encoding of each graph node consists of the encoding of its own cluster and its encoding within the cluster. The multi-layered cluster hierarchy is represented by a partitioning tree, where T represents the graph. The partitioning tree of the graph node clustering, with the root vertex of T. Represents a set of nodes Each leaf vertex in T Represents any node in V Each intermediate tree vertex Let G represent a cluster of graph nodes, and contain the graph nodes belonging to that cluster; use This represents the structural entropy of graph G.

[0014]

[0015] Indicates a cluster of connections A node and a cluster The number of edges of the other node. The volume of graph G is the sum of the degrees of the nodes in G. Indicates belonging to a cluster The volume of a node is the sum of its degree. Cluster The parent node in the partition tree; in the construction of the partition tree, the "combination" operation combines vertices. and The nodes represent clusters, when the vertices are clustered together. and When representing clusters, they aggregate into higher-level clusters; when the "combination" operation updates the partitioning tree of the pattern association graph, it correspondingly changes the magnitude of the entropy of the corresponding structure; let T and Representing the vertices respectively and The changes in structural entropy caused by the "combination" operation are as follows:

[0016]

[0017] It calculates structural entropy changes based on local structural information and uses it as a lightweight indicator of the tightness of connections between nodes in a graph; if This indicates that and "Combining" them into higher-level clusters can reduce entropy, that is... and The graph nodes contained therein are structurally more tightly connected; furthermore, unlike the "combine" operation, the "merge" operation directly merges... and The nodes in the T array are merged together without creating a parent node for them, allowing T and They represent points respectively and The structural entropy change caused by the "merge" operation is as follows:

[0018]

[0019] It also calculates structural entropy changes based on the local graph structure and serves as a lightweight indicator of the tightness of connections between nodes in the graph; if This indicates that and Merging graph nodes into a cluster can reduce entropy; using a greedy rule, select the cluster pair with the largest entropy reduction for merging, iterate until merging can no longer reduce the total structural entropy, and obtain the optimal partitioning scheme;

[0020] 2-2, Intra-cluster Common Sub-pattern Recognition: Each node in the pattern association graph represents a complex pattern. For each cluster obtained during the structural entropy minimization process, the event types, time requirements, and attribute requirements contained in all complex patterns within the cluster are extracted. Different common sub-pattern recognition rules apply to the logical requirements defined by the AND, SEQ, OR, and NOT pattern operators: For a pair of patterns both containing AND, the recognized common sub-pattern is the largest common subset of the two event type sets; for patterns both containing SEQ, the recognized common sub-pattern is the largest common sequence of the basic event types; for two patterns containing AND and SEQ respectively, the largest common subset of the basic event types is extracted; for event types associated with the OR pattern operator, the sub-pattern is identified through the union of the event types in OR; each event type associated with the NOT pattern operator is treated as a separate type for sub-pattern recognition.

[0021] Step 3 specifically includes:

[0022] 3-1, Residual Instance Identification: Candidate instances of complex patterns not covered by the shared structure of instances within the cluster in step 2 are defined as "residual" instances, and the focus is on the differential portion not covered by the shared sub-patterns within the cluster; for complex patterns... ,make This represents the set of basic event types required for it. The set of basic event types required to represent its parent point, then The difference part is composed of the complement. , and the requirements of this type of event for logic, time, and attributes;

[0023] 3-2, Cluster-level candidate list construction and similarity calculation: For any pair of clusters in the aggregation plan, add cluster pairs that have inter-cluster edges (i.e., the two endpoint nodes belong to two different clusters) to the cluster-level candidate list; for each cluster pair in the cluster-level candidate list... ,in and They respectively represent belonging to and The complex pattern is used to determine the similarity between two clusters by the number of inter-cluster edges between them. ;

[0024] 3-3, Prioritizing the Inspection of High-Similarity Cluster Pairs and Handling Common Sub-patterns: The differential compensation mechanism prioritizes the inspection of cluster pairs with higher similarity to identify more sub-patterns of common candidate instances within the constrained construction time. When inspecting each pair of sub-clusters with inter-cluster edges, for sub-clusters with common sub-patterns between their differential parts, the differential compensation mechanism creates a new parent point for the sub-cluster and marks it with the identified common sub-pattern. Essentially, given sufficient planned construction time, it is possible to inspect all cluster pairs with inter-cluster edges, while still maintaining a lightweight approach.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] Based on the multi-level clustering structure among complex patterns, a content-aware structural entropy minimization algorithm and differential compensation mechanism are proposed to solve the problem that the existing aggregation plan for common candidate instances among complex patterns takes too long to build and cannot support online detection due to the exponential solution space. This reduces the construction time of the aggregation plan and supports operation on emerging server-insensitive platforms. Attached Figure Description

[0027] Figure 1 The system architecture diagram for implementing this invention;

[0028] Figure 2 This is a flowchart of the present invention;

[0029] Figure 3 The graph shows the average processing time per data item under different workload scales.

[0030] Figure 4 The graph shows the peak memory overhead during the detection process under different workload scales. Detailed Implementation

[0031] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0032] This invention accelerates candidate instance aggregation in complex pattern detection by employing content-aware structural entropy minimization and differential compensation mechanisms, thereby supporting online detection while reducing instance size and improving detection efficiency. In early complex event processing systems, each instance of a complex pattern was created and stored independently. However, as pattern length rapidly increases, instance size also grows dramatically, leading to severe waste of computational and storage resources. To enable complex event processing systems to scale to an increasing number of complex patterns, this can be addressed by aggregating common candidate instances from multiple complex patterns. To find the optimal aggregation plan for common instances among complex patterns, existing methods identify aggregable instances by using a subset of the data items required for a given complex pattern, while utilizing combinatorial optimization or heuristics to construct the aggregation plan. However, the solution space of this aggregation plan increases rapidly with the length and number of patterns. With the development of mobile applications, pattern length (over 20) and number (up to thousands) are constantly increasing, especially when complex patterns are generated from generative large language models, making the construction of the aggregation plan a new scalability bottleneck. The high time cost necessitates offline construction of the aggregation plan, and because the data stream needs to be processed in real time, it also introduces a considerable downtime for complex event processing systems. At the same time, due to the high construction costs, the quality of offline built aggregation plans still faces trade-offs in terms of construction time, which can have a long-term impact on the performance of complex event processing systems.

[0033] In this invention, the following techniques are employed to overcome the aforementioned difficulties: First, complex patterns (as graph nodes) are connected to similar patterns to construct an association graph between complex patterns, and structural entropy minimization analysis is performed on this graph. The optimal clustering structure between complex patterns is selected by calculating the structural entropy changes resulting from clustering operations between complex patterns (graph nodes), thus obtaining the optimal partitioning scheme that maximizes the compactness of the internal cluster structure and the spatiality of the inter-cluster structure. The required event sequences and constraints for all patterns within a cluster are compared to identify common pattern fragments that can be shared among multiple patterns within the cluster. Second, for residual subsequent instances that cannot be covered by the shared structure within the cluster, a lightweight differential compensation mechanism is used to identify cross-cluster aggregation opportunities through cluster-level search. Based on this, a tree-like multi-layer aggregation plan is generated, enabling instance aggregation during real-time detection.

[0034] This invention proposes an online method for detecting complex patterns in mobile data streams, comprising:

[0035] Step 1: Obtain the set of complex patterns to be detected, and classify each complex pattern... Defined as a quadruple Each complex pattern matching instance is composed of data items (i.e., underlying events) from the mobile data stream. It matches. The set of basic event types required in the instance, is the maximum allowable time difference between any pair of basic events in an instance that matches . It represents the requirements for the attributes of basic events; refers to a logical operator that specifies the required logical relationship between basic events. Four common pattern logical operators include SEQ, AND, OR, and NOT. The SEQ operator requires that all required basic event types appear in an instance of a complex pattern, and the basic events of each type appear in the required chronological order. For example, for a complex pattern P that requires basic events of types A, B, and C, and A.t < B.t < C.t (A.t, B.t, and C.t represent the occurrence times of basic events of types A, B, and C respectively), the requirements for the logical relationship and event types in P can be represented by SEQ{A,B,C}. At the same time, for the three basic events that make up an instance that matches P, it should also satisfy C.t - A.t . The AND operator requires that all required basic event types appear within a time interval of , but there is no requirement for their order. For example, AND{A,B,C} can represent the requirement for the vehicle speed to be abnormal at positions A, B, and C within a certain time, but does not care about their order. The OR operator represents a set of alternative basic event types; taking the complex pattern of OR{A,B,C} as an example, if a basic event of any type in {A,B,C} appears and its attributes satisfy the requirements in, then such a basic event is a matching instance of OR{A,B,C}. The NOT operator requires that a specific type of event does not exist at a certain position in a complex pattern; using!E to represent the NOT operator on event type E, for example, SEQ{A,B,!C,D} can represent that within time, basic events of types A, B, and D appear, but there is no C type basic event that satisfies A.t < B.t < C.t < D.t. After extracting the characteristics such as the event types of the extraction patterns, the similarity between them is quickly calculated by counting the original event types commonly required between complex patterns. By connecting each complex pattern with its similar patterns to construct a pattern association graph, the patterns in the pattern association graph are used as graph nodes, and the patterns that share common sub-patterns in the association graph will form tightly connected clusters;

[0037] ​​​Step 2: Based on the pattern association graph constructed in Step 1, perform structural entropy minimization analysis. By calculating the structural entropy change of merging operations (i.e., merging a pair of nodes into a cluster, or merging a pair of smaller clusters into a larger cluster), select the optimal merging pair to obtain the optimal partitioning scheme that makes the internal structure of the clusters most compact and the inter-cluster structure most sparse. The structural entropy minimization process uses merging operations to automatically cluster similar complex patterns into clusters. It further compares the required basic event sequences and constraints of all patterns within a cluster, identifies common pattern fragments (i.e., sub-patterns) that can share candidate instances among multiple patterns within a cluster, and uses these sub-patterns to label the cluster.

[0038] Step 3: For the remaining candidate instances that cannot be covered by the intra-cluster shared structure in Step 2, a lightweight differential compensation mechanism is adopted to identify cross-cluster aggregation opportunities through cluster-level search. Cluster pairs with inter-cluster edges (the two endpoint nodes belong to two different clusters) are added to the cluster-level candidate list, and their similarity is calculated. By prioritizing the examination of cluster pairs with the highest similarity, sub-patterns that can generate more aggregateable instances can be identified within the constrained construction time. When examining each pair of clusters, if there is a common sub-pattern among the remaining candidate instances of their complex patterns, the differential compensation mechanism will create a new parent point for them and mark it with the identified common sub-pattern.

[0039] Step 4: Based on the intra-cluster instance aggregation plan constructed in Step 2 and the cross-cluster compensation aggregation plan generated in Step 3, a multi-level aggregation plan is generated to implement instance aggregation during real-time detection. First, the input movement event stream is processed, and the basic events of each type are inserted into a specific shareable buffer so that the detection process for each pattern can invoke the basic events in the buffer to create candidate instances. Then, in the multi-level aggregation plan, candidate instance creation for a node will not be performed until matching instances of all its parent nodes are found, thus avoiding the repeated creation and storage of instances of common sub-patterns at each level. Wherein:

[0040] Step 2 specifically includes:

[0041] (1) Minimizing structural entropy and optimal cluster partitioning: Graph nodes are encoded by dividing them into multiple clusters. The encoding of each graph node consists of the encoding of its own cluster and its encoding within the cluster. The multi-layered cluster hierarchy is represented by a partitioning tree, where T represents the graph. The partitioning tree of the graph node clustering, with the root vertex of T. Represents a set of nodes Each leaf vertex in T Represents any node in V Each intermediate tree vertex This represents a cluster of graph nodes in G, and contains the graph nodes belonging to that cluster. This represents the structural entropy of graph G.

[0042]

[0043] Indicates a cluster of connections A node and a cluster The number of edges of the other node. It is the volume of graph G (the sum of the degrees of the nodes in G). Indicates belonging to a cluster The volume of a node (the sum of the corresponding node degrees). Cluster The parent node in the partition tree. In partition tree construction, the "combine" operation can combine vertices. and The nodes (or clusters) represented are grouped into (higher-level) clusters. When the "combination" operation updates the partitioning tree of the related graph, it will correspondingly change the magnitude of the corresponding structural entropy. Let T and Representing the vertices respectively and The changes in structural entropy caused by the "combination" operation are as follows:

[0044]

[0045] It can calculate structural entropy changes based on local structural information and serve as a lightweight indicator of the tightness of connections between nodes in a graph. If This indicates that and "Combining" into (higher-level) clusters can reduce entropy, i.e. and The graph nodes contained therein are structurally more tightly connected. Furthermore, unlike the "combine" operation, the "merge" operation directly... and The nodes in the T array are merged together without creating a parent node for them, allowing T and They represent points respectively and The structural entropy change caused by the "merge" operation is as follows:

[0046]

[0047] It can also calculate structural entropy changes based on local graph structure and serve as a lightweight indicator of the tightness of connections between nodes in the graph. If This indicates that and Merging graph nodes into a cluster can reduce entropy. By using a greedy rule, the cluster pairs with the largest reduction in entropy are selected for merging, and this process is iterated until merging can no longer reduce the overall structural entropy, thus obtaining the optimal partitioning scheme.

[0048] (2) Common sub-pattern recognition within the cluster: Each node on the association graph represents a complex pattern. For each cluster obtained in the process of minimizing structural entropy, the event sequence (including event type) and constraints (time difference Time, attribute requirement R) of all complex patterns within the cluster are extracted. Different rules apply to the common subpattern identification for the logical requirements defined by the AND, SEQ, OR, and NOT operators: For a pair of patterns both containing AND, the identified common subpattern is the largest common subset of the two event type sets. For example, AND{A, B, C} and AND{A, B, D} identify the common subpattern as AND{A, B}. For patterns both containing SEQ, the common subpattern is identified as the largest common sequence of the basic event types. For example, SEQ{A, B, C, D} and SEQ{B, C, E} identify SEQ(B, C) as their common subpattern. For two patterns containing both AND and SEQ, the largest common subset of the basic event types is extracted. For example, instances matching SEQ{B, C, E} also match AND{B, C, E}. For event types associated with the OR operator, the subpattern is identified by the union of the event types in OR. For example, given SEQ{B, C, E} and AND{C, D, OR{E, F}}, the identified subpattern is AND{C, OR{E, F}}. Each event type associated with the NOT operator is treated as a separate type, such as for SEQ{A, !B, C, D} and SEQ{A, !B, C, E}, the common subpattern is identified as SEQ{A, !B, C}.

[0049] Step 3 specifically includes:

[0050] (1) Identification of aggregatable "residual" instances: Candidate instances of complex patterns not covered by the shared structure of instances within the cluster in step 2 are defined as "residual" instances, and the focus is on the differential parts not covered by the shared sub-patterns within the cluster. For complex patterns... ,make This represents the set of basic event types required for it. The set of basic event types required to represent its parent point, then The difference part is composed of the complement. This includes the logical, temporal, and attribute requirements of this type of event. For example, given the patterns SEQ {A,B,C,E} and SEQ {C,D,E,F}, their difference segments are SEQ {A,B} and SEQ {D,F}, respectively, and their common segment is SEQ {C,E}.

[0051] (2) Cluster-level candidate list construction and similarity calculation: For any pair of clusters in the aggregation plan, add the cluster pairs that have inter-cluster edges (the two endpoint nodes belong to two different clusters) to the cluster-level candidate list. For each cluster pair in the cluster-level candidate list... ,in and They respectively represent belonging to and Complex patterns, the number of inter-cluster edges between them is used as their similarity. .

[0052] (3) Prioritizing High-Similarity Cluster Pairs and Handling Common Sub-patterns: The differential compensation mechanism prioritizes checking cluster pairs with higher similarity to identify more sub-patterns of common candidate instances within the constrained construction time. When checking each pair of sub-clusters with inter-cluster edges, if a common sub-pattern exists between their differential parts, the differential compensation mechanism creates a new parent point for them and labels it with the identified common sub-pattern. Essentially, given sufficient planned construction time, it is possible to check all cluster pairs with inter-cluster edges, while still maintaining a lightweight approach.

[0053] In the specific implementation process, such as Figure 1 The system framework diagram shown (the first box from the left represents the complex mode of the input system, the second box from the left is a schematic diagram of building the aggregation plan in step B, the third box from the left is a schematic diagram of the differential compensation mechanism in step C, the fourth box from the left is a schematic diagram of real-time detection in step D, and the rightmost box represents the moving data flow of the input system) and Figure 2 The implementation method will be explained in detail below, following the flowchart shown:

[0054] This invention is achieved through the following technical solution: An online detection method for complex patterns in mobile data streams, comprising the following steps:

[0055] Step A: Obtain the set of complex patterns to be detected, extract features such as event types from the patterns, and quickly calculate their similarity by counting the common basic event types required between complex patterns. Construct an inter-pattern association graph by connecting each complex pattern with its similar patterns; patterns with common sub-patterns in the association graph form tightly connected clusters.

[0056] Step B: Perform structural entropy minimization analysis based on the constructed pattern association graph. Cluster the patterns by calculating the structural entropy changes brought about by the clustering operation to obtain the optimal clustering scheme. Further, compare the event sequences and constraints of all patterns in the cluster to identify common pattern fragments that can be shared among multiple patterns within the cluster.

[0057] Step C: For residual candidate instances that cannot be covered by intra-cluster shared structures, a lightweight differential compensation mechanism is employed to identify cross-cluster aggregation opportunities through cluster-level search. When examining each pair of cross-cluster complex patterns, if there is a common sub-pattern between their differential parts, the differential compensation mechanism creates a new parent point for them and marks it with the identified common sub-pattern;

[0058] Step D: Based on the intra-cluster instance aggregation plan and the cross-cluster compensation aggregation plan, a multi-level aggregation plan is generated to achieve instance aggregation during real-time detection. First, the input movement event stream is fed in, and the base events for each type are inserted into a specific shareable buffer so that the detection process for each pattern can invoke the base events in the buffer to create candidate instances. Then, in the multi-level aggregation plan, candidate instance creation for a node is not performed until matching instances for all its parent nodes are found, thus avoiding the redundant creation and storage of instances for common sub-patterns at each level.

[0059] Step B includes the following:

[0060] Step Ba: Graph nodes are encoded by dividing them into multiple clusters. The encoding of each graph node consists of the encoding of its own cluster and its encoding within the cluster. The multi-cluster hierarchy is represented by a partitioning tree, where T represents the graph. The partitioning tree of the graph node clustering, with the root vertex of T. Represents a set of nodes Each leaf vertex in T Represents any node in V Each intermediate tree vertex This represents a cluster of graph nodes in G, and contains the graph nodes belonging to that cluster. This represents the structural entropy of graph G.

[0061]

[0062] Indicates a cluster of connections A node and a cluster The number of edges of the other node. It is the volume of graph G (the sum of the degrees of the nodes in G). Indicates belonging to a cluster The volume of a node (the sum of the corresponding node degrees). Cluster The parent node in the partition tree. In partition tree construction, the "combine" operation can combine vertices. and The nodes (or clusters) represented are grouped into (higher-level) clusters. When the "combination" operation updates the partitioning tree of the related graph, it will correspondingly change the magnitude of the corresponding structural entropy. Let T and Representing the vertices respectively and The changes in structural entropy caused by the "combination" operation are as follows:

[0063]

[0064] It can calculate structural entropy changes based on local structural information and serve as a lightweight indicator of the tightness of connections between nodes in a graph. If This indicates that and "Combining" into (higher-level) clusters can reduce entropy, i.e. and The graph nodes contained therein are structurally more tightly connected. Furthermore, unlike the "combine" operation, the "merge" operation directly... and The nodes in the T array are merged together without creating a parent node for them, allowing T and They represent points respectively and The structural entropy change caused by the "merge" operation is as follows:

[0065]

[0066] It can also calculate structural entropy changes based on local graph structure and serve as a lightweight indicator of the tightness of connections between nodes in the graph. If This indicates that and Merging graph nodes into a cluster can reduce entropy. By using a greedy rule, the cluster pairs with the largest reduction in entropy are selected for merging, and this process is iterated until merging can no longer reduce the overall structural entropy, thus obtaining the optimal partitioning scheme.

[0067] Step Bb: For each cluster obtained by minimizing structural entropy, extract the event sequence (including event type) and constraints (time difference Time, attribute requirement R) of all complex patterns within the cluster. Different common sub-pattern recognition rules exist for the logical requirements defined by operators such as AND, SEQ, OR, and NOT. For a pair of patterns both containing AND, the identified common subpattern is the greatest common subset of the two event type sets. For example, AND{A, B, C} and AND{A, B, D} identify the common subpattern as AND{A, B}. For patterns both containing SEQ, the identified common subpattern is the greatest common sequence of the basic event types. For example, SEQ{A, B, C, D} and SEQ{B, C, E} identify SEQ(B, C) as their common subpattern. For two patterns containing AND and SEQ respectively, the greatest common subset of the basic event types is extracted. For example, instances matching SEQ{B, C, E} also match AND{B, C, E}. For event types associated with the OR operator, the subpattern is identified by the union of the event types in OR. For example, given SEQ{B, C, E} and AND{C, D, OR{E, F}}, the identified subpattern is AND{C, OR{E, F}}. Each event type associated with the NOT operator is treated as a separate type. For example, for SEQ{A, SEQ, OR{B, C, D}, OR{E, F}}, OR{B, OR{E, F}}. The common subpatterns of SEQ{A, !B, C, E} and SEQ{B, C, D} are identified as SEQ{A, !B, C}.

[0068] Step C includes the following:

[0069] Step Ca: For "residual" candidate patterns not covered by the intra-cluster aggregation plan, focus on the differential portions not covered by shared sub-patterns. For any pair of clusters in the aggregation plan, add the cluster pairs that have inter-cluster edges (the two endpoint nodes belong to two different clusters) to the cluster-level candidate list, and use the number of inter-cluster edges between them as their similarity.

[0070] Step Cb: The differential compensation mechanism prioritizes checking cluster pairs with higher similarity to identify more common sub-patterns within the constrained construction time. When checking each pair of sub-cluster pairs with inter-cluster edges, if there is a common sub-pattern between their differential parts, the differential compensation mechanism creates a new parent point for them and labels it with the identified common sub-pattern.

[0071] Figure 3This document compares the average processing time per data item of the present invention with existing online complex pattern detection algorithms in specific examples, under different workload scales. Specifically, (a) shows the processing time of the proposed method and the comparative method on a Vehicle mobile data stream for 100 and 500 complex patterns; (b) shows the processing time of the proposed method and the comparative method on a Vehicle mobile data stream for 1000 and 2000 complex patterns; (c) shows the processing time of the proposed method and the comparative method on a Sensor mobile data stream for 100 and 500 complex patterns; and (d) shows the processing time of the proposed method and the comparative method on a Sensor mobile data stream for 800 and 1000 complex patterns.

[0072] Figure 3 In this paper, PatternInsight represents the method proposed in this invention. PatternInsight (CASEM-2) represents an aggregation plan that uses content-aware structural entropy minimization to construct an aggregation plan with two levels of instance aggregation. PatternInsight (CAEM) represents an aggregation plan that uses content-aware structural entropy minimization until its stopping condition is met. PatternInsight (CAEM+VCE) represents an aggregation plan constructed using content-aware structural entropy minimization together with a differential compensation mechanism. SPASS represents the method proposed in the paper Scalable pattern sharing on event streams, and RW represents the method proposed in the paper Real-time multi-pattern detection over event streams. Vehicle and Sensor are two mobile data streams from the CityPulse platform. As can be seen from the figure, the method proposed in this invention has a short average processing time per unit data item and maintains low latency under different workload scales.

[0073] Figure 4This paper compares the peak memory overhead of the proposed method with existing online complex pattern detection algorithms in specific examples, under different workload scales. Among them, (a) shows the memory overhead of the proposed method and the comparison method when processing 100 and 500 complex patterns on the Vehicle data stream; (b) shows the memory overhead of the proposed method and the comparison method when processing 1000 and 2000 complex patterns on the Vehicle data stream; (c) shows the memory overhead of the proposed method and the comparison method when processing 100 and 500 complex patterns on the Sensor data stream; and (d) shows the memory overhead of the proposed method and the comparison method when processing 800 and 1000 complex patterns on the Sensor data stream.

[0074] Figure 4 In this paper, PatternInsight represents the method proposed in this invention. PatternInsight (CASEM-2) represents an aggregation plan with two-level instance aggregation constructed using content-aware structural entropy minimization. PatternInsight (CAEM) represents an aggregation plan constructed using content-aware structural entropy minimization until its stopping condition is met. PatternInsight (CAEM+VCE) represents an aggregation plan constructed using content-aware structural entropy minimization together with a differential compensation mechanism. SPASS represents the method proposed in the paper Scalable pattern sharing on event streams, and RW represents the method proposed in the paper Real-time multi-pattern detection over event streams. Vehicle and Sensor are two mobile data streams from the CityPulse platform. As can be seen from the figure, the peak memory overhead during the detection process of the method proposed in this invention is relatively small under different workload scales.

[0075] Those skilled in the art will understand that, besides implementing the system, apparatus, and its modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and its modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; modules for implementing various functions can also be considered as both software programs implementing the method and structures within the hardware component. Specific embodiments of this invention have been described above. It should be understood that this invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the substantive content of this invention. In the absence of conflict, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other.

Claims

1. A method for online detection of complex patterns in mobile data streams, characterized in that, Includes the following steps: Step 1: Obtain the set of complex patterns to be detected, and classify each complex pattern... Defined as a quadruple Each complex pattern matching instance is composed of data items, i.e., basic events, from the mobile data stream; where It matches. The set of basic event types required in the instance It is a match The maximum permissible time difference between any pair of basic events in an instance. This indicates the requirements for basic event properties; These refer to logical operators, which specify the logical relationships required between basic events; The logical operators in it include SEQ, AND, OR, and NOT; among which the SEQ logical operator requires that all the required basic event types appear in an instance of the complex pattern, and the basic events of each type appear in the required chronological order; the complex pattern P requires the basic events of types A, B, and C to appear, and A.t < B.t < C.t, where A.t, B.t, and C.t respectively represent the occurrence times of the basic events of types A, B, and C. Therefore, the requirements for the logical relationship and event types in the complex pattern P can be represented by SEQ{A, B, C}. At the same time, for the three basic events that make up an instance that matches the complex pattern P, it should also satisfy C.t - A.t ; The AND logical operator requires that all the required basic event types appear within a time interval of but there is no requirement for their order. AND{A, B, C} represents the requirement for the vehicle speeds that are abnormal at positions A, B, and C within a certain time, but does not care about their order; The OR logical operator represents a set of specified alternative basic event types; OR{A, B, C} means that when a basic event of any type in {A, B, C} appears and its attributes meet the requirements in , then such a basic event is a matching instance of OR{A, B, C}; The NOT logical operator requires that a specific type of event does not exist at a certain position in the complex pattern; Using!E to represent the NOT operator on the event type E, SEQ{A, B,!C, D} means that within the time of , the basic events of types A, B, and D appear, but there is no C-type basic event that satisfies A.t < B.t < C.t < D.t; After extracting the characteristics such as the event types of the extraction pattern, the similarity is quickly calculated by counting the original event types commonly required between the complex patterns; By connecting each complex pattern with its similar pattern to construct a pattern association graph, the patterns in the pattern association graph are used as graph nodes, and the patterns that share common sub-patterns in the association graph will form a tightly connected cluster; Step 2: Based on the pattern association graph constructed in Step 1, perform structural entropy minimization analysis. By calculating the structural entropy change of merging a pair of nodes into a cluster or merging a pair of clusters into a single cluster, select the optimal merging pair to obtain the optimal partitioning scheme that makes the internal structure of the clusters most compact and the inter-cluster structure most sparse. The structural entropy minimization process uses the merging operation to automatically cluster similar complex patterns into clusters. Further, it compares the required basic event sequences and constraints of all patterns within a cluster to identify common pattern fragments, i.e., sub-patterns, that can share candidate instances among multiple patterns within a cluster, and marks the cluster with these sub-patterns. This clustering process is used to construct an instance aggregation plan among complex patterns, i.e., to determine which common sub-pattern instances are aggregated among which complex patterns. For each cluster aggregated by complex patterns, instances of the subpatterns of that cluster are aggregated among the complex patterns contained in that cluster; Step 3: For the remaining candidate instances that cannot be covered by the intra-cluster shared structure in Step 2, a lightweight differential compensation mechanism is adopted to identify cross-cluster aggregation opportunities through cluster-level search; cluster pairs with inter-cluster edges are added to the cluster-level candidate list, where inter-cluster edges refer to edges where the two endpoint nodes belong to two different clusters respectively; the similarity between cluster pairs is calculated, and the cluster pairs with the highest similarity are checked first to identify sub-patterns that can generate more aggregateable instances within the constrained construction time; when checking each pair of clusters, if there is a common sub-pattern among the remaining candidate instances of their complex patterns, the differential compensation mechanism will create a new parent point for the pair of clusters and mark it with the identified common sub-pattern to update the aggregation plan; Step 4: Based on the intra-cluster instance aggregation plan constructed in Step 2 and the cross-cluster compensation aggregation plan generated in Step 3, implement instance aggregation during real-time detection; first, input the moving event stream, inserting the basic events of each type into a shareable buffer so that the detection process of each pattern can call the basic events in the buffer to create candidate instances; then, in the multi-level aggregation plan, candidate instance creation of a node will not be performed until matching instances of all its parent nodes appear, thereby avoiding the repeated creation and storage of instances of common sub-patterns at all levels; where: Step 2 specifically includes: 2-1, Structural Entropy Minimization and Optimal Cluster Partitioning: Graph nodes are encoded by dividing them into multiple layers of clusters. The encoding of each graph node consists of the encoding of its own cluster and its encoding within the cluster. The multi-layered cluster hierarchy is represented by a partitioning tree, where T represents the graph. The partitioning tree of the graph node clustering, with the root vertex of tree T. Represents a set of nodes Each leaf vertex in T Represents any node in V Each intermediate tree vertex Let G represent a cluster of graph nodes, and contain the graph nodes belonging to that cluster; use This represents the structural entropy of graph G. ； Indicates a cluster of connections A node and a cluster The number of edges of the other node. The volume of graph G is the sum of the degrees of the nodes in G. Indicates belonging to a cluster The volume of a node is the sum of its degree. Cluster The parent node in the partition tree; in the construction of the partition tree, the "combine" operation combines vertices. and The nodes represent clusters, when the vertices are clustered together. and When representing clusters, they aggregate into higher-level clusters; when the "combination" operation updates the partitioning tree of the pattern association graph, it will correspondingly change the magnitude of the entropy of the corresponding structure; let T and Representing the vertices respectively and The changes in structural entropy caused by the "combination" operation are as follows: ； It calculates structural entropy changes based on local structural information and uses it as a lightweight indicator of the tightness of connections between nodes in the graph; if This indicates that and "Combining" them into higher-level clusters can reduce entropy, that is... and The graph nodes contained therein are more structurally connected; furthermore, unlike the "combine" operation, the "merge" operation directly merges them... and The nodes in the T array are merged together without creating a parent node for them, allowing T and They represent points respectively and The structural entropy change caused by the "merge" operation is as follows: (The original text appears to be incomplete and contains several errors. A more accurate translation would require the full context.) ； It also calculates structural entropy changes based on the local graph structure and serves as a lightweight indicator of the tightness of connections between nodes in the graph; if This indicates that and Merging graph nodes into a cluster can reduce entropy; using a greedy rule, select the cluster pair with the largest entropy reduction for merging, iterate until merging can no longer reduce the total structural entropy, and obtain the optimal partitioning scheme; 2-2, Intra-cluster Common Sub-pattern Recognition: Each node in the pattern association graph represents a complex pattern. For each cluster obtained during the structural entropy minimization process, the event types, time requirements, and attribute requirements contained in all complex patterns within the cluster are extracted. Different common sub-pattern recognition rules apply to the logical requirements defined by the AND, SEQ, OR, and NOT pattern operators: For a pair of patterns both containing AND, the recognized common sub-pattern is the largest common subset of the two event type sets; for patterns both containing SEQ, the recognized common sub-pattern is the largest common sequence of the basic event types; for two patterns containing AND and SEQ respectively, the largest common subset of the basic event types is extracted; for event types associated with the OR pattern operator, the sub-pattern is identified through the union of the event types in OR; each event type associated with the NOT pattern operator is treated as a separate type for sub-pattern recognition. Step 3 specifically includes: 3-1, Residual Instance Identification: Candidate instances of complex patterns not covered by the shared structure of instances within the cluster in step 2 are defined as "residual" instances, and the focus is on the differential parts not covered by the shared sub-patterns within the cluster; for complex patterns... ,make This represents the set of basic event types required for it. The set of basic event types required to represent its parent point, then The difference part is composed of the complement. , and the requirements of this type of event for logic, time, and attributes; 3-2, Cluster-level candidate list construction and similarity calculation: For any pair of clusters in the aggregation plan, add cluster pairs that have inter-cluster edges (i.e., the two endpoint nodes belong to two different clusters) to the cluster-level candidate list; for each cluster pair in the cluster-level candidate list... ,in and They respectively represent belonging to and The complex pattern is defined as the number of inter-cluster edges between the two clusters, which is used as their similarity. ; 3-3, Prioritizing the Inspection of High-Similarity Cluster Pairs and Handling Common Sub-patterns: The differential compensation mechanism prioritizes the inspection of cluster pairs with higher similarity to identify more sub-patterns of common candidate instances within the constrained construction time. When inspecting each pair of sub-clusters with inter-cluster edges, for sub-clusters with common sub-patterns between their differential parts, the differential compensation mechanism creates a new parent point for the sub-cluster and marks it with the identified common sub-pattern. Essentially, given sufficient planned construction time, it is possible to inspect all cluster pairs with inter-cluster edges, while still maintaining a lightweight approach.

Images