A hotspot event identification sub-graph clustering and evolution tracking method

Abstract

The present application relates to big data processing technical field, specifically relates to a kind of hotspot event identification subgraph clustering and evolution tracking method.In distributed graph computing engine setting streaming data access buffer, network interaction data is converted into time-stamped dynamic edge sequence.Time decay function and the incremental clustering model of graph topological potential field combination are constructed, and the local update subgraph of the newly added edge and the degree variation adjacent node in current time sliding window is extracted.The topological potential energy change of each node in local update subgraph is calculated, and the node set exceeding threshold value is determined as active evolution area.The present application maps hotspot event aggregation evolution information to the corresponding physical shard of graph storage engine, triggers the memory index update of local subgraph structure, outputs node topological offset trajectory, reduces redundant operation and computing resource consumption, compresses response length, guarantees the running stability of system when sudden data is written, and helps to realize that hotspot event identification subgraph clustering data is more accurate and efficient write-in.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology, specifically to a method for identifying hot events through subgraph clustering and evolution tracking. Background Technology

[0002] In the field of big data processing, conventional technical solutions for identifying and tracking the evolution of online hotspot events typically employ distributed graph computing engines to transform network interaction data into a static graph structure. To capture the dynamic evolution of events, existing solutions generally use a time-slicing mechanism. This involves truncating the global relationship graph at fixed time intervals and performing full graph clustering calculations on all nodes and edges within the global graph at the end of each time slice. By extracting the global clustering results corresponding to each time slice and comparing the overall differences in regional structure between adjacent time slices, the evolutionary state of the hotspot event can be determined, and a tracking path can be generated.

[0003] Based on the specific implementation of the aforementioned conventional scheme, since a full re-clustering of the entire relational graph is required within each time slice, when the continuous accumulation of interaction data in the network leads to a large global graph size, the full computation must traverse all nodes and edges in the graph and recalculate the connectivity indicators between nodes. This global recalculation method generates a large number of redundant input-output operations and computational resource consumption related to the unchanged stable graph regions, resulting in a lag between the explosion of interaction data and the generation of evolutionary paths. This cannot meet the computational timeliness requirements for real-time tracking of the evolutionary paths of hot events in a dynamic interactive big data environment. Summary of the Invention

[0004] The purpose of this invention is to provide a method for identifying hotspot events through subgraph clustering and evolution tracking, which can solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A hotspot event identification subgraph clustering and evolution tracking method includes: setting up a streaming data access buffer in a distributed graph computing engine to convert received network interaction data into a dynamic edge sequence with timestamps; constructing an incremental clustering model combining a time decay function and a graph topological potential field; extracting newly added edges and adjacent nodes with changed degrees from the dynamic edge sequence within the current time sliding window to form a locally updated subgraph; calculating the change in topological potential energy of each node in the locally updated subgraph; determining the set of nodes whose change in topological potential energy exceeds a preset physical topological threshold as an active evolution region; mapping the set of nodes in the active evolution region to the corresponding physical shard of the graph storage engine; triggering the memory index update of the local subgraph structure of the corresponding physical shard; and outputting an event evolution path containing the node topological offset trajectory.

[0007] Preferably, converting the received network interaction data into a timestamped dynamic edge sequence includes: parsing the source node identifier, target node identifier, and interaction occurrence timestamp in the network interaction data; constructing an undirected edge skeleton in the streaming data access buffer using the source node identifier and target node identifier, and binding the interaction occurrence timestamp as an edge attribute of the undirected edge skeleton; and writing the undirected edge skeleton carrying the interaction occurrence timestamp into a distributed message queue in chronological order to form the timestamped dynamic edge sequence.

[0008] Preferably, within the current time window, extracting newly added edges and adjacent nodes with changed degrees from the dynamic edge sequence to form a locally updated subgraph includes: defining the current time window based on the current system time and according to a preset time span; filtering target edges from the dynamic edge sequence that fall within the current time window, identifying edges among the target edges that do not exist in the historical full graph as newly added edges; extracting the endpoints of the newly added edges, and obtaining all first-order neighbor nodes directly connected to the endpoints in the historical full graph; and combining the newly added edges, the endpoints, and the first-order neighbor nodes to generate the locally updated subgraph.

[0009] Preferably, the incremental clustering model combining the time decay function and the graph topological potential field includes: constructing the time decay function with the start and end times of the current time window as independent variables; obtaining the set of edges connecting any node in the local update subgraph within the current time window; calculating the product of the edge weight of each edge in the set and the time decay function at the timestamp corresponding to each edge; accumulating all products to generate the current topological potential value of any node in the graph topological potential field that integrates the time dimension.

[0010] Preferably, calculating the topological potential energy change of each node in the locally updated subgraph and determining the set of nodes whose topological potential energy change exceeds a preset physical topological threshold as an active evolution region includes: reading the historical topological potential energy value corresponding to the previous time window for each node in the locally updated subgraph from the graph storage engine; for each node, calculating the absolute value of the difference between the current topological potential energy value and the historical topological potential energy value as the topological potential energy change; traversing the topological potential energy change of all nodes in the locally updated subgraph, extracting nodes whose topological potential energy change is greater than the preset physical topological threshold, and marking the connected subgraph formed by the extracted nodes and the edges between the nodes as the active evolution region.

[0011] Preferably, mapping the node set of the active evolution region to the corresponding physical shard of the graph storage engine, and triggering the memory index update of the local subgraph structure of the corresponding physical shard, includes: obtaining the hash value set of all nodes in the active evolution region, calculating the geometric center hash value of the hash value set; locating the target physical shard to which the geometric center hash value belongs in the sharding routing table of the graph storage engine; sending an index update instruction to the target physical shard, writing the node attribute data and edge relationship data of the active evolution region into the memory data structure of the target physical shard, and reconstructing the local adjacency table index inside the target physical shard.

[0012] Preferably, after generating the locally updated subgraph, the method further includes: performing topology filtering on the locally updated subgraph based on the concept of communication network flow control; calculating the in-degree and out-degree values ​​of each endpoint in the locally updated subgraph; calculating the absolute value of the difference between the in-degree and out-degree values; when the absolute value of the difference exceeds a preset degree deviation threshold, determining the endpoint as an abnormal topology node; removing the abnormal topology node and the edges between the first-order neighbor nodes connected to the abnormal topology node from the locally updated subgraph to generate a filtered locally updated subgraph.

[0013] Preferably, when calculating the current topological potential energy value of any node in the graph topological potential field in the time dimension of fusion, the method further includes: real-time statistics of the total number of nodes and the total number of edges in the locally updated subgraph, calculating the ratio of the total number of edges to the total number of nodes as the current subgraph density coefficient; inputting the current subgraph density coefficient into a preset adjustment function, outputting a decay rate adjustment factor; multiplying the decay rate adjustment factor by the exponent base in the time decay function, updating the decay curve of the time decay function according to the product result, and recalculating the current topological potential energy value based on the updated decay curve.

[0014] Preferably, after marking the connected subgraph formed by the extracted nodes and the edges between the nodes as the active evolution region, the method further includes: extracting a central node from the active evolution region, wherein the central node is the node with the largest change in topological potential energy; querying the historical physical segment identifier sequence to which the central node belongs in a series of consecutive time windows; arranging the historical physical segment identifier sequence in chronological order, and combining it with the target physical segment identifier corresponding to the current time window to generate the node topological offset trajectory.

[0015] Preferably, before sending the index update instruction to the target physical shard, the method further includes: collecting the current memory occupancy rate and CPU utilization rate of the target physical shard; when the current memory occupancy rate exceeds a preset storage security threshold or the CPU utilization rate exceeds a preset computing security threshold, identifying adjacent physical shards in an idle state in the graph storage engine; splitting the node attribute data and the edge relationship data in the active evolution region into multiple data shards, and routing the multiple data shards to the target physical shard and the adjacent physical shards respectively for distributed index reconstruction.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0017] 1. To address the computational time lag issue caused by the global recalculation mentioned above, this invention constructs a time sliding window by setting up a streaming data access buffer. It extracts only newly added edges and adjacent nodes with changed degrees to form a locally updated subgraph, and combines the time decay function with the graph's topological potential field to calculate the change in the topological potential energy of the nodes. This processing method transforms the full clustering of the global graph into incremental clustering calculation for the locally updated subgraph, avoiding traversal of unchanged stable graph regions. This reduces redundant input / output operations and the consumption of central processing unit and memory resources during graph computation, and shortens the response time from data reception to output node topological offset trajectories.

[0018] 2. During the construction of local update subgraphs and the triggering of index updates, abnormal topology nodes are removed by statistically analyzing the difference between the in-degree and out-degree of nodes, thus eliminating the data interference of invalid nodes on subsequent clustering calculations. The current subgraph density coefficient is introduced to dynamically adjust the decay curve of the time decay function, so that the potential energy calculation can be adapted to subgraph structures with different densities. When mapping active evolution regions to physical shards, a node topology offset trajectory containing physical location migration records is generated by combining the historical physical shard identifier sequence. When the target physical shard load exceeds the safety threshold, the data is split and routed to adjacent idle physical shards for distributed index reconstruction, ensuring the operational stability of the graph storage engine during large-scale burst data writes. Attached Figure Description

[0019] Appendix Figure 1 This is a flowchart of the physical sharding mapping and distributed index update process of the present invention;

[0020] Figure 2 This is a flowchart illustrating the construction process of a dynamic relational subgraph for local updating of the subgraph in this invention.

[0021] Figure 3 This is a flowchart of the local update subgraph topology filtering process of the present invention;

[0022] Figure 4This is a flowchart of the calculation process for the time decay-topological potential incremental clustering model of the present invention;

[0023] Figure 5 This is a flowchart of the active evolution region determination and center node extraction process of the present invention;

[0024] Figure 6 This is a flowchart of the network interactive data streaming access and dynamic edge sequence generation process of the present invention. Detailed Implementation

[0025] The technical solution disclosed in this specific embodiment is applied to network data processing scenarios in a distributed graph computing environment, enabling real-time identification and full lifecycle evolution tracking of network hotspot events. The following content provides a complete and sufficient disclosure of the implementation details of the technical solution, enabling those skilled in the art to understand and reproduce the corresponding technical solution.

[0026] refer to Figure 1 In one embodiment, the distributed graph computing engine is built using a vertex-centric programming model. Internally, the engine sets up a streaming data access buffer with a circular buffer structure. The buffer's read and write pointers are updated atomically to accommodate data consistency requirements in multi-threaded concurrent write scenarios. The distributed graph computing engine receives real-time network interaction data from the platform. This data includes log data corresponding to various user interactions such as following, private messages, forwarding, commenting, liking, and mentioning. Each log entry contains fields such as source node identifier, target node identifier, interaction timestamp, interaction type, and interaction content hash value. The distributed graph computing engine transforms received network interaction data into a dynamic edge sequence with timestamps. Specifically, it performs format validation on the received network interaction data, removes invalid data with missing fields or illegal formats, extracts the source node identifier, target node identifier, and interaction timestamp from each valid data entry, constructs an edge structure using the source node identifier and target node identifier, and binds the interaction timestamp as an edge attribute to the edge structure. The bound edge structure is arranged according to the chronological order of the interaction occurrences, forming a dynamic edge sequence with timestamps. The dynamic edge sequence is continuously written to a distributed message queue for caching, providing time-consistent input data for subsequent graph computing processing.

[0027] refer to Figure 2The distributed graph computing engine constructs an incremental clustering model that combines a time decay function with the graph topological potential field. Within the current time sliding window, it extracts newly added edges and adjacent nodes with changed degrees from the dynamic edge sequence, forming a locally updated subgraph. Specifically, the current time sliding window is implemented using a sliding time window mechanism. Based on the current system time, the time range of the current time sliding window is defined according to a preset time span. The time span can be configured according to the real-time requirements of the business scenario. The sliding step size is proportional to the time span. Each time the window slides, the time range of the current time sliding window is redefined. The distributed graph computing engine pulls dynamic edge sequence data that falls within the current time sliding window from the distributed message queue and compares the pulled edge data with the edge set of the historical full graph. The historical full graph is a global relational graph structure that is persistently stored after the previous window is processed. During the comparison process, edges that do not exist in the edge set of the historical full graph are identified as newly added edges, and the two endpoints of the newly added edges are nodes with changed degrees. The distributed graph computing engine extracts the two endpoints of the newly added edge, queries the historical graph to obtain all first-order neighbor nodes directly connected to the two endpoints, and combines the newly added edge, the two endpoints and the corresponding first-order neighbor nodes to generate a locally updated subgraph. The locally updated subgraph only contains the local topology related to the newly added edge and does not involve stable regions in the global graph that have not undergone topological changes.

[0028] refer to Figure 5 The distributed graph computing engine calculates the change in topological potential energy of each node in the locally updated subgraph, and identifies the set of nodes whose topological potential energy change exceeds a preset physical topological threshold as active evolution regions. Specifically, in the incremental clustering model, the time decay function uses the start and end times of the current time window as independent variables and is constructed in an exponential decay form. The function expression is as follows:

[0029]

[0030] in, For timestamps The corresponding time decay coefficient, The preset decay rate constant, This is the end time of the current time window. The timestamps for the interactions corresponding to the connected edges, and satisfy , This represents the starting time point of the current time window. For any node in the locally updated subgraph, the incremental clustering model obtains the set of edges connecting that node within the current time window. It then calculates the product of the edge weight of each edge in the set and the function value of the time decay function at the corresponding timestamp of that edge. All product results are summed to generate the current topological potential value of that node in the graph topological potential field fused with the time dimension. The expression for calculating the current topological potential value is:

[0031]

[0032] in, For nodes The current topological potential energy value, For nodes The set of edges within the current time window. For nodes With nodes The edges connecting them, Connecting edges The edge weights, Connecting edges The corresponding timestamp of the interaction. For timestamps The corresponding time decay coefficient.

[0033] The distributed graph computing engine reads the historical topological potential energy values ​​of each node in the locally updated subgraph at the previous time window from the graph storage engine. These historical topological potential energy values ​​are bound to the node identifiers and stored in the node attribute data of the graph storage engine. For each node in the locally updated subgraph, the absolute value of the difference between the node's current topological potential energy value and its historical topological potential energy value is calculated as the change in topological potential energy for that node. The expression for calculating the change in topological potential energy is:

[0034]

[0035] in, For nodes The change in topological potential energy, For nodes The current topological potential value within the current time window. For nodes The historical topological potential energy value corresponding to the previous sliding window. The distributed graph computing engine traverses the topological potential energy change of all nodes in the locally updated subgraph, extracts nodes whose topological potential energy change is greater than a preset physical topological threshold. The preset physical topological threshold is determined based on the statistical quantile of the topological potential energy change of all nodes in the historical time period. The extracted nodes and the connected subgraph formed by the edges between the nodes are marked as active evolution regions. Each active evolution region corresponds to a potential hotspot event.

[0036] The distributed graph computing engine maps the node set of the active evolution zone to the corresponding physical shards of the graph storage engine, triggering memory index updates of the local subgraph structure of the corresponding physical shard, and outputting an event evolution path containing the node topology offset trajectory. Specifically, the graph storage engine adopts a consistent hashing sharding architecture, where each physical shard corresponds to a continuous hash value range on a hash ring. The sharding routing table stores the hash value range and access address information corresponding to each physical shard. The distributed graph computing engine obtains the hash value set of all nodes in the active evolution zone and calculates the geometric center hash value of the hash value set. The expression for calculating the geometric center hash value is:

[0037]

[0038] in, The geometric center hash value, The total number of nodes in the active evolutionary region. The first in the active evolutionary region The hash value of each node. This is for floor function operations. The distributed graph computing engine locates the target physical shard to which the geometric center hash value belongs in the sharding routing table, sends an index update command to the target physical shard, writes the node attribute data and edge relationship data of the active evolution zone into the memory data structure of the target physical shard, and rebuilds the local adjacency list index within the target physical shard. The index rebuilding process only applies to the nodes and edges corresponding to the active evolution zone and does not modify the indexes corresponding to the unchanged topology within the physical shard. The distributed graph computing engine extracts the topological position change information of nodes within the active evolution zone in different time windows, generates an event evolution path containing the topological offset trajectory of nodes, and completes the identification and evolution tracking of hot events.

[0039] In this embodiment, the topological features and potential energy calculation results of each node within the locally updated subgraph are recorded and tracked using the following table:

[0040] Table 1. Calculation Results of Topological Features and Potential Energy of Nodes in Locally Updated Subgraph

[0041]

[0042] This table is used to record the topological changes and potential energy calculation process of each node in the locally updated subgraph, providing a traceable quantitative basis for the determination of active evolution zones and ensuring the reproducibility of the node selection process.

[0043] In this embodiment, real-time access and standardized processing of network interaction data are achieved through a streaming data access buffer. A locally updated subgraph is constructed by extracting newly added edges and associated nodes based on a sliding time window. An incremental clustering model is constructed by combining a time decay function and the graph topological potential field. Potential energy calculation and region determination are performed only on the locally updated subgraph, transforming global full clustering into local incremental calculation, avoiding redundant traversal of stable graph regions. At the same time, local index updates of physical partitions are achieved through consistent hash mapping, shortening the response latency of data processing and completing the identification of hot events and the output of evolution paths.

[0044] In one embodiment, after receiving network interaction data, the distributed graph computing engine cleans and standardizes the data, parsing the source node identifier, target node identifier, and interaction timestamp. Specifically, the distributed graph computing engine performs format verification on the received network interaction data. The verification includes the completeness of three core fields: source node identifier, target node identifier, and interaction timestamp. It verifies whether the interaction timestamp conforms to the Unix millisecond-level standard numerical format and whether the source node identifier and target node identifier conform to the preset ID encoding rules, removing invalid data with missing fields or illegal formats. For valid data that passes the format verification, the distributed graph computing engine extracts the source node identifier, target node identifier, interaction timestamp, and interaction behavior type fields. In the streaming data access buffer, it constructs an undirected edge skeleton using the source node identifier and target node identifier. The undirected edge skeleton does not distinguish the direction of the source and target nodes. The unique edge identifier corresponding to the undirected edge skeleton is generated by concatenating the hash values ​​of the source node identifier and target node identifier in ascending order, avoiding the generation of duplicate edge structures for bidirectional interaction behaviors of the same pair of nodes. The distributed graph computing engine binds the interaction timestamp as an edge attribute to the undirected edge skeleton, and associates the interaction behavior type and interaction content hash value as additional edge attributes with the undirected edge skeleton. Edge attributes are stored in a key-value pair structure and uniquely bound to the edge identifier. The distributed graph computing engine writes the undirected edge skeletons carrying the interaction timestamps into a distributed message queue in chronological order. The distributed message queue adopts a multi-partition architecture, with each partition corresponding to an independent write thread. The undirected edge skeletons are partitioned and routed according to the hash value of the edge identifier, ensuring that the edge data corresponding to the same pair of nodes is written to the same partition, guaranteeing the temporal consistency of the edge data, and ultimately forming a dynamic edge sequence with timestamps.

[0045] The distributed graph computing engine uses the current system time as a baseline and defines a current time window based on a preset time span. It filters target edges from the dynamic edge sequence that fall within the current time window, identifying those not present in the historical graph as new edges. Specifically, the time span of the current time window can be configured according to the real-time requirements of the business scenario. In scenarios with high real-time requirements, the time span is set to 1 minute and the sliding step size to 10 seconds; in regular monitoring scenarios, the time span is set to 10 minutes and the sliding step size to 1 minute. Each time the window slides, the distributed graph computing engine uses the current system time as the end time point and traces back the preset time span to determine the start time point of the current time window, thus completing the definition of the current time window. The distributed graph computing engine uses a consumer group of a distributed message queue to batch retrieve data from the dynamic edge sequence according to the defined time range. The batch size of each retrieval can be dynamically adjusted according to the consumer's processing capacity. After retrieval, it filters out edge data with timestamps greater than or equal to the start time point and less than or equal to the end time point of the current time window, which are then used as target edges. The distributed graph computing engine compares the target edge with the edge set of the historical full graph. The historical full graph is the full graph structure that is persistently stored in the graph storage engine after the previous time window processing is completed. The edge identifier set of the historical full graph is pre-stored in a Bloom filter. The Bloom filter is deployed locally on each computing node of the distributed graph computing engine. The distributed graph computing engine uses the Bloom filter to quickly match the edge identifier of the target edge. If the edge identifier does not match in the Bloom filter, the target edge is determined to be a newly added edge, avoiding frequent remote queries to the graph storage engine and improving the identification efficiency of newly added edges.

[0046] The distributed graph computing engine extracts the endpoints of newly added edges and retrieves all first-order neighbor nodes directly connected to the endpoints from the historical full graph. It then combines the newly added edges, endpoints, and first-order neighbor nodes to generate a locally updated subgraph. Specifically, the distributed graph computing engine extracts the two endpoints of each newly added edge, forming an endpoint set. Through the batch query interface of the graph storage engine, it inputs all node identifiers from the endpoint set to retrieve a list of adjacent nodes corresponding to each endpoint. All nodes in the adjacent node list are the first-order neighbor nodes corresponding to that endpoint. All first-order neighbor nodes corresponding to all endpoints together constitute the first-order neighbor node set. The distributed graph computing engine combines the newly added edge set, endpoint set, and first-order neighbor node set to generate a locally updated subgraph. The node set of the locally updated subgraph includes the endpoint set and the first-order neighbor node set. The edge set of the locally updated subgraph includes the newly added edge set, historical edges between endpoints and first-order neighbor nodes, and historical edges between first-order neighbor nodes. This ensures that the locally updated subgraph contains the complete local topology related to the newly added edges, avoiding potential energy calculation errors caused by missing topology information.

[0047] refer to Figure 3 After generating the locally updated subgraph, the distributed graph computing engine performs topology filtering based on the concept of communication network flow control. It counts the in-degree and out-degree values ​​of each endpoint in the subgraph, calculates the absolute value of the difference between the in-degree and out-degree values, and when the absolute value of the difference exceeds a preset degree deviation threshold, the endpoint is identified as an abnormal topology node. The abnormal topology node and the edges between it and its first-order neighbors are removed from the subgraph, generating a filtered locally updated subgraph. Specifically, the in-degree value is the number of edges where the node is the target node, and the out-degree value is the number of edges where the node is the source node. The expression for calculating the in-degree value is:

[0048]

[0049] The expression for calculating the out-degree value is:

[0050]

[0051] in, For nodes The in-degree value, For nodes The out-degree value, For nodes In the local update of the full set of edges in the subgraph For nodes For source node, node The edge connecting to the target node. For nodes For source node, node The edge connecting to the target node. This is the cardinality operation for the set. The distributed graph computing engine calculates the absolute value of the difference between the in-degree and out-degree values ​​of each endpoint, expressed as:

[0052]

[0053] in, For nodes The absolute value of the degree difference is used to determine the degree deviation threshold. The preset degree deviation threshold is determined based on the statistical quantiles of the absolute values ​​of the degree differences of all endpoints in the locally updated subgraph. When the absolute value of the degree difference of a node exceeds the preset degree deviation threshold, the node is identified as an abnormal topology node. Abnormal topology nodes typically correspond to fake nodes such as marketing accounts or bot accounts with severely imbalanced in-degree and out-degree. These nodes can interfere with subsequent clustering calculation results. The distributed graph computing engine removes abnormal topology nodes from the locally updated subgraph, and also removes the edges between abnormal topology nodes and their first-order neighbors, while retaining the edge structure between first-order neighbors. This generates a filtered locally updated subgraph, avoiding damage to the normal topology structure.

[0054] refer to Figure 4 The distributed graph computing engine uses the start and end times of the current time window as independent variables to construct a time decay function that decreases exponentially over time. For any node in the locally updated subgraph, it obtains the set of edges connecting that node within the current time window, calculates the product of the edge weight of each edge in the set and the time decay function at the corresponding timestamp of each edge, and accumulates all products to generate the current topological potential value of that node in the graph topological potential field that incorporates the time dimension. Specifically, different interaction behaviors correspond to different edge weights, and the configuration of edge weights is standardized according to the following table:

[0055] Table 2 Interaction Behavior and Edge Weight Configuration Table

[0056]

[0057] This table provides a unified configuration standard for edge weights corresponding to different interaction behaviors, ensuring consistency in edge weight assignment and enabling the topological potential value to accurately reflect the closeness of interactions between nodes. The distributed graph computing engine matches the corresponding edge weight from the configuration table based on the interaction behavior type of the connection and calculates the node's current topological potential value using a time decay function.

[0058] When calculating the current topological potential energy value of any node in the graph topological potential field with fused time dimension, the distributed graph computing engine continuously calculates the total number of nodes and the total number of edges in the locally updated subgraph. The ratio of the total number of edges to the total number of nodes is used as the current subgraph density coefficient. This current subgraph density coefficient is input into a preset adjustment function, which outputs a decay rate adjustment factor. The decay rate adjustment factor is multiplied by the exponent base of the time decay function. The decay curve of the time decay function is updated based on the product result, and the current topological potential energy value is recalculated based on the updated decay curve. Specifically, the expression for calculating the current subgraph density coefficient is:

[0059]

[0060] in, The density coefficient of the current subgraph. To locally update the total number of edges in the subgraph, This updates the total number of nodes in a local subgraph. The preset adjustment function is a linear adjustment function, expressed as:

[0061]

[0062] in, As the decay rate adjustment factor, The preset slope coefficient, The preset intercept coefficient, and The value of is obtained by fitting historical relationship graph data. During the fitting process, it is ensured that the subgraph density coefficient and the decay rate adjustment factor are negatively correlated. That is, the higher the subgraph density, the smaller the decay rate adjustment factor, and the slower the time decay rate, which is suitable for the characteristics of frequent node interactions and longer validity of historical interaction information in dense subgraphs; conversely, the lower the subgraph density, the larger the decay rate adjustment factor, and the faster the time decay rate, which is suitable for the characteristics of shorter validity of historical interaction information in sparse subgraphs. The distributed graph computing engine updates the time decay function based on the decay rate adjustment factor. The updated time decay function expression is:

[0063]

[0064] in, This is the updated time decay function. The distributed graph computing engine replaces the time decay coefficient in the original topological potential value calculation expression with the calculation result of the updated time decay function, and recalculates the current topological potential value of the node to make the potential energy calculation result adapt to the density of the current subgraph.

[0065] The distributed graph computing engine reads the historical topological potential energy values ​​of each node in the locally updated subgraph from the graph storage engine, corresponding to the previous time window. For each node, it calculates the absolute value of the difference between the current topological potential energy value and the historical topological potential energy value, using this as the topological potential energy change. It iterates through the topological potential energy changes of all nodes in the locally updated subgraph, extracting nodes whose topological potential energy changes are greater than a preset physical topological threshold. The extracted nodes and the connected subgraph formed by the edges between them are marked as active evolution regions. Specifically, the distributed graph computing engine reads the historical topological potential energy values ​​of all nodes in the locally updated subgraph at once through the batch query interface of the graph storage engine, reducing the number of remote queries. The preset physical topological threshold can be dynamically adjusted according to the current subgraph density coefficient. The adjustment logic is that the higher the subgraph density coefficient, the higher the preset physical topological threshold, avoiding too many falsely identified nodes in dense subgraphs. The distributed graph computing engine extracts nodes whose topological potential energy changes exceed a preset physical topology threshold, forming a candidate node set. A depth-first search algorithm is used to traverse the candidate node set and identify the connectivity between nodes. Each connected component is an independent active evolution region, and each active evolution region corresponds to a potential hotspot event.

[0066] In this embodiment, a time-consistent dynamic edge sequence is constructed by cleaning and standardizing network interaction data. Newly added edges are quickly identified based on a Bloom filter. A locally updated subgraph containing complete local topological information is constructed. Abnormal topological nodes are removed by filtering out the in-degree difference, reducing the interference of invalid data on the calculation results. At the same time, the decay curve of the time decay function is dynamically adjusted based on the subgraph density coefficient, so that the topological potential energy calculation is adapted to subgraph structures with different densities, improving the accuracy of active evolution zone determination and providing a reliable quantitative basis for the accurate identification of hotspot events.

[0067] refer to Figure 6In a preferred embodiment, the distributed graph computing engine obtains the hash value set of all nodes in the active evolution zone, calculates the geometric center hash value of the hash value set, locates the target physical shard to which the geometric center hash value belongs in the sharding routing table of the graph storage engine, sends an index update instruction to the target physical shard, writes the node attribute data and edge relationship data of the active evolution zone into the memory data structure of the target physical shard, and reconstructs the local adjacency list index inside the target physical shard. Specifically, the hash value of a node is calculated by using a preset hash algorithm to calculate the node identifier. After the calculation is completed, the hash value is converted into a 32-bit unsigned integer to ensure that the hash value falls within the numerical range of the consistent hash ring. The sharding routing table stores the start and end values ​​of the hash value interval corresponding to each physical shard, as well as the access address, running status, and other information of the physical shard. The sharding routing table is synchronized in real time to all computing nodes of the distributed graph computing engine to ensure the global consistency of routing information. The distributed graph computing engine matches the corresponding hash value interval in the sharding routing table according to the geometric center hash value, and the physical shard corresponding to this interval is the target physical shard.

[0068] Before sending index update commands to the target physical shard, the distributed graph computing engine collects the current memory usage and CPU utilization of the target physical shard. When the current memory usage exceeds a preset storage safety threshold or the CPU utilization exceeds a preset computing safety threshold, the engine identifies idle adjacent physical shards in the graph storage engine. It then splits the node attribute data and edge relationship data in the active evolution zone into multiple data shards, and routes these data shards to the target physical shard and idle adjacent physical shards for distributed index reconstruction. Specifically, the distributed graph computing engine collects real-time running status data of the storage node where each physical shard resides through the node monitoring interface of the graph storage engine. The collected data includes metrics such as memory usage, CPU utilization, disk I / O utilization, and network bandwidth utilization. The collection frequency is set to once every 100ms to ensure the real-time nature of the status data. The preset storage security threshold and preset computation security threshold are set based on the hardware configuration and business capacity of the graph storage engine. When the current memory usage of the target physical shard exceeds the preset storage security threshold, or the CPU utilization exceeds the preset computation security threshold, the target physical shard is determined to be under high load and unable to handle the computation and storage pressure of this index update. On the consistent hash ring, the distributed graph computation engine identifies physical shards adjacent to the hash value range of the target physical shard, and filters out physical shards with memory usage below the preset idle threshold and CPU utilization below the preset idle computation threshold as adjacent physical shards in an idle state. The distributed graph computing engine divides the set of node hash values ​​in the active evolution zone into multiple contiguous sub-intervals. Each sub-interval corresponds to a data shard, and each data shard contains approximately the same number of nodes to ensure load balancing. Multiple data shards are routed to the target physical shard and its idle neighboring physical shards. Each physical shard is only responsible for processing the node and edge data within its corresponding data shard, rebuilding its local adjacency table index. After index rebuilding, each physical shard synchronizes the index update results to the global routing table to ensure that subsequent queries can correctly route to the corresponding data shard. The running status and routing decisions of each physical shard are recorded in the following table:

[0069] Table 3 Physical Fragmentation Operation Status and Routing Decision Table

[0070]

[0071] This table records the real-time running status of each physical shard, providing a quantitative basis for routing decisions and data shard allocation during the index update process, and ensuring the load balancing and operational stability of the graph storage engine.

[0072] When the current memory occupancy of the target physical shard does not exceed the preset storage safety threshold, and the CPU utilization does not exceed the preset computation safety threshold, the distributed graph computing engine directly sends an index update command to the target physical shard. This command writes the node attribute data and edge relationship data of the active evolution region into the target physical shard's memory data structure. The memory data structure uses a bidirectional adjacency list structure, with each node corresponding to one adjacency list entry. Each entry contains information such as the identifiers of all neighboring nodes, edge attributes, and edge timestamps. The adjacency list uses a skip list structure for indexing, with the skip list layer set to 8 levels to ensure that the time complexity of node queries remains stable at the logarithmic level. When the distributed graph computing engine rebuilds the local adjacency list index within the target physical shard, it only updates the nodes and edges corresponding to the active evolution region, without modifying the index structure of other unchanged nodes and edges within the physical shard, thus reducing the computational load of index updates.

[0073] After marking the connected subgraph formed by the extracted nodes and the edges between them as active evolution regions, the distributed graph computing engine extracts the central node from these regions. The central node is the node with the largest change in topological potential energy. If multiple nodes have the same change in topological potential energy, and all are at the maximum value, the node with the highest degree is selected as the central node. The distributed graph computing engine queries the historical physical shard identifier sequence to which the central node belongs within several consecutive preceding time windows. The number of consecutive preceding time windows can be configured according to business needs. Through the node historical trajectory query interface of the graph storage engine, the central node identifier and the corresponding time range are passed in to batch obtain the physical shard identifier to which the node belongs in each preceding time window. The distributed graph computing engine arranges the historical physical shard identifier sequence in chronological order and combines it with the target physical shard identifier corresponding to the current time window to generate a node topological offset trajectory containing physical location migration records. The expression for the node topological offset trajectory is:

[0074]

[0075] in, As the central node The node topology offset trajectory. This is the historical physical partition identifier to which the center node belongs within the preceding n time windows. This is the identifier of the target physical segment corresponding to the current time sliding window.

[0076] The distributed graph computing engine calculates the similarity between node sets corresponding to active evolutionary regions within adjacent time windows, based on the node set of the active evolutionary region. The similarity calculation expression is as follows:

[0077]

[0078] in, The similarity between the node sets corresponding to two active evolutionary regions. This refers to the set of nodes corresponding to the active evolutionary regions within the previous time window. This is the set of nodes corresponding to the active evolution zones within the current time window. When the similarity exceeds a preset event association threshold, it determines that the two active evolution zones correspond to different evolution stages of the same hot event. The topological offset trajectories of the nodes corresponding to the two active evolution zones are then spliced ​​with the regional structural change information to form a complete event evolution path. The event evolution path includes information such as the initiation time of the hot event, core propagation nodes, propagation range migration, and topological offset trends of key nodes, thus fully reconstructing the entire lifecycle evolution process of the hot event.

[0079] In this embodiment, a consistent hashing algorithm is used to achieve a precise mapping from active evolution regions to physical shards. Dynamic routing and distributed index reconstruction are achieved based on the real-time running status of physical shards, avoiding system performance degradation caused by excessive load on a single shard and ensuring system stability during large-scale burst data writes. At the same time, the node topology offset trajectory is generated by extracting the historical physical shard identifier sequence of the central node. Combined with the similarity calculation of the node set, the association of different evolution stages of the same event is realized, fully restoring the entire life cycle evolution process of hot events and achieving accurate identification and continuous tracking of hot events.

Claims

1. A method for identifying hotspot events through subgraph clustering and evolution tracking, characterized in that, include: In the distributed graph computing engine, a streaming data access buffer is set up to convert the received network interaction data into a dynamic edge sequence with timestamps. An incremental clustering model combining a time decay function and a graph topological potential is constructed. Within the current time sliding window, newly added edges and adjacent nodes with changed degrees are extracted from the dynamic edge sequence to form a locally updated subgraph. Calculate the change in topological potential energy of each node in the local update subgraph, and determine the set of nodes whose change in topological potential energy exceeds a preset physical topological threshold as an active evolution region. The node set of the active evolution region is mapped to the corresponding physical shard of the graph storage engine, triggering the memory index update of the local subgraph structure of the corresponding physical shard, and outputting the event evolution path containing the node topology offset trajectory; Constructing an incremental clustering model that combines the time decay function with the graph topological potential field includes: constructing the time decay function with the start and end times of the current time window as independent variables; For any node in the local update subgraph, obtain the set of edges connecting the arbitrary node within the current time window; Calculate the product of the edge weight of each edge in the edge set and the time decay function at the corresponding timestamp of each edge, and sum all the products to generate the current topological potential value of any node in the graph topological potential field with fused time dimension. Calculating the change in topological potential energy of each node in the local update subgraph, and determining the set of nodes whose change in topological potential energy exceeds the preset physical topological threshold as the active evolution region, includes: reading the historical topological potential energy value of each node in the local update subgraph corresponding to the previous time window from the graph storage engine; For each node, the absolute value of the difference between the current topological potential energy value and the historical topological potential energy value is calculated as the change in topological potential energy. Traverse the topological potential energy change of all nodes in the local update subgraph, extract the nodes whose topological potential energy change is greater than the preset physical topological threshold, and mark the connected subgraph formed by the extracted nodes and the edges between the nodes as the active evolution region. Mapping the node set of the active evolution region to the corresponding physical shard of the graph storage engine, and triggering the memory index update of the local subgraph structure of the corresponding physical shard, includes: obtaining the hash value set of all nodes in the active evolution region, and calculating the geometric center hash value of the hash value set; The expression for calculating the geometric center hash value is: ； in, The hash value of the geometric center. The total number of nodes in the active evolutionary region. The first in the active evolutionary region The hash value of each node. This is a floor function; Locate the target physical shard to which the geometric center hash value belongs in the sharding routing table of the graph storage engine; Send an index update instruction to the target physical shard, write the node attribute data and edge relationship data of the active evolution region into the memory data structure of the target physical shard, and rebuild the local adjacency table index inside the target physical shard.

2. The hotspot event identification subgraph clustering and evolution tracking method according to claim 1, characterized in that, The received network interaction data is converted into a dynamic edge sequence with timestamps, including parsing the source node identifier, target node identifier, and interaction timestamp in the network interaction data; In the streaming data access buffer, an undirected edge skeleton is constructed using the source node identifier and the target node identifier, and the interaction timestamp is bound as an edge attribute of the undirected edge skeleton. The undirected edge skeletons carrying the timestamps of the interactions are written into a distributed message queue in chronological order to form the dynamic edge sequence with timestamps.

3. The hotspot event identification subgraph clustering and evolution tracking method according to claim 2, characterized in that, Within the current time window, newly added edges and adjacent nodes with changed degrees are extracted from the dynamic edge sequence to form the local update subgraph, including: defining the current time window based on the current system time and according to a preset time span; Target edges that fall within the current time window are selected from the dynamic edge sequence, and edges that do not exist in the historical full graph are identified as newly added edges. Extract the endpoints of the newly added edges, and obtain all first-order neighbor nodes directly connected to the endpoints in the historical full graph; The newly added edge, the endpoint, and the first-order neighbor node are combined to generate the locally updated subgraph.

4. The hotspot event identification subgraph clustering and evolution tracking method according to claim 3, characterized in that, After generating the local update subgraph, the method further includes: performing topology filtering on the local update subgraph based on the concept of communication network flow control, and counting the in-degree and out-degree values ​​of each endpoint in the local update subgraph; the in-degree value is the number of edges connected to the node as the target node, and the out-degree value is the number of edges connected to the node as the source node. Calculate the absolute value of the difference between the in-degree value and the out-degree value. When the absolute value of the difference exceeds a preset degree deviation threshold, the endpoint is determined to be an abnormal topology node. The abnormal topology node and the edges between the first-order neighbor nodes connected to the abnormal topology node are removed from the local update subgraph to generate a filtered local update subgraph.

5. The hotspot event identification subgraph clustering and evolution tracking method according to claim 4, characterized in that, When calculating the current topological potential energy value of any node in the graph topological potential field in the time dimension of fusion, the method further includes: real-time statistics of the total number of nodes and the total number of edges in the locally updated subgraph, and calculating the ratio of the total number of edges to the total number of nodes as the density coefficient of the current subgraph. The current subgraph density coefficient is input into a preset adjustment function, and the decay rate adjustment factor is output. The decay rate adjustment factor is multiplied by the exponent base of the time decay function, the decay curve of the time decay function is updated based on the product result, and the current topological potential value is recalculated based on the updated decay curve.

6. The hotspot event identification subgraph clustering and evolution tracking method according to claim 5, characterized in that, After marking the connected subgraph formed by the extracted nodes and the edges between the nodes as the active evolution region, the method further includes: extracting the center node from the active evolution region, wherein the center node is the node with the largest change in topological potential energy. Query the historical physical shard identifier sequence to which the central node belongs within a series of preceding time windows; The historical physical fragment identifier sequence is arranged in chronological order and combined with the target physical fragment identifier corresponding to the current time window to generate the node topology offset trajectory.

7. The hotspot event identification subgraph clustering and evolution tracking method according to claim 6, characterized in that, Before sending the index update instruction to the target physical partition, the method further includes: collecting the current memory occupancy and CPU utilization of the target physical partition; When the current memory occupancy rate exceeds a preset storage security threshold or the central processing unit utilization rate exceeds a preset computing security threshold, adjacent physical fragments that are in an idle state are identified in the graph storage engine. The node attribute data and the edge relationship data in the active evolution region are split into multiple data shards, and the multiple data shards are respectively routed to the target physical shard and the adjacent physical shard for distributed index reconstruction.

Images