A method for calculating connected component increments based on multiple graphs

Abstract

This invention discloses a method for incremental calculation of connected components based on multiple graphs. By renumbering the connected components and connecting them to form a graph, new connected components are calculated. This method achieves the goal of merging different graphs with the same points and then calculating the connected components, thus solving the problem of long calculation time and low efficiency caused by recalculating connected components when merging graphs.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer network traffic analysis, and in particular relates to a method for calculating connected component increments based on multiple graphs. Background Art

[0002] In the process of network traffic analysis, when analyzing network data based on large-scale traffic data, it is often necessary to recalculate the connected components when solving the network attribution judgment problem of network data from different sources and the node change problem caused by network effectiveness.

[0003] When analyzing network data, determining whether data obtained from different channels and devices belongs to the same network requires detailed data analysis. This often prevents complete network attribution when calculating connectivity. Therefore, when determining that two sets of network data belong to the same physical network, the connectivity components must be recalculated. The typical approach is to obtain all eligible protocol data and recalculate, but this recalculation consumes significant space and time resources.

[0004] Furthermore, because network connectivity is time-sensitive, changes to certain nodes in the original network data often affect connectivity, especially when network addresses are reallocated. In this case, obtaining new connectivity data often requires removing data from before the network status change. In this case, the common approach is to obtain all eligible protocol data and recalculate, which is also time-consuming and wastes resources.

[0005] In summary, for the above two scenarios, existing technologies obtain connectivity by obtaining all qualified protocol data and then recalculating it. Such calculations are all full-scale calculations, which greatly consumes space and time resources. Summary of the Invention

[0006] In view of this, the present invention provides a connected component increment calculation method based on multiple graphs, which realizes merging existing connected component data to obtain a new graph and then calculating new connected components.

[0007] The present invention provides a method for calculating connected component increments based on multiple graphs, comprising the following steps:

[0008] Collect existing connected component IDs to form a connected component set, and the IDs of connected components contained in different graph IDs are all different; perform an aggregation operation on the connected component set, collect the graph IDs corresponding to all connected component IDs, and form (compontID, Set(graph ID)); let the nodes with only one data in Set(graph ID) connect to themselves to form a self-connected edge, and let the nodes with multiple data in Set(graph ID) be interconnected to form an interconnected edge, thereby forming a new graph; perform connected component calculation on the new graph to obtain a data pair of Ip and the new connected component ID.

[0009] Furthermore, the aggregation operation on the connected component set is performed by using the reduceByKey method of the big data processing framework Spark.

[0010] Furthermore, after the connected components of the new graph are calculated to obtain a data pair of IP and a new connected component ID, the data pair of IP and the new connected component ID is stored as (IP, new connected component ID, new graph ID).

[0011] Furthermore, the process of storing the data pair of Ip and the new connected component ID as (IP, new connected component ID, new graph ID) is divided into time stages for storage, that is, the connected components within the time stage are first calculated, and then the connected components are merged and then the connected components of all data are calculated.

[0012] Furthermore, the time period is one week.

[0013] Beneficial effects:

[0014] 1. The present invention renumbers the connected components, connects them to form a graph, and then calculates new connected components. This achieves the calculation of connected components after merging different graphs containing the same points. This solves the problem of needing to recalculate connected components when merging graphs, resulting in long calculation time and low efficiency.

[0015] 2. The present invention records the connected component data within a set time, and when it is necessary to remove certain redundant data, the data of that time is merged and calculated, avoiding the need to re-acquire all edges and recalculate, reducing the time and space occupied by a single calculation, and making it more convenient and efficient to remove redundant data. DETAILED DESCRIPTION

[0016] The present invention is described in detail with reference to the following examples.

[0017] The present invention relates to the following definitions and methods:

[0018] Graph ID: used to mark the index of the graph. All data containing the same graph ID belongs to this graph. Graph theory-related algorithms can be calculated on this graph. The data type is integer.

[0019] Connected component ID (componentID): An index value used to mark the uniqueness of each connected component in the graph. All nodes within the same connected component ID belong to the same graph and are interconnected. The data type is integer.

[0020] reduceByKey: An aggregation method in Spark that merges the values ​​for each key using an associative and commutative reduction function.

[0021] The technical principle of the multi-graph-based connected component increment calculation method provided by the present invention is:

[0022] Assuming that two nodes n1 and n2 are interconnected in a certain graph before merging, then these two nodes are still interconnected in the graph after merging; Assuming that two nodes n1 and n2 cannot be connected in graph G1 and belong to connected components c1 and c2 respectively, and there is a graph Gk, and there is a connected component c3 in graph Gk that contains these two nodes, then after merging, all nodes contained in c1, c2, and c3 are interconnected; Simplifying the connected components in the graph into nodes, and simplifying the edges between nodes into self-connected edges of connected components, will not cause information loss when calculating connected components during merging, and a large number of points and edges in multiple graphs can be simplified into connected components, as well as edges between connected components. The processing logic of the present invention meets the basic idea of ​​splitting and calculating large-scale data, so the implementation method adopts the idea based on the big data framework, and includes the operation method of part of the big data processing framework Spark in the implementation process to simplify the difficulty.

[0023] The present invention provides a method for calculating connected component increments based on multiple graphs, which specifically includes the following steps:

[0024] Step 1: Collect existing connected component IDs and ensure that the connected component IDs contained in different graph IDs are different.

[0025] Collect the existing connected component IDs (componentID) and renumber the componentIDs under different graph IDs so that they are not repeated. Specifically, for the existing connected component set, convert all (componentID, graph ID) into (componentID, Set (graph ID)), where each Set has only one value.

[0026] Step 2: For the existing connected component set, perform a reduceBykey operation to find the IPs that appear in multiple graphs, and obtain a set of (compontID, Set(graph ID)).

[0027] Specifically, by executing the reduceByKey(fun(set1,set2)=>set1+set2) collection operation, we obtain a set (compontID,Set(graph ID)), which collects the graph IDs corresponding to all componentIDs. The reduceByKey method aggregates all data with the same componentID and merges the two sets using a reduction function. For example, merging ((Ip1,(1)) and (Ip1,(2)) yields ((Ip,(1,2)).

[0028] Specifically, we perform a reduceByKey operation on the key-value pairs consisting of the connected components (IP, set(componentID, graph ID)) to find all IPs that appear in multiple graphs. The result of the reduceByKey operation is the set of all (componentID, graph ID) pairs corresponding to the same IP. For any IP in the data, the corresponding set contains all connected component IDs and graph IDs where this IP appears.

[0029] Then, take the maximum numerical value of the componentID in all graphs, recorded as Cmax, and then renumber the repeated componentIDs in different graphs. Since the maximum value of the current connected component is Cmax, the value greater than Cmax can be guaranteed to be unique during renumbering. The renumbered dictionary is broadcast to all computing nodes and applied to the entire distributed cluster.

[0030] After the above operation, a set of (IP, componentID, graph ID) is obtained. At this time, the componentIDs corresponding to different graph IDs are different.

[0031] Step 3: Connect all nodes in the Set that have only one piece of data to themselves, forming a self-connected edge. Flatmap the nodes in the Set that have multiple pieces of data into multiple (componentID, graph ID) interconnected edges, thus forming a new graph.

[0032] Specifically, after renumbering the connected components, recalculate the graphs in the existing set of connected components to obtain new connected components and store them. For each IP, connect all the nodes in the value set to itself to form a self-connected edge. Expand the nodes with multiple data in the value set into edges that interconnect multiple (componentID, graph ID). The significance of this is that since all componentIDs in the set represent a connected component, the interconnection relationship between IPs within a connected component is manifested as a self-connected edge. The connection relationship between IPs that do not belong to the same componentID is manifested as an interconnected edge between two componentIDs. After this transformation, a new graph is formed.

[0033] For example, for a piece of data ("192.168.1.100", ((5, "Figure 1"), (8, "Figure 2"), (10100, "Figure 3"))), the reorganized edges are ((5->5), (8->8), (10100->10100), (5->8), (5->10100)). At this time, 5 edges can ensure the correct connectivity transformation of this graph, and the connection relationship of the new graph can fully represent the connection relationship of all IP nodes.

[0034] Step 4: Perform a connected component calculation on this new graph to obtain (Ip, new connected component ID), update the database, and store it as (IP, new connected component ID, new graph ID).

[0035] In addition, the previously saved data of (IP, componentID, graph ID) is retained for a period of time. When a rollback operation is required, delete (IP, new connected component ID, new graph ID) to restore it. When the update is confirmed, delete all the data corresponding to the old graph ID. Update the database and store it as (IP, new connected component ID, new graph ID). At this time, the calculation of the merged connected components has been completed.

[0036] The merged graph ID has no impact on the calculation. It is only used as an index to identify the entire graph and has uniqueness. All IP data containing this graph ID belongs to this graph. For example: (Figure 1, Figure 2) are merged into Figure 3. Then, for the data, the original data should be a group of (IP, connected component id1, "Figure 1") and a group of (IP, connected component id2, "Figure 2"). The final result data is a group of (IP, connected component id3, "Figure 3"), and "Figure 3" plus the two IP fields form a "unique index".

[0037] In addition, to further improve the storage efficiency, the present invention adopts a hierarchical storage method.

[0038] The hierarchical storage method is based on the merging of connected components. By storing and calculating connected components in stages to address data timeliness, it avoids the need to recalculate the entire connected component due to data updates and deletions, significantly improving computational efficiency. The specific steps are as follows: Calculate the connected components within a set timeframe, merge the connected components, and then calculate all connected components. When redundant data needs to be removed, retrieve the correct data and merge it, significantly improving performance compared to recalculation.

[0039] For example, when the hierarchical storage method is based on the merging of connected components, the data is divided into stages by week, and the connected components are stored and calculated to address the timeliness of the data. This avoids the full calculation of connected components caused by data updates and deletions, and greatly improves computing efficiency. The specific steps are as follows:

[0040] S1, calculate the connected components of one week;

[0041] S2. After merging the connected components as described above, calculate all connected components;

[0042] S3. When it is necessary to remove some redundant data, the correct data can be retrieved and merged.

[0043] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for calculating connected component increments based on multiple graphs, characterized in that: The following steps are involved: For the network data to be analyzed, the existing connected component IDs are collected to form a connected component set, and the IDs of the connected components contained in different graph IDs are all different; the connected component set is aggregated, the graph IDs corresponding to all the connected component IDs are collected, and all (compontID, graph ID) are converted into (compontID, Set (graph ID)); for the existing connected component set, a reduceBykey operation is performed to find out the IPs that appear in multiple graphs, and a set of (compontID, Set (graph ID)) is obtained; the nodes in Set (graph ID) with only one piece of data are connected to themselves to form a self-connected edge, and the nodes in Set (graph ID) with multiple pieces of data are interconnected to form an interconnected edge, thereby forming a new graph; the connected components of the new graph are calculated to obtain a data pair of IP and the new connected component ID; After the connected components of the new graph are calculated to obtain a data pair of IP and a new connected component ID, the data pair of IP and the new connected component ID is stored as (IP, new connected component ID, new graph ID).

2. The method for calculating connected component increment according to claim 1, characterized in that: The method of performing the aggregation operation on the connected component set is implemented by adopting the reduceByKey method of the big data processing framework Spark.

3. The method for calculating connected component increment according to claim 1, characterized in that: The process of storing the data pair of Ip and the new connected component ID as (IP, new connected component ID, new graph ID) is to store them in time stages, that is, first calculate the connected components within the time stage, then merge the connected components and calculate the connected components of all the data.

4. The method for calculating connected component increment according to claim 3, characterized in that: The time period is one week.