A network measurement data management method based on dynamic multi-copy index

Abstract

The application relates to the field of databases and discloses a network measurement data management method based on dynamic multi-copy indexes, which comprises the following steps: network measurement data collection, dynamic multi-copy index construction, dynamic multi-copy index query and update and the like; the application creates dynamic multi-copies with hot spot data and stores the dynamic multi-copies in an index structure based on a log-structured merge tree, realizes the function of multi-key query of network measurement data which cannot be completed by traditional key-value storage, improves the storage and query efficiency of network measurement data, reduces the index space overhead caused by query of different fields when a key-value database based on a log-structured merge tree stores network measurement data, and solves the storage bottleneck of existing network measurement data.

Description

Technical Field

[0001] This invention relates to the field of databases, and more specifically to a method for managing network measurement data based on dynamic multi-replica indexes. Background Technology

[0002] Network measurement collects essential network traffic statistics, which help network operators and data centers adjust network strategies, improve network performance, detect network anomalies promptly, and prevent network attacks. Network traffic measurement extracts or records network data at network nodes such as routers or switches, processing it into stream information and recording it. This method effectively extracts information from network traffic, but it consumes significant CPU and memory resources. As the bandwidth of data centers and backbone networks has generally increased to 10 to 100 Gbps, the amount of network measurement data generated by network traffic measurement has also increased, requiring greater data processing capabilities. Existing network measurement data processing frameworks struggle to handle the data generated by network traffic measurement under high bandwidth conditions. Currently, measurements of high-bandwidth network systems typically employ sampling methods to reduce the resources required to manage measurement data, ensuring that the data processing rate matches the network speed.

[0003] However, sampling-based methods do not process every packet flowing through the network; instead, they monitor a subset of packets based on probability or matching rules. This approach results in a loss of accuracy in the measurement data, especially in high-speed networks. To meet performance requirements, administrators must set sampling rates below one percent or even one-thousandth, leading to lower reliability of the measurement results and making more sophisticated network analysis tasks difficult.

[0004] Key-value stores are a new type of data storage system that organizes data into key-value pairs. Access is achieved by using a unique key to locate the corresponding value in the storage system. Log Structure Merge Tree (LSM-Tree) is a commonly used data structure for key-value stores. By encoding data updates into ordered blocks and writing them sequentially to external storage, LSM-Tree can handle high-rate data updates. Simultaneously, LSM-Tree hierarchically manages these ordered blocks, organizing them into a structure where they are ordered within each layer but unordered between layers. It periodically merges ordered blocks, resulting in a finite number of ordered layers. Through binary search within these ordered layers, it maintains high data access efficiency. Key-value databases built using LSM-Tree are well-suited for handling high-throughput data. However, because data can only be accessed by key, they are unsuitable for complex analytical tasks and are therefore difficult to use for managing network measurement data. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a network measurement data management method based on a dynamic multi-replica index. By storing network measurement data in a key-value store based on a log merging structure tree, and constructing a dynamic multi-replica index according to the real-time popularity of the data, the method maintains a high update rate for the key-value store while enabling access to different keys of the network measurement data.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] A network measurement data management method based on dynamic multi-replica index includes the following steps:

[0008] Step 1: Collection of Network Measurement Data; Collect network measurement data by deploying passive measurement devices on network nodes such as routers and switches; Each piece of network measurement data should include at least the following data fields: timestamp, source IP address, destination IP address, number of network packets, and total size; Depending on the type of network being measured, the source IP address and destination IP address can be IPv4 or IPv6 addresses; The data collected by the measurement devices is transmitted to the control plane node used for data storage and management via a separate network connection.

[0009] Step 2: Create a dynamic multi-replica index; based on the data fields that the user needs to query, create or update the dynamic multi-replica index, and store the network measurement data in the log structure merge tree corresponding to the dynamic multi-replica index.

[0010] Specifically, in step two, when creating or updating a dynamic multi-replica index based on the data fields that the user needs to query, the dynamic multi-replica index includes multiple log structure merge trees, one of which is the primary tree and the remaining log structure merge trees are secondary trees; one of the data fields is set as the primary key, and one or more of the remaining data fields are set as secondary keys, with the number of secondary keys being the same as the number of secondary trees;

[0011] Each network measurement data point is stored as a key-value pair in each index entry of the main tree. Hotspot data exists in the network measurement data of the main tree, and a copy of each hotspot data point is stored as a key-value pair in each index entry of the secondary tree. The key of each index entry in the main tree is the primary key, and the value is the network measurement data corresponding to the primary key. The key of each index entry in the secondary tree is the secondary key corresponding to that secondary tree, and the value is the network measurement data corresponding to the secondary key.

[0012] Each subtree corresponds to a Bloom filter; when storing a copy of each hot data point as a key-value pair in the subtree's index entry, the key of the added index entry is inserted into the Bloom filter corresponding to that subtree.

[0013] Step 3: Based on the user's query request, query the network measurement data in the dynamic multi-replica index and return the retrieved network measurement data to the user.

[0014] Specifically, when querying network measurement data in the dynamic multi-replica index based on a user's query request and returning the retrieved network measurement data to the user, the following steps are included:

[0015] Step 3A: When a user initiates a query request, the user enters the keyword P in data field N. The Bloom filter corresponding to data field N is used to confirm whether the keyword P to be searched exists in the subtree Q corresponding to data field N. If so, the user queries the subtree Q with the keyword P as the key and returns the value of the index entry corresponding to the keyword as the network measurement data obtained from the query to the user. If not, proceed to Step 3B.

[0016] Step 3B: Scan the entire main tree with the keyword P as the key, and send the value of the index entry corresponding to the keyword back to the user as the network measurement data obtained from the query. Since the Bloom filter has an error probability, even if the Bloom filter confirms that the subtree contains the keyword to be queried, it may not be able to find the corresponding index entry in the subtree. In this case, a search is also performed in the main tree.

[0017] Step 3C: Based on the user's query request, update the hot data obtained by classifying based on the secondary key; if the non-hot data corresponding to the secondary key M becomes hot data, then create an index entry for the newly added hot data in the subtree corresponding to the secondary key M.

[0018] In the preferred scheme, if the Bloom filter corresponding to data field N contains the keyword P, but the network measurement data cannot be obtained from tree Q, then the entire main tree is scanned with the keyword P as the key, and the value of the index entry corresponding to the keyword is sent back to the user as the obtained network measurement data.

[0019] Step 4: Perform popularity statistics on the retrieved network measurement data and update the dynamic multi-replica index based on the popularity statistics.

[0020] The hot data statistics method used when building a dynamic multi-replica index is as follows: Hot data exists in the network measurement data of the main tree, categorized by each secondary key. This hot data, categorized by secondary key M, is stored in the index entries of the secondary tree corresponding to secondary key M. When counting hot data, all network measurement data is categorized according to secondary key M, and the number of data entries in each category is counted. Based on the user-set hot data threshold, network measurement data with a number of entries greater than the threshold is considered hot data for secondary key M, while network measurement data with a number of entries less than or equal to the threshold is considered non-hot data for secondary key M.

[0021] When performing popularity statistics on the retrieved network measurement data, the influence of query frequency on popularity is added to the hot data statistics method used when building a dynamic multi-replica index.

[0022] The data fields N, secondary key M, key P, and letters contained in tree Q in this invention are only for the purpose of describing the logical correspondence between the technical solutions and do not refer to any specific data.

[0023] Compared with the prior art, the beneficial technical effects of the present invention are:

[0024] This invention creates dynamic multiple copies of hot data and stores them in an index structure based on a log structure merge tree. This enables multi-key querying of network measurement data, which is impossible with traditional key-value stores, and improves the storage and query efficiency of network measurement data. At the same time, it reduces the index space overhead of key-value databases based on log structure merge trees when storing network measurement data for querying different fields, thus solving the storage bottleneck of existing network measurement data. Attached Figure Description

[0025] Figure 1 This is a flowchart of the network measurement data management method based on dynamic multi-replica index of the present invention;

[0026] Figure 2 This is an architectural diagram of Embodiment 1 of the present invention;

[0027] Figure 3 This is an example diagram of the network measurement data collected by this invention;

[0028] Figure 4 This is a schematic diagram of the data stored in the tree in the dynamic multi-replica index of this invention;

[0029] Figure 5 This is a schematic diagram of the structure of the dynamic multi-replica index of the present invention;

[0030] Figure 6 This is a flowchart of the query process for the dynamic multi-copy index of this invention. Detailed Implementation

[0031] The network measurement data management method based on dynamic multi-replica index of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] Example 1

[0033] The network measurement data management method based on dynamic multi-replica index in this invention aims to meet the performance requirements for data storage and querying of network measurement data in high-performance network environments with bandwidths exceeding 10 gigabits per second. It also addresses the limitation of key-value stores, which prevent querying across different keys when storing network measurement data. Based on a log structure merging tree, this invention designs a dynamic multi-replica index to store a small portion of the data, categorized by the field to be queried, within different log structure merging trees according to access frequency. This achieves low-latency querying across different fields while leveraging the write performance of the log structure merging tree.

[0034] In this embodiment, a network traffic measurement task is deployed on a soft router node. The network measurement data is sent to a control plane node that uses a dynamic multi-replica index. The control plane node then builds the index, stores and manages the data, and responds to user query requests.

[0035] Figure 1 The flowchart of the network measurement data management method based on dynamic multi-replica index of the present invention is as follows: Figure 1 As shown, the present invention includes the following steps:

[0036] Step 1: Obtain network measurement data through measurement devices deployed on network nodes and send it to control plane nodes. The network devices include routers or switches. The network measurement data includes timestamps, source IP addresses, destination IP addresses, source ports, destination ports, number of packets, and total size.

[0037] Step 2: Control the plane nodes to periodically collect network measurement data sent by the network nodes, create or update dynamic multi-replica index entries according to the data fields that the user needs to query, and store the network measurement data into the log structure merge tree corresponding to the dynamic multi-replica index;

[0038] Step 3: Based on the user's query request, retrieve data from the dynamic multi-replica index and return the required data to the user;

[0039] Step 4: Based on the retrieved data, update the popularity statistics of the dynamic multi-replica index, and update the dynamic multi-replica index according to the popularity statistics. Create or delete new network measurement data replicas in the dynamic multi-replica index.

[0040] Figure 2 This is a system architecture diagram used in this embodiment. Figure 2 As shown, the measurement device deployed on the network node collects the packet data flowing through the network node via interface 1 in the network under test, and processes it into network measurement data that needs to be recorded; the network node transmits the network measurement data to the control plane node through interface 2, which is independent of the network under test.

[0041] The control plane node collects network measurement data through the network interface card connected to the network node. After the data management module establishes or updates the dynamic multi-replica index entries, it writes the dynamic multi-replica index, the network measurement data to be recorded, and the copies therein to the storage device. At the same time, the data management module is also responsible for responding to user query requests. Using the dynamic multi-replica index structure, it finds and reads the network measurement data required by the user from the storage device, returns it to the user, and updates the data popularity according to the user's query request, updating the dynamic multi-replica index entries accordingly, and creating new data copies for hot data.

[0042] Figure 3 This demonstrates possible scenarios for data collected by control plane nodes. Network measurement data includes five common network measurement fields: timestamp, source IP address, destination IP address, packet count, and total size. In a preferred embodiment, the network measurement data also includes fields collected by other measurement devices, such as source port, destination port, protocol type, and packet header information. There are no restrictions on the data format, as long as the data can be parsed to the required packet format.

[0043] In this example, timestamps are chosen as the primary key and source IP addresses as the secondary key. All network measurement data is stored in the main tree using timestamps as the key; while some hot data uses source IP addresses as the key, and copies of these hot data are stored in the secondary tree. Although this embodiment only uses source IP addresses as secondary keys, the invention does not limit the number of secondary keys, as long as they can be parsed or calculated from the data. For example, any number of fields can be selected as secondary keys, or a combination of multiple fields can be chosen, such as <source IP address, destination IP address>. Each secondary key corresponds to a log structure merge tree in the dynamic multi-replica index. Figure 4 This demonstrates the structure of the dynamic multi-replica index in this example.

[0044] After the control plane nodes collect the raw message data, they will use the following steps to build a dynamic multi-replica index:

[0045] Step 2A: Classify the network measurement data according to the data fields to be searched. In this embodiment, the data fields to be searched are the primary key "timestamp" and the secondary key "source IP address". Since the network measurement data is collected by time, the timestamps in this embodiment are already ordered and unique, so this step can be skipped. For the source IP address, the source IP addresses can be arranged in a column, and data with the same source IP address can be merged into one network measurement data. For example... Figure 3 In the data, the three network measurement data entries numbered 2, 5, and 9 have the same source IP address. These three network measurement data entries will be classified into the same category of data with their source IP address 192.168.1.4 as the key.

[0046] Step 2B: Generate index entries corresponding to the primary key and secondary key respectively. In the primary key, each network measurement data point corresponds to an index entry in the main tree, i.e. Figure 3 Each data entry corresponds to a main tree index entry. The key of each index entry in the main tree is a timestamp, and the value is the complete network measurement data. For example, the network measurement data with data number 1 corresponds to the following main tree index entry:<key=2100,value=(2100,192.168.1.8,192.168.1.11,3,3163)> The log structure corresponding to the secondary key is merged into a sub-tree. To save space and ensure search efficiency, the sub-tree index entries do not contain all data, only copies of the most popular data. Therefore, it is necessary to first calculate the popularity of each type of network measurement data after categorizing by the secondary key, and then store copies of the more popular network measurement data into the sub-tree. The key of the index entry in the sub-tree is the data field to be searched; in this embodiment, it is the source IP address, and the value is a copy of all network measurement data indexed by the search data field. For example, Figure 4 This displays the contents of two secondary key index entries with keys 192.168.1.4 and 192.168.1.12 respectively. The popularity calculation method involves weighting the number of packets, total size, total number of entries, or other fields in the data according to user-defined thresholds. The specific steps for popularity calculation and index entry selection are as follows:

[0047] Step 2B1: Calculate the popularity value for each type of data according to the weights and calculation methods preset by the user. In this embodiment, the sum of the total number of packages and the number of data entries for a given data type will be used as the popularity value for that type. For example... Figure 3 , Figure 4 The popularity value of the class corresponding to the source IP address 192.168.1.4 is 16+3=19; the popularity value of the class corresponding to 192.168.1.12 is 10+3=13.

[0048] Step 2 (B2): Based on a user-preset threshold, select data classes with a popularity value greater than the threshold to create index entries in the subtree. A lower threshold results in more subtree index entries but faster searches; a higher threshold results in less subtree index entries but lower query performance. For example, in this embodiment, if the threshold is set to 12, then data classes 192.168.1.4 and 192.168.1.12 will be indexed in the subtree.

[0049] Step 2 (B3) updates the Bloom filter corresponding to the subtree. Since the subtree index only contains a small portion of the data, it cannot respond to all queries. However, the log structure merging tree itself cannot determine which data is not in the tree, requiring additional records to quickly confirm which data can be found in the subtree, avoiding the overhead of useless queries. Therefore, in this invention, each subtree is accompanied by a Bloom filter to record all data in the subtree. The Bloom filter supports two operations: insertion and query. The insertion operation inserts a key into the set maintained by the Bloom filter, and the query operation checks whether a key exists in the set maintained by the Bloom filter. Therefore, after adding index entries to the subtree in Step 2 (B2), this step requires inserting the keys of the added index entries into the Bloom filter.

[0050] For network measurement data with indexed entries, users can choose to query using either the primary key or the secondary key. Figure 6 Here is a flowchart of the query process, with the specific steps as follows:

[0051] Step 3A: Determine whether the data field queried by the user is a primary key. If it is a primary key, directly read the corresponding index entry from the main tree, obtain the network measurement data corresponding to the keyword through the query interface of the log structure merging tree, and return it to the user.

[0052] Step 3B: If the data field queried by the user is a secondary key, the Bloom filter of the secondary tree should be accessed first. If the result of querying the keyword through the Bloom filter is that the keyword is not in the corresponding secondary tree, then there is no need to query the secondary tree. Instead, all network measurement data in the main tree should be traversed to find the corresponding index entry for the keyword to be queried. Because the main tree stores all network measurement data, the query can return the correct result. Querying the secondary key in the main tree is less efficient because there is no corresponding index, but since hot data is stored in the secondary tree, the probability of such a query occurring is low and it does not affect the overall system efficiency.

[0053] Step 3C: If the user's query keyword appears in the Bloom filter, the query interface of the log structure merge tree is used to query network measurement data from the secondary tree. Most queries targeting secondary keys can be completed quickly using this method. However, because Bloom filters have a certain false positive rate, the query from the secondary tree may not find any network measurement data. In this case, it is necessary to return to the main tree to query the required network measurement data.

[0054] Step 4A: Update the popularity value based on the retrieved network measurement data. Specifically, this involves adding the impact of query frequency on the popularity calculation method described above. For example, in this embodiment, if network measurement data from the source IP address 192.168.1.15 is continuously queried (in... Figure 3In the example, there is only one data entry with the number 15. After 10 queries, the popularity value of this entry will change from the original popularity value of 3 to 10+3=13, which exceeds the threshold set by the user.

[0055] Step 4B: If the query is about the secondary key and the queried keyword is not in the secondary tree, then fall back to the network measurement data retrieved from the main tree. If the updated popularity value of the queried network measurement data exceeds the user-defined threshold, create new index entries for these network measurement data entries with popularity values ​​exceeding the threshold; that is, create new copies of the data in the secondary tree. The method for creating new copies is the same as in the step of creating a dynamic multi-replica index.

[0056] This invention improves the storage and query efficiency of network measurement data by using the above-mentioned method for collecting, calculating, indexing, and storing network measurement data based on dynamic multi-replica indexes. At the same time, it reduces the index space overhead caused by querying different fields when storing network measurement data in a key-value database based on a log structure merging tree, thus solving the storage bottleneck of existing network measurement data.

[0057] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0058] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for managing network measurement data based on a dynamic multi-replica index, comprising the following steps: Step 1: Acquire network measurement data through the measurement devices deployed on the network nodes and send it to the control plane nodes; Step 2: Based on the data fields that the user needs to query, create or update the dynamic multi-replica index: The dynamic multi-replica index includes multiple log structure merge trees, one of which is the master tree and the remaining log structure merge trees are slave trees; Set one of the data fields as the primary key, and set one or more of the remaining data fields as secondary keys, with each secondary key corresponding to a sub-tree; Each network measurement data entry is stored as a key-value pair in each index entry of the main tree. Hotspot data exists within the network measurement data in the main tree; a copy of each hotspot data entry is stored as a key-value pair in each index entry of the secondary tree. The key of each index entry in the main tree is the primary key, and the value is the network measurement data corresponding to the primary key. The key of each index entry in the secondary tree is the secondary key corresponding to that secondary tree, and the value is the network measurement data corresponding to the secondary key. The network measurement data is stored in the log structure merge tree corresponding to the dynamic multi-replica index. Step 3: Based on the user's query request, query the network measurement data in the dynamic multi-replica index and return the retrieved network measurement data to the user; Step 4: Perform popularity statistics on the retrieved network measurement data and update the dynamic multi-replica index based on the popularity statistics.

2. The network measurement data management method based on dynamic multi-replica index according to claim 1, characterized in that, Each network measurement data entry includes the following data fields: timestamp, source IP address, destination IP address, number of network packets, and total size; where the source IP address and destination IP address are either IPv4 or IPv6 addresses.

3. The network measurement data management method based on dynamic multi-replica index according to claim 1, characterized in that: The network nodes in step one include routers and switches; the network measurement data obtained at the network nodes is transmitted to the control plane nodes through independent network connections.

4. The network measurement data management method based on dynamic multi-replica index according to claim 1, characterized in that: Each subtree corresponds to a Bloom filter; when storing a copy of each hot data point as a key-value pair in the subtree's index entry, the key of the added index entry is inserted into the Bloom filter corresponding to that subtree.

5. The network measurement data management method based on dynamic multi-replica index according to claim 1 or 4, characterized in that: The network measurement data in the main tree contains hotspot data statistically obtained based on each secondary key. The hotspot data obtained based on the secondary key M is stored in the index entry of the sub-tree corresponding to the secondary key M. When statistically analyzing hotspot data, all network measurement data are classified according to the secondary key M, and the number of network measurement data in each category is counted. Based on the hotspot data threshold set by the user, the network measurement data of each category with a number of data greater than the hotspot data threshold is regarded as hotspot data of the secondary key M, and the network measurement data of each category with a number of data less than or equal to the hotspot data threshold is regarded as non-hotspot data of the secondary key M.

6. The network measurement data management method based on dynamic multi-replica index according to claim 4, characterized in that: Step three, which involves retrieving network measurement data from the dynamic multi-replica index based on the user's query request and returning the retrieved network measurement data to the user, includes the following steps: Step 3A: When a user initiates a query request, the user enters the keyword P in data field N. The Bloom filter corresponding to data field N is used to confirm whether the keyword P to be searched exists in the subtree Q corresponding to data field N. If so, the user queries the subtree Q with the keyword P as the key and returns the value of the index entry corresponding to the keyword as the network measurement data obtained from the query to the user. If not, proceed to Step 3B. Step 3B: Scan the entire main tree with the keyword P as the key, and send the value of the index entry corresponding to the keyword back to the user as the network measurement data obtained from the query; Step 3C: Based on the user's query request, update the hot data obtained by classifying based on the secondary key; if the non-hot data corresponding to the secondary key M becomes hot data, then create an index entry for the newly added hot data in the subtree corresponding to the secondary key M.

7. The network measurement data management method based on dynamic multi-replica index according to claim 6, characterized in that: If the Bloom filter corresponding to data field N contains the keyword P, but no results are found when querying tree Q with the keyword P as the key, then the entire main tree is scanned with the keyword P as the key, and the value of the index entry corresponding to the keyword is sent back to the user as the network measurement data obtained from the query.

Images