Method for fast matching IP address range

Abstract

The application belongs to the technical field of IP address matching, and provides a method for quickly matching IP address range. The method can flexibly and efficiently match IP address range and process large-scale IP matching. The application adopts interval tree to quickly store and retrieve IP range, and adopts hash table to store metadata. An optimized search algorithm is developed to quickly locate target IP node in the interval tree, and the search direction is determined by comparison. Parallel processing technology is introduced to improve processing speed. A dynamic update mechanism is designed to ensure data consistency. A conflict resolution strategy is adopted to optimize the performance of the hash table, and an automatic conflict resolution strategy is set to process overlapping IP range.

Description

Technical Field

[0001] This invention belongs to the field of IP address matching technology, and specifically relates to a method for quickly matching IP address ranges. Background Technology

[0002] With the rapid development of the Internet, IP address management has become increasingly complex. In large-scale networks, the rapid and accurate matching and processing of IP address ranges is crucial for network security and operations. Existing IP address matching methods are often inefficient when processing large amounts of data and struggle to meet real-time requirements.

[0003] Existing technologies for IP address matching often employ a one-to-one comparison method, which is inefficient across large IP address ranges and struggles to cope with dynamically changing network environments. Furthermore, existing methods also have shortcomings when handling overlapping IP address ranges.

[0004] 1. Inefficient: Comparing IP addresses one by one is slow and cannot meet the needs of large-scale networks.

[0005] 2. Lack of flexibility: It is difficult to adapt to changes in network structure and dynamic adjustments of IP addresses.

[0006] 3. High error rate: Prone to errors when processing complex or overlapping IP address ranges.

[0007] 4. Poor scalability: Existing algorithms are difficult to extend to larger-scale network environments. Summary of the Invention

[0008] To address the technical problems of inefficiency and difficulty in handling large-scale IP address ranges in existing IP address matching methods, this invention proposes a fast IP address range matching method, providing a more flexible and efficient IP address range matching method capable of handling large-scale IP address matching.

[0009] The specific plan is as follows:

[0010] A method for quickly matching IP address ranges.

[0011] Step 1: Construct the data structure: The data structure includes a range tree and a hash table; the range tree is used to store a set of IP address ranges; the hash table is used to store the metadata of the IP address range, i.e., the index information, which includes the start IP, end IP, and network mask;

[0012] Step 2: IP address preprocessing: Standardize the input IP addresses and convert them into a uniform format;

[0013] Step 3: Quickly search and locate the target IP node: Set the time complexity of the interval search algorithm to O(log n), use the interval search algorithm to locate the target IP node in the interval tree, find the node that overlaps with the given interval in the interval tree, and determine which interval the IP address to be matched falls into; handle the overlapping IP address range based on the automatic conflict resolution strategy;

[0014] Step 4: Introduce a parallel processing mechanism in the fast search process: perform matching operations on at least one IP address simultaneously; the parallel processing mechanism is as follows: use a bitmap to represent the IP address space, apply the divide-and-conquer strategy to parallel processing, decompose large problems into smaller problems, solve them on different processing units; and utilize multi-threading technology to create a thread pool for dynamic multi-threading management.

[0015] Step 5: Matching and Verification: Use a hash table to locate the specific IP address range and perform matching and verification;

[0016] Step 6: Output Results: Output the matching results, including the network segment to which the IP address belongs and related attribute information;

[0017] Step 7: Introduce a dynamic update mechanism: When the IP address range changes, update the interval tree based on the retrieval tagging technique and update the hash table based on the average time complexity.

[0018] Preferably, the storage method of the interval tree in step 1 is as follows: the network administrator uses the interval tree to maintain a set of IP address ranges, and each IP address range is stored as an interval on a node of the tree; when it is necessary to quickly locate the subnet to which an IP address belongs or to calculate the number of available IP addresses in a certain subnet, the interval tree is used for querying; each node of the interval tree includes: data, interval attributes, and additional information.

[0019] Preferably, the IP address after standardization in step 2 is formatted as follows: it contains four groups of decimal numbers, each group ranging from 0 to 255, and the groups of numbers are separated by periods (.).

[0020] Preferably, the method for quickly searching and locating the target IP node in step 3 is as follows: the search direction is determined by comparing the interval where the given target IP node is located with the left endpoint and the maximum endpoint in the right subtree of the node interval; if the left endpoint of the target interval is less than or equal to the left endpoint of the current node and less than or equal to the maximum endpoint in the right subtree, then the left subtree is searched; otherwise, the right subtree is searched; this process continues until a node overlapping with the target interval is found or the end of the tree is reached, and finally the interval in which the IP address to be matched falls is determined.

[0021] Preferably, in step 4, the parallel processing technique steps are as follows:

[0022] S41: Bitmap representation of IP address space: Define and create a bitmap; convert IP addresses to bitmap indices; IP address translation;

[0023] S42: Implementation of the divide-and-conquer strategy:

[0024] S421 Task Decomposition: Divide the IP address range, break down a large range of IP addresses into smaller ranges, and obtain sub-tasks;

[0025] S422 handles subtasks: performing a matching operation within a certain subrange, checking if the IP address exists;

[0026] S423: Create a thread pool and perform parallel matching: Use Python's `concurrent.futures` module to manage the thread pool; set the maximum number of threads, submit tasks, and collect results; run parallel processing: call the matching function: set the IP addresses to be matched based on the initialized bitmap; perform parallel matching;

[0027] S44: Performance Optimization: Dynamically Adjust Thread Pool Size: Dynamically adjust the number of threads based on system load and task complexity; ensure that each thread handles the same number of tasks to avoid overloading any thread.

[0028] Preferably, step 5 involves using a hash table to locate a specific range of IP addresses and performing matching verification as follows:

[0029] S51: Construct a hash table: Convert the IP address and subnet mask into binary form; apply a logical AND operation to the binary representation of the IP address and subnet mask; determine the length of the network prefix by calculating the number of non-zero bits in the subnet mask; construct the hash key by combining the network prefix length and the result of the AND operation.

[0030] S52: IP Address Translation: Converts the IP address to be queried into the form of a network address and subnet mask for comparison with the hash key;

[0031] S53: Match Verification: Calculate the hash value of the IP address to be queried using a hash function, and look up the corresponding key in the hash table; if a matching key is found, verify whether the IP address does indeed belong to the range, and return the associated value information.

[0032] Preferably, in step 7, the dynamic update mechanism is implemented as follows:

[0033] S71: The interval tree implements interval updates through the lazy marking technique of nodes; the lazy marking technique is as follows: when an interval needs to be updated, the update operation is passed to the corresponding node. If the child nodes of the node have not been updated, the update operation to be performed is recorded in the node mark. When subsequent query or update requests reach the child nodes, the update is completed using the already recorded node mark.

[0034] S72: Update the hash table: When the IP address range changes, use the hash table to perform insertion, deletion and modification of key-value pairs to ensure that the average time complexity of the operation is O(1); periodically check the load factor of the hash table to make the average time complexity of the operation O(1), use open addressing or chaining as a collision resolution strategy, and rehash when it degenerates to O(n).

[0035] Preferably, the automated conflict resolution strategy in step 3 is as follows: perform conflict detection, use a search algorithm to locate whether the IP is duplicated with an existing IP in the MySQL database on the server side, if duplicates are found, remove duplicate and overlapping IPs, store the remaining IPs in the MySQL database, and prompt the interface that there is a duplicate IP: XXX.

[0036] Beneficial effects:

[0037] This invention provides a method for quickly matching IP address ranges. This method offers a more flexible and efficient IP address range matching scheme, effectively addressing the needs of large-scale IP address matching. First, an interval tree is used as the primary data structure, enabling rapid storage and retrieval of IP address ranges. Simultaneously, a hash table is used to store metadata of the IP address range, such as the starting IP, ending IP, and network mask, for fast access. When it is necessary to quickly locate the subnet to which an IP address belongs or to calculate the number of available IP addresses within a subnet, efficient queries can be performed using the interval tree. Third, this invention develops an optimized search algorithm that can quickly locate the node containing the target IP in the interval tree, thereby quickly returning the matching IP address range. Specifically, the search direction is determined by comparing the interval containing the given target IP node with the left endpoint and the maximum endpoint in the right subtree of the node's interval. If the left endpoint of the target interval is less than or equal to the left endpoint of the current node and less than or equal to the maximum endpoint in the right subtree, the left subtree is searched; otherwise, the right subtree is searched. This process continues until a node overlapping with the target interval is found or the end of the tree is reached. This invention effectively locates target IP nodes through interval trees and interval search algorithms, providing fast lookup performance. Fourth, this invention introduces parallel processing technology, allowing simultaneous matching operations on multiple IP addresses, significantly improving processing speed and efficiency. Fifth, this invention designs a dynamic update mechanism that can quickly update the interval tree and hash table when the IP address range changes, ensuring data consistency and accuracy. Sixth, to maintain the performance of the hash table, it adopts conflict resolution strategies such as open addressing or chaining, and performs rehashing when necessary to optimize hash distribution. Seventh, this invention sets up an automated conflict resolution strategy: when processing overlapping IP address ranges, an automated conflict resolution strategy is developed to ensure the correctness and reliability of the matching results.

[0038] This method achieves rapid IP address matching through optimized data structures and search algorithms to meet real-time processing requirements. The combination of interval trees and hash tables reduces the error rate and improves matching accuracy. This method is highly flexible and adaptable, capable of adapting to network environments of varying sizes and handling dynamic changes in network structure and IP address ranges. It exhibits strong scalability, supporting IP address matching in large-scale network environments and horizontally scaling as the network grows. Security was fully considered during the design process, ensuring the security and privacy of IP address data. Furthermore, this method supports complex query operations, including range overlap detection and subnetting. Parallel processing and algorithm optimization improve the efficiency of computational resource utilization and reduce system energy consumption. Attached Figure Description

[0039] Figure 1 Flowchart of a method for quickly matching IP address ranges. Detailed Implementation

[0040] The present invention will be further described below with reference to the embodiments and accompanying drawings.

[0041] like Figure 1 As shown, a method for quickly matching IP address ranges is described.

[0042] Step 1: Construct the data structure: The data structure includes a range tree and a hash table; the range tree is used to store a set of IP address ranges; the hash table is used to store the metadata of the IP address range, i.e., the index information, which includes the start IP, end IP, and network mask;

[0043] Storage Method: In IP address management, range trees can be used to store a set of IP address ranges. For example, a network administrator can use a range tree to maintain a set of IP address segments, with each segment stored as a range on a node in the tree. When it's necessary to quickly locate the subnet to which an IP address belongs or to calculate the number of available IP addresses within a subnet, a range tree can be used for efficient lookups.

[0044] Step 2: IP address preprocessing: Standardize the input IP addresses and convert them into a uniform format;

[0045] Step 3: Perform a fast search: Set the time complexity of the interval search algorithm to O(log n), use the interval search algorithm to locate the target IP node in the interval tree, find the node that overlaps with the given interval in the interval tree, and determine which interval the IP address to be matched falls into.

[0046] Interval search algorithm: The search algorithm commonly used to locate a target IP node in an interval tree is the interval search algorithm. An interval search algorithm is an algorithm that finds nodes in an interval tree that overlap with a given interval. An interval tree is a data structure extended from a red-black tree. Each node contains not only data but also an interval attribute and additional information, such as the maximum value of the endpoints of all intervals in the subtree.

[0047] The interval search algorithm has a time complexity of O(log n) because each iteration eliminates at least half of the search space, thus ensuring logarithmic time efficiency. In applications such as network routing and IP address lookup, interval trees and interval search algorithms can effectively locate target IP nodes and provide fast lookup performance.

[0048] Step 4: Performance Optimization: Introduce a parallel processing mechanism into the fast search process, performing matching operations on multiple IP addresses simultaneously. This parallel processing mechanism includes using bitmaps to represent the IP address space, as IP addresses are fixed-length numbers, suitable for fast matching using bit operations. Furthermore, a divide-and-conquer strategy can be used for parallel processing, breaking down large problems into smaller ones and solving them on different processing units; and multithreading technology is utilized by creating a thread pool for dynamic multithreaded management.

[0049] Step 5: Matching and Verification: Use a hash table to locate the specific IP address range and perform matching and verification;

[0050] Step 6: Output Results: Output the matching results, including the network segment to which the IP address belongs and related attribute information;

[0051] Step 7: Introduce a dynamic update mechanism: When the IP address range changes, update the interval tree and hash table in real time;

[0052] Range trees achieve high efficiency in range updates through lazy propagation of nodes. When a range needs to be updated, the update operation is propagated to the corresponding node and, if necessary, down to child nodes. This method reduces unnecessary node updates, thereby improving the performance of update operations. When IP address ranges change, range trees are used to quickly locate the affected nodes and apply update operations.

[0053] Updating a hash table typically involves inserting, deleting, or modifying key-value pairs. When the range of IP addresses changes, new IP addresses are added to the hash table or old IP addresses are deleted. The design of hash tables ensures that the average time complexity of these operations is O(1), but it can degenerate to O(n) in the worst case. To maintain the performance of hash tables, collision resolution strategies such as open addressing or chaining are used, and rehashing is performed when necessary to optimize the hash distribution.

[0054] Preferably, the storage method for storing a set of IP address ranges in step 1 using the interval tree is as follows: the network administrator uses the interval tree to maintain a set of IP address segments, with each IP address segment stored as an interval on a node of the tree; when it is necessary to quickly locate the subnet to which an IP address belongs or to calculate the number of available IP addresses in a certain subnet, the interval tree is used for querying; each node of the interval tree includes: data, interval attributes, and additional information.

[0055] Preferably, the method for fast searching in step 3 is as follows: the search direction is determined by comparing the interval containing the given target IP node with the left endpoint and the maximum endpoint in the right subtree of the node interval; if the left endpoint of the target interval is less than or equal to the left endpoint of the current node, and less than or equal to the maximum endpoint in the right subtree, then the left subtree is searched; otherwise, the right subtree is searched. This process continues until a node overlapping with the target interval is found or the end of the tree is reached.

[0056] Preferably, in step 4, the parallel processing technique steps are as follows:

[0057] S41: Bitmap representation of IP address space:

[0058] 1.1: Bit Initialization Map: Defines and creates a bitmap;

[0059] 1.2: Setting and Clearing IP Addresses: Converting IP addresses to bitmap indexes;

[0060] 1.3: IP Address Translation: IP to Index;

[0061] S42: Implementation of the divide-and-conquer strategy:

[0062] 2.1 Task decomposition: Divide the IP address range into smaller ranges to obtain sub-tasks;

[0063] 2.2 Processing subtasks: Perform a matching operation within a certain sub-range to check if the IP address exists;

[0064] S43: Multithreading Techniques and Thread Pools

[0065] 3.1 Creating a thread pool: Use Python's `concurrent.futures` module to manage the thread pool; set the maximum number of threads, submit tasks, and collect results;

[0066] 3.2 Parallel Processing: Calling the matching function: Setting the IP addresses to be matched based on the initialized bitmap; Executing parallel matching;

[0067] S44: Performance Optimization

[0068] 1.3 Dynamically adjust the thread pool size: dynamically adjust the number of threads based on the system load and task complexity;

[0069] 1.4 Task allocation strategy: Ensure that each thread handles the same number of tasks to avoid overloading any thread.

[0070] Preferably, in step 7, the dynamic update mechanism is implemented as follows:

[0071] S71: The interval tree implements interval updates through the lazy tagging technique of nodes; when an interval needs to be updated, the update operation is passed to the corresponding node and, if necessary, passed down to the child nodes.

[0072] S72: When the IP address range changes, use a hash table to perform insertion, deletion and modification of key-value pairs to ensure that the average time complexity of the operation is O(1); periodically check the load factor of the hash table to make the average time complexity of the operation O(1), and use open addressing or chaining as a conflict resolution strategy. When it degenerates to O(n), rehashing is performed.

[0073] Automated conflict resolution strategy: When dealing with overlapping IP address ranges, develop automated conflict resolution strategies to ensure the correctness and reliability of matching results.

[0074] Preferably, the automated conflict resolution strategy is as follows: perform conflict detection, use a search algorithm to locate whether the IP is duplicated with an existing IP in the database, if duplicates are found, remove duplicate and overlapping IPs, add the remaining IPs to the database, and prompt the interface that there is a duplicate IP: XXX.

[0075] The above-described specific embodiments are preferred embodiments of the present invention and are not intended to limit the present invention. Any other changes or equivalent substitutions made without departing from the technical solution of the present invention are included within the protection scope of the present invention.

Claims

1. A method for quickly matching IP address ranges, characterized in that, Step 1: Construct the data structure: The data structure includes a range tree and a hash table; the range tree is used to store a set of IP address ranges; the hash table is used to store the metadata of the IP address range, i.e., the index information, which includes the start IP, end IP, and network mask; Step 2: IP address preprocessing: Standardize the input IP addresses and convert them into a uniform format; Step 3: Quickly search and locate the target IP node: Set the time complexity of the interval search algorithm to O(log n), use the interval search algorithm to locate the target IP node in the interval tree, find the node that overlaps with the given interval in the interval tree, and determine which interval the IP address to be matched falls into; handle the overlapping IP address range based on the automatic conflict resolution strategy; Step 4: Introduce a parallel processing mechanism in the fast search process: perform matching operations on at least one IP address simultaneously; the parallel processing mechanism is as follows: use a bitmap to represent the IP address space, apply a divide-and-conquer strategy to parallel processing, decompose large problems into smaller problems, solve them on different processing units; and utilize multi-threading technology to create a thread pool for dynamic multi-threading management. Step 5: Matching and Verification: Use a hash table to locate the specific IP address range and perform matching and verification; Step 6: Output Results: Output the matching results, including the network segment to which the IP address belongs and related attribute information; Step 7: Dynamically update the range tree and hash table: When the IP address range changes, update the range tree based on lazy marking and update the hash table based on average time complexity.

2. The method for quickly matching IP address ranges according to claim 1, characterized in that, The storage method of the interval tree described in step 1 is as follows: The network administrator uses the interval tree to maintain a set of IP address ranges, with each IP address range stored as an interval on a node of the tree; when it is necessary to quickly locate the subnet to which an IP address belongs or to calculate the number of available IP addresses in a certain subnet, the interval tree is used for querying; Each node of the interval tree includes: data, interval attributes, and additional information.

3. A method for quickly matching IP address ranges according to claim 1, characterized in that, In step 2, the standardized IP address format is as follows: it contains four groups of decimal numbers, each group ranging from 0 to 255, and the groups of numbers are separated by a period (.).

4. A method for quickly matching IP address ranges according to claim 1, characterized in that, The method for quickly searching and locating the target IP node in step 3 is to determine the search direction by comparing the interval where the given target IP node is located with the left endpoint and the maximum endpoint in the right subtree of the node interval. If the left endpoint of the target interval is less than or equal to the left endpoint of the current node, and less than or equal to the maximum endpoint in the right subtree, then search the left subtree; Otherwise, search the right subtree; This process continues until a node overlapping with the target interval is found or the end of the tree is reached, ultimately determining which interval the IP address to be matched falls into.

5. A method for quickly matching IP address ranges according to claim 1, characterized in that, In step 4, the parallel processing technique steps are as follows: S41: Bitmap representation of IP address space: Define and create a bitmap; convert IP addresses to bitmap indices; IP address translation; S42: Implementation of the divide-and-conquer strategy: S421 Task Decomposition: Divide the IP address range, break down a large range of IP addresses into smaller ranges, and obtain sub-tasks; S422 handles subtasks: performing a matching operation within a certain subrange, checking if the IP address exists; S423: Create a thread pool and perform parallel matching: Use Python's concurrent.futures module to manage the thread pool; Set the maximum number of threads, submit the task, and collect the results; run parallel processing: call the matching function: set the IP addresses to be matched based on the initialized bitmap; perform parallel matching; S44: Performance Optimization: Dynamically Adjust Thread Pool Size: Dynamically adjust the number of threads based on system load and task complexity; ensure that each thread handles the same number of tasks to avoid overloading any thread.

6. A method for quickly matching IP address ranges according to claim 1, characterized in that, Step 5: Using a hash table to locate a specific range of IP addresses and perform matching verification. S51: Construct a hash table: Convert the IP address and subnet mask into binary form; apply a logical AND operation to the binary representation of the IP address and subnet mask; determine the length of the network prefix by calculating the number of non-zero bits in the subnet mask; construct the hash key by combining the network prefix length and the result of the AND operation. S52: IP Address Translation: Converts the IP address to be queried into the form of a network address and subnet mask for comparison with the hash key; S53: Match Verification: Calculate the hash value of the IP address to be queried using a hash function, and look up the corresponding key in the hash table; if a matching key is found, verify whether the IP address does indeed belong to the range, and return the associated value information.

7. A method for quickly matching IP address ranges according to claim 1, characterized in that, In step 7, the dynamic update mechanism is implemented as follows: S71: The interval tree implements interval updates through the lazy marking technique of nodes; the lazy marking technique is as follows: when an interval needs to be updated, the update operation is passed to the corresponding node. If the child nodes of the node have not been updated, the update operation to be performed is recorded in the node mark. When subsequent query or update requests reach the child nodes, the update is completed using the already recorded node mark. S72: Update the hash table: When the IP address range changes, use the hash table to perform insertion, deletion and modification of key-value pairs to ensure that the average time complexity of the operation is O(1); periodically check the load factor of the hash table to make the average time complexity of the operation O(1), use open addressing or chaining as a collision resolution strategy, and rehash when it degenerates to O(n).

8. A method for quickly matching IP address ranges according to claim 1, characterized in that, The automated conflict resolution strategy described in step 3 is as follows: perform conflict detection, use a search algorithm to locate whether the IP address is duplicated with an existing IP address in the MySQL database on the server side, remove duplicate and overlapping IP addresses if duplicates are found, store the remaining IP addresses in the MySQL database, and display a message on the interface indicating that a duplicate IP address exists: XXX.

Images