A high-speed network data mass rule searching and matching method

Abstract

The application discloses a high-speed network data mass rule searching and matching method, characterized in that the method comprises the following steps: analyzing received rules, and extracting ip addresses and rule id number information in the rules; adding the extracted ip addresses and rule id number information to a radixtree database; performing matching detection on received high-speed network data packets; and according to a hit rule id, querying a processing action corresponding to the rule id, and performing corresponding processing on the received high-speed network data packets. The technical scheme solves the problems of low performance and wasted memory space of the existing rule matching method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication network, in particular to a high-speed network data large capacity rule retrieval matching method. BACKGROUND

[0002] In the field of information security, network data transmitted at high speed on the communication network needs to be analyzed and processed. For example, IPS / IDS network security equipment can identify and analyze data extraction of different protocols in multiple layers according to different types of processing rules issued by the equipment, find behaviors such as attacks and intrusions, and then perform special processing on data traffic according to the processing method specified in the rules, such as alarm, ignore, discard, etc. The log system can record the relevant information of these behaviors for easy viewing at any time.

[0003] In the well-known Suricata traffic analysis and detection framework, the general method for rules is to parse the issued rules, extract the ip address information in the rules, and store the ip address information in the radixtree database node. The corresponding rule id information is also stored, and the id has an internal number starting from 1 and increasing to the maximum. However, an ip address may be the same in multiple rules, and the ip address of a node in the database may correspond to multiple different ids, so a bit mask method is used to store id number information. According to the total number of rules, a bit mask data string of corresponding size is applied, and the corresponding position 1 represents the id number valid according to the id number, for example, id 1 is the first position 1, id 2 is the second position 1, and so on, id n is the n-th position 1, and the other 0 positions represent invalid id numbers. In this way, each node node has ip address information and such a bit mask data string.

[0004] When network communication data is received later, the source ip address and destination ip address of the data packet are first extracted, and whether there is a matching node data in the radixtree database is queried according to the ip address. If found, the corresponding bit mask data string is taken out. Then the bit mask data strings of the source ip and the destination ip are compared from the first bit, and all the bits are compared to the last bit, and the id number corresponding to the bit that is all 1 is found. The result may not be, may be one, several or many. If the valid rule id number is found, the corresponding rule is found, and the data packet is further processed according to the other provisions in the rule.

[0005] The existing method of using bit mask data string to store corresponding id number has relatively large defects when the number of rules is relatively small, but when high-speed network data analysis and million-level large capacity rule processing are performed:

[0006] 1. Low performance: when high-speed network data analysis is performed, and ten million-level data packets need to be processed per second, if the rule capacity is million-level, the bit mask data string of source ip and destination ip needs to be compared for each data packet, and the first bit needs to be traversed to the last 100 million bit, which needs to be traversed 100 million times to obtain the final effective id number result. If each packet is traversed by million times, ten million packets per second multiplied by million is ten trillion. This scheme is very low in efficiency and basically infeasible.

[0007] 2. Waste of memory space: million-level large capacity rules, each bit mask data string needs 100k bytes of space. Each node in the radix tree database needs this space, even if it only contains one effective rule id number. If the million rules are all scattered and not repeated, each node has only one effective id number, and 100k multiplied by 100 million requires 100G of memory space, plus other memory consumption, which requires very high memory for the system, causing waste of memory space. SUMMARY

[0008] The present application provides a high-speed network data large capacity rule retrieval matching method, which solves the problems of low performance and waste of memory space of the existing rule matching method.

[0009] The embodiment of the present application provides a high-speed network data large capacity rule retrieval matching method, which comprises the following steps:

[0010] S1: analyzing the received rules, and extracting ip address and rule id number information in the rules;

[0011] S2: adding the extracted ip address and rule id number information to the radix tree database; in the radix tree database, each node corresponds to an ip address, and the ip address is used as a key to retrieve the node corresponding to the rule id number, and the rule id number is stored in ascending order according to the array storage mode;

[0012] S3: performing matching detection on the received high-speed network data packet; extracting the source IP address and the destination IP address of the high-speed network data packet, and querying the corresponding node in the radix tree database through the source IP address and the destination IP address; if the corresponding node is not found, the data packet does not need to be processed by rules; if the corresponding node is found, and the rule id values in the two nodes corresponding to the source IP address and the destination IP address are the same, the data packet needs to be processed by rules;

[0013] S4: querying the processing action corresponding to the rule id according to the hit rule id, and performing corresponding processing on the received high-speed network data packet.

[0014] In some embodiments, in step S2, the IP address is taken as the key key, and the node corresponding to the rule id number is searched, specifically including:

[0015] If the corresponding node is not searched, a new node needs to be created;

[0016] If the corresponding node is searched, the rule id number array structure corresponding to the searched node is taken out, and the new rule id number is added to the array structure.

[0017] In some embodiments, the new node is created specifically including: first allocating the space required by the rule id number array structure, filling in the rule id number, and then adding the IP address as the key key and the rule id number array structure as the value to the database.

[0018] In some embodiments, the new rule id number is added to the array structure, specifically including: comparing the new rule id number with the rule id number at the last position of the array structure, if the new rule id number is smaller than the rule id number at the current position in the array structure, the rule id number at the current position in the array structure is moved one position back, then the new rule id number is compared with the rule id number at the previous position, until the new rule id number is larger than the current rule id number, and the new rule id number is added to the position left after the current rule id number.

[0019] In some embodiments, in step S3, the rule id value comparison process in the two node nodes corresponding to the source ip address and the destination ip address specifically includes: first judging which array structure in the rule id number array structure in the two node nodes has less rule id number, using binary method, using each rule id number in the array structure with less number to search each id number in the array structure with more number, and if all can be searched, it is considered that the two array structures have common rule id.

[0020] Compared with the prior art, the beneficial effects of the present application are:

[0021] 1. The bit mask data string of rule id code is changed to an array composed of id numbers, which saves memory space. For a million-level large capacity rule, the original 100G memory may be required, and now only 4M memory is needed, which greatly saves memory space.

[0022] 2. By pre-sorting the rule id number array and storing it in the radix tree database, binary search is used for matching, which can obtain the matching result through a small number of queries, instead of the original 1 million times of loop traversal. The processing performance is exponentially improved, and the network processing speed is qualitatively changed. Millions of data packets per second can match millions of processing rules. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0024] Figure 1 The rule analysis flowchart of the present application;

[0025] Figure 2 The data matching flowchart of the present application;

[0026] The implementation, functional characteristics and advantages of the present application will be further described with reference to the embodiments and the drawings. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application.

[0029] The embodiment provides a high-speed network data mass rule retrieval matching method, which comprises the following steps:

[0030] S1: Analyzing the received rule, extracting the ip address and rule id number information in the rule;

[0031] S2: Adding the extracted ip address and rule id number information to the radixtree database; in the radixtree database, each node corresponds to an ip address, and the ip address is used as a key to search the node corresponding to the rule id number, and the rule id number is stored in ascending order in an array storage mode; that is, the node corresponding to an ip address has multiple rule id numbers.

[0032] In step S2, the ip address is used as a key to search the node corresponding to the rule id number, and the specific steps include:

[0033] If the corresponding node is not searched, a new node needs to be created; the creation of the new node specifically includes: first, allocating the space required by the rule id number array structure, filling in the rule id number, and then adding the ip address as a key and the rule id number array structure as a value to the database.

[0034] If the corresponding node is searched, the rule id number array structure corresponding to the searched node is taken out, and the new rule id number is added to the array structure; the specific steps include: comparing the new rule id number with the rule id number at the last position of the array structure, if the new rule id number is smaller than the rule id number at the current position in the array structure, the rule id number at the current position in the array structure is moved one position back, and then the new rule id number is compared with the rule id number at the previous position, until the new rule id number is larger than the current rule id number, and the new rule id number is added to the position vacated after the current rule id number.

[0035] Here, each rule id number is stored using only 4 bytes, which saves a large amount of memory space. The maximum required memory is changed from 100G to only 4M.

[0036] S3: performing matching detection on the received high-speed network data packet; extracting the source IP address and the destination IP address of the high-speed network data packet, and querying the corresponding node in the radix tree database through the source IP address and the destination IP address; if the corresponding node is not found, the data packet does not need to be processed by rules; if the corresponding node is found, and the rule id values in the two node corresponding to the source IP address and the destination IP address are the same, the data packet needs to be processed by rules;

[0037] In step S3, the rule id value comparison process in the two node corresponding to the source IP address and the destination IP address includes: first, judging which array structure body in the two node has less rule id number, using binary method, using each rule id number in the array structure body with less number to search each id number in the array structure body with more number, if all can be searched, it is considered that the two array structure bodies have common rule id.

[0038] S4: according to the hit rule id, querying the processing action corresponding to the rule id, and processing the received high-speed network data packet.

[0039] It should be noted that the embodiment effectively solves the problem of memory space, and uses an unsigned integer array to store multiple rule id numbers, instead of using bit mask data string, each rule id number is stored by an unsigned integer variable, which occupies 4 bytes, so that 1 million rule id numbers need to occupy space of 1 million times 4 bytes, only 4M bytes of memory space is enough. Moreover, no matter how many id numbers the different node in the radix tree database contains, the total memory space will not change or increase, only corresponding to the total number of rules. 4M saves a huge amount of memory space compared with 100G, and reduces the occupation of system memory resources.

[0040] Further, after 1 million rules are parsed, the rule id number is stored in the radix tree database using an array, and the rule id number in the array is sorted in ascending order at the same time of storage.

[0041] After parsing the received network data packet, the source IP and destination IP addresses are obtained, and then the radix tree database is queried respectively. If there is relevant information, the corresponding node is obtained, and the corresponding rule id number array structure body is obtained from the node. At this point, two arrays are obtained, and it is judged which array has fewer elements. The array is used to query the array with more elements to find the common matching rule id number. Binary search is used, and the precondition of this method is that the values in the array are sequentially arranged. The sorting has been done in the previous step when adding to the radix tree database. The time complexity of binary search is the logarithm of n with a base of 2. From a 1 million array, whether a certain id number matches can be queried at most 20 times, which is the number of the most extreme case of the most queries. For the case of fewer array elements, the result can be queried a few times. Compared with starting from the first position and traversing 100 million times for comparison, the matching efficiency is greatly improved.

[0042] The above is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A method for searching and matching of high-speed network data mass rules, characterized in that, The method comprises the following steps: S1: parsing the received rule, extracting the ip address and rule id number information in the rule; S2: adding the extracted ip address and rule id number information to the radixtree database; in the radixtree database, each node corresponds to an ip address, and the ip address is used as a key to search for the node corresponding to the rule id number, and the rule id number is stored in ascending order according to the array storage mode; S3: matching detection is performed on the received high-speed network data packet; the source ip address and destination ip address of the high-speed network data packet are extracted, and the corresponding node in the radixtree database is searched through the source ip address and destination ip address; if the corresponding node is not found, the data packet does not need to be processed by the rule; if the corresponding node is found, and the rule id values in the two nodes corresponding to the source ip address and destination ip address are the same, the data packet needs to be processed by the rule; S4: according to the hit rule id, the processing action corresponding to the rule id is queried, and the received high-speed network data packet is processed correspondingly; In step S3, the rule id value comparison process in the two nodes corresponding to the source ip address and destination ip address specifically includes: first, judging which array structure body in the rule id number array structure body in the two nodes has fewer rule id numbers, using the binary method, using each rule id number in the array structure body with fewer numbers to search for each id number in the array structure body with more numbers, if all can be searched, it is considered that the two array structure bodies have common rule ids.

2. The method of claim 1, wherein the high-speed network data mass rule search matching method is characterized by, In step S2, the ip address is used as a key to search for the node corresponding to the rule id number, which specifically includes: If the corresponding node is not searched, a new node needs to be created; If the corresponding node is searched, the rule id number array structure body in the searched node is taken out, and the new rule id number is added to the array structure body.

3. The method of claim 2, wherein the high-speed network data mass rule search matching method is characterized by, Creating a new node specifically includes: first, allocating the space required by the rule id number array structure body, filling in the rule id number, and then adding the ip address as the key and the rule id number array structure body as the value to the database.

4. The method of claim 2, wherein the high-speed network data mass rule search matching method is characterized by, Adding the new rule id number to the array structure body specifically includes: comparing the size of the new rule id number with the rule id number at the last position of the array structure body, if the new rule id number is smaller than the rule id number at the current position in the array structure body, the rule id number at the current position in the array structure body moves one position back, and then the new rule id number is compared with the rule id number at the previous position, until the new rule id number is larger than the current rule id number, and the new rule id number is added to the position left after the current rule id number.

Images