A fast classification and identification method for Internet traffic based on optimized DFA regular expression matching

Abstract

The present invention discloses a method for rapid classification and identification of Internet traffic based on optimized DFA regular expression matching. The method first collects Internet protocol load feature strings, pre-processes them, and then uses a regular expression matching data structure to store relevant feature information. Secondly, a deterministic finite state machine regular matching startup mechanism based on hash verification quickly determines whether to start regular matching, and then uses a fast matching structure update and failure backtracking mechanism to improve the probability of rapid matching success with subsequent traffic messages and compress the number of matches. The method can quickly achieve feature matching of network traffic in a high-speed network environment by constructing an efficient regular expression matching data structure and matching algorithm, thereby completing the identification and classification of Internet traffic types, and effectively solving the problem that the current method is difficult to process high-speed network traffic.

Description

Technical Field

[0001] The present invention belongs to the technical field of cyberspace security, and in particular relates to a method for quickly classifying and identifying Internet traffic based on optimized DFA regular expression matching. Background Art

[0002] The rapid development of computer networks has brought unimaginable speed and convenience to our lives, making our lives more colorful. Fast, convenient, instant, and everyone's participation have become the typical characteristics of modern networks. However, with the rapid development of the network, many problems have also arisen. With the popularization of networking, this problem is becoming increasingly serious.

[0003] From the perspective of network security, various network attacks emerge in an endless stream, such as zombie networks, computer viruses, computer Trojans, worms, application software vulnerabilities, software backends, etc., which seriously threaten computer network services and information security. Network attacks may cause information leakage, data loss, online payment theft and other losses. Network traffic classification and identification can play a good preventive role in this. First, classify and identify network traffic, limit network traffic of known malicious attacks, and avoid network intrusion. Secondly, establish a normal network traffic model, and find anomalies by comparing unknown network data with normal network traffic models, and then focus on abnormal traffic. The classification and identification of network traffic can accurately realize intrusion detection and anomaly detection, and can also shield suspicious network traffic.

[0004] Among existing technologies, regular expressions are widely used in network traffic identification due to their powerful representation capabilities. They can express complex matching rules in a simple way, making the load-based matching method highly accurate. However, it is precisely because of the precise matching characteristics of regular expressions that the recognition speed is relatively slow. When network traffic surges, traditional regular expression-based matching methods often consume a lot of computing resources and have high requirements on machine performance, thus limiting their application in high-speed network environments.

[0005] Although the traditional deterministic finite state machine (DFA) regular expression matching method can ensure the correctness of the match, it is often inefficient when processing large amounts of network traffic because it needs to match character by character. Especially when faced with complex regular expressions, the deterministic finite state machine method may generate a large number of state transitions, further reducing the matching efficiency. Summary of the invention

[0006] In order to solve the problems existing in the prior art, the present invention proposes a method for rapid classification and identification of Internet traffic based on optimized DFA regular expression matching.

[0007] The technical solution of the present invention is as follows:

[0008] A method for rapid classification and identification of Internet traffic based on optimized DFA regular expression matching, comprising:

[0009] Step 1) Collect Internet protocol payload feature strings and pre-process them;

[0010] Step 2) Construct a regular expression matching data structure combining a hash table, a linked list, and a deterministic finite state machine;

[0011] Step 3) According to the regular expression matching data structure, a hash table with multiple entry states is set to store deterministic finite state machine information of the regular expression corresponding to the known Internet protocol payload feature string;

[0012] Step 4) A deterministic finite state machine regular matching startup mechanism based on hash verification is used to extract substrings from the captured unknown Internet traffic message data, and the corresponding hash value of the substring is calculated. Then, the state mapping result in the hash table is searched according to the hash value. If it is mapped to a valid state of the deterministic finite state machine, regular matching is started;

[0013] Step 5) A fast matching structure update and invalidation backtracking mechanism is adopted. Once a byte match is found to be unsuccessful during the string matching process, the match is immediately determined to have failed, and the Internet traffic message data is determined to be an unknown Internet protocol. The matching pointer is retraced to the position where the regular match was successfully started last time, and the pointer continues to move backward; if the regular expression of a certain Internet protocol is successfully matched, the Internet traffic message data is determined to be the corresponding Internet protocol, and the regular expression matching data structure is updated, and the linked list element corresponding to the string is moved to the first position of the entire linked list.

[0014] Furthermore, the Internet protocol payload characteristic character string collected in step 1) includes different types of Internet protocol payload characteristic character strings, covering the protocol characteristic field formats and contents above the transport layer.

[0015] Furthermore, the preprocessing process in step 1) includes data cleaning, data standardization and data labeling.

[0016] Furthermore, in step 2), the regular expression matching data structure uses a hash method to achieve fast access to the finite state machine. The specific construction method includes:

[0017] Step 2-1) Read according to the regular expression of the known Internet protocol, select the first three bytes as the data source for calculating the hash value, and insert the string with the same hash value calculation result into the linked list in the form of each entry in the hash table;

[0018] Step 2-2) Each element in the linked list has a pointer to a deterministic finite state machine, which is associated with one or more Internet protocols. The pointer can be used to match the deterministic state machine, so that the field to be matched is output to be a certain Internet protocol or does not match all Internet protocols.

[0019] Furthermore, the deterministic finite state machine information storage process in step 3) specifically includes:

[0020] Step 3-1) Read the regular expressions of known Internet protocol payload characteristic strings one by one, select the first three characters of the regular expression, and use the hash algorithm to calculate the hash value of the string, find the corresponding linked list in the hash table, create a linked list element for the string, and insert it to the end of the linked list;

[0021] Step 3-2) Create an associated deterministic finite state machine for the newly inserted linked list element. The starting state of the deterministic finite state machine is a state in which the string is successfully matched. The construction of the remaining states and state transitions of the deterministic finite state machine is completed in a traditional manner.

[0022] Furthermore, the deterministic finite state machine regular matching startup mechanism based on hash verification in step 4) specifically includes:

[0023] Step 4-1) Read the corresponding message of the captured unknown traffic data, extract the application layer data of the message, then read three characters in the order of the application layer data, calculate the hash value of the string, and determine whether there is a linked list at the corresponding position of the hash table. If there is a linked list, match the strings of the linked list elements one by one to see if they are equal to the string. If they are equal, start regular matching;

[0024] Step 4-2) When the regular expression matching starts, it first enters the starting state of the deterministic finite state machine corresponding to the linked list element, and reads the data content of the message application layer data after the string character by character, and then performs regular expression matching of the corresponding deterministic finite state machine.

[0025] Furthermore, the fast matching structure update and failure backtracking mechanism in step 5) specifically includes:

[0026] Step 5-1) When using the regular expression matching data structure to classify Internet traffic, if the deterministic finite state machine regular matching start requirements are met, the application layer data is read character by character, and the corresponding deterministic finite state machine is used to match the data. If a character match fails, the matching process is immediately exited, the Internet traffic message data is determined as an unknown Internet protocol, and the matching string is backtracked to the position of the string that triggered the deterministic finite state machine regular matching start, and then the content of the subsequent messages is read one by one;

[0027] Step 5-2) If the application layer data successfully matches the regular expression of a certain Internet protocol, the message is determined to be the corresponding Internet protocol and the regular expression matching data structure is updated, and the linked list element corresponding to the string is moved to the first position of the entire linked list.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] In view of the current problems of low efficiency and high storage space requirements for Internet traffic matching, the present invention proposes a method for rapid classification and identification of Internet traffic based on optimized DFA regular expression matching. The present invention first collects Internet protocol payload feature strings, pre-processes them, and then uses a regular expression matching data structure to store relevant feature information. Secondly, a deterministic finite state machine regular matching startup mechanism based on hash verification quickly determines whether to start regular matching, and then uses a fast matching structure update and failure backtracking mechanism to increase the probability of successful rapid matching with subsequent traffic messages and compress the number of matches. The method of the present invention can quickly achieve feature matching of network traffic in a high-speed network environment by constructing an efficient regular expression matching data structure and matching algorithm, thereby completing the identification and classification of Internet traffic types, and effectively solving the problem that current methods are difficult to process for high-speed network traffic. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of a regular expression matching data structure in an embodiment;

[0031] Figure 2 This is a flow chart of a method for rapid classification and identification of Internet traffic based on optimized DFA regular expression matching in an embodiment. DETAILED DESCRIPTION

[0032] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent modifications of the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0033] Example

[0034] The invention discloses a method for quickly classifying and identifying Internet traffic based on optimizing DFA regular expression matching, comprising:

[0035] Step 1) Collect Internet protocol payload feature strings and pre-process them;

[0036] Step 2) Construct a regular expression matching data structure combining a hash table, a linked list, and a deterministic finite state machine;

[0037] Step 3) According to the regular expression matching data structure, a hash table with multiple entry states is set to store deterministic finite state machine information of the regular expression corresponding to the known Internet protocol payload feature string;

[0038] Step 4) A deterministic finite state machine regular matching startup mechanism based on hash verification is used to extract substrings from the captured unknown Internet traffic message data, and the corresponding hash value of the substring is calculated. Then, the state mapping result in the hash table is searched according to the hash value. If it is mapped to a valid state of the deterministic finite state machine, regular matching is started;

[0039] Step 5) A fast matching structure update and invalidation backtracking mechanism is adopted. Once a byte match is found to be unsuccessful during the string matching process, the match is immediately determined to have failed, and the Internet traffic message data is determined to be an unknown Internet protocol. The matching pointer is retraced to the position where the regular match was successfully started last time, and the pointer continues to move backward; if the regular expression of a certain Internet protocol is successfully matched, the Internet traffic message data is determined to be the corresponding Internet protocol, and the regular expression matching data structure is updated, and the linked list element corresponding to the string is moved to the first position of the entire linked list.

[0040] Furthermore, the Internet protocol payload characteristic character string collected in step 1) includes different types of Internet protocol payload characteristic character strings, covering the protocol characteristic field formats and contents above the transport layer.

[0041] Furthermore, the preprocessing process in step 1) includes data cleaning, data standardization and data labeling.

[0042] Furthermore, in step 2), the regular expression matches the data structure (for example, Figure 1 As shown in Figure 1, the hash method is used to achieve fast access to the finite state machine, thereby improving the speed and efficiency of subsequent state matching and solving the problem of low efficiency of the traditional DFA method when processing large amounts of traffic. The specific construction method includes:

[0043] Step 2-1) Read according to the regular expression of the known Internet protocol, select the first three bytes as the data source for calculating the hash value, and insert the string with the same hash value calculation result into the linked list in the form of each entry in the hash table;

[0044] Step 2-2) Each element in the linked list has a pointer to a deterministic finite state machine, which is associated with one or more Internet protocols. The pointer can be used to match the deterministic state machine, so that the field to be matched is output to be a certain Internet protocol or does not match all Internet protocols.

[0045] Furthermore, the deterministic finite state machine information storage process in step 3) specifically includes:

[0046] Step 3-1) Read the regular expressions of known Internet protocol payload characteristic strings one by one, select the first three characters of the regular expression, and use the hash algorithm to calculate the hash value of the string, find the corresponding linked list in the hash table, create a linked list element for the string, and insert it to the end of the linked list;

[0047] Step 3-2) Create an associated deterministic finite state machine for the newly inserted linked list element. The starting state of the deterministic finite state machine is a state in which the string is successfully matched. The construction of the remaining states and state transitions of the deterministic finite state machine is completed in a traditional manner.

[0048] Furthermore, the deterministic finite state machine regular matching startup mechanism based on hash verification in step 4) specifically includes:

[0049] Step 4-1) Read the corresponding message of the captured unknown traffic data, extract the application layer data of the message, then read three characters in the order of the application layer data, calculate the hash value of the string, and determine whether there is a linked list at the corresponding position of the hash table. If there is a linked list, match the strings of the linked list elements one by one to see if they are equal to the string. If they are equal, start regular matching;

[0050] Step 4-2) When the regular match is started, first enter the start state of the deterministic finite state machine corresponding to the linked list element, and read the data content of the message application layer data after the string character by character, and then perform regular expression matching of the corresponding deterministic finite state machine. This matching mechanism can quickly determine whether to start regular matching through hash verification, effectively reducing the number of matches.

[0051] Furthermore, the fast matching structure update and failure backtracking mechanism in step 5) specifically includes:

[0052] Step 5-1) When using the regular expression matching data structure to classify Internet traffic, if the deterministic finite state machine regular matching start requirements are met, the application layer data is read character by character, and the corresponding deterministic finite state machine is used to match the data. If a character match fails, the matching process is immediately exited, the Internet traffic message data is determined as an unknown Internet protocol, and the matching string is backtracked to the position of the string that triggered the deterministic finite state machine regular matching start, and then the content of the subsequent messages is read one by one;

[0053] Step 5-2) If the application layer data successfully matches the regular expression of a certain Internet protocol, the message is determined to be the corresponding Internet protocol and the regular expression matching data structure is updated, and the linked list element corresponding to the string is moved to the first position of the entire linked list, thereby increasing the probability of rapid successful matching with subsequent traffic messages and compressing the number of matches.

[0054] In contrast, the Internet traffic rapid classification and identification method based on optimized deterministic finite state machine regular expression matching proposed in the present invention can effectively solve the problems of low efficiency and state space explosion of traditional deterministic finite state machine methods when processing large amounts of traffic, while maintaining accuracy and reducing the demand for resources.

[0055] This example simulates the implementation of the Internet traffic rapid classification and identification method based on optimized DFA regular expression matching of the present invention. The simulated operating environment is an Intel-Windows architecture PC, a PC with a Core eight-core CPU with a hardware main frequency of 2.5GHz or above, a memory ≥8GB, a hard disk of 500GB, and runs the Windows 10 operating system.

[0056] like Figure 2 As shown, it starts from step S101, reads characteristic character strings in the form of Internet protocol regular expressions one by one, and then goes to step S102;

[0057] In step S102, the first three characters of the characteristic character string are extracted, and the hash value of the character string is calculated, and then the process goes to step S103;

[0058] In step S103, according to the calculated hash value, the corresponding position is found in the hash table, and it is checked whether there is an associated linked list, and then the process goes to step S104;

[0059] In step S104, it is determined whether the associated linked list exists, if yes, go to step S105, otherwise go to step S106;

[0060] In step S105, the strings of the elements in the linked list are matched one by one, and then the process goes to step S107;

[0061] In step S106, a linked list is created, the first linked list element is added, and is associated with the DFA of the regular expression of the Internet protocol signature, and then the process goes to step S110;

[0062] In step S107, it is determined whether there is an element in the linked list that successfully matches the string. If yes, it goes to step S108; otherwise, it goes to step S109;

[0063] In step S108, the DFA of the Internet protocol signature in the expression is merged with the DFA associated with the element, and then the process goes to step S110;

[0064] In step S109, a new element is created in the linked list and associated with the DFA of the Internet protocol signature regular expression, and then the process goes to step S110;

[0065] In step S110, it is determined whether all Internet protocols have been read, if so, it goes to step S111, otherwise it goes to step S101;

[0066] In step S111, the Internet traffic message data captured in the network is read, and then the process goes to step S112;

[0067] In step S112, the application layer data content in the message is extracted, and the application layer data content is read character by character after the first three characters, and then the process goes to step S113;

[0068] In step S113, the hash values ​​of the three characters are calculated, and the corresponding positions are found in the hash table according to the hash values, and then the process goes to step S114;

[0069] In step S114, determine whether the corresponding position is associated with the linked list, if yes, go to step S115, otherwise go to step S123;

[0070] In step S115, read each element in the linked list one by one, and then go to step S116;

[0071] In step S116, determine whether the string matches the string stored in the element, if yes, go to step S117, otherwise go to step S118;

[0072] In step S117, the DFA associated with the element is read, and the DFA regular expression matching process is started, and then the process goes to step S120;

[0073] In step S118, it is determined whether the application layer data of the message has been read. If yes, it goes to step S119, otherwise it goes to step S123;

[0074] In step S119, read the last three character strings of the message application layer data, and then go to step S113;

[0075] In step S120, determine whether the DFA matching result is successful, if yes, go to step S121, otherwise go to step S122;

[0076] In step S121, the message is determined to be a corresponding Internet protocol, and then the process goes to step S123;

[0077] In step S122, the message is determined to be an unknown Internet protocol, and then the process goes to step S123;

[0078] In step S123, it is determined whether the analysis process is finished, if so, it goes to step S124, otherwise it goes to step S111;

[0079] In step S124, the analysis is completed and the program is exited.

[0080] In summary, the Internet traffic rapid classification and identification method based on optimized DFA regular expression matching of the present invention can quickly realize feature matching of network traffic in a high-speed network environment by constructing an efficient regular expression matching data structure and matching algorithm, thereby completing the identification and classification of Internet traffic types, and effectively solving the problem that current methods are difficult to process high-speed network traffic.

[0081] The implementation modes of the present invention are described in detail above in conjunction with the embodiments, but the present invention is not limited to the above-mentioned implementation modes. For ordinary technicians in this technical field, after knowing the contents recorded in the present invention, they can make several equivalent changes and substitutions thereto without departing from the principle of the present invention, and these equivalent changes and substitutions should also be regarded as belonging to the protection scope of the present invention.

[0082] The above description is only a specific implementation of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a technician familiar with the technical field within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention.

Claims

1. A method for rapid classification and identification of Internet traffic based on optimized DFA regular expression matching, characterized in that: include: Step 1) Collect Internet protocol payload feature strings and pre-process them; Step 2) constructs a regular expression matching data structure that combines a hash table, a linked list, and a deterministic finite state machine; in step 2), the regular expression matching data structure uses a hash method to achieve fast access to the finite state machine, and the specific construction method includes: Step 2-1) Read according to the regular expression of the known Internet protocol, select the first three bytes as the data source for calculating the hash value, and insert the string with the same hash value calculation result into the linked list in the form of each entry in the hash table; Step 2-2) Each element in the linked list has a pointer to a deterministic finite state machine, which is associated with one or more Internet protocols. The pointer can be used to match the deterministic finite state machine, so that the output field to be matched is a certain Internet protocol or does not match all Internet protocols; Step 3) According to the regular expression matching data structure, a hash table of multiple entry states is set to store the deterministic finite state machine information of the regular expression corresponding to the known Internet protocol payload feature string; the deterministic finite state machine information storage process in step 3) specifically includes: Step 3-1) Read the regular expressions of known Internet protocol payload characteristic strings one by one, select the first three characters of the regular expression, and use the hash algorithm to calculate the hash value of the string, find the corresponding linked list in the hash table, create a linked list element for the string, and insert it to the end of the linked list; Step 3-2) Create an associated deterministic finite state machine for the newly inserted linked list element. The initial state of the deterministic finite state machine is the state of successful string matching. The construction of the remaining states and state transitions of the deterministic finite state machine is completed in a traditional way. Step 4) A deterministic finite state machine regular matching startup mechanism based on hash verification is used to extract substrings from the captured unknown Internet traffic message data, and the corresponding hash value of the substring is calculated. Then, the state mapping result in the hash table is searched according to the hash value. If it is mapped to a valid state of the deterministic finite state machine, regular matching is started; Step 5) A fast matching structure update and invalidation backtracking mechanism is adopted. Once a byte match is found to be unsuccessful during the string matching process, the match is immediately determined to have failed, and the Internet traffic message data is determined to be an unknown Internet protocol. The matching pointer is retraced to the position where the regular match was successfully started last time, and the pointer continues to move backward; if the regular expression of a certain Internet protocol is successfully matched, the Internet traffic message data is determined to be the corresponding Internet protocol, and the regular expression matching data structure is updated, and the linked list element corresponding to the string is moved to the first position of the entire linked list.

2. The method for rapid classification and identification of Internet traffic based on optimized DFA regular expression matching according to claim 1 is characterized in that: The Internet protocol payload characteristic character string collected in step 1) includes different types of Internet protocol payload characteristic character strings, covering the format and content of protocol characteristic fields above the transport layer.

3. The method for rapid classification and identification of Internet traffic based on optimized DFA regular expression matching according to claim 1 is characterized in that: The preprocessing process in step 1) includes data cleaning, data standardization and data labeling.

4. The method for rapid classification and identification of Internet traffic based on optimized DFA regular expression matching according to claim 1 is characterized in that: The deterministic finite state machine regular matching startup mechanism based on hash verification in step 4) specifically includes: Step 4-1) Read the unknown Internet traffic message data, extract the application layer data of the message, then read three characters in the order of the application layer data, calculate the hash value of the string, and determine whether there is a linked list at the corresponding position of the hash table. If there is a linked list, match the strings of the linked list elements one by one to see if they are equal to the string. If they are equal, start regular matching; Step 4-2) When the regular expression matching starts, it first enters the starting state of the deterministic finite state machine corresponding to the linked list element, and reads the data content of the message application layer data after the string character by character, and then performs regular expression matching of the corresponding deterministic finite state machine.

5. The method for rapid classification and identification of Internet traffic based on optimized DFA regular expression matching according to claim 1 is characterized in that: The fast matching structure update and failure backtracking mechanism in step 5) specifically includes: Step 5-1) When using the regular expression matching data structure to classify Internet traffic, if the deterministic finite state machine regular matching start requirements are met, the application layer data is read character by character, and the corresponding deterministic finite state machine is used to match the data. If a character match fails, the matching process is immediately exited, the Internet traffic message data is determined as an unknown Internet protocol, and the matching string is backtracked to the position of the string that triggered the deterministic finite state machine regular matching start, and then the content of the subsequent messages is read one by one; Step 5-2) If the application layer data successfully matches the regular expression of a certain Internet protocol, the message is determined to be the corresponding Internet protocol and the regular expression matching data structure is updated, and the linked list element corresponding to the string is moved to the first position of the entire linked list.

Images