Network data packet analysis system

Abstract

The invention relates to the technical field of networks, in particular to a network data packet analysis system, and provides a system for network data packet analysis, which comprises a protocol decoding engine, a graph optimization engine and a graph traversal engine, in a protocol decoding engine, a plain text concise language specification is adopted to describe basic logic and logic combination of formal languages, description of logic combination and complex nested structures is supported, high expression ability and good readability are achieved, the design is concise enough, the manufacturing difficulty of decoding scripts and the total number of codes can be greatly reduced, and the decoding efficiency is improved. In the graph optimization engine and the graph traversal engine, grammar logic of an original graph structure processed by the protocol decoding engine is converted into graph logic by adopting a graph algorithm to realize performance optimization during operation, and flexible analysis of different types of protocols is supported for a multi-graph engine architecture; and a graph structure designed in a classified manner comprises a character string graph, a regular graph and the like, and a standardized graph traversal mechanism is realized by unifying a multi-graph traversal engine.

Description

Technical Field

[0001] This invention relates to the field of network technology, specifically a network packet parsing system. Background Technology

[0002] Existing protocol decoding technologies typically consist of a protocol decoding engine and a protocol decoder. The protocol decoding engine is usually fixed, while the protocol decoder is developed separately according to the actual decoding requirements of different protocols; usually, the development time of the decoding engine of a mature product is negligible compared to the cumulative development time of the decoder.

[0003] like Figure 1 As shown, developing a protocol decoder using the existing process typically requires 5-20 person-days for a complete cycle. If requirements change, the process needs to be repeated multiple times. Manual coding consumes most of the time, resulting in excessively long development cycles and causing other problems.

[0004] 1. Inability to adapt to the rapid growth and changes in protocols.

[0005] The types of network protocols are constantly growing rapidly, and protocol versions may also be updated, but the development cycle of protocol decoders is too long, and they usually only support decoding of common protocols; this often leads to the need to customize a batch of protocol decoders for a project, which results in long project cycles and high costs.

[0006] 2. Unable to meet the needs of different users

[0007] Different users may have completely different needs for decoding results. For the same protocol, some customers want the decoding results to be as detailed as possible, while others have specific field requirements. Decoding more detailed results may prevent the data from being stored in the database. Even if a decoder for a certain protocol has been implemented, it is still necessary to develop one to meet different needs.

[0008] 3. Unable to adapt to specific industry needs

[0009] Security industry clients need to decode specific protocols, but cannot provide information such as the original data packets and protocol format of the protocol; on-site development is required, and the cycle is too long, which cannot meet the timeliness requirements of security countermeasures.

[0010] The entire network packet parsing process consists of two parts: a compilation module and an interpretation and execution module. The compilation module transforms the decoding script into an interpreter, which includes lexical analysis, grammar analysis, and other functions. The interpretation and execution module then traverses and decodes the network packets through the decoder obtained above to obtain the final decoding result.

[0011] Overall, traditional decoder processes have long development cycles, poor scalability, and low decoding performance. The runtime performance of most products that support decoding is not ideal, and popular open-source tools are no exception. Nowadays, network bandwidth is growing rapidly, sometimes requiring real-time processing of traffic of 40Gbps or even higher. At the same time, decoding technology is widely used in other fields, such as providing metadata for big data platforms. When faced with TB or even PB-level data files to be decoded, decoding performance is undoubtedly the most important competitive indicator. Summary of the Invention

[0012] The purpose of this invention is to provide a network data packet parsing system to address the aforementioned problems.

[0013] The technical solution adopted in this invention is as follows: a network packet parsing system, including a compilation module and an interpretation and execution module, wherein the compilation module is used to generate a decoder for network packet parsing, and the compilation module is equipped with a protocol decoding engine and a graph optimization engine;

[0014] The protocol decoding engine is used for interpreter language generation, lexical analysis, and grammar analysis for protocol decoding;

[0015] The graph optimization engine is used for graph optimization and decoder construction.

[0016] The interpretation and execution module is used to decode network data packets generated by the compiler module and output the decoding results. The interpretation and execution module is equipped with a graph traversal engine.

[0017] The graph traversal engine is used for graph traversal and decoding.

[0018] Furthermore, the generation of the interpretation language for the protocol decoding includes the following steps:

[0019] Read the protocol specifications of network data packets, describe the protocol specifications of network data packets in a formal language, and abstract the logic of the formal language to obtain basic logical units;

[0020] Define the specification of the interpreted language used to describe the basic logic units, set the lexical and grammatical rules for the decoding protocol script, and perform lexical and grammatical parsing.

[0021] The interpretation language specification is encapsulated in the network protocol decoding engine, and the engine architecture is unified.

[0022] Furthermore, in the lexical parsing, a character type mapping table is first established to classify and map the characters in the decoding protocol script. Then, the input stream is traversed in character order, and the characters are processed according to their types. The identified lexical units are encapsulated as Token objects, and each Token contains a start position, an end position, a type identifier, and an associated semantic action.

[0023] The grammar parsing uses a top-down recursive descent parsing method to parse the token sequence according to predefined grammar rules.

[0024] Furthermore, the lexical method includes the composition and writing method of words, sentences, and segments in the decoding protocol script. The words are composed of one or more consecutive characters, including special characters, string constants, numeric constants, regular expression constants, keywords, and identifiers.

[0025] The special characters consist of one or two consecutive punctuation marks and are used to segment words, construct words, construct sentences, construct semantics, and act as operators;

[0026] The regular expression constant is used to define complex words, and the syntax is defined as regex(regexExpr);

[0027] The keywords are reserved strings used for guiding words, data types, decoding methods, and special references;

[0028] The identifier is used to identify the name of a user-defined item, including identifying field names, variable names, and grammar names;

[0029] The sentence consists of multiple words and is a linguistic unit used to express a complete semantic meaning. The sentence includes annotation statements and definition statements.

[0030] The comment statements begin with # and end with \n;

[0031] The definition statement begins with a statement definition introductory word and ends with a semicolon, and includes the definition of the decoding field, the definition of the grammar, the definition of the root grammar, the definition of the external logic, and the definition of the decoding mode.

[0032] The segment contains one or more definition statements of the same kind, and the segments include, in order, a variable definition segment, a grammar definition segment, a root grammar definition segment, an external logic definition segment, and a decoding mode definition segment;

[0033] The variable definition section includes the decoded field definition statement and the int variable definition statement;

[0034] The grammar definition section includes at least one line of grammar definition statements.

[0035] Furthermore, the graph optimization includes the following steps:

[0036] Obtain the original graph structure from the network protocol decoding engine;

[0037] The original graph structure is classified by a graph algorithm engine, and the graph structure type is detected to select an optimization method, perform constraint checks, and generate a decoder.

[0038] The graph traversal uses a graph traversal engine to classify the original graph structure, performs state transitions on the graph, and outputs the decoding result at the receiving state of the graph.

[0039] Furthermore, the graph structure types include structure graphs, string graphs, regular graphs, sequence graphs, branch graphs, loop graphs, context graphs, decision graphs, and trie graphs;

[0040] Structure diagrams are used for fast decoding of fixed-length data structures;

[0041] The string graph is used for string matching;

[0042] The regular graph is used for deterministic finite automata and efficient pattern matching;

[0043] The sequence diagram is used to execute multiple subgraphs sequentially;

[0044] The branch graph is used to traverse each subgraph sequentially, back off fields and operations with side effects on failures, and continue traversing the next subgraph; if successful, it returns directly.

[0045] The cycle graph is used to support conditional loops and maximum number of iterations;

[0046] The context association graph is used to associate the decoded values ​​from the preceding text and determine which path to traverse based on the values.

[0047] The decision graph is based on a state machine to implement complex conditional branching logic;

[0048] The trie graph is used for efficient multi-pattern string matching.

[0049] Furthermore, the optimization methods include trie optimization, serial optimization, parallel optimization, and recursive optimization;

[0050] When the detection results of the graph structure type involve string branches, a trie optimization is performed by constructing a TireGraph.

[0051] When the detection results of the graph structure type involve sequential execution, serial optimization is performed through sequence merging.

[0052] When the detection results of the graph structure type involve parallel branches, parallel optimization is performed by merging branches;

[0053] When the detection results of the graph structure type involve nested structures, recursive optimization is performed through subgraph optimization.

[0054] Furthermore, the concatenation optimization refers to merging graph nodes that are executed sequentially and have no intermediate side effects into a single operation. Concatenation optimization must simultaneously satisfy the following conditions: the two graph nodes must be in a sequential execution relationship, neither graph has a decoding field, neither graph has a runtime expansion length of the grammar, and the graph type supports the merging operation.

[0055] The series optimization includes series connection and series merging;

[0056] The series connection refers to connecting two graphs together to form a new graph;

[0057] The term "serial merging" refers to the ability of the second graph to be merged into the first graph.

[0058] Graph structure types suitable for concatenation optimization include string and string graphs, regular graph and string graphs, regular graph and regular graphs, sequence graphs and sequence graphs;

[0059] The cascade optimization includes the following steps:

[0060] S1. Optimize the subgraphs of the sequence graph sequentially;

[0061] S2. Connect the previously optimized graph to the currently optimized subgraph;

[0062] S3. Connection failed, return to S1 to continue optimizing the next subgraph;

[0063] S4. The concatenation is successful. The newly concatenated graph is then concatenated and merged with the remaining subgraphs.

[0064] S5. After optimization, if the number of subgraphs decreases, replace the original subgraphs; if the number of subgraphs is equal to the original number of subgraphs, but attributes have been merged, also replace the original subgraphs.

[0065] The parallel optimization refers to merging multiple branches with string prefixes into a decision graph structure by combining the subgraphs of the branch graph. Parallel optimization must simultaneously satisfy the following conditions: it must be in a branch graph, the number of subgraphs of the branch graph must be greater than or equal to 1, the subgraphs of the branch graph are a sequence, and the first subgraph of the sequence is one of the string graph, regular graph, or decision graph.

[0066] The parallel optimization includes parallel connection and parallel combination;

[0067] The parallel connection refers to connecting two graphs together to form a new graph;

[0068] The term "joint union" refers to the fact that the second graph can be merged into the first graph.

[0069] Graph structure types suitable for parallel optimization include string graphs and string graphs, string graphs and regular graphs, string graphs and decision graphs, regular graphs and decision graphs, and branch graphs and branch graphs.

[0070] The parallel optimization includes the following steps:

[0071] A1. Traverse the subgraphs of the branch graph and optimize accordingly;

[0072] A2. If the optimized subgraph set is empty, add the optimized subgraph to the set and return to A1; if the optimized subgraph set is not empty, iterate through the set and merge the subgraphs.

[0073] A3. Union and Merge: If the graph in the set is a decision graph, then the optimized subgraphs in A2 are unioned and merged into the decision graph. If the merge is successful, it means that the current subgraph has been successfully optimized. Then return to A1 to continue optimizing the remaining subgraphs.

[0074] A4. Parallel connection: If the graph in the set is not a decision graph, then connect this graph in parallel with the optimized graph in A2 to create a new graph. If successful, replace the current graph; otherwise, return to A3 to continue.

[0075] A5. After optimization, if the number of subgraphs decreases, replace the original subgraphs.

[0076] Furthermore, the graph traversal engine starts from the root graph and performs traversal decoding from top to bottom according to the characteristics of the graph structure type, including single-packet mode decoding and stream mode decoding.

[0077] The single-packet mode decoding is for stateless single-packet protocols such as UDP and TCP, and the state is cleared and reset each time it is decoded.

[0078] The streaming mode decoding is designed for protocols that require cross-packet transmission, meaning protocols that cannot be fully decoded because a single packet cannot be transmitted completely. When data is insufficient, the relevant state needs to be saved in the decoding context. When the next decoded data arrives, the state is restored to the position of the last traversal and the traversal continues.

[0079] Furthermore, the graph traversal engine includes single-packet mode decoding and streaming mode decoding;

[0080] The single-packet mode decoding is used for stateless single-packet protocols. The state is cleared and reset each time it is decoded, and the state only changes within a single packet.

[0081] The streaming mode decoding is used for protocols that require cross-packet transmission. If a single packet cannot be transmitted completely, decoding cannot be completed. When there is insufficient data, the state is saved in the decoding context. When the next decoded data arrives, the state is restored to the position of the last traversal and the traversal continues.

[0082] The beneficial effects of the present invention include at least one of the following;

[0083] 1. A system for network packet parsing is provided, which includes a protocol decoding engine, a graph optimization engine, and a graph traversal engine. The protocol decoding engine uses a concise plain text language specification to describe the basic logic of the formal language and the combination of logic, supports the description of logical combinations and complex nested structures, has strong expressive power and good readability, and is sufficiently concise in design. The difficulty of creating and reading decoding scripts and the total amount of code will be greatly reduced, and the development cycle will be greatly shortened. In the graph optimization engine and the graph traversal engine, the original graph structure processed by the protocol decoding engine is converted into graph logic using graph algorithms to achieve runtime performance optimization. For the multi-graph engine architecture, it supports flexible parsing of different types of protocols, and the categorized graph structures include string graphs, regular expression graphs, sequence graphs, branch graphs, and loop graphs. The unified multi-graph traversal engine implements a standardized graph traversal mechanism.

[0084] 2. Simultaneously, in protocol decoding, each network protocol is treated as a formal language, abstracted to describe most protocols, and a concise interpretation language specification is designed to describe the basic logic and combinations of formal languages. Plain text protocol decoding scripts replace traditional hard-coded decoders, and a complete three-level grammar system of words, sentences, and segments is constructed, supporting rich grammar definitions: structure grammar, general grammar, context-related grammar, etc., and complex decoding logic expression: supporting condition judgment, loop control, variable operation, etc.

[0085] 3. Decoders generated using interpreted languages ​​can solve the problem of rapid protocol growth and changes, reducing decoder development time to 1 / 3 to 1 / 4 of the original time, greatly improving development speed, reducing the technical requirements for developers, eliminating the need for qualifications such as C++, and the number of decoders developed in six months exceeds the total of the past few years, effectively coping with the rapid growth of protocols.

[0086] 4. Graph compilation optimization can significantly reduce the number of graphs generated by the script-based Chinese decoding method. Without compromising correctness, performance can be greatly improved by reducing memory accesses, cache misses, and function jumps. This makes the decoding script less dependent on the skill level of the decoding script developer, thus greatly enhancing decoding performance. Graph optimization starts from the root graph and proceeds top-down, optimizing subgraphs. The most important optimizations, namely chaining and paralleling, are described here. Attached Figure Description

[0087] Figure 1 Flowchart for developing a decoder for an existing protocol;

[0088] Figure 2 This is a flowchart of a network packet parsing system;

[0089] Figure 3 A diagram showing the relationship between the protocol decoding engine, graph optimization algorithm, and graph traversal algorithm.

[0090] Figure 4 Example diagrams for classifying graph types;

[0091] Figure 5 Optimize the flowchart for the diagram;

[0092] Figure 6 Graph traversal flowchart;

[0093] Figure 7 Here are some example diagrams showing the process before and after cascade optimization;

[0094] Figure 8 Example diagrams before and after another series optimization;

[0095] Figure 9 Example diagram before parallel optimization;

[0096] Figure 10 This is an example diagram after parallel optimization;

[0097] Figure 11 Example diagram of group field;

[0098] Figure 12 Create an example diagram for the TCP script;

[0099] Figure 13 For TCP protocol data streams;

[0100] Figure 14 This is the result of decoding the TCP protocol data stream;

[0101] Figure 15 This is the related decoding grammar graph above;

[0102] Figure 16 For TCP protocol data streams with Options;

[0103] Figure 17 This is the decoding result of a TCP protocol data stream with Options.

[0104] Figure 18 This is a zero-copy flowchart;

[0105] Figure 19 This is a diagram illustrating memory management.

[0106] Figure 20 This is a schematic diagram of a network packet parsing device. Detailed Implementation

[0107] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described in the accompanying drawings can generally be arranged and designed in various different configurations.

[0108] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0109] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined with each other.

[0110] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0111] like Figure 2 and Figure 3 As shown, a network packet parsing system includes a compilation module and an interpretation and execution module. The compilation module is used to generate a decoder for network packet parsing, and the compilation module is equipped with a protocol decoding engine and a graph optimization engine.

[0112] The protocol decoding engine is used for interpreter language generation, lexical analysis, and grammar analysis for protocol decoding;

[0113] The graph optimization engine is used for graph optimization and decoder construction.

[0114] The interpretation and execution module is used to decode network data packets generated by the compiler module and output the decoding results. The interpretation and execution module is equipped with a graph traversal engine.

[0115] The graph traversal engine is used for graph traversal and decoding.

[0116] The purpose of this design is to provide a system for network packet parsing, which includes a protocol decoding engine, a graph optimization engine, and a graph traversal engine. The protocol decoding engine uses a concise plain text language specification to describe the basic logic of the formal language and the combination of logic, supporting the description of logical combinations and complex nested structures. It has strong expressive power and good readability, and its design is sufficiently concise. The difficulty of creating and reading decoding scripts, as well as the total amount of code, will be significantly reduced, and the development cycle will be greatly shortened. In the graph optimization and graph traversal engines, the original graph structure processed by the protocol decoding engine is transformed into graph logic using graph algorithms to achieve runtime performance optimization. For the multi-graph engine architecture, it supports flexible parsing of different types of protocols, and the categorized graph structures include string graphs, regular expression graphs, sequence graphs, branch graphs, and loop graphs. A unified multi-graph traversal engine implements a standardized graph traversal mechanism.

[0117] Meanwhile, in this embodiment, as Figure 4 As shown, graph structure types include structure graphs, string graphs, regular graphs, sequence graphs, branch graphs, loop graphs, context graphs, decision graphs, and trie graphs, among which structure graphs are used for fast decoding of fixed-length data structures;

[0118] The string graph is used for string matching;

[0119] The regular graph is used for deterministic finite automata and efficient pattern matching;

[0120] The sequence diagram is used to execute multiple subgraphs sequentially;

[0121] The branch graph is used to traverse each subgraph sequentially, back off fields and operations with side effects on failures, and continue traversing the next subgraph; if successful, it returns directly.

[0122] The cycle graph is used to support conditional loops and maximum number of iterations;

[0123] The context association graph is used to associate the decoded values ​​from the preceding text and determine which path to traverse based on the values.

[0124] The decision graph is based on a state machine to implement complex conditional branching logic;

[0125] The trie graph is used for efficient multi-pattern string matching.

[0126] like Figure 5 As shown, the optimization methods include trie optimization, serial optimization, parallel optimization, and recursive optimization;

[0127] When the detection results of the graph structure type involve string branches, a trie optimization is performed by constructing a TireGraph.

[0128] When the detection results of the graph structure type involve sequential execution, serial optimization is performed through sequence merging.

[0129] When the detection results of the graph structure type involve parallel branches, parallel optimization is performed by merging branches;

[0130] When the detection results of the graph structure type involve nested structures, recursive optimization is performed through subgraph optimization.

[0131] The purpose of this design is to significantly reduce the number of graphs generated by the script-based Chinese method through compilation optimization. Without affecting correctness, performance can be greatly improved by reducing memory accesses, cache misses, and function jumps. This makes the decoding script no longer dependent on the skill level of the decoding script developer, thus greatly improving decoding performance. After these optimizations, a complex and large graph can be transformed into an equivalent smaller graph, which can significantly improve traversal performance through algorithms.

[0132] Serial optimization and parallel optimization are the two most important aspects of the entire graph optimization process. Serial optimization refers to merging sequentially executed graph nodes without intermediate side effects into a single operation, thereby reducing the overhead of graph traversal. Serial optimization is divided into serial connection and serial merging. Serial connection refers to connecting two graphs together to form a new graph; serial merging refers to merging the second graph into the first graph.

[0133] Graph structure types suitable for concatenation optimization include string and string graphs, regular graph and string graphs, regular graph and regular graphs, sequence graphs and sequence graphs;

[0134] The cascade optimization includes the following steps:

[0135] S1. Optimize the subgraphs of the sequence graph sequentially;

[0136] S2. Connect the previously optimized graph to the currently optimized subgraph;

[0137] S3. Connection failed, return to S1 to continue optimizing the next subgraph;

[0138] S4. The concatenation is successful. The newly concatenated graph is then concatenated and merged with the remaining subgraphs.

[0139] S5. After optimization, if the number of subgraphs decreases, the original subgraphs are replaced; if the number of subgraphs is equal to the original number of subgraphs, but the attributes are merged, the original subgraphs are also replaced.

[0140] Taking parsing HTTP response headers as an example, its grammar is as follows:

[0141] field RspCode = string;

[0142] parser RespVer = "HTTP / 1.1";

[0143] parser Space = " ";

[0144] parser RspCode = "200";

[0145] Parser Phrase = "OK";

[0146] parser MimeEnd = "\r\n";

[0147] parser Resp= RespVer Space RspCode Phrase MimeEnd;

[0148] root = Resp;

[0149] Here, Resp is a sequence, and RespVer, Space, RspCode, Phrase, and MimeEnd are string graphs used to match the string "HTTP / 1.1 200 OK\r\n". RspCode has field attributes that need to be output, and the operations are performed in the following order:

[0150] 1) Optimize RespVer. RespVer is a string graph, which is atomic and cannot be optimized further. The optimized graph is itself. If it is a general grammar such as sequence, loop, or branch, then the optimized graph will be returned as a new graph.

[0151] 2) Serial connection, optimize Space. Similar to RespVer, Space is atomic and cannot be optimized further. Return to itself, and then use the optimized graph in 1) to serially connect with the current Space. Since RespVer and Space have no fields and runtime expansion length attributes, they can be serially connected together to form a new string graph with the content "HTTP / 1.1". Collect this new graph.

[0152] 3) Serial merging and optimizing RspCode: Use the optimized graph from step 2) to serialize and merge with RspCode. Since RspCode has field attributes, it cannot be optimized and merged. Therefore, it can only be kept as is and the RspCode is collected.

[0153] 4) Optimize the Phrase. Use the optimized graph RspCode from 3) to concatenate the Phrase. Since RspCode has fields, it cannot be concatenated.

[0154] 5) Optimize MimeEnd by using Phare to concatenate and connect MimeEnds. They can be merged together to form a new string graph containing "OK\r\n", which can then be collected.

[0155] 6) Since the number of new images collected is less than the number of original images, the original images can be replaced, such as... Figure 7 As shown, the sequence Resp, which ultimately consists of 5 string graphs, is optimized into a sequence with only 3 string graphs, as follows. Figure 8 As shown, if RspCode does not need to output, it will eventually be optimized into a single string graph.

[0156] Parallel optimization refers to the subgraphs of a branch graph, merging multiple branches with string prefixes into a decision graph structure. A decision graph is an unordered branch graph. Simply put, a decision graph is an abstraction based on a regular state machine, adding extra fields and subgraphs at the endpoint. Decision graphs reduce redundant matching operations and lower the cost of repeated matching caused by branch backtracking. Parallel optimization requires the following conditions to be met simultaneously: it must be within a branch graph; the number of subgraphs in the branch graph must be greater than or equal to one; the subgraphs in the branch graph must be sequences; and the first subgraph of the sequence must be one of a string graph, a regular graph, or a decision graph.

[0157] The parallel optimization includes parallel connection and parallel combination;

[0158] The parallel connection refers to connecting two graphs together to form a new graph;

[0159] The term "joint union" refers to the fact that the second graph can be merged into the first graph.

[0160] Graph structure types suitable for parallel optimization include string graphs and string graphs, string graphs and regular graphs, string graphs and decision graphs, regular graphs and decision graphs, and branch graphs and branch graphs.

[0161] The parallel optimization includes the following steps:

[0162] A1. Traverse the subgraphs of the branch graph and optimize accordingly;

[0163] A2. If the optimized subgraph set is empty, add the optimized subgraph to the set and return to A1; if the optimized subgraph set is not empty, iterate through the set and merge the subgraphs.

[0164] A3. Union and Merge: If the graph in the set is a decision graph, then the optimized subgraphs in A2 are unioned and merged into the decision graph. If the merge is successful, it means that the current subgraph has been successfully optimized. Then return to A1 to continue optimizing the remaining subgraphs.

[0165] A4. Parallel connection: If the graph in the set is not a decision graph, then connect this graph in parallel with the optimized graph in A2 to create a new graph. If successful, replace the current graph; otherwise, return to A3 to continue.

[0166] A5. After optimization, if the number of subgraphs decreases, replace the original subgraphs.

[0167] In practical execution, taking the parsing of HTTP request headers as an example, the grammatical expression is as follows:

[0168] field URI = string;

[0169] field ReqVer = string;

[0170] parser URI = regex([^\x20\x0d\x0a]*);

[0171] parser Space = " ";

[0172] parser ReqVer = regex(HTTP / 1.\d?);

[0173] parser MimeEnd = "\r\n";

[0174] parser Req = Space URI Space ReqVer MimeEnd;

[0175] parser Methods = "GET" | "POST" | "PUT" | "DELETE";

[0176] parser FirstLine = Methods Req;

[0177] root = FirstLine;

[0178] Here, Methods is a branch, and each subgraph of the branch is a string graph:

[0179] 1) Optimize the GET request to return an optimized graph;

[0180] 2) The set of optimized subgraphs is empty. Add the optimized graph from step 1) to this set.

[0181] 3) Optimize POST to return an optimized subgraph;

[0182] 4) The optimized subgraph set is not empty. There is one subgraph, namely GET. Since GET is not a decision graph, we now connect GET and POST in parallel to create a decision graph, and then replace GET with the decision graph.

[0183] 5) Continue optimizing PUT. If the set of optimized subgraphs is not empty and there is only one decision graph, then the decision graph will be merged and combined using PUT.

[0184] 6) Continue with DELETE, as in 5), the decision graph will combine and merge the DELETE operations;

[0185] 7) Finally, the Methods branch graph, after decision graph optimization, is reduced from four string graphs to a single decision graph, as shown below. Figure 9 and Figure 10 As shown.

[0186] Previously, parsing HTTP request headers required comparing strings one by one. If it was DELETE, it would require multiple function jumps and string comparisons through the four graphs of GET, POST, PUT, and DELETE to achieve a successful match. Now, with the optimization of the decision graph, the internal state machine only needs to match the DELETE string once to get the result. The algorithm complexity has been improved from O(M*N) to O(N).

[0187] In this embodiment, as Figure 6 As shown, the graph traversal engine starts from the root graph and performs traversal decoding from top to bottom according to the characteristics of the graph structure type, including single-packet mode decoding and stream mode decoding.

[0188] The script defaults to single-packet mode decoding. Stream mode decoding will only be performed if flowMode is written in the appropriate place in the script. Single-packet mode decoding is for stateless single-packet protocols such as UDP and TCP. The state is cleared and reset each time it is decoded, and the state only changes within a single packet.

[0189] In streaming mode decoding, protocols that require cross-packet transmission are typically not fully transmitted in a single packet, resulting in incomplete decoding. Therefore, when data is insufficient, the relevant state needs to be saved in the decoding context. When the next data arrives, the state is restored to the position of the last traversal and the traversal continues.

[0190] Here, we take the TELNET protocol as an example. The TELNET protocol is a streaming decoding protocol, and the syntax for matching TELNET protocol usernames is as follows:

[0191] field TelnetUserName= string;

[0192] parser TelnetUserName= regex([^\xff\x0d]*);

[0193] parser LoginPrompt= regex((Username: )|(login: )|(localhost login: ));

[0194] parser Login = LoginPrompt TelnetUserName;

[0195] root = Login;

[0196] flowMode;

[0197] The `LoginPrompt` grammar is a regular expression; a match is considered successful if it matches any one of the following three: "Username: ", "login: ", or "localhost login: ". Suppose we have a scenario where the current TELNET login prompt is "Username: ". Since TELNET uses stream decoding, the only remaining data is "User". The rest of the data will be in the next packet. Traditional decoders in the industry typically handle this by caching all the data from the current TELNET protocol decoding and then reassembling it in the next packet before starting the decoding process again. While this approach can achieve successful decoding, it undoubtedly wastes performance.

[0198] This embodiment provides streaming mode decoding to solve this problem. The regular expression graph in this embodiment implements streaming mode decoding. When "User" is successfully matched, but due to insufficient data, the regular expression matching is not successful and the endpoint of the regular expression has not been reached, the current regular expression matching state and function stack information are directly saved. The current regular expression graph is restored to continue traversal during the next decoding iteration. There are two recovery methods: stackless coroutine recovery and stacked coroutine recovery.

[0199] When there is insufficient data for a stackless coroutine, the function stack is exited level by level, and the necessary recovery information on the function stack is saved in the decoding context recovery stack. The specific method is as follows:

[0200] 1) Push the regular expression state information of LoginPrompt onto the stack, and then pop out of the LoginPrompt regular expression graph function stack;

[0201] 2) Push the current state of Login onto the stack, for example, if the current incomplete state is the first subgraph, then exit the function stack of the Login sequence graph;

[0202] 3) Inform the caller that decoding is incomplete.

[0203] Once the external system has prepared the subsequent "name:" and data, decoding begins again. At this point, the engine restores the stack information based on the decoding context, from top to bottom, back to the position of the last decoding. The specific restoration method is as follows:

[0204] 1) Enter the function of the Login sequence graph, retrieve the current subgraph that should be restored from the recovery stack, and pop it from the stack;

[0205] 2) Enter the LoginPrompt regular expression function, retrieve the state of the last regular expression match from the recovery stack and pop it from the stack, and continue matching from "name: ".

[0206] For coroutines with stacks, the functionality is similar to that of coroutines without stacks; both can resume traversal from the last execution position. The specific method is as follows:

[0207] 1) Suspend the current function and switch the function stack frame to the decoding entry point.

[0208] 2) Inform the caller that decoding is incomplete.

[0209] Once the external system has prepared the subsequent "name:" and data, decoding begins again:

[0210] 1) Directly switch to the suspended function and restore to the last suspended function stack frame;

[0211] 2) Continue matching from "name: ".

[0212] Because stacked coroutines preserve the complete function stack, they do not need to separately save any information related to the function stack. When the function stack is deep, suspending and restoring the function stack is far more efficient than that of stackless coroutines.

[0213] Not only the regular graphs exemplified above support streaming mode, but all the graphs designed in this embodiment support streaming mode, and their behavior is as follows:

[0214] 1) When there is sufficient data, the behavior is consistent with single-packet decoding;

[0215] 2) When data is insufficient, record the current matching position and save the function stack using either a stackless or stacked coroutine.

[0216] 3) When the next packet arrives, restore the function stack to the position where data was insufficient using either a stackless or stacked coroutine, and continue matching from the position after the data shortage. Note that the method of saving and restoring the function stack must be consistent.

[0217] This approach offers significant advantages: it allows for a return to the previous position with minimal cost, enabling continued decoding. This not only improves decoding performance but also reduces memory consumption and allows for concurrent decoding of more streams.

[0218] In this embodiment, the generation of the interpretation language for protocol decoding includes the following steps:

[0219] Read the protocol specifications of network data packets, describe the protocol specifications of network data packets in a formal language, and abstract the logic of the formal language to obtain basic logical units;

[0220] Define the specification of the interpreted language used to describe the basic logic units, set the lexical and grammatical rules for the decoding protocol script, and perform lexical and grammatical parsing.

[0221] The interpretation language specification is encapsulated in the network protocol decoding engine, and the engine architecture is unified.

[0222] The purpose of this design is to provide an interpretation language suitable for protocol decoding. It adopts a concise plain text language specification to describe the basic logic of formal languages ​​and the combination of logic. It supports the description of logical combinations and complex nested structures, has strong expressive power and good readability. The design is concise enough, which will greatly reduce the difficulty of creating and reading decoding scripts, as well as the total amount of code, and will greatly shorten the development cycle.

[0223] Meanwhile, in this embodiment, a character type mapping table is first established in the lexical parsing process to classify and map the characters in the decoding protocol script. Then, the input stream is traversed in character order, and the characters are processed according to their types. The identified lexical units are encapsulated as Token objects, and each Token contains a start position, an end position, a type identifier, and an associated semantic action.

[0224] The grammar parsing uses a top-down recursive descent parsing method to parse the token sequence according to predefined grammar rules.

[0225] Furthermore, lexical analysis includes the composition and writing methods of words, sentences, and segments in the decoding protocol script.

[0226] The purpose of this design is to treat each network protocol as a formal language in protocol decoding, abstracting it to describe the vast majority of protocols. A concise interpreted language specification describes the basic logic and combinations of these formal languages. Plain text protocol decoding scripts replace traditional hard-coded decoders, constructing a complete three-level grammar system of words, sentences, and segments. It supports rich grammatical definitions: structured grammars, general grammars, context-related grammars, etc., and complex decoding logic expressions: supporting conditional judgments, loop control, variable operations, etc. Decoders generated using an interpreted language can solve the problem of rapid protocol growth and changes, reducing decoder development time to 1 / 3 to 1 / 4 of the original, greatly improving development speed, lowering the technical requirements for developers (no longer requiring C++ qualifications), and enabling the development of more decoders in six months than in the previous few years combined, effectively coping with the rapid growth of protocols.

[0227] In practical implementation, the interpretation language provided in this application can also be called FPDL, or Fast Protocol Decode Language. In its specific execution, a word consists of one or more consecutive characters, including special characters, string constants, numeric constants, regular expression constants, keywords, and identifiers.

[0228] The following example illustrates this: the character is (TAB), the HEX (hexadecimal) is 0x09, the category is word segmentation, it is used for horizontal tabs, and WS (white space).

[0229] The character is (new line), HEX (hexadecimal) is 0x0A, it is categorized as segment words and construction statements, used for newline characters, WS and construction comment lines, and marks the end position of the line.

[0230] For string constants, use a pair of double quotes to define a string constant, and the backslash is the escape character.

[0231] For numeric constants, this embodiment supports integer type numeric constants with a value range of [0, 0x7FFFFFFF], and also supports hexadecimal numeric constants, such as 0x20 being equivalent to the numeric constant 32.

[0232] For regular expression constants, the following example is provided: the metasyntax is *+?{m,n}, and the example is (xyz)*.

[0233] , is used to represent loop syntax, and its meaning is consistent with *+?{m,n} in the decoding protocol script;

[0234] The metasyntax is [], the example is [xyz], used to represent a character set, the character to be matched is any character in the character set. [xyz] matches the character x, y or z.

[0235] For keywords that are internally reserved strings and used as guide words, data types, decoding methods, special references, etc., the following examples are provided: the keyword is "field", the category is a guide word for defining a decoding field, and it is used to define a general decoding field;

[0236] The keyword is regex, the category is segmentation, the word guide is used to construct regular expression constants.

[0237] For the identifier part, the identifier is used to identify the names of user-defined items such as field names, variable names, and grammar names. A valid identifier should conform to the regular expression: [a-zA-Z_][a-zA-Z0-9_]*.

[0238] In this embodiment, a sentence is composed of multiple words and is a linguistic unit used to express a complete semantic meaning. A sentence includes comment statements and definition statements.

[0239] The comment statements begin with # and end with \n;

[0240] The definition statement begins with a statement definition leader and ends with a semicolon, and includes the definition of the decoding field, the definition of the grammar, the definition of the root grammar, the definition of the external logic, and the definition of the decoding mode.

[0241] The decoding fields are divided into structure fragment decoding fields, general decoding fields, group fields, context-related decoding fields, and extended decoding fields.

[0242] In the structure fragment decoding field, a part of the structure is defined as a decoding field. It is necessary to define the offset and length OffsetLength of this field relative to the structure.

[0243] OffsetLength is divided into ByteOffsetLength and BitOffsetLength:

[0244] The definition syntax of ByteOffsetLengt:

[0245] byteoffset, bytelength

[0246] The syntax for defining BitOffsetLength:

[0247] byteoffset: bitoffset, bitlength

[0248] The syntax for defining fields in a structure fragment decoding method:

[0249] structfield field identifier = structure identifier [OffsetLength], data type, decoding method;

[0250] The decoding methods for structure fragment decoding fields and general decoding fields can be omitted, and the default is literal interpretation; the decoding action of structure fragment decoding fields depends on the structure grammar, and the grammar of the structure needs to be defined before the field can be decoded.

[0251] Such as structfield DstAddress = eth2[0, 6], string, MAC; # byteofflen

[0252] A decoding field named DstAddress is defined, located at offset 0 bytes in the eth2 structure, with a length of 6 bytes, a data type of string, and a decoding method of MAC.

[0253] Another example is structfield IHL = iphdr1[0: 4, 4], int; # bitofflen

[0254] A decoding field named IHL is defined, located at offset 0 bytes 4 bits in the structure iphdr1, with a length of 4 bits and an integer data type. No decoding method is defined.

[0255] For general decoded fields, the definition syntax is as follows:

[0256] field identifier = data type, decoding method;

[0257] Generally, the decoding action of a field depends on the grammar with the same name. A grammar with the same name needs to be defined for the field to be decoded. In specific implementations, for example, `field Name = string, DOMAIN;` # Domain name field in the DNS protocol.

[0258] A decoding field named Name is defined, with a data type of string and a decoding method of DOMAIN (domain name decoding);

[0259] For group fields such as Figure 11 As shown, the left side is the decoding script, the upper right side is the protocol data stream, and the lower right side is the decoding result. lv and lvs are defined as groupfields, and the decoding result presents a hierarchical tree structure.

[0260] In this embodiment, lv is a sequence grammar and lvs is a cyclic grammar; generally, the decoding field can also be defined as a groupfield. The definition and decoding result related to lvs are shown in the figure.

[0261] The above-mentioned associated decoding field is used as a control decoding field. The correct decoding field is selected and the decoding action is executed through the logical expression of the runtime variable. The definition syntax of the above-mentioned associated decoding field is as follows:

[0262] relatedfield field identifier = if logical expression 1 field identifier 1;

[0263] else if logical expression 2 field identifier 2;

[0264] ...;

[0265] else if logical expression N field identifier N;

[0266] else field identifier N+1;

[0267] In specific implementation, the following are included:

[0268] field Name = string, DOMAIN;

[0269] field Host = string, IPADDR;

[0270] field Type = int, N2H;

[0271] field HostV6= string,IPV6ADDR;

[0272] field Ignore = string,HEX;

[0273] relatedfield RDData= if ( Type = 2 || Type = 5 || Type = 12 ) Name

[0274] else if (Type = 1) Host

[0275] else if (Type = 28) HostV6;

[0276] else Ignore;

[0277] In this embodiment, whether RDData is decoded into a domain name, IP address, or IPv6 is determined by the runtime decoding result of Type.

[0278] For extended decoded fields used to assign field properties, the `invisible` or `regist` keyword is added before the definition statement of the structure fragment decoded field or the general decoded field. The definition syntax of the `invisible` keyword is as follows:

[0279] Invisible structure fragments or general decoded field definition statements;

[0280] The syntax for defining the regist keyword is:

[0281] regist struct fragment or general decode field definition statement

[0282] The grammar definition section includes structure grammars, general grammars, context-dependent grammars, and pre-declared grammars.

[0283] The structure grammar uses the length of the structure to define a continuous block of data, and the structure is divided into fixed-length structures and runtime-length structures. The syntax for defining a fixed-length structure is as follows:

[0284] The struct is a unit of length for identifying a positive integer.

[0285] The syntax for defining the runtime structure is as follows:

[0286] The struct is a structure identifier, and the variable identifier is the length unit.

[0287] The general grammar includes simple grammars and combinational logic grammars. The syntax of the general grammar is defined as follows:

[0288] parser grammar identifier = definition statement;

[0289] The definition statement of the simple grammar is a regular expression constant or a string constant;

[0290] The combinational logic grammar consists of multiple simple grammars arranged according to one of the basic logics of a formal language, and includes sequence grammar, branching grammar, and loop grammar. The basic logics of the formal language include sequence, branching, and looping. The syntax of the sequence grammar is defined as follows:

[0291] Grammar identifier 1 Space grammar identifier 2 … Space grammar identifier n;

[0292] The syntax for defining the branching grammar is as follows:

[0293] Grammar identifier 1 | Grammar identifier 2 … | Grammar identifier n;

[0294] The syntax for the loop grammar is defined as follows:

[0295] (Grammar identifier) ​​Loop count definition;

[0296] The above-mentioned associative grammar selects the correct grammar parsing branch based on the execution result of the logical expression of runtime variables. The definition syntax of the above-mentioned associative grammar is as follows:

[0297] relatedparser grammar identifier = if logical expression 1 grammar identifier 1;

[0298] else if logical expression 2 grammar identifier 2;

[0299] ...;

[0300] else if logical expression N grammar identifier N;

[0301] else grammar identifier N+1;

[0302] The pre-declared grammar uses the `declare` keyword to identify mutual reference dependencies between grammars. The syntax definition of the pre-declared grammar is as follows:

[0303] declare grammar identifier;

[0304] The root grammar definition is used to define the entry point for grammar parsing. The syntax for defining the root grammar is as follows:

[0305] root = grammatical name;

[0306] The external logic definition is used to execute logical expressions during grammar expansion, and then execute specified actions based on the result of the logical expressions. The syntax for defining the external logic definition is as follows:

[0307] if logical expression 1 executes action 1;

[0308] else if logical expression 2 executes action 2;

[0309] ...;

[0310] else if logical expression N executes action N;

[0311] on grammar identifier;

[0312] The decoding modes include streaming mode and multiple decoding mode. Streaming mode is used for streaming decoding, where the data required to complete one full decoding operation consists of multiple data blocks. The definition syntax for streaming mode is:

[0313] flowMode;

[0314] The multiple decoding mode is used to support generating multiple independent protocol decoding results from a single decoding operation. The syntax for defining the multiple decoding mode is as follows:

[0315] whileMode.

[0316] At the same time, a segment contains one or more definition statements of the same kind. The segments include, in order, variable definition segment, grammar definition segment, root grammar definition segment, external logic definition segment, and decoding mode definition segment.

[0317] The variable definition section includes the decoded field definition statement and the int variable definition statement;

[0318] The grammar definition section includes at least one line of grammar definition statements;

[0319] In some use cases, the root grammar definition section includes a single root grammar definition statement.

[0320] In some use cases, the external logic definition section includes external logic definition statements;

[0321] In some use cases, the decoding mode definition section includes a single line of decoding mode definition statements.

[0322] Taking decoding script generation as an example, decoding script generation needs to be carried out according to the RFC specification of the public protocol or the specification of the private protocol. All protocol scripts are composed of multiple segments, and each segment consists of one or more definition statements of the same type. The script is written in a segmented manner, from top to bottom: variable definition statement, field definition statement, grammar definition statement, root grammar definition statement, assignment definition statement, and decoding mode definition statement. The keywords of the statements must start with the introductory keywords in Table 3, such as field, structfield, groupfield, relatedfield, invisible, regist, extern, int, parser, struct, relatedparser, declare, root, flowMode, and whileMode.

[0323] The steps involved are as follows:

[0324] B1. Organize the protocol structure; clarify what components the protocol consists of, such as a fixed-length header, a variable-length payload, etc.

[0325] B2. Determine the structure grammar; if the length is fixed, use the structure grammar; if the length is not fixed and a loop is needed until a certain condition is met, then a loop grammar is needed; if further judgment is needed based on the already decoded value, then the above-mentioned contextual grammar should be used.

[0326] B3. Determine the decoding mode; if it is a streaming mode or multiple decoding mode, the corresponding keywords need to be added; single packets do not require processing.

[0327] B4. Determine the output fields and decoding methods; whether a field is output depends on the user's needs. If decoding is required, a field or structfield needs to be defined. If a hierarchy needs to be generated, a groupfield needs to be defined. If decoding is required but not output, it needs to be marked as invisible.

[0328] like Figures 12 to 17 As shown, taking TCP script creation as an example, many protocols have fixed-length headers, such as ETH2, IP, TCP, UDP, DNS, etc. The header consists of multiple fields, and the structure or detailed description of the header can usually be found in RFC documents. Select a protocol to decode, such as TCP, the TCP protocol header is defined in RFC 793.

[0329] The variable part begins with Options; therefore, the fixed-length TCP header starts from SourcePort and ends with UrgentPointer. First, according to... Figure 12 This yields the TCP header (fixed-length portion) to be 20 bytes long, along with the offset and length of each field within the TCP header. Then, the TCP protocol decoding script can be written, requiring the use of four definition statements: structure fragment decoding fields, structure grammar, sequence grammar, and root grammar.

[0330] 1. Structure fragment decoding field definition:

[0331] structfield SourcePort = TCPHeader[0,2],int,N2H;

[0332] This statement defines SourcePort as a fragment of the TCPHeader structure, located at offset 0 bytes within the TCPHeader, with a length of 2 bytes, and requiring the use of the general decoding method N2H to convert the network byte order to the host byte order;

[0333] By comparing the definitions of DataOffset, understand the difference between ByteOffsetLength and BitOffsetLength:

[0334] structfield DataOffset = TCPHeader[12:0,4],int;

[0335] Then, based on the offset and length of each field in the TCP header, each field is defined as a structure fragment decoding field;

[0336] 2. Definition of Structure Grammar:

[0337] struct TCPHeader 20;

[0338] This statement defines TCPHeader as a 20-byte structure.

[0339] 3. Sequence Grammar Definition:

[0340] parser TCP = TCPHeader ignoretoend;

[0341] This statement defines TCP as a sequence grammar, consisting of two subgrammars, TCPHeader and ignoretoend, arranged sequentially.

[0342] ignoretoend is a special structure grammar that indicates that the data after it should be ignored. It is equivalent to regex(.*) but has higher execution performance.

[0343] 4. Definition of Root Grammar

[0344] root = TCP;

[0345] This statement defines the root grammar as TCP, and each script needs to define a root grammar as the entry point for parsing;

[0346] Then, tools are needed to verify the script's correctness, and finally, the script is used to decode the TCP protocol data stream, such as... Figure 13 A data stream is provided, and the result is... Figure 14 The decoding result.

[0347] Since Options were omitted in the example above, this embodiment will decode Options (according to RFC793);

[0348] Options consist of multiple Options, which need to be defined first. Each Option consists of OptionKind, OptionLength, and OptionData, which also need to be defined one by one.

[0349] 1. Define OptionKind, OptionLength, OptionData

[0350] field OptionKind = int;

[0351] field OptionLength = int;

[0352] field OptionData = string,HEX; # Decode as a hexadecimal string

[0353] int OptionDataLength = OptionLength – 2; # Subtract the lengths of OptionKind and OptionLength

[0354] struct OptionKind 1; # OptionKind is one byte long

[0355] struct OptionLength 1; # OptionLength is one byte in length

[0356] struct OptionData OptionDataLength; #The length of OptionLength is OptionDataLength

[0357] 2. Define Option

[0358] RFC793 defines two formats for Options:

[0359] Case 1: A single octet of option-kind.

[0360] Case 2: An octet of option-kind, an octet of option-length, and the actual option-data octets.

[0361] The format to which Option should be decoded is determined by the (runtime) decoding result of OptionKind; the associated grammar mentioned above needs to be used:

[0362] parser OptionCase2 = OptionLength OptionData;

[0363] relatedparser OptionCase = if (OptionKind > 1) OptionCase2

[0364] else nop; # OptionCase1

[0365] parser Option = OptionKind OptionCase;

[0366] 3. Define Options

[0367] Options is a loop consisting of multiple Options. The loop terminates under two constraints: the loop terminates when OptionKind = 0 is obtained after decoding, or the number of characters consumed by Options is equal to the length of the TCP variable-length portion.

[0368] The length of the variable-length portion (i.e., from Options to the end) is defined as follows:

[0369] int VariableLength = DataOffset * 4 - 20; # Consistent with common syntax, supports +-* / &|()

[0370] Then we need to use the C-class loop control of the loop grammar in the general grammar:

[0371] parser Options = Option(Options.bytes < VariableLength && OptionKind!= 0);

[0372] 4. Define the TCP variable-length portion

[0373] The variable-length portion of TCP consists of Options and Padding;

[0374] Padding is used for alignment. If you want to observe whether the padding contains data, you need to decode the padding. In this example, you can use ignoretoend instead.

[0375] parser TCPVariable = Options ignoretoend;

[0376] This completes the decoding of Options. If you want the TCP decoding results to appear more hierarchical, you need to define some group fields:

[0377] structfield Flags = TCPHeader[13:2,6],int;

[0378] groupfield Flags;

[0379] groupfield Options;

[0380] groupfield option;

[0381] The grammar used during the verification process Figure 15 As shown, and input a containing Figure 16 The TCP protocol data stream with Options, the final result is as follows Figure 17 As shown.

[0382] In this embodiment, the fields are first categorized, and the basic field types are defined as follows:

[0383] 1) StringRefField: A string reference field that supports zero-copy;

[0384] 2) StringPartRefField: Partial string field, supports start position and length expressions;

[0385] 3) IntTypeField: The abstract base class for integer type fields;

[0386] Integer field specialization:

[0387] 1) BitIntField: Bit field, supports bit offset and bit length;

[0388] 2) IntXField: Unsigned integer field (8 / 16 / 24 / 32 / 40 / 48 / 56 / 64 bits);

[0389] 3) IntXN2HField: An unsigned integer field for converting network byte order to host byte order (8 / 16 / 24 / 32 / 40 / 48 / 56 / 64 bits);

[0390] 4) `AsciiToIntField`: Converts an ASCII string to an integer;

[0391] Protocol-specific fields:

[0392] 1) DomainField: Domain name field, supports DNS domain name resolution;

[0393] 2) MacField: MAC address field;

[0394] 3) IPAddrField: IPv4 address field;

[0395] 4) IPv6AddrField: IPv6 address field;

[0396] Encoding conversion field:

[0397] 1) HEXField: Hexadecimal encoded field;

[0398] 2) Base64Field: Base64 encoded field;

[0399] 3) .Base64DecodeField: Base64 decoded field;

[0400] 4) UrlDecodeField: URL decoding field;

[0401] Advanced processing fields:

[0402] 1) UTF16Field: UTF-16 encoded field;

[0403] 2) CharacterEntityDecodeField: Character entity decoding field;

[0404] Then, the field decoding execution mechanism is set up. Each field implements the virtual function DecodeField(), providing specific decoding logic. The function signature is as follows:

[0405] virtual void DecodeField(const uint8_t* buffer, int32_t length,DecodedField* decodedField, FastDecodeContext* context) = 0;

[0406] Decoding process:

[0407] 1) Field Recognition: Based on the field definitions in the decoding script, create corresponding field class instances;

[0408] 2) Parameter binding: Bind field parameters using DecodeFieldParam;

[0409] 3) Polymorphic Invocation: At runtime, a DecodedField is obtained from the field pool, and the specific decoding implementation is invoked through a virtual function;

[0410] 4) Output the result: Store the decoding result in DecodedField.

[0411] In order to improve the overall decoding efficiency, such as Figure 18 As shown, an intelligent hybrid memory management strategy is adopted, which automatically selects the optimal memory processing method based on field type and processing requirements, enabling zero-copy scenarios:

[0412] 1) When dealing with simple string fields that do not require conversion;

[0413] 2) When data can directly reference the original buffer;

[0414] Implementation: decodeField->m_stringValue = buffer;

[0415] Controlled copy scenarios:

[0416] 1) Numeric type fields require byte order conversion;

[0417] 2) Fields that require formatting or encoding conversion;

[0418] 4) Calculate the generated field values ​​(such as checksums);

[0419] 4) Bit field and unaligned data processing;

[0420] Implementation: decodedField->MallocDataBuffer(length); FPDE_MEMCPY(...);

[0421] This hybrid strategy maintains high performance while ensuring the accuracy and flexibility of data processing.

[0422] like Figure 19 As shown, a network packet parsing system with heavy memory management is provided. The core of memory management is the field pool DecodedFieldPool, which adopts an efficient object pool pattern.

[0423] Pool configuration parameters:

[0424] 1) Default number of fields: 4096 (DEFAULT_FIELD_COUNT);

[0425] 2) Recycling: When the pool is full, it is recycled from the beginning.

[0426] 3) Pool full detection: Detected using the m_full flag and the IsFull() method;

[0427] 4) Error Reporting: Configurable to throw an exception when the pool is full;

[0428] Field lifecycle management:

[0429] 1) Retrieve a field from the pool;

[0430] 2) Roll back to the last field;

[0431] 3) Shrink the memory of the field pool;

[0432] Pool recovery mechanism:

[0433] 1) Use CallableProxy to implement recovery points;

[0434] 2) Supports rolling back and restoring field positions;

[0435] 3) `InitRestoreProxy()` initializes the recovery proxy;

[0436] 4) `UpdateAliveFieldsPosition()` updates the position of the active field;

[0437] Field buffer management strategy: Each DecodedField has its own data buffer management.

[0438] Buffer configuration:

[0439] 1) Default size: 64 bytes (DEFAULT_BUFFER_SIZE);

[0440] 4) Dynamic expansion: Automatically expands as needed;

[0441] 3) Smart Shrink: Automatically shrinks back to the default size when the size exceeds the default limit;

[0442] Distributor hierarchy:

[0443] Supports multi-level memory allocation strategies:

[0444] 1) The system's default allocator;

[0445] 2) Custom allocator;

[0446] 3). PMR Allocator (C++17);

[0447] Memory alignment optimization:

[0448] 1) Use MAX_ALIGNMENT = alignof(std::max_align_t) to ensure optimal alignment;

[0449] 2) Supports memory alignment requirements of the SIMD instruction set.

[0450] like Figure 20 The diagram shown is a structural schematic of an electronic device according to an embodiment of this application. The electronic device includes a memory and a processor. The memory stores computer-readable instructions, and the processor executes the computer-readable instructions stored in the memory to implement the network packet parsing system described in any of the above embodiments.

[0451] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A network data packet parsing system, characterized in that, It includes a compilation module and an interpretation and execution module. The compilation module is used to generate a decoder for parsing network packets. The compilation module is equipped with a protocol decoding engine and a graph optimization engine. The protocol decoding engine is used for interpreter language generation, lexical analysis, and grammar analysis for protocol decoding; The graph optimization engine is used for graph optimization and decoder construction. The interpretation and execution module is used to decode network data packets generated by the compiler module and output the decoding results. The interpretation and execution module is equipped with a graph traversal engine. The graph traversal engine is used for graph traversal and decoding.

2. The network packet parsing system according to claim 1, characterized in that, The generation of the interpretation language for the protocol decoding includes the following steps: Read the protocol specifications of network data packets, describe the protocol specifications of network data packets in a formal language, and abstract the logic of the formal language to obtain basic logical units; Define the specification of the interpreted language used to describe the basic logic units, set the lexical and grammatical rules for the decoding protocol script, and perform lexical and grammatical parsing. The interpretation language specification is encapsulated in the network protocol decoding engine, and the engine architecture is unified.

3. The network packet parsing system according to claim 2, characterized in that, In the lexical parsing process, a character type mapping table is first established to classify and map the characters in the decoding protocol script. Then, the input stream is traversed in character order, and the characters are processed according to their types. The identified lexical units are encapsulated as Token objects, and each Token contains a start position, an end position, a type identifier, and an associated semantic action. The grammar parsing uses a top-down recursive descent parsing method to parse the token sequence according to predefined grammar rules.

4. A network packet parsing system according to claim 3, characterized in that, The lexical method includes the composition and writing method of words, sentences, and segments in the decoding protocol script. The words are composed of one or more consecutive characters, including special characters, string constants, numeric constants, regular expression constants, keywords, and identifiers. The special characters consist of one or two consecutive punctuation marks and are used to segment words, construct words, construct sentences, construct semantics, and act as operators; The regular expression constant is used to define complex words, and the syntax is defined as regex(regexExpr); The keywords are reserved strings used for guiding words, data types, decoding methods, and special references; The identifier is used to identify the name of a user-defined item, including identifying field names, variable names, and grammar names; The sentence consists of multiple words and is a linguistic unit used to express a complete semantic meaning. The sentence includes annotation statements and definition statements. The comment statements begin with # and end with \n; The definition statement begins with a statement definition introductory word and ends with a semicolon, and includes the definition of the decoding field, the definition of the grammar, the definition of the root grammar, the definition of the external logic, and the definition of the decoding mode. The segment contains one or more definition statements of the same kind, and the segments include, in order, a variable definition segment, a grammar definition segment, a root grammar definition segment, an external logic definition segment, and a decoding mode definition segment; The variable definition section includes the decoded field definition statement and the int variable definition statement; The grammar definition section includes at least one line of grammar definition statements.

5. A network packet parsing system according to claim 1, characterized in that, The graph optimization includes the following steps: Obtain the original graph structure from the network protocol decoding engine; The original graph structure is classified by a graph algorithm engine, and the graph structure type is detected to select an optimization method, perform constraint checks, and generate a decoder. The graph traversal uses a graph traversal engine to classify the original graph structure, performs state transitions on the graph, and outputs the decoding result at the receiving state of the graph.

6. A network packet parsing system according to claim 5, characterized in that, The graph structure types include structure graphs, string graphs, regular graphs, sequence graphs, branch graphs, loop graphs, context graphs, decision graphs, and trie graphs; Structure diagrams are used for fast decoding of fixed-length data structures; The string graph is used for string matching; The regular graph is used for deterministic finite automata and efficient pattern matching; The sequence diagram is used to execute multiple subgraphs sequentially; The branch graph is used to traverse each subgraph sequentially, back off fields and operations with side effects on failures, and continue traversing the next subgraph; if successful, it returns directly. The cycle graph is used to support conditional loops and maximum number of iterations; The context association graph is used to associate the decoded values ​​from the preceding text and determine which path to traverse based on the values. The decision graph is based on a state machine to implement complex conditional branching logic; The trie graph is used for efficient multi-pattern string matching.

7. A network packet parsing system according to claim 6, characterized in that, The optimization methods include trie optimization, serial optimization, parallel optimization, and recursive optimization; When the detection results of the graph structure type involve string branches, a trie optimization is performed by constructing a TireGraph. When the detection results of the graph structure type involve sequential execution, serial optimization is performed through sequence merging. When the detection results of the graph structure type involve parallel branches, parallel optimization is performed by merging branches; When the detection results of the graph structure type involve nested structures, recursive optimization is performed through subgraph optimization.

8. A network packet parsing system according to claim 7, characterized in that, The concatenation optimization refers to merging sequentially executed graph nodes without intermediate side effects into a single operation. Concatenation optimization must simultaneously satisfy the following conditions: the two graph nodes must be sequentially executed, neither graph has a decoding field, neither graph has a runtime expansion length of the grammar, and the graph type supports the merging operation. The series optimization includes series connection and series merging; The series connection refers to connecting two graphs together to form a new graph; The term "serial merging" refers to merging the second graph into the first graph. Graph structure types suitable for concatenation optimization include string and string graphs, regular graph and string graphs, regular graph and regular graphs, sequence graphs and sequence graphs; The cascade optimization includes the following steps: S1. Optimize the subgraphs of the sequence graph sequentially; S2. Connect the previously optimized graph to the currently optimized subgraph; S3. Connection failed, return to S1 to continue optimizing the next subgraph; S4. The concatenation is successful. The newly concatenated graph is then concatenated and merged with the remaining subgraphs. S5. After optimization, if the number of subgraphs decreases, replace the original subgraphs; if the number of subgraphs is equal to the original number of subgraphs, but attributes have been merged, also replace the original subgraphs. The parallel optimization refers to merging multiple branches with string prefixes into a decision graph structure by combining the subgraphs of the branch graph. Parallel optimization must simultaneously satisfy the following conditions: it must be in a branch graph, the number of subgraphs of the branch graph must be greater than or equal to 1, the subgraphs of the branch graph are a sequence, and the first subgraph of the sequence is one of the string graph, regular graph, or decision graph. The parallel optimization includes parallel connection and parallel combination; The parallel connection refers to connecting two graphs together to form a new graph; The term "joint union" refers to merging the second graph into the first graph. Graph structure types suitable for parallel optimization include string graphs and string graphs, string graphs and regular graphs, string graphs and decision graphs, regular graphs and decision graphs, and branch graphs and branch graphs. The parallel optimization includes the following steps: A1. Traverse the subgraphs of the branch graph and optimize accordingly; A2. If the optimized subgraph set is empty, add the optimized subgraph to the set and return to A1; if the optimized subgraph set is not empty, iterate through the set and merge the subgraphs. A3. Union and Merge: If the graph in the set is a decision graph, then the optimized subgraphs in A2 are unioned and merged into the decision graph. If the merge is successful, it means that the current subgraph has been successfully optimized. Then return to A1 to continue optimizing the remaining subgraphs. A4. Parallel connection: If the graph in the set is not a decision graph, then connect this graph in parallel with the optimized graph in A2 to create a new graph. If successful, replace the current graph; otherwise, return to A3 to continue. A5. After optimization, if the number of subgraphs decreases, replace the original subgraphs.

9. A network data packet parsing system according to claim 5, characterized in that, The graph traversal engine starts from the root graph and performs traversal decoding from top to bottom according to the characteristics of the graph structure type, including single-packet mode decoding and stream mode decoding. The single-packet mode decoding is for stateless single-packet protocols such as UDP and TCP, and the state is cleared and reset each time it is decoded. The streaming mode decoding is designed for protocols that require cross-packet transmission, meaning protocols that cannot be fully decoded because a single packet cannot be transmitted completely. When data is insufficient, the relevant state needs to be saved in the decoding context. When the next decoded data arrives, the state is restored to the position of the last traversal and the traversal continues.

10. A network data packet parsing system according to claim 9, characterized in that, The graph traversal engine includes single-packet mode decoding and streaming mode decoding; The single-packet mode decoding is used for stateless single-packet protocols. The state is cleared and reset each time it is decoded, and the state only changes within a single packet. The streaming mode decoding is used for protocols that require cross-packet transmission. If a single packet cannot be transmitted completely, decoding cannot be completed. When there is insufficient data, the state is saved in the decoding context. When the next decoded data arrives, the state is restored to the position of the last traversal and the traversal continues.

Images