An automated extraction method and system for network communication protocol syntax information

Abstract

The application relates to a kind of network communication protocol syntax information's automated extraction method and system, comprising: all protocol analysis files are preprocessed, including extracting protocol name, extracting protocol parser and being associated with protocol name, extracting the hierarchical relationship between protocol parser;The protocol to be analyzed selected by user is handled, including extracting all field information of the protocol to be analyzed, expanding the content of main analysis function, extracting all packet types and the constituent field information thereof, splicing the segment of each packet type to generate complete packet type.The application extracts protocol syntax information from the in-depth analysis of the network communication protocol analysis file of Wireshark, and stores in a structured manner.Compared with other network communication protocol syntax information extraction methods, the application has wide adaptability, high automation and high efficiency, and at the same time has high accuracy, can effectively save labor, time cost.

Description

Technical Field

[0001] This invention relates to an automated method and system for extracting network communication protocol syntax information, belonging to the field of information security technology. Background Technology

[0002] Information security is a crucial aspect of modern societal security. The widespread application of various network communication protocols has made information exchange within networks complex and diverse. These protocols essentially define a series of rules, field meanings, and possible value ranges for the exchange of data during communication. These rules are essential for the correct execution of network communication and are commonly referred to as the syntax structure of the network communication protocol. Information security maintainers and researchers need to understand the syntax structure of the network communication protocols they use when performing tasks such as network maintenance and software testing. This allows them to set targeted protection rules or construct security test files to discover potential security vulnerabilities. Wireshark is an open-source network protocol analysis tool that can parse protocol syntax structures by calling protocol parsing plugins that follow certain standards. These protocol parsing plugins are typically written in C, and these C language files are usually called protocol parsing files. These protocol parsing files are developed and maintained collaboratively by numerous Wireshark users online, resulting in a vast number of network communication protocols that Wireshark can parse. Since the protocol parsing file contains detailed syntactic information about the protocol, including the definition of protocol fields, the encoding method of fields, the location of fields, and the structure of protocol data packets, if this syntactic information can be extracted, it will provide an important data source for application scenarios such as vulnerability mining based on fuzzing and network security protection.

[0003] Currently, there are two main methods for extracting syntactic information from network communication protocols: manual methods and artificial intelligence (AI) methods. Manual methods require researchers and testers to first read the protocol specification document, understand its syntactic structure, and then manually extract the protocol syntactic information. Manual methods are characterized by high accuracy, but poor versatility and low automation, i.e., low efficiency. AI methods use artificial intelligence technologies such as natural language processing and machine learning to intelligently analyze network communication protocol specification documents, thereby automatically extracting protocol syntactic information. AI methods typically rely on some machine learning algorithm and model, which can automate the extraction of protocol syntactic information to a certain extent. However, the effectiveness of this method depends on a large amount of high-quality labeled data to train the model, and obtaining high-quality labeled data is not easy, so the accuracy of the obtained protocol syntactic information is usually low. In addition, AI methods usually have certain requirements on the format of network communication protocol specification documents. Different protocol specifications are often formulated by different standardization organizations, vendor alliances, or even a single vendor. The formats of these protocol specification documents formulated by different organizations vary greatly, which limits the versatility of AI methods. Overall, the characteristics of AI methods are: low accuracy, limited versatility, and high automation. Summary of the Invention

[0004] To address the shortcomings and deficiencies in current methods for extracting syntactic information from network communication protocols, this invention provides an automated method for extracting syntactic information from network communication protocols. This method combines the high accuracy of manual methods with the high degree of automation of artificial intelligence methods, while also mitigating the poor versatility of the former two methods. In other words, this method is characterized by high accuracy, strong versatility, and high degree of automation.

[0005] This invention is based on protocol parsing files provided by Wireshark, extracting protocol syntax information from these files. The framework of this invention consists of two modules: a preprocessing module and a main parsing module. The preprocessing module's main task is to perform preliminary processing on all protocol parsing files, extracting the protocol name information, protocol parser information, and the hierarchical relationship between protocol parsers for each protocol. The main parsing module's main task is to process a specific protocol to be parsed, extracting its field information. If the hierarchical relationship of the protocol to be parsed is above the transport layer TCP / UDP protocol, then it extracts the field information of that protocol and its lower layers up to all layers above TCP / UDP.

[0006] The technical solution of this invention is as follows:

[0007] An automated method for extracting network communication protocol syntax information includes:

[0008] All protocol parsing files are preprocessed, including extracting protocol names, extracting protocol parsers and associating them with protocol names, and extracting the hierarchical relationships between protocol parsers;

[0009] The process involves processing the protocol selected by the user, including extracting all field information of the protocol, expanding the content of the main parsing function, extracting all data packet types and their constituent field information, and concatenating the segments of each data packet type to generate a complete data packet type.

[0010] According to a preferred embodiment of the present invention, the protocol name and the protocol parser are extracted and associated with the protocol name based on the protocol registration mechanism of the Wireshark protocol parsing plugin;

[0011] The hierarchical relationship between parsers is extracted based on the maintenance mechanism of the Wireshark parser table and by traversing the explicit call relationship between upper and lower level parsers.

[0012] According to a preferred embodiment of the present invention, extracting the protocol name includes:

[0013] Read each Wireshark protocol parsing file sequentially, extract and save all information related to the protocol name, including the protocol name, protocol filter name, protocol unique identifier, and protocol handle, and write this information related to the protocol name and the corresponding protocol parsing file name into the protocol list.

[0014] According to a preferred embodiment of the present invention, extracting the protocol parser and associating it with the protocol name includes:

[0015] Extract all information related to the protocol parser, including the protocol parser's filter name, main parsing function name, corresponding protocol handle, and protocol parser handle, and write it into the parser table;

[0016] The protocol list is searched based on the extracted protocol resolver handles. The protocol resolvers are associated with the corresponding protocol names, and an attribute with a global or local value is added to each protocol resolver, depending on whether the protocol resolver has been globally registered.

[0017] The associations and attributes are written together into the resolver table.

[0018] According to a preferred embodiment of the present invention, extracting the hierarchical relationship between protocol parsers includes:

[0019] The hierarchical relationship between different protocol parsers is the calling relationship between different protocol parsers. The network communication protocols corresponding to different protocol parsers with calling relationships are in a hierarchical relationship. The hierarchical relationship of each protocol parser is extracted and written into the parser table. After processing all protocol parsing files, the hierarchical relationship of each protocol parser in the parser table is improved.

[0020] According to a preferred embodiment of the present invention, all field information of the protocol to be parsed is extracted, including:

[0021] For a selected protocol to be parsed, the preprocessed protocol list is searched based on the protocol name, the corresponding protocol parsing file name is obtained and the protocol parsing file is read. All field information, including field identifier, name, filter name, data type, cardinality, and optional value information, is extracted from the protocol parsing file and saved to the field list as standard fields.

[0022] According to a preferred embodiment of the present invention, expanding the content of the main analytic function refers to: locating the main analytic function and improving its content; specifically including:

[0023] The parser table obtained from the preprocessing is searched based on the name of the protocol to be parsed to obtain the name of the main parsing function; the position of the main parsing function is located and its body is extracted as the program block to be parsed; the program block to be parsed is then normalized, specifically by sequentially traversing each line of the program block while performing the following expansion operations:

[0024] If a sub-function call exists, the definition of the function is searched in the current protocol parsing file. If the definition is found, all the code of the function is copied to the calling statement. If it is not found, no operation is performed.

[0025] If it is a code that reads data packets from a buffer, then replace the offset information in it with the corresponding real value;

[0026] If it is field parsing code, replace the field offset information with the corresponding actual value and replace the field length information with the actual value. If it cannot be replaced with an actual value, it means that the length of the current field is related to the value of a previous field. Set the length information of that field to the form of the expression "value of field".

[0027] If an if / else if / else statement is followed by only one line of statements without curly braces, then add curly braces to that line of statements. For if / else if statements that lack an else statement, add a complete else statement with an empty else statement. Repeat this process for the expanded parsed block until all function calls in the parsed block are completed, and all offset information related to data packet buffer reading, as well as offset and length information related to field parsing, are replaced, resulting in a complete parsed block.

[0028] According to a preferred embodiment of the present invention, extracting all data packet types of the protocol to be parsed and all field information contained in each data packet type includes:

[0029] The obtained code blocks to be parsed are divided into different types of basic blocks according to their different structures, including ordinary basic blocks, branch basic blocks, branch item basic blocks, and return basic blocks. A branch basic block refers to the code block of the entire branch structure, including conditional structure basic blocks and selection structure basic blocks. A branch item basic block refers to the code block of each branch in the branch structure, including if basic blocks, else if basic blocks, else basic blocks, case basic blocks, and default basic blocks. Specifically, function basic blocks are considered as branch basic blocks. Return basic blocks include function return basic blocks and function termination basic blocks. Ordinary basic blocks refer to code blocks other than branch basic blocks, branch item basic blocks, and return basic blocks. Specifically, loop structures such as do…while loops, while loops, and for loops are considered ordinary basic blocks.

[0030] During the parsing process, each basic path of the program block to be parsed is regarded as a parsing path of a data packet type. Based on this, all data packet types are extracted, and all field information that makes up each data packet type is extracted, including the field identifier, variable name, offset, length, and byte order. The field identifier is matched with standard fields to complete the field name and optional values.

[0031] Extract field information from all basic blocks contained in each basic path of the program block to be parsed, and store the field information in different basic block lists. At the same time, establish the hierarchical relationship between these basic block lists, and concatenate all basic block lists in each basic path to obtain the field list corresponding to all data packet types.

[0032] According to a preferred embodiment of the present invention, during the parsing process, a global list is used to store the list of all basic blocks and the field information they contain.

[0033] Create and maintain a stack to store all the basic block lists, called a list stack; the basic block list at the top of the stack is used to store the field information extracted from the current basic block, called the top stack list.

[0034] Parse the basic blocks in the block to be parsed:

[0035] First, create a global list `packet_all` to store all the basic block lists and their field information, and add the global list `packet_all` to the list stack as the initial top list of the stack;

[0036] Then, the following processing is performed on the next basic block:

[0037] If it is a normal basic block, a new list is added to the top list of the stack. The new list has only one element, "0", which represents the type of the current list. The new list is added to the list stack, and the top list of the stack is the newly pushed list. Field information is extracted from the current basic block and saved in dictionary form. It is then matched with standard fields to improve the extracted field information, which is stored in the top list of the stack. The loop basic block is treated as being composed of identical sub-data blocks. The field type (i.e., the number of sub-data blocks) and field length (i.e., the length of the sub-data blocks) are analyzed and extracted according to the loop condition. After the current basic block is traversed, the top list of the stack is popped.

[0038] If it is a basic function block, two new lists are added sequentially to the top list of the stack. The first new list has only one element "2" and the second new list has only one element "5". "5" is used to mark the end of the function. The first new list is pushed onto the list stack. The contents of the basic function block are traversed sequentially and the field information is extracted and saved to the top list of the stack. After the current basic function block is traversed, the top list of the stack is popped.

[0039] If it is an if or switch block, a new list is added to the top of the stack, containing only one element "1", and then the new list is pushed onto the stack. For an if block, the operation of creating a new list is repeated, this time containing only one element "2". The field information extracted from the if block is saved to the top of the stack. After the current if block is traversed, the top of the stack is popped.

[0040] If it is an else if block, case block, default block, for loop block, or while loop block, a new list is added to the top of the stack. The new list contains only one element, "2". The field information extracted from the current block is saved to the top of the stack. For for loop blocks or while loop blocks, they are traversed only once without entering the loop. After the current block is traversed, the top of the stack is popped.

[0041] If it is an else block, a new list is added to the top list of the stack. The new list has only one element "3". The new list is pushed onto the list stack. The field information extracted from the else block is saved to the top list of the stack. When the else block is finished, the top list of the stack is popped twice.

[0042] If it is a function return block, i.e. a return statement, then a new list is added to the list at the top of the stack, and the new list has only one element "4";

[0043] After sequentially traversing all the blocks of code to be parsed, a list will be obtained containing a segment of all data packet types and their constituent fields of the protocol to be parsed.

[0044] According to a preferred embodiment of the present invention, concatenating segments of each data packet type to generate a complete data packet type includes:

[0045] For each basic block list in the obtained global list packet_all, the non-normal basic block lists are expanded and concatenated into new lists in different ways until all basic block lists are normal basic block lists or function-ending basic block lists. All basic block lists of each packet type are then merged to obtain the complete packet type.

[0046] Further preferredly, for a basic block list with a type value of "0", the list itself is selected for concatenation with the preceding and following segments during concatenation; for basic block lists with type values ​​of "2" and "3", where the elements are in a sequential relationship, they are concatenated with the preceding and following segments during concatenation; for a basic block list with a type value of "1", where the elements are in a parallel relationship, each element is selected for concatenation with the preceding and following lists during concatenation; for a basic block list with a type value of "4", the first basic block list with a type value of "5" is searched for, and the basic block lists with type values ​​of "4" and "5" and the basic block lists between them are deleted; for a basic block with a type value of "5", the list itself is selected for concatenation with the preceding and following segments during concatenation.

[0047] After concatenation, a segment list containing all data packet types and their constituent fields is obtained. The elements in the segment list of each data packet type are merged, and only dictionary elements, i.e. fields, are retained; finally, all data packet types of the protocol and their constituent fields are obtained.

[0048] An automated system for extracting network communication protocol syntax information includes:

[0049] The preprocessing module is used to preprocess all protocol parsing files, including extracting protocol names, extracting protocol parsers and associating them with protocol names, and extracting the hierarchical relationship between protocol parsers;

[0050] The main parsing module is used to process the protocol to be parsed selected by the user, including extracting all field information of the protocol to be parsed, expanding the content of the main parsing function, extracting all data packet types and their constituent field information, and concatenating the segments of each data packet type to generate a complete data packet type.

[0051] The beneficial effects of this invention are:

[0052] This invention extracts protocol syntax information from Wireshark's network communication protocol parsing files through in-depth analysis and stores it in a structured manner. Compared to other methods for extracting network communication protocol syntax information, this invention has broad applicability, high automation and efficiency, and high accuracy, effectively saving labor and time costs. Attached Figure Description

[0053] Figure 1 This is a general framework diagram of the automatic extraction method for network communication protocol syntax information of the present invention:

[0054] Figure 2 This is a schematic diagram of the protocol parsing file preprocessing process of the present invention;

[0055] Figure 3 This is a schematic diagram illustrating the process of the present invention for processing the protocol to be parsed selected by the user;

[0056] Figure 4 This is a schematic diagram of the final list structure;

[0057] Figure 5 To Figure 4 A flowchart illustrating the process of concatenating lists. Detailed Implementation

[0058] The present invention will be further described below with reference to the accompanying drawings, but is not limited thereto.

[0059] Example 1

[0060] An automated method for extracting network communication protocol syntax information, such as Figure 1 As shown, it includes:

[0061] All protocol parsing files are preprocessed, including extracting protocol names, extracting protocol parsers and associating them with protocol names, and extracting the hierarchical relationships between protocol parsers;

[0062] The process involves processing the protocol selected by the user, including extracting all field information of the protocol, expanding the content of the main parsing function, extracting all data packet types and their constituent field information, and concatenating the segments of each data packet type to generate a complete data packet type.

[0063] Example 2

[0064] The method for automatically extracting network communication protocol syntax information according to Embodiment 1 differs in that:

[0065] The protocol name and protocol parser are extracted based on the protocol registration mechanism of the Wireshark protocol parsing plugin and associated with the protocol name.

[0066] The hierarchical relationship between parsers is extracted based on the maintenance mechanism of the Wireshark parser table and by traversing the explicit call relationship between upper and lower level parsers.

[0067] Extract the protocol name, including:

[0068] Read each Wireshark protocol parsing file sequentially, extract and save all information related to the protocol name, including the protocol name, protocol filter name, protocol unique identifier, and protocol handle, and write this information related to the protocol name and the corresponding protocol parsing file name into the protocol list.

[0069] Extract the protocol parser and associate it with the protocol name, including:

[0070] Extract all information related to the protocol parser, including the protocol parser's filter name, main parsing function name, corresponding protocol handle, and protocol parser handle, and write it into the parser table;

[0071] The protocol list is searched based on the extracted protocol resolver handles. The protocol resolvers are associated with the corresponding protocol names, and an attribute with a global or local value is added to each protocol resolver, depending on whether the protocol resolver has been globally registered.

[0072] The associations and attributes are written together into the resolver table.

[0073] Extract the hierarchical relationship between protocol parsers, including:

[0074] The hierarchical relationship between different protocol parsers is the calling relationship between different protocol parsers. The network communication protocols corresponding to different protocol parsers with calling relationships are in a hierarchical relationship. The hierarchical relationship of each protocol parser is extracted and written into the parser table. After processing all protocol parsing files, the hierarchical relationship of each protocol parser in the parser table is improved.

[0075] Extract all field information of the protocol to be parsed, including:

[0076] For a selected protocol to be parsed, the preprocessed protocol list is searched based on the protocol name, the corresponding protocol parsing file name is obtained and the protocol parsing file is read. All field information, including field identifier, name, filter name, data type, cardinality, and optional value information, is extracted from the protocol parsing file and saved to the field list as standard fields.

[0077] Expanding the content of the main analytic function refers to: locating the main analytic function and improving its content; specifically including:

[0078] The parser table obtained from the preprocessing is searched based on the name of the protocol to be parsed to obtain the name of the main parsing function; the position of the main parsing function is located and its body is extracted as the program block to be parsed; the program block to be parsed is then normalized, specifically by sequentially traversing each line of the program block while performing the following expansion operations:

[0079] If a sub-function call exists, the definition of the function is searched in the current protocol parsing file. If the definition is found, all the code of the function is copied to the calling statement. If it is not found, no operation is performed.

[0080] If it is a code that reads data packets from a buffer, then replace the offset information in it with the corresponding real value;

[0081] If it is field parsing code, replace the field offset information with the corresponding actual value and replace the field length information with the actual value. If it cannot be replaced with an actual value, it means that the length of the current field is related to the value of a previous field. Set the length information of that field to the form of the expression "value of field".

[0082] If an if / else if / else statement is followed by only one line of statements without curly braces, then add curly braces to that line of statements. For if / else if statements that lack an else statement, add a complete else statement with an empty else statement. Repeat this process for the expanded parsed block until all function calls in the parsed block are completed, and all offset information related to data packet buffer reading, as well as offset and length information related to field parsing, are replaced, resulting in a complete parsed block.

[0083] Extract all packet types of the protocol to be parsed and all field information contained in each packet type, including:

[0084] The obtained code blocks to be parsed are divided into different types of basic blocks according to their different structures, including ordinary basic blocks, branch basic blocks, branch item basic blocks, and return basic blocks. Branch basic blocks refer to the code block of the entire branch structure, including conditional structure (if / else if / else) basic blocks and selection structure (switch) basic blocks. Branch item basic blocks refer to the code block of each branch in the branch structure, including if basic blocks, else if basic blocks, else basic blocks, case basic blocks, and default basic blocks. Specifically, function basic blocks are considered as branch basic blocks. Return basic blocks include function return basic blocks and function termination basic blocks. Ordinary basic blocks refer to code blocks other than branch basic blocks, branch item basic blocks, and return basic blocks. Specifically, loop structures such as do…while loops, while loops, and for loops are considered ordinary basic blocks.

[0085] During the parsing process, each basic path of the program block to be parsed is regarded as a parsing path of a data packet type. Based on this, all data packet types are extracted, and all field information that makes up each data packet type is extracted, including the field identifier, variable name, offset, length, and byte order. The field identifier is matched with standard fields to complete the field name and optional values.

[0086] Extract field information from all basic blocks contained in each basic path of the program block to be parsed, and store the field information in different basic block lists. At the same time, establish the hierarchical relationship between these basic block lists, and concatenate all basic block lists in each basic path to obtain the field list corresponding to all data packet types.

[0087] During the parsing process, a global list is used to store a list of all basic blocks and their contained field information.

[0088] Create and maintain a stack to store all the basic block lists, called a list stack; the basic block list at the top of the stack is used to store the field information extracted from the current basic block, called the top stack list.

[0089] Parse the basic blocks in the block to be parsed:

[0090] First, create a global list `packet_all` to store all the basic block lists and their field information, and add the global list `packet_all` to the list stack as the initial top list of the stack;

[0091] Then, the following processing is performed on the next basic block:

[0092] If it is a normal basic block, a new list is added to the top list of the stack. The new list has only one element, "0", which represents the type of the current list. The new list is added to the list stack, and the top list of the stack is the newly pushed list. Field information is extracted from the current basic block and saved in dictionary form. It is then matched with standard fields to improve the extracted field information, which is stored in the top list of the stack. The loop basic block is treated as being composed of identical sub-data blocks. The field type (i.e., the number of sub-data blocks) and field length (i.e., the length of the sub-data blocks) are analyzed and extracted according to the loop condition. After the current basic block is traversed, the top list of the stack is popped.

[0093] If it is a basic function block, two new lists are added sequentially to the top list of the stack. The first new list has only one element "2" and the second new list has only one element "5". "5" is used to mark the end of the function. The first new list is pushed onto the list stack. The contents of the basic function block are traversed sequentially and the field information is extracted and saved to the top list of the stack. After the current basic function block is traversed, the top list is popped from the stack.

[0094] If it is an if or switch block, a new list is added to the top of the stack, containing only one element "1", and then the new list is pushed onto the stack. For an if block, the operation of creating a new list is repeated, this time containing only one element "2". The field information extracted from the if block is saved to the top of the stack. After the current if block is traversed, the top of the stack is popped.

[0095] If it is an else if block, case block, default block, for loop block, or while loop block, a new list is added to the top of the stack. The new list contains only one element, "2". The field information extracted from the current block is saved to the top of the stack. For for loop blocks or while loop blocks, they are traversed only once without entering the loop. After the current block is traversed, the top of the stack is popped.

[0096] If it is an else block, a new list is added to the top list of the stack. The new list has only one element "3". The new list is pushed onto the list stack. The field information extracted from the else block is saved to the top list of the stack. When the else block is finished, the top list of the stack is popped twice.

[0097] If it is a function return block, i.e. a return statement, then a new list is added to the list at the top of the stack, and the new list has only one element "4";

[0098] After sequentially traversing all the blocks of code to be parsed, a list will be obtained containing a segment of all data packet types and their constituent fields of the protocol to be parsed.

[0099] Segments of each data packet type are concatenated to generate a complete data packet type, including:

[0100] For each basic block list in the obtained global list packet_all, the non-normal basic block lists are expanded and concatenated into new lists in different ways until all basic block lists are normal basic block lists or function-ending basic block lists. All basic block lists of each packet type are then merged to obtain the complete packet type.

[0101] For a list of basic blocks with a type value of "0", the list itself is selected for concatenation with the preceding and following segments during the concatenation process; for lists of basic blocks with types of "2" and "3", where the elements are in a sequential relationship, they are concatenated with the preceding and following segments during the concatenation process; for a list of basic blocks with a type value of "1", where the elements are in a parallel relationship, each element is selected for concatenation with the preceding and following lists during the concatenation process; for a list of basic blocks with a type value of "4", the first list of basic blocks with a type value of "5" is searched for, and the lists of basic blocks with types of "4" and "5" and the basic block lists between them are deleted; for a basic block with a type value of "5", the list itself is selected for concatenation with the preceding and following segments during the concatenation process;

[0102] After concatenation, a segment list containing all data packet types and their constituent fields is obtained; the elements in the segment list of each data packet type are merged, and only dictionary elements, i.e., fields, are retained; finally, all data packet types of the protocol and their constituent fields are obtained.

[0103] Example 3

[0104] The method for automatically extracting network communication protocol syntax information according to Embodiment 1 differs in that:

[0105] like Figure 2 As shown, the protocol name and protocol parser are extracted according to the protocol registration mechanism of the Wireshark protocol parsing plugin and associated with the protocol name;

[0106] The hierarchical relationship between parsers is extracted based on Wireshark's parser table maintenance mechanism (i.e., lower-layer protocols generate parser tables, and upper-layer protocols add entries to their parser tables) and by traversing the explicit call relationship between upper and lower-layer parsers (lower-layer parsers explicitly specify upper-layer parsers by calling the find_dissector() function).

[0107] Extract the protocol name, including:

[0108] Each Wireshark protocol parsing file is read sequentially, searching for the protocol registration function `proto_register_protocol()`. This function is usually located within the `proto_register_XXX()` function, where "XXX" represents the corresponding protocol name. A single protocol parsing file may register multiple protocols simultaneously. The registration function for each protocol is parsed, extracting and saving all information related to the protocol name, including the protocol name, the protocol filter name, the protocol's unique identifier, and the protocol handle. The returned protocol handle is also extracted for associating with the protocol parser. The extracted protocol name information and the corresponding protocol parsing file name are saved to the protocol list `proto_list`.

[0109] Extract the protocol parser and associate it with the protocol name, including:

[0110] The protocol parsing file searches for the global resolver registration function `register_dissector()` and the local resolver registration function `create_dissector_handle()`. These two functions are usually located in `proto_register_XXX()` or `proto_reg_handoff_XXX()`, where "XXX" is the corresponding protocol name. Information about each parameter is extracted: the resolver's filter name, the main parsing function name, the corresponding protocol handle, and the protocol resolver handle. This information is then written to the resolver table for resolvingr association.

[0111] Based on the extracted protocol resolver handles, the protocol list is searched, and each protocol resolver is associated with its corresponding protocol name (including long and short protocol names). Depending on whether the protocol resolver is globally registered, an attribute with a global or local value is added to each resolver. The associations and attributes are then written to the resolver table. Specifically, for resolvers registered using `register_dissector()`, a global attribute `global` is added; for resolvers registered using `create_dissector_handle()`, a local attribute `local` is added. The extracted protocol handles are searched in the obtained protocol table `proto_list`, and the resolvers are associated with their corresponding protocol names. The extracted resolver information and association information are then saved to the resolver table `dissect_list`.

[0112] Extract the hierarchical relationship between protocol parsers, including: the hierarchical relationship between different protocol parsers, i.e., the calling relationship between different protocol parsers; different protocol parsers with calling relationships correspond to network communication protocols in a hierarchical relationship; extract the hierarchical relationship of each protocol parser and write it into the parser table, and complete the hierarchical relationship of each protocol parser in the parser table after processing all protocol parsing files. When processing a protocol to be parsed that is above the TCP / UDP protocol, parse the protocol parsing file corresponding to each parser in sequence according to its hierarchical relationship to achieve complete parsing of the protocol to be parsed. Specifically, this includes:

[0113] Determine the hierarchical relationship of each parser and add an upper or lower property to the parser. If the `heur_dissector_add()` or `dissector_add_unit_with_preference()` function is called in the parser registration function `proto_reg_handoff_XXX()`, it indicates that the current protocol parser is a higher-level parser and the corresponding protocol is a higher-level protocol. Extract the "parsing table name" parameter from these two functions, and create a parser with this "parsing table name" as the lower-level parser corresponding to the current parser. Add a `lower` attribute to this parser, with the attribute value being "parsing table name". If the `register_dissector_table()` or `register_heur_dissector_list()` function is called in the protocol registration function `proto_register_XXX()` to create a "parsing table name", or if the `find_dissector()` function is called, it indicates that the current parser is a lower-level parser and the corresponding protocol is a lower-level protocol. Furthermore, the parser parameter in the `find_dissector()` function is a higher-level parser and the corresponding protocol is a higher-level protocol. Therefore, add an `upper` attribute to this parser, with the attribute value being "parsing table name". For example, in the protocol parser registration function `proto_reg_handoff_tpkt()` in the protocol parser file `packet-tpkt.c`, `find_dissector(“ositp”)` is called, indicating that the parser `dissect_tpkt` is a lower-level parser, meaning the tpkt protocol is a lower-level protocol, while the parser represented by the parameter "ositp" is an upper-level parser, meaning the corresponding COTP protocol is an upper-level protocol. Therefore, an attribute "lower": "ositp" is added to the parser `dissect_tpkt`. This invention parses the upper and lower layer protocols and corresponding parsers for all protocols above the transport layer (TCP / UDP), storing this information in the corresponding protocol parser attribute of the parser table. After extracting the hierarchical relationship of all parsers, the upper and lower layer relationships of each parser are further refined based on the current parser table `dissect_list`. For example, based on the attribute "lower": "ositp" of the parser dissect_tpkt, the attribute "upper": "tpkt" is added to the parser dissect_ositp. Subsequent parsing of a specific protocol will then proceed according to the hierarchical relationship stored in the parsing table dissect_list.

[0114] like Figure 3 As shown, extract all field information of the protocol to be parsed, including:

[0115] For a selected protocol to be parsed, the preprocessed protocol list is searched based on the protocol name, the corresponding protocol parsing file name is obtained and the protocol parsing file is read, all field information of the protocol is extracted from the protocol parsing file, and all fields contained in the protocol and their field information are extracted from the "header fields" data structure maintained in the Wireshark protocol parsing plugin, including the field identifier, name, filter name, data type, cardinality, and optional value information, and saved to the field list.

[0116] Specifically, it includes:

[0117] The user selects an option containing a protocol name and a parser name. Based on the selected protocol name, the preprocessed protocol table `proto_list` is searched to find the corresponding protocol parser file name and read it. Each line is read sequentially, and the protocol registration function `proto_register_XXX()` is parsed, where `XXX` is the protocol name. An array of structures of type `hf_register_info` is extracted, and information for each field is extracted from the array, including the field identifier, field name, possible values, and data type. Furthermore, if a field has an attribute of the form `VALS(XXX)`, where `XXX` is an array storing field information, each element of the array representing all possible values ​​for that field; this field is often a packet classification field. All field information is saved to the field list `field_list` as standard fields, and then further processing is performed. The data definitions for the field information are shown in Table 1.

[0118] Table 1

[0119] name Field Name identifier Field identifier variable_name field variable name offset Field offset type Field type length Field length endian byte order

[0120] Expanding the content of the main analytic function refers to: locating the main analytic function and improving its content; specifically including:

[0121] Based on the parser name selected by the user, a search is performed in the preprocessed parser table `dissect_list` to obtain the corresponding main parsing function name `dissect_XXX`, where `XXX` is the protocol name. The location of the main parsing function in the protocol file to be parsed is then determined, and its function body is extracted as the program block to be parsed. Next, the program block to be parsed is standardized by sequentially traversing each line of the block while performing the following expansion operations:

[0122] If it is a code that reads data packets from a buffer, then the offset information in it is replaced with the actual value. For example, in "tvb_get_ntohs(tvb,offset+4)", "offset+4" is replaced with "4" (in this case, offset = 0).

[0123] If the field parsing code is "proto_tree_add_uint" or "proto_tree_add_item", then replace the offset and length parameters with the actual values ​​of the offset and length. If the length is in the form of "len-1", it means that the current length parameter is determined by the "len" field, and replace the length parameter with "value of(len)-1".

[0124] If an if / else if / else statement consists of only one line of code and does not use curly braces "{}", then curly braces "{}" are added to that line. Additionally, it checks if the if / else if statement ends without an else statement; if it does, an empty else statement is added.

[0125] If a sub-function call exists, such as "call_dissector_" or "dissect_", then the definition of the function is searched in the current parsing file and all the code of the function is extracted and then inserted under the call statement. If the definition of the function is not found, no operation is performed and the process continues to the next line.

[0126] Repeat the entire process until all lines of the parsed block have been processed, resulting in a standardized parsed block.

[0127] Extract all packet types of the protocol to be parsed and all field information contained in each packet type. Analyze the program structure and control flow transfer of the protocol parsing file, identify all execution paths to obtain a basic path set, and treat each basic path in the basic path set as a packet type to obtain all packet types. Traverse all statements contained in the parsing process of each packet type and trace the generation process of the Wireshark protocol tree to obtain all fields contained in each packet type, as well as the identifier, variable name, offset, length, and byte order information of each field. This includes:

[0128] The obtained code blocks to be parsed are divided into different types of basic blocks according to their different structures, including ordinary basic blocks, branch basic blocks, branch item basic blocks, and return basic blocks. Branch basic blocks refer to the code block of the entire branch structure, including conditional structure (if / else if / else) basic blocks and selection structure (switch) basic blocks. Branch item basic blocks refer to the code block of each branch in the branch structure, including if basic blocks, else if basic blocks, else basic blocks, case basic blocks, and default basic blocks. Specifically, function basic blocks are considered as branch basic blocks. Return basic blocks include function return basic blocks (i.e., function return statements) and function termination basic blocks. Ordinary basic blocks refer to code blocks other than branch basic blocks, branch basic blocks, branch item basic blocks, and return basic blocks. Specifically, loop structures such as do…while loops, while loops, and for loops are considered ordinary basic blocks.

[0129] During the parsing process, each basic path of the program block to be parsed is regarded as a parsing path of a data packet type. Based on this, all data packet types are extracted, and all field information that makes up each data packet type is extracted, including the field identifier, variable name, offset, length, and byte order. The field identifier is matched with standard fields to complete the field name and optional values.

[0130] Extract field information from all basic blocks contained in each basic path of the program block to be parsed, and store the field information in different basic block lists. At the same time, establish the hierarchical relationship between these basic block lists, and concatenate all basic block lists in each basic path to obtain the field list corresponding to all data packet types.

[0131] During the parsing process, a global list is used to store a list of all basic blocks and their contained field information.

[0132] This invention stores all basic block lists in a nested manner in a global list to represent the hierarchical relationship between the basic block lists. Specifically, sub-lists are used as elements in the parent list, and each sub-list represents a level in the hierarchy. The global list serves as the final output list after the parsing process is complete.

[0133] The basic block list is used to store field information extracted from the target basic block. The global list is a list that stores all basic block lists. During the parsing process, there will be multiple basic block lists, each corresponding to the field information extracted from a different basic block.

[0134] A stack is created and maintained to store the list of all basic blocks; this is called the list stack. The list of basic blocks at the top of the stack stores the field information extracted from the current basic block; this is called the top stack list. Different stack operations are performed based on the type of the basic block. The type values ​​and stack operations for different types of basic blocks are shown in Table 2.

[0135] Table 2

[0136]

[0137]

[0138] Parse the basic blocks in the block to be parsed:

[0139] First, create a global list `packet_all` to store all the basic block lists and their field information, and add the global list `packet_all` to the list stack as the initial top list of the stack;

[0140] Then, the following processing is performed on the next basic block:

[0141] When processing each basic block, a list ['X'] is added to the top list of the stack_list, where "X" is the type value of the current basic block. The type values ​​of each basic block are shown in Table 1. Specifically, if it is an if basic block, a list ['1'] is added to the top list of the stack, and this list ['1'] is added to the stack_list. Then, a list ['2'] is added to the current top list ['1'], and then the list ['2'] is added to the stack_list. If it is a function basic block, two lists ['2'] and ['5'] are added to the top list of the stack_list, and only the list ['2'] is put into the stack_list.

[0142] The system extracts field information from the current basic block and stores 'field1' and 'field2' in dictionary form. Then, it matches these fields with the preprocessed field list `field_all` to complete the field names and data types of 'field1' and 'field2'. Finally, 'field1' and 'field2' are saved to the top list of the stack, which is now ['X', 'field1', 'field2']. Specifically, for loop-based basic blocks, the loop count is used as the length attribute of the data block, and the change in the loop variable is used as the data type attribute. After the current basic block is traversed, a pop operation is performed based on the type value of the top list (i.e., the first element of the top list), indicating the end of processing for the current basic block. The specific number of pop operations is shown in Table 1.

[0143] Repeat this step until the last line of the block to be parsed is reached. This step ends and a list named packets_all is obtained, which stores all data packet types of the current protocol and the segments of their constituent fields.

[0144] Segments of each data packet type are concatenated to generate a complete data packet type.

[0145] First, all lists are expanded in different ways according to their types until all elements in the list `packet_all` are lists of ordinary basic blocks or lists of function-ending basic blocks, i.e., list types "0" or "5". At this point, each element in `packet_all` corresponds to a segment of a data packet type and its constituent fields. Then, the segments of each data packet type are concatenated, retaining only dictionary-type elements (fields). Finally, each element in the list `packet_all` represents a data packet type and its constituent fields. For example... Figure 4 As shown, each letter represents a different list of basic blocks, and the superscript numbers indicate their type. A, D, G, J, M, N, O, Q, and R are ordinary basic block lists; B and H are branch basic block lists; C, E, I, L, and P are branch item basic blocks, where E is a function basic block; K and F are termination basic blocks, where K is a function return basic block and F is a function termination basic block. The global list `packets_all` stores three basic block lists A, B, and R; the branch basic block list B stores two basic block lists C and P, and so on.

[0146] For branch blocks, the stored fields are in a parallel relationship; during concatenation, each element is selected and concatenated with the preceding and following segments. For other types of blocks, the stored fields are in a sequential relationship; during concatenation, they are directly concatenated with the preceding and following segments. For example... Figure 5 As shown:

[0147] If B is the basic block of the branch, then each of its elements C and P is selected and concatenated with the preceding and following segments to obtain two branches A\C\R and A\P\R.

[0148] C and P are basic blocks of branch items. Their elements are in order. They are directly concatenated to obtain branches A\D\E\F\O\R and A\P\R. At this point, branches A\P\R are all ordinary basic blocks. The concatenation of this branch is complete.

[0149] Continue processing branches A\D\E\F\O\R, where E is the basic block of the branch item, and directly concatenate them to obtain branches A\D\G\H\N\F\O\R;

[0150] H is the basic branch block. Selecting I and L respectively and splicing them together will result in branches A\D\G\I\N\F\O\R and A\D\G\L\N\F\O\R.

[0151] I and J are the basic blocks of the branch item. They are directly concatenated to obtain branches A\D\G\J\K\N\F\O\R and A\D\G\M\N\F\O\R. At this point, branches A\D\G\M\N\F\O\R are both ordinary basic blocks and function termination basic blocks. This branch concatenation is complete.

[0152] Continue processing branches A\D\G\J\K\N\F\O\R. K is a function return basic block, so search for the first function end basic block F, and delete K, F and the segments between them to get branch A\D\G\J\O\R. At this point, branches A\D\G\J\O\R are all ordinary basic blocks, and this branch concatenation is complete.

[0153] After the concatenation is completed, three branches are obtained: A\P\R, A\D\G\M\N\F\O\R, and A\D\G\J\O\R. Then, the elements of each branch are merged, and only the elements of type dictionary are kept. Finally, the three branches correspond to three data packet types, and each element is a field that makes up the data packet type.

[0154] Example 4

[0155] An automated system for extracting network communication protocol syntax information includes:

[0156] The preprocessing module is used to preprocess all protocol parsing files, including extracting protocol names, extracting protocol parsers and associating them with protocol names, and extracting the hierarchical relationship between protocol parsers;

[0157] The main parsing module is used to process the protocol to be parsed selected by the user, including extracting all field information of the protocol to be parsed, expanding the content of the main parsing function, extracting all data packet types and their constituent field information, and concatenating the segments of each data packet type to generate a complete data packet type.

Claims

1. A method for automatically extracting network communication protocol syntax information, characterized in that, include: All protocol parsing files are preprocessed, including extracting protocol names, extracting protocol parsers and associating them with protocol names, and extracting the hierarchical relationships between protocol parsers; The process involves processing the protocol selected by the user, including extracting all field information of the protocol to be parsed, expanding the content of the main parsing function, extracting all data packet types and their constituent field information, and concatenating the field information of each data packet type to generate a complete data packet type. Expanding the content of the main analytic function refers to: locating the main analytic function and improving its content; specifically including: The parser table obtained from the preprocessing is searched based on the name of the protocol to be parsed to obtain the name of the main parsing function; the position of the main parsing function is located and the body of the main parsing function is extracted as the program block to be parsed; the program block to be parsed is then standardized. Extract all data packet types and their constituent fields, including: The obtained program blocks to be parsed are divided into different types of basic blocks according to their different structures, including ordinary basic blocks, branch basic blocks, branch item basic blocks, and return basic blocks. During the parsing process, each basic path of the program block to be parsed is regarded as a parsing path of a data packet type. Based on this, all data packet types are extracted, and all field information of the data packets that make up each data packet type is extracted. Extract field information from all basic blocks contained in each basic path of the program block to be parsed, and store the field information in different basic block lists. At the same time, establish the hierarchical relationship between these basic block lists, and concatenate all basic block lists in each basic path to obtain the field list corresponding to all data packet types.

2. The method for automatically extracting network communication protocol syntax information according to claim 1, characterized in that, The protocol name and protocol parser are extracted based on the protocol registration mechanism of the Wireshark protocol parser plugin and associated with the protocol name; the hierarchical relationship between parsers is extracted based on the maintenance mechanism of the Wireshark parser table and by traversing the explicit call relationship between upper and lower level parsers.

3. The method for automatically extracting network communication protocol syntax information according to claim 1, characterized in that, Extract the protocol name, including: Read each Wireshark protocol parsing file sequentially, extract and save all information related to the protocol name, including the protocol name, the protocol filter name, the protocol's unique identifier, and the protocol handle, and write this information related to the protocol name and the corresponding protocol parsing file name into the protocol list; Extract the protocol parser and associate it with the protocol name, including: Extract all information related to the protocol parser, including the protocol parser's filter name, main parsing function name, corresponding protocol handle, and protocol parser handle, and write it into the parser table; The protocol list is searched based on the extracted protocol resolver handles. The protocol resolvers are associated with the corresponding protocol names, and an attribute with a global or local value is added to each protocol resolver, depending on whether the protocol resolver has been globally registered. The associations and attributes are written together into the resolver table.

4. The method for automatically extracting network communication protocol syntax information according to claim 1, characterized in that, Extract the hierarchical relationship between protocol parsers, including: The hierarchical relationship between different protocol parsers is the calling relationship between different protocol parsers. The network communication protocols corresponding to different protocol parsers with calling relationships are in a hierarchical relationship. The hierarchical relationship of each protocol parser is extracted and written into the parser table. After processing all protocol parsing files, the hierarchical relationship of each protocol parser in the parser table is improved.

5. The method for automatically extracting network communication protocol syntax information according to claim 1, characterized in that, Extract all field information of the protocol to be parsed, including: For a selected protocol to be parsed, the preprocessed protocol list is searched based on the protocol name, the corresponding protocol parsing file name is obtained and the protocol parsing file is read. All field information, including field identifier, name, filter name, data type, cardinality, and optional value information, is extracted from the protocol parsing file and saved to the field list as standard fields.

6. The method for automatically extracting network communication protocol syntax information according to claim 1, characterized in that, Iterate through each line of the block to be parsed sequentially, while performing the following expansion operations: If a subfunction call exists, search for the definition of the subfunction in the current protocol parsing file. If the definition is found, copy all the code of the subfunction to the calling statement. If it is not found, do not perform any operation. If it is a code that reads data packets from a buffer, then replace the offset information in it with the corresponding real value; If it is field parsing code, replace the field offset information with the corresponding actual value and replace the field length information with the actual value. If it cannot be replaced with an actual value, it means that the length of the current field is related to the value of a previous field. Set the length information of that field to the form of the expression "value of field". If an if / else if / else statement is followed by only one line of statements without curly braces, then add curly braces to that line of statements. For if / else if statements that lack an else statement, add a complete else statement with an empty else statement. Repeat this process for the expanded parsed block until all function calls in the parsed block are completed, and all offset information related to data packet buffer reading, as well as offset and length information related to field parsing, are replaced, resulting in a complete parsed block.

7. The method for automatically extracting network communication protocol syntax information according to claim 1, characterized in that, A branch block is the code block that constitutes the entire branch structure, including conditional structure blocks and selection structure blocks; a branch item block is the code block for each branch in the branch structure, including if, else if, else, case, and default blocks; a return block includes function return blocks and function termination blocks; a regular block is the code block other than branch blocks, branch item blocks, and return blocks. All field information includes the field identifier, variable name, offset, length, and byte order. The field identifier is matched with standard fields to complete the field name and optional values.

8. The method for automatically extracting network communication protocol syntax information according to claim 1, characterized in that, During the parsing process, a global list is used to store a list of all basic blocks and their contained field information; Create and maintain a stack to store all the basic block lists, called a list stack; the basic block list at the top of the stack is used to store the field information extracted from the current basic block, called the top stack list; Parse the basic blocks in the block to be parsed: First, create a global list `packet_all` to store all the basic block lists and their field information, and add the global list `packet_all` to the list stack as the initial top list of the stack; Then, the following processing is performed on the next basic block: If it's a regular basic block, a new list is added to the top of the stack. This new list contains only one element, "0," which represents the type of the current list. The new list is then added to the list stack, making the top list the newly pushed list. Field information is extracted from the current basic block and saved as a dictionary. This dictionary is then matched with standard fields to refine the extracted field information, which is then stored in the top list. Looping basic blocks are treated as being composed of identical sub-data blocks. Based on the loop condition, the field types and lengths are analyzed and extracted. After the current basic block is traversed, the top list is popped from the stack. If it is a basic function block, two new lists are added to the top list of the stack in sequence. The first new list has only one element "2" and the second new list has only one element "5". "5" is used to mark the end of the function. The first new list is pushed onto the list stack. The contents of the basic function block are traversed sequentially and the field information is extracted and saved to the top list of the stack. After the current basic function block is traversed, the top list of the stack is popped. If it is an if or switch block, a new list is added to the top of the stack, containing only one element "1", and then the new list is pushed onto the stack. For an if block, the operation of creating a new list is repeated, this time containing only one element "2". The field information extracted from the if block is saved to the top of the stack. After the current if block is traversed, the top of the stack is popped. If it is an else if block, case block, default block, for loop block, or while loop block, a new list is added to the top of the stack. The new list contains only one element, "2". The field information extracted from the current block is saved to the top of the stack. For for loop blocks or while loop blocks, they are traversed only once without entering the loop. After the current block is traversed, the top of the stack is popped. If it is an else block, a new list is added to the top list of the stack. The new list has only one element "3". The new list is pushed onto the list stack. The field information extracted from the else block is saved to the top list of the stack. When the else block is finished, the top list of the stack is popped twice. If it is a function return block, i.e., a return statement, then a new list is added to the list at the top of the stack, and the new list has only one element "4"; After sequentially traversing all the blocks of code to be parsed, a list will be obtained containing a segment of all data packet types and their constituent fields of the protocol to be parsed.

9. A method for automatically extracting network communication protocol syntax information according to any one of claims 1-8, characterized in that, Segments of each data packet type are concatenated to generate a complete data packet type, including: For each basic block list in the obtained global list packet_all, the non-normal basic block lists are expanded and concatenated into new lists in different ways until all basic block lists are normal basic block lists or function-ending basic block lists. All basic block lists of each data packet type are then merged to obtain the complete data packet type. For a list of basic blocks with a type value of "0", the list itself is selected for concatenation with the preceding and following segments. For lists of basic blocks with types of "2" and "3", where the elements are in a sequential relationship, the list is concatenated with the preceding and following segments. For a list of basic blocks with a type value of "1", where the elements are in a parallel relationship, each element is selected for concatenation with the preceding and following lists. For a list of basic blocks with a type value of "4", the first list of basic blocks with a type value of "5" is searched for, and the lists of basic blocks with types of "4" and "5" and the basic block lists between them are deleted. For a basic block with a type value of "5", the list itself is selected for concatenation with the preceding and following segments. After concatenation, a segment list containing all data packet types and their constituent fields is obtained. The elements in the segment list of each data packet type are merged, and only dictionary elements, i.e. fields, are retained; finally, all data packet types of the protocol and their constituent fields are obtained.

10. An automated system for extracting network communication protocol syntax information, characterized in that, include: The preprocessing module is used to preprocess all protocol parsing files, including extracting protocol names, extracting protocol parsers and associating them with protocol names, and extracting the hierarchical relationship between protocol parsers; The main parsing module is used to process the protocol to be parsed selected by the user, including extracting all field information of the protocol to be parsed, expanding the content of the main parsing function, extracting all data packet types and their constituent field information, and concatenating the field information of each data packet type to generate a complete data packet type; Expanding the content of the main analytic function refers to: locating the main analytic function and improving its content; specifically including: The parser table obtained from the preprocessing is searched based on the name of the protocol to be parsed to obtain the name of the main parsing function; the position of the main parsing function is located and the body of the main parsing function is extracted as the program block to be parsed; the program block to be parsed is then standardized. Extract all data packet types and their constituent fields, including: The obtained program blocks to be parsed are divided into different types of basic blocks according to their different structures, including ordinary basic blocks, branch basic blocks, branch item basic blocks, and return basic blocks. During the parsing process, each basic path of the program block to be parsed is regarded as a parsing path of a data packet type. Based on this, all data packet types are extracted, and all field information of the data packets that make up each data packet type is extracted. Extract field information from all basic blocks contained in each basic path of the program block to be parsed, and store the field information in different basic block lists. At the same time, establish the hierarchical relationship between these basic block lists, and concatenate all basic block lists in each basic path to obtain the field list corresponding to all data packet types.

Images