Automatic extraction method of network protocol design knowledge based on RFC document

Abstract

The application discloses a network protocol design knowledge automatic extraction method based on RFC documents, and comprises the following steps: acquiring a chapter list of the RFC document and a line list in each chapter; acquiring a structured description in the chapter list and the line list to generate a key line list; analyzing the structured description in the key line list to obtain a formatted field list containing partial information and / or an undefined name list of an automaton; supplementing the formatted field list containing partial information and / or the undefined name list of the automaton based on chapter content to which a key line corresponding to the structured description belongs; and obtaining a network protocol design knowledge extraction result based on the formatted field list containing complete information and the automaton migration list. The application can automatically extract relevant network protocol design information.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a design understanding technology for network protocols, specifically a method for automatically extracting network protocol design knowledge based on RFC documents. Background Technology

[0002] Network protocols are pre-agreed "agreements" between computers to enable communication over a network. As long as the same protocol is followed, communication is possible. Network protocols play a crucial role as the infrastructure of the Internet, making their analysis particularly important. Network protocol analysis mainly falls into two categories: dynamic testing and static analysis. Both methods require domain knowledge to guide the construction of protocol message structures. RFC (Request for Comments) documents are a series of numbered standards that record Internet specifications, protocols, and processes. Basic Internet communication protocols are detailed in their corresponding RFC documents. Analyzing network protocols requires researchers to read numerous RFC documents to acquire domain knowledge. Dynamic testing methods need to extract information about the structural composition of protocol messages, while static testing methods need to obtain specific information such as the length and type of protocol message fields. However, due to the long drafting process of RFC documents, the involvement of multiple organizations, and the wide variety of protocols they contain, the descriptions in RFC documents are often inconsistent and difficult for human readers to understand, posing significant challenges to network protocol analysis. Summary of the Invention:

[0003] To address the aforementioned issues, this invention proposes an automatic extraction method for network protocol design knowledge based on RFC documents. By analyzing and understanding the structured representations and natural language descriptions in RFC documents, relevant network protocol design information can be automatically extracted.

[0004] A method for automatically extracting network protocol design knowledge based on RFC documents, the method comprising:

[0005] Based on the txt format of the RFC document, obtain the list of chapters of the RFC document and the list of lines in each chapter;

[0006] Obtain the structured descriptions of the chapter list and the line list to generate a key line list; wherein, the structured descriptions include: message structure and / or automata;

[0007] The structured description in the key row list is parsed to obtain a formatted field list containing partial information and / or an undefined name list of the automaton; wherein, the information in the formatted field list includes: chapter name, field name, field type, field length, field value, and field description;

[0008] Given the structured description as a message structure, based on the chapter content to which the corresponding key line of the structured description belongs, update the field length, field value, and field type to obtain a formatted field list containing complete information;

[0009] In the case of the structured description being an automaton, the chapter content to which the key line corresponding to the structured description belongs is obtained, and the name in the undefined name list is used to match the chapter content. The obtained automaton migration information is then stored in the automaton migration list.

[0010] By combining the formatted field list containing complete information and the ingress automaton migration list, the network protocol design knowledge extraction result is obtained.

[0011] Furthermore, the message structure includes one or more of the following: graphic format, pseudo-C code format, and ASN.1 format.

[0012] The process of obtaining the structured descriptions of the chapter list and the row list includes:

[0013] When the message structure is in the form of a graphic, after using the regular expression r'^\+-+\+$' to match and identify the starting line of the structured description, continue to traverse the line list, and when a single empty line is encountered, take the current line as the ending line;

[0014] When the message structure is in pseudo-C code form, the regular expression r'^\s*(struct|enum|typedef struct)\s+\w*\s*\{' is used to match and identify the starting line, and the regular expression r'\}[^;]*;' is used to match and identify the ending line, and the curly braces between the starting line and the ending line are correctly closed.

[0015] When the message structure is in ASN.1 form, after matching and identifying the starting line using the regular expressions r'\w+\s+(OBJECT-GROUP|MODULE-COMPLIANCE|OBJECT IDENTIFIER|OBJECT-TYPE|SEQUENCE)' and r'\w+\s+::=\s+\w+\s\{', the line list is traversed. When the current line ends with a right curly brace and the next line is empty, the current line is taken as the ending line.

[0016] Furthermore, the structured description in the key row list is parsed to obtain a formatted field list containing partial information, including:

[0017] For message structures in the form of images and text, obtain the key lines containing the "|" symbol or the ":" symbol, and use the corresponding "|" symbol or the ":" symbol as a delimiter to cut the key lines to obtain several fields and the field names of the fields;

[0018] Based on the variable-length field symbols contained in the field and the space placeholders for the field in the diagram contained in the field name, the field length of each field is obtained and then stored in the formatted field list; wherein, if the field name contains the field length information in the "()" symbol, the field length is updated using the field length information and then stored in the formatted field list.

[0019] After formatting and updating the field names, they are stored in a formatted field list; the formatting and updating includes: removing redundant whitespace characters and extra information contained in the "()" and "[]" symbols in the field names, and restoring the abbreviations contained in the field names to full word descriptions;

[0020] Retrieve the chapter name corresponding to the key row and store it in the formatted field list;

[0021] And / or,

[0022] For a message structure in pseudo-C code form, the type of information described by the current line is determined based on the value of the first element after the current line in the key line list; the information type includes: structured information or enumerated information.

[0023] If the information type described in the preceding line is structured information, then the regular expression r'[\w\:]+\s+[\*\w\d\[\]]+;' is used to match the line describing the field information, the field length contained in the "<>" symbol, and the chapter name corresponding to the line describing the field information. After obtaining the field name, field type, field length, and chapter name respectively, they are stored in the formatted field list.

[0024] If the information type described in the preceding line is enumerated information, then the regular expression r'[\w\d]+\([\w\d\.]+\)' is used to match the lines describing the field information to obtain the field value information. After obtaining the field name, field type and chapter name by matching the field corresponding to the field value information, the information is stored in the formatted field list.

[0025] And / or,

[0026] For message structures in ASN.1 format, the field types are obtained based on the "SYNTAX" keyword in the key line list and stored in the formatted field list;

[0027] Based on the information in the first row of the key row list, determine whether the text description of the key row is nested information;

[0028] If it is nested information, the field name is obtained based on the starting line of the key line text starting with the current field name. Then, the parent field name is obtained based on the content of UP_FIELD in the "::={UP_FIELD}" format description in the ending line of the key line. Finally, the field name and the parent field name are stored in the formatted field list.

[0029] If the information is not nested, the information in the first line will be used as the parent field name for parsing each field in the subsequent parsing. When traversing each subsequent key line, "FIELD_NAME FIELD_TYPE," will be used to obtain the field name, and the field name and the parent field name will be stored in the formatted field list.

[0030] Retrieve the chapter name corresponding to the key line and store it in a formatted field list.

[0031] Furthermore, the updated field length includes:

[0032] Obtain the chapter corresponding to the chapter name, and re-divide the chapter content according to sentences to obtain the updated chapter content;

[0033] Generate a pattern and field name to match the field description based on the field name, match the updated chapter content, and if the match is successful, use the current line and several lines after the current line as the field description.

[0034] The updated chapter content is matched based on the field name, and if the match is successful, the sentences containing the field name are analyzed for part-of-speech tags to obtain sentences containing verbs + cardinality.

[0035] The word corresponding to the cardinality is used as the numerical value of the field length, and the unit of the numerical value is obtained based on the next character of the cardinality;

[0036] Update the field length in the formatted field list based on the numerical value of the field length and the unit of the numerical value.

[0037] Furthermore, the values ​​that the update field can take include:

[0038] Obtain the chapter corresponding to the chapter name, and re-divide the chapter content according to sentences to obtain the updated chapter content;

[0039] The updated chapter content is matched based on the field name, and if a match is successful, the presence of modal verbs or comparative words in the sentence is determined.

[0040] In the presence of modal verbs or comparison words, perform part-of-speech analysis on sentences containing field names to obtain sentences containing MUST(NOT)be(set to)NUM, less than NUM, greater than NUM, or is equal to NUM, are not equal to NUM;

[0041] Convert the modal verbs or comparison words into corresponding symbols and extract the values ​​from MUST(NOT)be(set to)NUM, less than NUM, greater than NUM, or is equal to NUM, are not equal to NUM.

[0042] Based on the symbol and the value, update the field values ​​in the formatted field list.

[0043] Furthermore, the updated field type includes:

[0044] Based on the field description information in the structured description, including protocol message field names, field types, and field lengths, the field types are deduced and the formatted field list is updated.

[0045] or,

[0046] Based on the field names and field lengths in the formatted field list, deduce the field types and update the formatted field list;

[0047] or,

[0048] Based on the field names in the formatted field list, deduce the field types and update the formatted field list;

[0049] or,

[0050] Based on the field lengths in the formatted field list, deduce the field types and update the formatted field list.

[0051] Furthermore, the automaton includes one or more of the following: a list of text and a dotted-line diagram.

[0052] The step of obtaining the structured descriptions of the chapter list and the line list to generate the key line list includes:

[0053] When the automaton is in the form of a list of text, if the current line matches one of the regular expressions [r"State\s+Event\s+Action\s+New State",r"\|\sState\s\|"], it is identified as the starting line. In subsequent traversals, if the current line is an empty line and the starting line is not None, it is identified as the ending line.

[0054] In the case of a dotted-line graph automaton, if the current line matches one of the regular expressions [r'^[\|\+]+---->',r'<----[-\|\+]+$'], it is identified as the starting line. In subsequent traversals, if the current line is an empty line and the starting line is not None, it is identified as the ending line.

[0055] Furthermore, parsing the structured description in the key row list to obtain the list of undefined names for the automaton includes:

[0056] Use spaces to separate key lines to obtain a list of words;

[0057] Iterate through the list of words;

[0058] If a word that is all uppercase or a word that is a noun can be obtained, then add the word that is all uppercase and the word that is a noun to the list of undefined names.

[0059] If words that are all uppercase or words whose part of speech is noun are not retrieved, the regular expression [r'(Event\d+):(\w*)',r'(\w*)[sS]tate:',r'State\(s\):(\w+)'] is used to match the state name and event name of the automaton and add them to the list of undefined names.

[0060] Further, in the case where the structured description is an automaton, the chapter content to which the key line corresponding to the structured description belongs is obtained, and the names in the undefined name list are matched against the chapter content. The obtained automaton migration information is then stored in the automaton migration list, including:

[0061] Obtain the chapter content to which the key line corresponding to the structured description belongs;

[0062] The chapter content is re-divided into sentences to obtain the updated chapter content;

[0063] If the updated chapter content contains a sentence that includes a keyword representing a state transition, and the first half and the second half of the keyword each contain a name from the list of undefined names, then the sentence is determined to contain automaton transition information that changes the state. The sentence is then divided into automaton transition information quadruples and stored in the automaton transition list. The automaton transition information quadruples include: original state, event, action, and destination state.

[0064] If the updated chapter content contains a sentence that includes a keyword representing a state transition, and the first or second half of the keyword contains a name from the list of undefined names, then the sentence is determined to contain automaton transition information that does not change the state. After splitting the sentence into automaton transition information quadruples, the information is stored in the automaton transition list.

[0065] Furthermore, the method also includes:

[0066] Based on the knowledge extraction results of the network protocol design, detection rules and / or fuzz test configuration files are generated.

[0067] Compared with existing technologies, the core of the network protocol design knowledge automatic extraction method of the present invention is to identify five common structured descriptions of message structure and automata in documents, and extract key information such as representative message field names and automata state names. On this basis, combined with pattern matching and natural language extraction technology, richer and more comprehensive design knowledge is further extracted from the above structure and other texts in the document that use natural language descriptions. Finally, complete design knowledge including protocol message field names, field types, field lengths and automata information is formed, which is used to guide rule generation in static detection process, input generation in fuzzing process, etc., and provides domain knowledge support for code defect detection and vulnerability mining of protocol software, thereby improving detection and mining efficiency. Attached Figure Description

[0068] Figure 1 Flowchart of a method for automatically extracting network protocol design knowledge based on RFC documents.

[0069] Figure 2 Example image for illustration.

[0070] Figure 3 Example diagram describing pseudo-C code form.

[0071] Figure 4 Example diagram describing ASN.1 format.

[0072] Figure 5 Example image of a text list description format.

[0073] Figure 6 Example diagram of the description form of the dot-line graph automaton.

[0074] Figure 7 The fuzz test configuration file represents this.

[0075] Figure 8 Example diagram of automata test sequences.

[0076] Figure 9 The first structured description in RFC 4271.

[0077] Figure 10 Example of generating a fuzz test configuration file. Detailed Implementation

[0078] To make the above features and advantages of the present invention more apparent and understandable, the technical solution of the present invention will be further described below through specific embodiments.

[0079] The implementation steps of this invention are mainly divided into five stages: preprocessing stage, structured description recognition stage, key information extraction stage, text understanding stage, and knowledge representation stage.

[0080] Preprocessing stage:

[0081] 1. RFC document preprocessing

[0082] First, the RFC document in TXT format is obtained from the RFC-editor website (RFC-editor is an officially maintained RFC document editing website). Then, the document is traversed, and the header and footer of the RFC document are removed. Generally, the lines near the header and footer of the RFC document have a special byte "0x0c", which can be used for matching. Then, multiple regular expressions (e.g., r'^[0-9.]{1,8}.?+') are used to match the title lines in the remaining RFC document, cutting the RFC document into multiple chapters, resulting in a chapter list. Each element in the chapter list is a chapter, and each chapter is a line list, with each element in the line list being an original line in the document.

[0083] Structured description identification stage:

[0084] Structured description methods can be broadly divided into two categories: message structure and automata. Message structure can be further divided into three forms: text and graphics, pseudo-C code, and ASN.1. Automata can be divided into two forms: text list and dot-line graph.

[0085] 2. Structured description method for identifying message structure

[0086] Iterate through the list of chapters and the list of lines in each chapter obtained in step 1, and attempt to match and identify all forms of structured description text. Identify based on the summarized characteristics, and only identify the first example. If the identification is successful, set the structured description mode of the current RFC document. If the identification fails, it means that the current RFC document does not contain the content to be extracted.

[0087] The specific identification steps for different message structure description styles are as follows:

[0088] a. Graphical format: A chart or graph describing the information composed of "+" and "-" symbols in the message field (e.g., Figure 2The code uses the regular expression r'^\+-+\+$' to match and identify the starting line, records the index of the starting line in the current chapter, and continues to traverse the line list in the chapter. When a single empty line (i.e., containing only a newline character '\n') is encountered, the current line is recorded as the ending line. The code then sets the structured description of the current document's message structure to a graphical format.

[0089] b. Pseudo-C code format: This refers to key lines of text that describe message field composition information in pseudo-C code format (e.g., ...). Figure 3 The regular expression `r'^\s*(struct|enum|typedef struct)\s+\w*\s*\{'` matches the start line, and the regular expression `r'\}[^;]*;' matches the end line. It also needs to consider whether the curly braces `{}` in the key lines are correctly closed; a stack-like approach is used to handle this. The structured description of the current document's message structure is set to pseudo-C code.

[0090] c. ASN.1 format: This refers to the key line text describing the message field composition information in ASN.1 format (e.g., ...). Figure 4 Using the regular expression r'\w+\s+(OBJECT-GROUP|MODULE-COMPLIANCE|OBJECT)

[0091] IDENTIFIER|OBJECT-TYPE|SEQUENCE)' and r'\w+\s+::=\s+\w+\s\{' match the start line. If the current line ends with a right curly brace "}" and the next line is blank, it is identified as the end line. Set the structured description mode of the current document's message structure to ASN.1 format.

[0092] 3. Recognizing the structured description method of automata information

[0093] The specific steps for identification, based on different descriptive styles of automata information, are as follows:

[0094] a. Text list format: For cases involving text list format (e.g.) Figure 5 If the current line matches one of the regular expressions [r"State\s+Event\s+Action\s+New State",r"\|\sState\s\|"], it is identified as the starting line of the automaton description. The process continues to traverse the next line of the current chapter. If the current line is empty and the starting line is not None, it is identified as the ending line. The structured description of the automaton in the current document is set to a text list format.

[0095] b. Dotted line graph format: This refers to the dotted line graph format (example...). Figure 6 If the current line matches one of the regular expressions [r'^[\|\+]+---->', r'<----[-\|\+]+$'], it is identified as the starting line of the automaton's dot-line diagram representation. The process continues traversing the next line of the current chapter. If the current line is empty and the starting line is not None, it is identified as the ending line. The structured representation of the automaton for the current document is set to a dot-line diagram.

[0096] 4. Extract structured descriptions

[0097] Iterate through the list of chapters and the list of lines within each chapter obtained in step 1. Using the characteristics of the structured description of the document identified in step 2, match and identify the starting and ending lines of the structured description, and record the chapters in which the structured description is located. Add the key line description information to the corresponding key line list in turn, and finally obtain a list of key lines with different description styles.

[0098] Key information extraction stage:

[0099] 5. Extract message structure information

[0100] Traverse the list of key lines in the message structure, and parse the corresponding network protocol design information according to different description styles. For field description information including protocol message field name, field type, and field length, store it in a formatted field according to a standardized format. Each field is stored in the form of a six-tuple: <section name, field name, field type, field length, field value, field description>. The derivation of some field types, field lengths, and field descriptions will be described in the next stage. Parse the key line text information according to the structured description method of the RFC document obtained in step 2. The specific steps are as follows:

[0101] a. Graphical Format: Traverse the list of key lines obtained in step 3 and parse the content of each line. If the current line starts with the symbol "+-" or is an empty line, skip the parsing, as these are usually separators. If the current line contains "|" or ":", use that symbol as a separator to cut the current line and obtain the multiple fields described in the current line. Then traverse the multiple fields obtained, first determine if it is a variable-length field, that is, check if the current line contains symbols such as "variable", " / ", or "~". If so, set the size of the current field to "-1", indicating that the length of the field is uncertain; otherwise, it is a fixed-length field. These field names include their space placeholders in the graph. The actual meaningful length of the field can be calculated based on its length, mainly according to "size = (len(field) + 3) / / BIT_SBL_SIZE", where BIT_SBL_SIZE is the length of a character in the current graph description style where "+-" represents 1 bit (e.g., if "+-" represents 1 bit, then BIT_SBL_SIZE is 2). Next, the field names are further verified. Redundant whitespace characters are removed, and any extra information contained in parentheses or square brackets ("()" or "[]") is removed. If parentheses contain information describing the field length (e.g., 2 octets, 4 bits), this information is identified, calculated, and the current field size is updated. The current field size is then set to be immutable (by setting a flag). Abbreviations in the field names are then processed and restored to their full word descriptions. Finally, the field names are updated, and the chapter containing the current key line is written as the field's chapter name into the aforementioned six-tuple, resulting in a preliminary field. This preliminary field is then added to the formatted field list.

[0102] By extracting structured information in the form of images and text, we can currently obtain information such as "chapter name", "field name", and "field length".

[0103] b. Pseudo-C code form: Iterate through the list of key lines obtained in step 3 and parse the content of each line. First, use the information in the first line to determine whether the key line text describes structured information or enumerated information. Because structured information contains field information, while enumerated information contains field value information, we can use a space to split the value of the first element after the current line to determine this. If it describes structured information, iterate through each line and use the regular expression r'[\w\:]+\s+[\*\w\d\[\]]+;' to match the lines describing field information. Generally, they have the form "TYPE FIELD_NAME;", which allows us to identify the field type and field_name. In addition, there may be cases where the field length is indicated by the "<>" symbol, in which case the field size is also obtained. If the description is of enumeration information, use the regular expression r'[\w\d]+\([\w\d\.]+\)' to match the described enumeration information. The key characteristic is "ENUM_NAME(VALUE)". Similarly, extract the values ​​and then match the fields corresponding to these enumeration values ​​at the end of the key lines (e.g., "}HandshakeType;"). Finally, set the corresponding field name, field value, and field type, and add the field to the formatted field list.

[0104] By extracting structured information in pseudo-C code form, we can currently obtain information such as "chapter name", "field name", "field length", and "field type".

[0105] c. ASN.1 format: Iterate through the list of key lines obtained in step 3 and parse the content of each line. First, using the information in the first line, determine whether the key line text description is nested. If it contains the ASN.1 keyword from any of the following: "OBJECT-GROUP", "MODULE-COMPLIANCE", "OBJECT IDENTIFIER", or "OBJECT-TYPE", then it is nested. If nested, it describes a relationship between two fields. The starting line of the key line text begins with the current field name, which can be used to obtain the field name. Iterate through each line. If the keyword "SYNTAX" exists, the current line describes the type of the current field, which is easy to extract. Finally, if the ending line of the key line contains a description in the form "::={UP_FIELD}", extract the content of UP_FIELD as the parent field of the current field, set it as the section_name of the current field, and add the current field to the formatted field list. If it is not nested, it describes a field and its contained field information. Take the information from the starting line as the parent field of each subsequent field, that is, set it to the section_name of each subsequent field. Then, traverse each line. The characteristic of the field is described as "FIELD_NAME FIELD_TYPE,". Then, obtain the corresponding information through simple separation to get a field. Add the field to the formatted field list.

[0106] By extracting the structured information in ASN.1 format, we can currently obtain information such as "chapter name", "field name", and "field type".

[0107] 6. Extract automata information

[0108] If step 3 yields the list of key lines for the automaton, iterate through the list and parse the content of each line. Based on certain characteristics in the automaton description, identify names that might represent events, states, or actions, and add them to the automaton's undefined names. Specifically, use spaces to split the current line, obtaining a list of words. Then, iterate through this list; if a word is all uppercase, it might be a state name; if the current word is a noun, it might be an event or action name. Add these words to the automaton's undefined names.

[0109] If the list of key lines for the automaton is empty, the regular expression [r'(Event\d+):(\w*)',r'(\w*)[sS]tate:',r'State\(s\):(\w+)'] is used to match the state name and event name of the automaton and add them to the undefined name list of the automaton.

[0110] Text comprehension stage:

[0111] Steps 5 and 6 yield a series of formatted field lists and a list of undefined names for the automata. However, the six-tuple information for each field is incomplete. Therefore, this step involves further refining the corresponding fields and deriving the automata migration information. The input to this stage of the invention is the formatted field list and the chapter content to which the field belongs; the output is the refined formatted field list or the automata migration list. For broader matching, the chapter content can be expanded to larger chapters or even the entire RFC document.

[0112] 7. Derive field length using text description information

[0113] Iterating through the formatted field list obtained in step 5, for each field's corresponding chapter, this invention first re-divides the chapter according to sentences. All lines within a chapter are then processed by replacing all newline characters with spaces, assembling the content into complete sentence information. Then, natural language processing tools are used to split this information into individual sentences and store them within the chapter.

[0114] Iterate through each field in the formatted field list, using the field name followed by "is", ":\n", or "" as a pattern to match the field description. Iterate through each line in the input chapter. If the pattern exists in the current line or the current line contains the field name, a match is successful. Then, the text content of the current line and the next five lines is added to the formatted field list as the field's description. Any extra or incomplete sentences are discarded. The field description information is only used to help deduce information for other fields.

[0115] Next, it iterates through each sentence in the section and description of the field. If a sentence contains a field name, it is used as the initial analysis point. The length of a field in a statement typically appears in the form of is / are / length of / length / oflength+num. This invention takes statements containing these keywords as input and uses NLP technology to analyze the part-of-speech tagging of the statements, identifying statements containing "VBZ CD" (verb + cardinality). The word corresponding to CD represents the message length. Simultaneously, to determine the unit of the extracted length, this invention analyzes whether the following character is ['-octet', '-bit', 'octet', 'bit', 'octets', 'bits', 'bytes']. This invention stores the message field length in bits. For octet and byte, the value is multiplied by 8 before storage. If there is no unit after the value, this invention assumes it is described as bytes and therefore also stores the value multiplied by 8. In this way, this invention obtains the length corresponding to the field and then updates the corresponding values ​​in the formatted field list.

[0116] 8. Derive field values

[0117] When describing field values ​​in RFC text, modal verbs ("MUST", "MUST NOT") or comparison terms ("is less than", "is equal to") are typically used. Therefore, this invention iterates through each statement in the chapter; if a statement contains a field name and also includes a modal verb or comparison term, it is considered a statement to be extracted. Next, this invention uses part-of-speech tagging to generate a list of parts of speech for the statement. If it includes part-of-speech combinations as shown in Table 1, then this invention converts the modal verbs and comparison terms into corresponding symbols, and then extracts the values ​​from CD. Finally, a list of values ​​is generated based on the symbols. For example: if field <= 3, then values ​​= [0, 1, 2, 3].

[0118] Table 1. List of NLP Part-of-Speech Analysis

[0119]

[0120] 9. Deducing Field Types

[0121] There are three methods to infer field types. First, the field type can be easily deduced using the text description information obtained in step 5. If obvious field types such as "octet", "string", "boolean", or "integer" exist in the text description, then the field type is set to that type. Furthermore, after obtaining more complete information about the field name and length, the field type can be inferred from these two pieces of information. Second, the field type can be inferred from the field name. If the field name contains "domain name" or "owner name", the field type can be set to "dnsname". If the field name contains "time" or "ttl", or "ip" or "ipv6", the field type can be set to "timestamp" or "ip" respectively. Third, the field type can be inferred from the field length. If the field length is "8", "16", "32", or "64", the field type can be set to "byte", "word", "dword", or "qword" respectively. The priority of these methods for inferring field types decreases from 1 to 3, with the method of inferring the field type by field length having the lowest priority.

[0122] 10. Derive automaton migration information

[0123] Step 5 yields a series of undefined automaton names. These names are used as keywords to match possible automaton transition description statements in the paragraph description information to identify an automaton transition (i.e., original state, event, action, and destination state). The transition information is then stored in the automaton transition list.

[0124] Iterate through the list of chapters obtained in step 1, combining each line within each chapter, and segment the paragraphs using NLTK (a third-party library, Natural Language Toolkit). Then, match each normal natural language statement. If the statement contains keywords that might indicate state transitions, such as {"receive", "transfer", "change", "remain"}, further check if there is an automaton with an undefined name in both the part before and after the keyword. If so, it's an automaton transition message that changes state; otherwise, it's likely an automaton transition message that doesn't change state. Then, further segment the statement according to English grammar, storing each part in an automaton transition information quadruple, and then add it to the automaton transition list.

[0125] Knowledge expression stage:

[0126] The formatted field list and automaton migration list obtained through steps 7, 8, 9, and 10 constitute the final network protocol design information. The formatted field list can be converted into JSON format and stored in a file for later use. Current applications include generating detection rules and generating fuzzy testing configuration files, as detailed below:

[0127] 11. Detection rule generation

[0128] By utilizing the information of fields in the formatted field list (field name, field length, field value), a series of detection rules in the form of "chk_bf(cond,op)" can be generated. Here, "cond" represents the condition of the rule, generally corresponding to the field's value range or length limit. If the condition is met, the correct operation is defined as "op". For example, "((hold>3&&hold==0),use(hold))" means that when using the hold field, this invention needs to determine whether its value is 0 or greater than or equal to 3.

[0129] 12. Generating fuzz test configuration files

[0130] By utilizing information about fields in a formatted field list (field name, field length, field type, field value), this invention can automatically generate test configuration files for BooFuzz ​​(a general-purpose network protocol fuzzing framework) for fuzz testing. The mutation primitive to be selected for the current field is determined based on the field length or field type; for example, s_bytes() represents a field of multiple bytes. Figure 7 As shown, the first field is "Marker", and by default, the value length is insufficient to be padded with "\x00". s_byte() represents a field of one byte size. Figure 7 The field "Optional Parameters Length" is 1 byte in size, and the field value is represented by the byte string of its "value" parameter. The value of the fuzz test is represented by the "fuzz_values" parameter.

[0131] By utilizing the transition information (original state, destination state) of the automaton, a sequence of automata for fuzz testing can be generated. For example... Figure 8 As shown, the test sequence of the automaton is obtained by using the automaton transition information "State Machine Record(state='Connect',event='['CollisionDetectEstablishedState',…,'Event 12','DelayOpenTimer_Expires']',action='None',new_state='OpenSent')".

[0132] Example:

[0133] The following is an example of extracting network protocol design knowledge from the BGP protocol RFC 4271.

[0134] In step 1, RFC 4271 is preprocessed to remove redundant headers, footers and other unimportant information, resulting in the preprocessed file "rfc4271.txt.preprocessed". Then, the preprocessed file is matched with chapters to divide it into chapters, resulting in the chapter list "sections".

[0135] In step 2, the list of sections obtained in step 1 is traversed, and the following section, "4.1. Message Header Format," of RFC 4271 is identified: Figure 9 The message structure description in graphical form is then set to graphical form, thus ending the traversal.

[0136] In step 3, no structured description of automata information was identified, so the value was left blank.

[0137] In step 4, based on the graphic description obtained in step 2, step 3.a is performed to extract the structured description in the graphic form, identify the start and end lines of each structured description, record the chapter it belongs to, and add it to the message structure key line list.

[0138] In step 5, firstly, according to the RFC structured description method, the key information of the message structure key line list described in the graphic form in step 5.a is extracted, the key line list is traversed, the content of each line is parsed, the field information is obtained in turn, and stored in the formatted field list.

[0139] In step 6, since the structured description of the automaton information is empty, the more general regular expression [r'(Event\d+):(\w*)',r'(\w*)[sS]tate:',r'State\(s\):(\w+)'] is used directly during the traversal of the key rows to match the state name and event name of the automaton and add them to the undefined name of the automaton.

[0140] In step 7, the formatted field list obtained in step 4 is traversed, and the field length is further inferred and understood based on the text description in the field information and the chapter in which it belongs. The following example shows the text description and corresponding field part displayed during the inference process: "[+]Field:Error Code Description:Error Code:This 1-octetunsigned integer indicates the type of NOTIFICATION."

[0141] In step 8, the value of the field is inferred, and the numerical value in the text description is extracted using NLP part-of-speech analysis as a candidate value for the field.

[0142] In step 9, the type of the formatted field is inferred based on the priority of the three inference methods, and the field type is updated accordingly. For example, in the text description of the field "Withdrawn Routes" in "[+]Current fieldname:Withdrawn Routes,field size:-1\n[+]Field:Withdrawn Routes Description:Withdrawn Routes:This is a variable-length field that contains a list of IP address prefixes for the routes that are being withdrawn from service.", if the text description of the field "Withdrawn Routes" contains keywords such as "IP address", then this field is considered to be describing IP addresses, and its field type is set to "ip", with a length of "-1", which represents multiple IP addresses.

[0143] In step 10, using the undefined name of the automaton obtained in step 6, the chapters are traversed to find descriptions of the automaton, identify automaton transition information, and add it to the automaton transition list. Below is an example of adding automaton transition information to the automaton transition list: "[+]Added unchanged state machine transition:StateMachine Record(state='Established',event='['CollisionDetectEstablishedState']',action='None',new_state='Established')".

[0144] In steps 11 and 12, the formatted field list obtained in steps 7, 8, and 9 and the automaton migration list obtained in step 10 are used to generate the corresponding JSON format file. Then, according to the designed application method, the detection rules and fuzz test configuration files are generated as needed. Figure 10 Here is an example of a fuzzing configuration file obtained according to RFC 4271:

[0145] Similarly, it can also generate such as Figure 10 The automaton test sequence is shown.

[0146] Although the present invention has been disclosed above with reference to embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent substitutions made by those skilled in the art to the technical solutions of the present invention should be covered within the protection scope of the present invention, which is defined by the claims.

Claims

1. An automatic extraction method of network protocol design knowledge based on RFC documents, characterized in that, The method includes: Based on the txt format of the RFC document, obtain the list of chapters of the RFC document and the list of lines in each chapter; Obtain the structured descriptions of the chapter list and the line list to generate a key line list; wherein, the structured descriptions include: message structure and / or automata; The structured description in the key row list is parsed to obtain a formatted field list containing partial information and / or an undefined name list of the automaton; wherein, the information in the formatted field list includes: chapter name, field name, field type, field length, field value, and field description; Given the structured description as a message structure, based on the chapter content to which the corresponding key line of the structured description belongs, update the field length, field value, and field type to obtain a formatted field list containing complete information; In the case of the structured description being an automaton, the chapter content to which the key line corresponding to the structured description belongs is obtained, and the name in the undefined name list is used to match the chapter content. The obtained automaton migration information is then stored in the automaton migration list. By combining the formatted field list containing complete information and the ingress automaton migration list, the network protocol design knowledge extraction result is obtained.

2. The method of claim 1, wherein, The message structure includes one or more of the following: graphic format, pseudo-C code format, and ASN.1 format; The process of obtaining the structured descriptions of the chapter list and the row list includes: When the message structure is in the form of a graphic, after using the regular expression r'^\+-+\+$' to match and identify the starting line of the structured description, continue to traverse the line list, and when a single empty line is encountered, take the current line as the ending line; When the message structure is in pseudo-C code form, the regular expression r'^\s*(struct|enum|typedef struct)\s+\w*\s*\{' is used to match and identify the starting line, and the regular expression r'\}[^;]*;' is used to match and identify the ending line, and the curly braces between the starting line and the ending line are correctly closed. When the message structure is in ASN.1 form, after matching and identifying the starting line using the regular expressions r'\w+\s+(OBJECT-GROUP|MODULE-COMPLIANCE|OBJECT IDENTIFIER|OBJECT-TYPE|SEQUENCE)' and r'\w+\s+::=\s+\w+\s\{', the line list is traversed. When the current line ends with a right curly brace and the next line is empty, the current line is taken as the ending line.

3. The method of claim 2, wherein, The process of parsing the structured description in the key row list yields a formatted field list containing partial information, including: For message structures in image and text format, obtain the key lines containing the "|" symbol or the ":" symbol, and use the corresponding "|" symbol or the ":" symbol as a delimiter to split the key lines to obtain several fields and the field names of those fields; Based on the variable-length field symbols contained in the field and the space placeholders for the field in the diagram contained in the field name, the field length of each field is obtained and then stored in the formatted field list; wherein, if the "()" symbol in the field name contains field length information, the field length is updated using the field length information and then stored in the formatted field list. After formatting and updating the field names, they are stored in a formatted field list; the formatting and updating includes: removing redundant whitespace characters and extra information contained in the "()" and "[]" symbols in the field names, and restoring the abbreviations contained in the field names to full word descriptions; Retrieve the chapter name corresponding to the key row and store it in the formatted field list; And / or, For a message structure in pseudo-C code form, the type of information described by the current line is determined based on the value of the first element after the current line in the key line list; the information type includes: structured information or enumerated information; If the information type described in the preceding line is structured information, then the regular expression r'[\w\:]+\s+[\*\w\d\[\]]+;' is used to match the line describing the field information, the field length contained in the "<>" symbol, and the chapter name corresponding to the line describing the field information. After obtaining the field name, field type, field length, and chapter name, they are stored in the formatted field list. If the information type described in the preceding line is enumerated information, then the regular expression r'[\w\d]+\([\w\d\.]+\)' is used to match the lines describing the field information to obtain the field value information. After obtaining the field name, field type and chapter name by matching the field value information, the information is stored in the formatted field list. And / or, For message structures in ASN.1 format, the field types are obtained based on the "SYNTAX" keyword in the key line list and stored in the formatted field list; Based on the information in the first row of the key row list, determine whether the text description of the key row is nested information; If it is nested information, the field name is obtained based on the starting line of the key line text starting with the current field name. Then, the parent field name is obtained based on the content of UP_FIELD in the "::={UP_FIELD}" format description in the ending line of the key line. Finally, the field name and the parent field name are stored in the formatted field list. If the information is not nested, the information in the first line will be used as the parent field name for parsing each field in the subsequent parsing. When traversing each subsequent key line, "FIELD_NAME FIELD_TYPE," will be used to obtain the field name, and the field name and the parent field name will be stored in the formatted field list. Retrieve the chapter name corresponding to the key line and store it in a formatted field list.

4. The method of claim 2, wherein, Update field length, including: Obtain the chapter corresponding to the chapter name, and re-divide the chapter content according to sentences to obtain the updated chapter content; Generate a pattern and field name to match the field description based on the field name, match the updated chapter content, and if the match is successful, use the current line and several lines after the current line as the field description. The updated chapter content is matched based on the field name, and if the match is successful, the sentences containing the field name are analyzed for part-of-speech tags to obtain sentences containing verbs + cardinality. The word corresponding to the cardinality is used as the numerical value of the field length, and the unit of the numerical value is obtained based on the next character of the cardinality; Update the field length in the formatted field list based on the numerical value of the field length and the unit of the numerical value.

5. The method of claim 2, wherein, Update field values, including: Obtain the chapter corresponding to the chapter name, and re-divide the chapter content according to sentences to obtain the updated chapter content; The updated chapter content is matched based on the field name, and if a match is successful, the presence of modal verbs or comparative words in the sentence is determined. In the presence of modal verbs or comparison words, perform part-of-speech analysis on sentences containing field names to obtain sentences containing MUST(NOT)be(set to)NUM, less than NUM, greater than NUM, or is equal to NUM, are not equal to NUM; Convert the modal verbs or comparison words into corresponding symbols and extract the values ​​from MUST(NOT)be(set to)NUM, less than NUM, greater than NUM, or is equal to NUM, are not equal to NUM. Based on the symbol and the value, update the field values ​​in the formatted field list.

6. The method of claim 2, wherein, Update field types, including: Based on the field description information in the structured description, including protocol message field names, field types, and field lengths, the field types are deduced and the formatted field list is updated. or, Based on the field names and field lengths in the formatted field list, deduce the field types and update the formatted field list; or, Based on the field names in the formatted field list, deduce the field types and update the formatted field list; or, Based on the field lengths in the formatted field list, deduce the field types and update the formatted field list.

7. The method of claim 1, wherein, The automaton includes one or more of the following: a text list and a dot-line diagram; The step of obtaining the structured descriptions of the chapter list and the line list to generate the key line list includes: When the automaton is in the form of a list of text, if the current line matches one of the regular expressions [r"State\s+Event\s+Action\s+New State",r"\|\sState\s\|"], it is identified as the starting line. In subsequent traversals, if the current line is an empty line and the starting line is not None, it is identified as the ending line. In the case of a dotted-line graph automaton, if the current line matches one of the regular expressions [r'^[\|\+]+---->',r'<----[-\|\+]+$'], it is identified as the starting line. In subsequent traversals, if the current line is an empty line and the starting line is not None, it is identified as the ending line.

8. The method of claim 7, wherein, The parsing of the structured description in the key row list yields a list of undefined names for the automaton, including: Use spaces to separate key lines to obtain a list of words; Iterate through the list of words; If a word that is all uppercase or a word that is a noun can be obtained, then add the word that is all uppercase and the word that is a noun to the list of undefined names. If words that are all uppercase or words whose part of speech is noun are not retrieved, the regular expression [r'(Event\d+):(\w*)',r'(\w*)[sS]tate:',r'State\(s\):(\w+)'] is used to match the state name and event name of the automaton and add them to the list of undefined names.

9. The method of claim 8, wherein, In the case of the structured description being an automaton, the chapter content to which the key line corresponding to the structured description belongs is obtained, and the names in the undefined name list are matched against the chapter content. The resulting automaton migration information is then stored in the automaton migration list, including: Obtain the chapter content to which the key line corresponding to the structured description belongs; The chapter content is re-divided into sentences to obtain the updated chapter content; If the updated chapter content contains a sentence that includes a keyword indicating a state transition, and the first half and the second half of the keyword each contain a name from the list of undefined names, then the sentence is determined to contain automaton transition information that changes the state. The sentence is then divided into automaton transition information quadruples and stored in the automaton transition list. The automaton transition information quadruples include: original state, event, action, and destination state. If the updated chapter content contains a sentence that includes a keyword representing a state transition, and the first or second half of the keyword contains a name from the list of undefined names, then the sentence is determined to contain automaton transition information that does not change the state. After splitting the sentence into automaton transition information quadruples, the information is stored in the automaton transition list.

10. The method of any one of claims 1-9, wherein, Also includes: Based on the knowledge extraction results of the network protocol design, detection rules and / or fuzz test configuration files are generated.

Images