Descriptable network traffic metadata extraction method, device, equipment, medium and product

Abstract

The invention discloses a descriptable network flow metadata extraction method, device, equipment, medium and product, and relates to the field of network space security content security direction.The method comprises the steps that an XML description file is written according to a protocol application interaction process and description specifications; the XML description file comprises protocol state description and data extraction requirement description; based on the XML description file, whether an external extension function needs to be called is judged, and if yes, an external extension library is compiled for function extension; if not, compiling the XML description file into a state node tree file through a description compiler, and generating a user dynamic function library file; executing processing functions carried by the state node tree file and the user dynamic function library file by utilizing a message processing automaton, and outputting domain data extracted by a current message; the domain data is network flow metadata. The flow metadata analysis method and device can meet the flow metadata analysis requirements in different scenes.

Description

Technical Field

[0001] The present application relates to the field of cyberspace security content security, and in particular to a describable network traffic metadata extraction method, device, equipment, medium and product. Background Art

[0002] Network traffic data extraction technology has always been an important data support in the field of network security. Network traffic restoration or feature extraction technology relies on parsing the domain carried by the protocol and extracting content data. The application of this type of technology plays a core role in cyberspace security. In recent years, with the development and application of artificial intelligence (AI) models in the field of network security, complex and diverse network traffic data features have also become indispensable data requirements. In general, network traffic data extraction technology can be applied to traffic visualization, network forensics, network monitoring, AI model security testing and other fields.

[0003] How to extract metadata or features from network traffic in real time and in multiple dimensions? Traditional network traffic metadata extraction methods and systems are often implemented in a hard-coded manner. The most direct problem is that as network applications become more complex and changeable, the back-end system's data requirements change, and it is impossible to flexibly and quickly meet the needs of variable metadata extraction and analysis. Summary of the Invention

[0004] The purpose of this application is to provide a describable network traffic metadata extraction method, device, equipment, medium and product to solve the problem of not being able to flexibly and quickly meet the needs of variable metadata extraction and analysis.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a describable method for extracting network traffic metadata, comprising:

[0007] Write an XML description file according to the protocol application interaction process and description specifications; the XML description file includes a protocol status description and a data extraction requirement description;

[0008] Based on the XML description file, determining whether it is necessary to call an external extension function and determining a first determination result;

[0009] If the first judgment result is yes, compile the external extension library to expand the function;

[0010] If the first judgment result is no, compiling the XML description file into a state node tree file through a description compiler, and compiling to generate a user dynamic function library file;

[0011] The message processing automaton is used to execute the processing functions carried by the state node tree file and the user dynamic function library file, and output the domain data extracted from the current message; the domain data is network traffic metadata.

[0012] In a second aspect, the present application provides a describable network traffic metadata extraction device, comprising:

[0013] An XML description file writing module is used to write an XML description file according to the protocol application interaction process and description specifications; the XML description file includes a protocol status description and a data extraction requirement description;

[0014] A first judgment module is used to judge whether it is necessary to call an external extension function based on the XML description file and determine a first judgment result;

[0015] A function extension module, configured to compile an external extension library to perform function extension if the first judgment result is yes;

[0016] A compiling module, configured to compile the XML description file into a state node tree file and generate a user dynamic function library file by using a description compiler if the first judgment result is no;

[0017] The domain data extraction module is used to use the message processing automaton to execute the processing functions carried by the state node tree file and the user dynamic function library file, and output the domain data extracted from the current message; the domain data is network traffic metadata.

[0018] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described describable network traffic metadata extraction methods.

[0019] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described describable network traffic metadata extraction methods.

[0020] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described describable network traffic metadata extraction methods.

[0021] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0022] While meeting the above-mentioned metadata extraction performance and requirements, this application designs a describable and scalable network traffic metadata extraction method. By writing an XML description file that covers the protocol status description and data extraction requirement description, the metadata extraction description of the required protocols and applications can be updated dynamically in real time. The XML description file is compiled by a description compiler to generate a state node tree file and a user dynamic function library file. The message processing automaton is used to output the domain data extracted from the current message. This domain data is the network traffic metadata, so that the metadata in various protocols can be extracted and output according to the description requirements, flexibly and quickly meeting the variable metadata extraction and analysis requirements, so as to meet the traffic metadata analysis requirements in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 This is the core functional structure diagram of this application;

[0025] Figure 2 This is a diagram of an application environment of a method for extracting network traffic metadata that can be described in an embodiment of the present application;

[0026] Figure 3 A flowchart of a describable method for extracting network traffic metadata provided by one embodiment of the present application;

[0027] Figure 4 A flowchart of the description compilation phase of this application;

[0028] Figure 5 This is a diagram of the XML description syntax architecture of this application;

[0029] Figure 6 This is the state node data structure diagram of this application;

[0030] Figure 7 This is the micro-operation node data structure diagram of this application; wherein, Figure 7 (a) is the mircostate node data structure diagram; Figure 7 (b) is the mircofunction node data structure diagram; Figure 7 (c) in the figure is the data structure diagram of the sofunction node;

[0031] Figure 8 This is the data structure diagram of the intermediate jump node of this application; wherein, Figure 8 (a) in the figure is the data structure diagram of the nextstate node; Figure 8 (b) is the varnode node data structure diagram;

[0032] Figure 9 This is the data storage structure diagram of this application; wherein, Figure 9 (a) is the data storage structure diagram of the var_data structure; Figure 9 (b) is the data storage structure diagram of the string_search_pattern structure; Figure 9 (c) is the data storage structure diagram of the multi_search_pattern structure;

[0033] Figure 10 This is the relationship diagram between the protocol description and the state node tree structure of this application;

[0034] Figure 11 This is the organizational structure chart of the message processing automation machine for this application;

[0035] Figure 12 This is a flow chart of the message automation engine operation of this application;

[0036] Figure 13 This is the message automaton state traversal sub-flowchart of this application;

[0037] Figure 14 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0039] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0040] In order to solve the problem that traditional commercial network traffic metadata extraction methods and systems are implemented in a hard-coded manner, that is, the processing logic is determined during the program writing process, and new extraction requirements cannot be added in real time. The most direct problem is that as network applications become complex and changeable, it is impossible to meet the needs of flexible and rapid updates of variable metadata extraction and analysis. Under the premise of meeting the needs of flexible and extended metadata extraction, this application designs a describable and extensible network protocol metadata extraction method, which can realize real-time dynamic updates of new protocol metadata extraction descriptions and metadata extraction requirements, and output metadata extraction from various protocols based on the description requirements to meet different traffic feature analysis scenarios, especially to meet the needs of different feature data analysis of AI.

[0041] The core functional components designed in this application include a description language, a description compiler, a message data processing state machine engine, etc., wherein the user needs to use the description language of this application to write the protocol data description according to the protocol specification, and the description compiler performs syntax checking and compilation to generate a binary state node tree. The message data processing state machine loads the state node tree. The state node carries various designed data processing atomic micro-operations. The state machine only needs to execute the various atomic micro-operations carried in sequence according to the deterministic node sequence and convert the metadata format until it encounters the termination node. The metadata extraction of a session is completed or the processing of a message data is completed, and the metadata required to be extracted by the user can be output. By applying this application, a set of network traffic metadata extraction systems that can describe and extend multi-protocol support can be implemented.

[0042] The application of this application to implement a network traffic metadata or feature data extraction system requires the integration of large-scale flow table management and scalable application identification functions. The large-scale flow table mainly provides network traffic session management, including network traffic preprocessing, TCP session reassembly and order preservation, and provides the required session node storage and query space. Application identification mainly identifies session protocols and applications, thereby applying this application for data extraction. Currently, there are many mature designs and applications in this area of technology, which will not be discussed in depth here, but only as a prerequisite technology application for this application.

[0043] To apply this application, you first need to write an XML description of the network protocol or application, describe the protocol status and metadata extraction requirements, write XML description files for each protocol and application, and then compile and process to generate a state node tree file before it can be used by the message data processing state machine engine. After the description compiler compiles it into a binary protocol state node, for some protocols with complex data processing, it is necessary to write an external micro function file to expand the external processing function, and compile it into a user dynamic function library file through GCC, which is called by the message data processing state machine engine during the protocol processing process. The core functional structure of this application is as follows: Figure 1 shown.

[0044] The present application embodiment provides a describable method for extracting network traffic metadata, such as Figure 2 As shown, the method is executed by a computer device, specifically, it can be executed by a computer device such as a terminal or a server alone, or it can be executed by a terminal and a server together. In the embodiment of the present application, as shown in FIG. Figure 3 As shown, the method includes the following steps.

[0045] S1: Write an XML description file based on the protocol application interaction process and description specifications; the XML description file includes a description of the protocol status and a description of the data extraction requirements. The description specification is a description grammar and rules provided by this application, and users need to rely on the grammar and rules to write XML description files.

[0046] S2: Based on the XML description file, determine whether it is necessary to call an external extension function. If so, execute S3; if not, execute S4.

[0047] S3: If the first judgment result is yes, compile the external extension library to expand the function.

[0048] S4: If the first judgment result is no, compile the XML description file into a state node tree file through a description compiler, and compile and generate a user dynamic function library file.

[0049] S5: Utilize the message processing automaton to execute the processing functions carried by the state node tree file and the user dynamic function library file, and output the domain data extracted from the current message; the domain data is network traffic metadata.

[0050] In an exemplary embodiment, before S3, the step further includes: utilizing the externally implemented C language function and using a GCC compiler to compile it into a user dynamic function library.

[0051] In an exemplary embodiment, S4 specifically includes the following steps.

[0052] Construct a state node data structure, a micro-operation node data structure, an intermediate jump node data structure and a data data storage data structure; the state node data structure includes the state information in the XML description file; the micro-operation node data structure includes multiple node types, and the multiple nodes include mircostate nodes, mircofunction nodes and sofunction nodes; the intermediate jump node data structure includes nextstateu nodes and vardata nodes; the data data storage data structure includes data representation structures of var_data, string_search_pattern and multi_search_pattern.

[0053] During the compilation process, the tags, operations and required parameter data of the XML description file are used to generate state nodes, micro-operation nodes and intermediate jump nodes that conform to the state node data structure description to construct a finite state tree.

[0054] A state node tree file is constructed according to the finite state tree, and the required user dynamic function library file is generated.

[0055] In practical applications, such as Figure 4 As shown, in order to achieve the describability and extensibility of network traffic protocol data extraction, the design implementation process in the design description compilation phase is as follows.

[0056] Step 1: Write an XML description file. Write a description file according to the protocol application interaction process and description specifications to obtain the protocol data extraction XML description file. After completion, jump to step 2 for processing.

[0057] In this step, XML description grammar rules are designed to define various tags and attributes to describe the logic and data parameters extracted by the protocol. The protocol description grammar is composed of a series of tags and attributes to define variables, constants, and states. The states describe the logic and required data of atomic functions, built-in functions, and external extended functions. For example, the variable subtag is used to define variables, the constvalue subtag is used to define constants, mircostate, mirfuncion, so_function and other tags are used to describe the operation of message and data processing, switch and case are used to describe the multi-mode branch processing of messages, etc. The specific description language system is as follows: Figure 5 shown.

[0058] The atomic operations related to message processing in this solution are implemented according to the network protocol specification. Atomic operations mean that when a protocol domain is distributed across two messages, the current processing context will be saved after the current message is processed, and processing will continue when the next message arrives. The message processing function is described in the XML description using the mircostate tag. During the processing process, the message processing pointer stops at the position where the current processing ends. The atomic operations implemented in this solution are shown in Table 1.

[0059] Table 1 Message processing atomic operation semantics

[0060]

[0061] The message processing automaton engine is designed with some session-related data processing operations, such as saving data to the flow table data register, reading the flow table data register, and obtaining the message direction within a session cycle. In the XML description, the relevant operations are described using the mirfunction tag, as shown in Table 2.

[0062] Table 2 Microfunction operation semantics table

[0063] Serial number Identifier describe 0 engine_save_data_to_flow Save data to flow table register 1 engine_load_data_from_flow Export data from flow table registers 2 engine_check_init_dir Determine whether the message direction is a flow direction 3 engine_payload_len_get Get the message payload length 4 engine_payload_current_get Get the current pointer of the message 5 engine_payload_current_leftlen_get Get the current unprocessed length of the message 6 engine_check_multi_value Comparison of multiple integer branch data

[0064] Step 2. During the protocol and application description process in step 1, some domain data cannot be implemented using the atomic operations supported by the above engine or the implementation is very complex. For example, DNS domain name extraction requires jumping to other locations in the message and calculating. In this case, the method name can be described, and external functions can be extended to implement the changed method. Therefore, an external extension library is designed here to implement complex message domain data processing and transmission through a unified interface. It can process general data not related to message scanning, and jump to step 3 after completion.

[0065] Function files are written in C language and compiled into user dynamic extension library files using GCC compiler. The files are indexed and executed by the engine through the function name. They are usually used to process some special data calculations and logical operations. User-defined functions have a unified parameter format and are named with so_user_[funname]. The specific parameter definitions are shown in Table 3.

[0066] Table 3 External function parameter list

[0067] parameter illustrate inpara Input parameter list inparanum Number of input parameters outpara Output parameter list outparanum Number of output parameters buff Temporary feature storage bufflen Temporary feature storage length

[0068] Step 3: Use the description compiler to compile the XML description file obtained in the above step 1 into a state node tree file. After completion, jump to step 4 for processing.

[0069] The compilation process of this step is to generate a state node tree file from the XML description file. The file consists of some nodes that store status, micro-operations, and the required parameter information of micro-operations. The state node tree is stored in a continuous storage space. The node types generated after compilation include state node, mircostate node, mircofunction node, sofunction node, nextstate, varnode node, data representation structure, etc. The following description will replace the specific data and structure names. The data structures of various operation nodes are described in detail as follows.

[0070] The state node data structure mainly stores the state information in the description, which mainly carries the next state address after the current processing state is completed, as well as various micro-operation information carried in the current processing state. When executing to the current processing state, all micro-operation functions in the current processing state need to be executed in sequence. The state node is defined as follows Figure 6 shown.

[0071] 2. Micro-operation node data structure, mainly including mircostate node, mircofunction node, sofunction node types. mircostate carries atomic operations related to message data domain scanning, can process cross-messages during execution, can define domain processing width limit fields, and can also carry required quantitative or variable parameter data, as well as domain single-mode matching and multi-mode matching. The mircofuntion node mainly has some built-in functions that are not related to message processing, such as data that needs to be temporarily stored in the session, etc. In addition, multi-integer value multi-mode comparison branch processing is also designed. Sofunction mainly processes externally defined data structures and parameters, and provides them to the message processing automaton in the form of external libraries for some special extraction or calculation methods. Various micro-node definitions are as follows: Figure 7 shown.

[0072] 3. The intermediate jump node data structure mainly includes nextstate and vardata nodes. They do not carry specific message data processing functions and only serve as connection nodes, the next processing node after the current processing; the varnode node carries the variable or constant data type, length, value, allocation address and other information required by the protocol, and is fixedly stored in the data storage space of the protocol. The relevant definitions are as follows: Figure 8 shown.

[0073] 4. The data storage data structure mainly includes data representation structures such as var_data, string_search_pattern, and multi_search_pattern, which are all embedded in the data field of the micro-operation node. The var_data structure describes the type of the variable and the storage variable index. The value can be a fixed constant value or the index of the related variable. The storage varnode node of the variable needs to be found according to the index; the string_search_pattern structure describes the parameters required by the single-mode matching algorithm, including the string and next table information required by the single-mode matching KMP algorithm; the multi_search_pattern structure describes the state and acceptance table required by the AC state machine of the multi-mode matching algorithm, and converts the data into linear address storage. This structure only needs to record the number of matching patterns, the current matching protocol number, and the relative address of the nextstate node after the match is successful. Related definitions are as follows Figure 9 shown.

[0074] During the compilation process, the XML description of the label and operation, the required parameter data to generate the above-described state, micro-operation, data and other nodes, forming a finite state tree from the root to the terminal node, after address conversion, stored in continuous memory, when compiling and generating the state micro-operation node, if the micro-operation does not have a success or failure jump state description, the default failure jumps to state 0, and the success executes the next micro-operation. If there is a need to jump to other states or branch jump state descriptions in the middle, a nextstate node will be inserted to carry the jump state index information. The organization of the compilation protocol state tree is as follows Figure 10 shown.

[0075] Step 4: At this point, the protocol application description and file compilation are completed, and a specific state node tree file and a user dynamic function library file are generated for subsequent message processing engine loading and use.

[0076] In an exemplary embodiment, S5 specifically includes the following steps.

[0077] The message processing automaton is used to load the state node tree file and the user dynamic function library file into the message processing automaton engine.

[0078] The message after the front flow table and application identification is delivered to the engine for processing, the flow table flow node message processing context is initialized, and the message payload data is scanned.

[0079] Based on the flow processing state of the message processing automaton, it is determined whether the current processing state of the current payload data is the initial state of the protocol, and a second determination result is determined.

[0080] If the second judgment result is yes, the initial state is executed, the next state is introduced into the current processing state, and it is determined whether the current processing state is a terminal state, and the third judgment result is determined.

[0081] If the third judgment result is yes, the domain data extracted from the current message is output.

[0082] If the third judgment result is no, all state nodes are traversed and the state sub-process is executed.

[0083] Determine whether the message atomic operation in the current processing state processing process requires cross-message processing to determine the fourth judgment result.

[0084] If the fourth judgment result is yes, the session context information of the flow table is saved to the engine session context address storage space.

[0085] If the fourth judgment result is no, the current processing state is completed, the next state node is imported, and the process returns to "judging whether the message atomic operation in the current processing state needs cross-message processing, and determining the fourth judgment result".

[0086] If the second judgment result is no, the current processing sub-state is restored from the stream context.

[0087] In an exemplary embodiment, the message processing automaton is used to load the state node tree file and the user dynamic function library file into the engine, which specifically includes the following steps.

[0088] The message processing automaton is used to load the parameter definition information of the state node tree file into the data storage address space, and the state node is loaded into the protocol state tree storage address space.

[0089] Load the user dynamic function library file into the engine.

[0090] In an exemplary embodiment, traversing all state nodes and executing state sub-processes specifically include:

[0091] Read the micro-operation node and success / failure jump information under the current processing state node, and determine whether the currently processed micro-operation node is a mircostate node to determine a fifth determination result.

[0092] If the fifth judgment result is yes, parse the var_data structure carried in the micro-operation node, calculate the parameter indirect address space, fill the parameter list, and execute the message processing atomic operation carrying function, scan the corresponding field of the message, determine whether the message scanning is completed, and determine the sixth judgment result.

[0093] If the sixth judgment result is yes, set a cross-message identification bit in the flow table, save the current processing status information, wait for the next message to be processed and resume processing, and determine whether the current domain data needs to be extracted to determine the seventh judgment result.

[0094] If the seventh judgment result is yes, the current domain data is extracted from the data storage area in the message processing automaton engine, and it is determined whether the current import node is the nextstate node to determine the eighth judgment result.

[0095] If the eighth judgment result is yes, the next state node address is calculated according to the next state index carried in the nextstate node, and the current node traversal is now completed.

[0096] If the eighth judgment result is no, read the next micro-operation node and jump to the step of "determining whether the currently processed micro-operation node is a mircostate node".

[0097] If the result of the seventh judgment is no, jump to the step of "judging whether the current imported node is a nextstate node".

[0098] If the sixth judgment result is no, it is determined whether the current operation is successfully executed, and a tenth judgment result is determined.

[0099] If the result of the tenth judgment is yes, jump to the step of "judging whether it is necessary to extract the current domain data".

[0100] If the tenth judgment result is no, the process enters the termination state.

[0101] If the fifth judgment result is no, it is determined whether the currently processed micro-operation node is a mirfunction node, and a ninth judgment result is determined.

[0102] If the ninth judgment result is yes, parse the var_data structure carried in the micro-operation node, calculate the parameter indirect address space, fill the parameter list, execute the micro-function function, call the common data processing function built into the engine, and jump to the step "Is the current operation executed successfully?"

[0103] If the ninth judgment result is no, it is determined whether the currently processed micro-operation node is a sofunction node, and an eleventh judgment result is determined.

[0104] If the eleventh judgment result is yes, search for the loaded user dynamic function library function function, parse the var_data structure carried in the micro-operation node, calculate the parameter indirect address space, fill in the parameter list, and execute the user-defined function, and jump to the step "determine whether the current operation is executed successfully".

[0105] If the eleventh judgment result is no, jump to the step of “reading the next micro-operation node”.

[0106] In practical applications, in order to efficiently extract data from messages, a processing flow for the message data processing stage is designed. The specific implementation is described as follows.

[0107] The message data processing process is mainly based on the message processing automaton engine. When the message processing automaton is initialized, the state node tree file is loaded into the state node storage memory. The message processing automaton is driven by the message. When a session starts, it starts from the starting state described by the protocol application, scans the message data, and obtains the data storage address and micro-operation node from the state node in the memory. The message automaton processing structure design is as follows: Figure 11 shown.

[0108] As described above, the message processing automaton engine's operation involves reading state nodes and the functions carried by the micro-nodes within them, executing processing logic, and accessing data. After executing the micro-operations under the current processing state node, it jumps to the next state node according to the description in the nextstate node. While traversing and processing messages, the node carrying information about whether the current domain data needs to be extracted when processing the message domain at the current micro-operation node. If so, the domain data is converted and stored in the message processing automaton engine's memory. After a message is processed, the required metadata is output to the user.

[0109] The data access addresses designed for the message processing automaton engine are divided into three categories: engineaddr_base, constaddr_base, and variableaddr_base. Each type of storage space stores different types of data. Engineaddr_base is the engine's built-in temporary storage space. Under the micro-operation node, the parameters are carried in the var_data structure. The var_data data structure is organized, where the flag field identifies the type of the variable, and the data field indicates the storage sequence index value or fixed value of the data. Four data types, consttype, variabletype, valuetype, and enginetype, are defined to carry different data access address spaces.

[0110] When calculating parameter values, all parameters are uniformly converted into the varobject(type, len, value) structure to support the parameter list, where the type field identifies whether the variable is a fixed value or address type, the len field identifies the length of the variable, and the value field stores the specific fixed value or address.

[0111] For consttype and variabletype data, you need to find the data storage address space base address protoroot.var_addr of the variable node of the current protocol or application and start addressing from this space. Sizevarnode is the size of the varnode node. Therefore, the address varnodeaddr of the current variable varnode node is calculated as shown in the following formula.

[0112] varnodeaddr=protoroot.var_addr+var_data.data*sizevarnode

[0113] Among them, var_data.data is the indirect index value of varnode in the data storage address space, and the varnode address is addressed according to this index value.

[0114] The data storage node address varnodeaddr calculated above can access the varnode node of the current variable. From the information carried in the varnode node, the required data value can be indirectly addressed or calculated. The data value calculation formula is as follows.

[0115]

[0116] Among them, varobject.value is the parameter value carried by the micro-operation; varnodeaddr.addr is the offset address of the varobject.value parameter on its base address; var_data.flag is the type of the variable organized by the var_data data structure.

[0117] The message automation process is as follows: Figure 12 The specific steps are as follows:

[0118] Step 1: Parse and load the state node tree file generated in the description compilation phase, load the parameter definition information into the data storage address space, load the state node into the protocol state tree storage address space, and jump to step 2.

[0119] Step 2: Initialize and load the user dynamic function library file generated in the description compilation phase. The implementation method of loading the extension library can improve flexibility by feature update. Jump to step 3.

[0120] Step 3: After the pre-flow table and application identification, the message data enters the engine, initializes the message processing context, and initially scans the protocol field from the first payload data of the message, and jumps to step 4.

[0121] Step 4: The message processing automaton engine determines whether the current payload processing is in the initial state of the protocol. If so, the current processing state is the initial state and the process jumps to step 6; otherwise, the process jumps to step 5.

[0122] Step 5: Export the saved flow context from the message processing engine session context address space, restore the current flow to the previous message processing state, and then jump to step 6.

[0123] Step 6: Determine whether the current processing state is the termination state (state 0). If not, jump to step 7. Otherwise, end the session data extraction process of the current flow and jump to step 10.

[0124] Step 7: Execute the state node traversal processing sub-process. The specific processing process will be described later. Jump to step 8.

[0125] Step 8: Determine whether the message atomic operation is completed during the current processing state and whether cross-message processing is required. If not, complete the current processing state and import the next state node to jump to step 6; otherwise, jump to step 9.

[0126] Step 9: Save the current flow processing context information to the engine session context address storage space, restore the state when the next message arrives, and jump to step 10.

[0127] Step 10: The user can export the data extracted from the current message from the engine's storage space. At this point, the current message data extraction is completed, and the state machine operation ends on the current message.

[0128] The state node traversal sub-process processing in step seven is the process of traversing the state nodes in the protocol state tree address space and executing micro-operations. The compiled state nodes and the micro-operations they carry are stored continuously in the memory. Starting from the initial node, the micro-operation nodes carried in the execution state are executed until the termination state. Four types of micro-operation node traversals are designed in this application, such as mircostate node, mircofunction node, sofunction node, and nextstate node. Various types of micro-operation nodes are organized in continuous memory in the order described in the state space, and execution starts from the first micro-operation to the end of the last micro-operation. The state node traversal processing flow is as follows: Figure 13 The specific steps are as follows:

[0129] Step 1: Read the micro-operation node and success / failure jump information under the current processing status node, and jump to step 2.

[0130] Step 2: Determine whether the currently processed micro-operation node is a mircostate node. If so, jump to step 3 to import the parameters and apply the parameter data and jump to step 4 to execute. Otherwise, jump to step 8.

[0131] Step 3: Parse the parameter var_data structure carried in the micro-operation node. Based on the type and data information carried in var_data, the parameter indirect address space or value can be calculated, that is, the parameter list varobject can be filled for use in subsequent steps.

[0132] Step 4: Execute the functions carried by the message processing atomic operation and scan the corresponding fields of the message. If the message scan is completed, set the cross-message flag and save the current processing status information. Resume processing when the next message arrives. If the execution is successful, jump to step 5, otherwise enter the termination state.

[0133] Step 5: Determine whether it is necessary to extract data in the current domain, that is, whether to describe extracting the current domain data. If so, jump to step 6, otherwise jump to step 7.

[0134] Step 6: Extract the current domain data to the data storage area in the message processing automation engine, and jump to step 7.

[0135] Step 7: Determine whether the current import node is the nextstate node. If so, jump to step 13; otherwise, jump to step 12.

[0136] Step 8: Determine whether the currently processed micro-operation node is a micro-function execution node. If so, proceed to step 3 for parameter import processing and then to step 9 for micro-function execution. Otherwise, jump to step 10.

[0137] Step 9: Execute the micro-function function and call the common data processing function built into the engine. If the execution is successful, jump to step 5, otherwise enter the termination state.

[0138] Step 10: Determine whether the currently processed micro-operation node is an external dynamic expansion function execution node. If so, proceed to step 3 for parameter import processing and then jump to step 10; otherwise, jump to step 12.

[0139] Step 11: Search the function in the loaded user dynamic function library through the dlsym system call, apply the parameters exported in step 3 and execute it. If the execution is successful, jump to step 5, otherwise enter the termination state.

[0140] Step 12: Read the next micro-operation node information and jump to step 2.

[0141] Step 13: Calculate the next state node address based on the next index carried in the nextstate node. This completes the traversal of the current node.

[0142] In practical applications, this application can extract protocol domain data from network traffic data in real time and with high performance, serving as the foundation for content restoration systems. Based on the restored metadata, network monitoring, forensics, privacy protection, and other applications can also be used as feature data support for training and classification of large network security models. This application significantly improves performance and flexibility, making it highly valuable for real-time analysis of backbone network traffic and playing a crucial role in ensuring network security.

[0143] Based on the same inventive concept, embodiments of the present application also provide a describable network traffic metadata extraction device for implementing the aforementioned describable network traffic metadata extraction method. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the describable network traffic metadata extraction device provided below can be found in the above-mentioned limitations of the describable network traffic metadata extraction method and will not be repeated here.

[0144] In an exemplary embodiment, a describable network traffic metadata extraction apparatus is provided, comprising:

[0145] An XML description file writing module is used to write an XML description file according to the protocol application interaction process and description specifications; the XML description file includes a protocol status description and a data extraction requirement description;

[0146] The first judgment module is used to judge whether it is necessary to call an external extension function based on the XML description file and determine a first judgment result.

[0147] The function extension module is used to compile an external extension library to perform function extension if the first judgment result is yes.

[0148] A compiling module is used to compile the XML description file into a state node tree file through a description compiler and generate a user dynamic function library file if the first judgment result is no.

[0149] The domain data extraction module is used to use the message processing automaton to execute the processing functions carried by the state node tree file and the user dynamic function library file, and output the domain data extracted from the current message; the domain data is network traffic metadata.

[0150] In an exemplary embodiment, a computer device is provided, such as Figure 14As shown, the computer device can be a server or a terminal. The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store describable network traffic metadata extraction data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a describable network traffic metadata extraction method is implemented.

[0151] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the above method when executing the computer program.

[0152] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the above method when executed by a processor.

[0153] In an exemplary embodiment, a computer program product is provided, including a computer program, which implements the above method when executed by a processor.

[0154] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0155] In this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0156] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0157] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0158] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A describable network traffic metadata extraction method, characterized in that: include: Write XML description files according to the protocol application interaction process and description specifications; The XML description file includes a protocol status description and a data extraction requirement description; Based on the XML description file, determining whether it is necessary to call an external extension function and determining a first determination result; If the first judgment result is yes, compile the external extension library to expand the function; If the first judgment result is no, compiling the XML description file into a state node tree file through a description compiler, and compiling to generate a user dynamic function library file; The message processing automaton is used to execute the processing functions carried by the state node tree file and the user dynamic function library file, and output the domain data extracted from the current message; the domain data is network traffic metadata.

2. The describable network traffic metadata extraction method according to claim 1, characterized in that: Compile external extension libraries to expand functions, which previously also included: Utilize the external C language function and use GCC compiler to compile it into an external user dynamic function library.

3. The describable network traffic metadata extraction method according to claim 1, characterized in that: The XML description file is compiled into a state node tree file by a description compiler, and the compilation generates a user dynamic function library file, specifically including: Construct a state node data structure, a micro-operation node data structure, an intermediate jump node data structure, and a data data storage data structure; the state node data structure carries the state information in the XML description file; the micro-operation node data structure includes multiple node types, the multiple nodes include mircostate nodes, mircofunction nodes, and sofunction nodes; the intermediate jump node data structure includes a nextstate node and a vardata node; the data data storage data structure includes data representation structures of var_data, string_search_pattern, and multi_search_pattern; During the compilation process of the compiler, the tags, operations and required parameter data of the XML description file are used to generate state nodes, micro-operation nodes and intermediate jump nodes that conform to the state node data structure description, and a finite state tree is constructed; A state node tree file is constructed according to the finite state tree, and the required user dynamic function library file is generated.

4. The describable network traffic metadata extraction method according to claim 1, characterized in that: Utilizing the message processing automaton to execute the processing functions carried by the state node tree file and the user dynamic function library file, and outputting the domain data extracted from the current message, specifically including: Using the message processing automaton, the state node tree file and the user dynamic function library file are loaded into the message processing automaton engine; Passing the message after the pre-flow table and application identification to the engine for processing, initializing the flow table flow node message processing context, and starting to scan the message payload data; Based on the flow processing state of the message processing automaton, determining whether the current processing state of the current payload data is the initial state of the protocol, and determining a second judgment result; If the second judgment result is yes, executing the initial state, importing the next state into the current processing state, judging whether the current processing state is the terminal state, and determining the third judgment result; If the third judgment result is yes, output the domain data extracted from the current message; If the third judgment result is no, traverse all state nodes and execute the state sub-process; Determine whether the message atomic operation in the current processing state processing process requires cross-message processing, and determine a fourth judgment result; If the fourth judgment result is yes, the context information of the current flow processing is saved to the engine session context address storage space; If the fourth judgment result is no, the current processing state is completed, the next state node is imported, and the process returns to "determining whether the message atomic operation in the current processing state requires cross-message processing, and determining the fourth judgment result"; If the second judgment result is no, the current processing sub-state is restored from the session context of the flow table.

5. The describable network traffic metadata extraction method according to claim 4, characterized in that: Using the message processing automaton, the state node tree file and the user dynamic function library file are loaded into the engine, specifically including: Using the message processing automaton, the parameter definition information of the state node tree file is loaded into the data storage address space, and the state node is loaded into the protocol state tree storage address space; Dynamically load the user dynamic function library file into the engine.

6. The describable network traffic metadata extraction method according to claim 4, characterized in that: Traverse all state nodes and execute state sub-processes, including: Read the micro-operation node and the success and failure jump information under the current processing state node, and determine whether the currently processed micro-operation node is a mircostate node, and determine a fifth judgment result; If the result of the fifth judgment is yes, parse the var_data structure carried by the micro-operation node, calculate the parameter indirect address space, fill the parameter list, execute the message processing atomic operation carrying function, scan the corresponding field of the message, determine whether the message scanning is completed, and determine the sixth judgment result; If the sixth judgment result is yes, save the current processing state information, wait for the next message to be processed and resume processing, and determine whether the current domain data needs to be extracted, and determine the seventh judgment result; If the seventh judgment result is yes, extract the current domain data from the data storage area in the message processing automaton engine, and determine whether the current import node is the nextstate node, and determine the eighth judgment result; If the result of the eighth judgment is yes, the next state node address is calculated according to the next state index carried by the nextstate node, and the current node traversal is now completed; If the eighth judgment result is no, read the next micro-operation node and jump to the step of "determining whether the currently processed micro-operation node is a mircostate node"; If the result of the seventh judgment is no, jump to step "determine whether the current imported node is the nextstate node"; If the sixth judgment result is no, determining whether the current operation is successfully executed, and determining a tenth judgment result; If the result of the tenth judgment is yes, jump to the step of "determining whether it is necessary to extract the current domain data"; If the tenth judgment result is no, entering the termination state; If the fifth judgment result is no, determining whether the currently processed micro-operation node is a mirfunction node, and determining a ninth judgment result; If the ninth judgment result is yes, parse the var_data structure carried by the micro-operation node, calculate the parameter indirect address space, fill the parameter list, execute the micro-function function, call the common data processing function built into the engine, and jump to the step of "determining whether the current operation is executed successfully"; If the ninth judgment result is no, determining whether the currently processed micro-operation node is a sofunction node, and determining an eleventh judgment result; If the eleventh judgment result is yes, search for the loaded user dynamic function library function, parse the var_data structure carried by the micro-operation node, calculate the parameter indirect address space, fill the parameter list, and execute the user-defined function, and jump to the step of "determining whether the current operation is executed successfully"; If the eleventh judgment result is no, jump to step "read the next micro-operation node".

7. A describable network traffic metadata extraction device, characterized in that: include: An XML description file writing module is used to write XML description files according to the protocol application interaction process and description specifications; The XML description file includes a protocol status description and a data extraction requirement description; A first judgment module is used to judge whether it is necessary to call an external extension function based on the XML description file and determine a first judgment result; A function extension module, configured to compile an external extension library to perform function extension if the first judgment result is yes; A compiling module, configured to compile the XML description file into a state node tree file and generate a user dynamic function library file by using a description compiler if the first judgment result is no; The domain data extraction module is used to use the message processing automaton to execute the processing functions carried by the state node tree file and the user dynamic function library file, and output the domain data extracted from the current message; the domain data is network traffic metadata.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the describable network traffic metadata extraction method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the describable network traffic metadata extraction method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the describable network traffic metadata extraction method according to any one of claims 1 to 6 is implemented.

Images