System and method for analyzing interface traffic log message on network shunting equipment
By designing a traffic log message analysis system on the interface on the network shunt device, using JSON format parsing configuration files and memory databases, the problem of inconvenience in traffic log parsing in the existing technology is solved, and efficient and flexible log analysis is achieved.
Patent Information
- Application Number
- CN202411496266.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-25
- Publication Date
- 2025-02-11
AI Technical Summary
In the analysis of traffic logs, the existing technology has problems such as unintuitive interface, complicated information extraction, and inability to independently analyze application layer content, which leads to inconvenience in user use.
Design an interface traffic log message analysis system on network shunt devices. By loading and parsing JSON format parsing configuration files, dynamically adjusting the application layer parsing method, supporting custom message parsing mode and data filtering format, using in-memory database storage and Provides a simple query interface.
The simplified log message parsing process is implemented, which significantly shortens the time for log analysis, improves the flexibility and scalability of use, and makes the log analysis process easier and easier to manage.
Smart Images
Figure CN119420827A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of flow log message analysis, and in particular to a system and method for analyzing interface flow log messages on a network shunting device. Background Art
[0002] Usually, the traffic log parsing method is to parse according to the protocol layer, and parse out each layer of the log data message, including the physical layer, data link layer, network layer, transport layer (see Table 1) and application layer. At the same time, a certain method is used to parse out the detailed content of each field at each layer so that users can analyze the behavior of network communication.
[0003] Table 1: Data link layer, network layer, and transport layer field information.
[0004] The traffic log interface data message studied in the present invention is an Ethernet IPv4 message, and the transport layer is UDP, the port number is P, and the application layer content includes the service protocol number, service source IP address, service destination IP address, service source port, service destination port, application layer general protocol, message body length, HTTP protocol field, account information, server host name, protocol identification code, etc., see Table 2.
[0005] Table 2: Traffic log interface message protocol layer structure information.
[0006] The commonly used technical solution in the prior art is to use third-party general tool software such as Wireshark and tcpdump to parse the traffic log. The general tool software can parse and display information of each protocol layer, including the data frame overview of the physical layer, the Ethernet frame header information of the data link layer, the IP packet header information of the network layer, the TCP / UDP data segment header information of the transport layer, and some application layer information, such as the HTTP protocol. In order to help users quickly locate the data packets of interest, the general tool software also provides a filter function. These filters can filter data packets according to specific fields of the data packets (such as source / destination IP address, port number, protocol number, etc.), reduce the display of redundant information, and enable users to focus more on analyzing key data.
[0007] Although some general-purpose tool software is powerful, its interface may not be intuitive and friendly enough for non-professionals, and it takes some time to get familiar with it. In addition, common graphical interface general-purpose tool software uses different windows to display log analysis information. When you need to extract multiple traffic logs, you need to view information in multiple windows. Information extraction is relatively discrete, information comparison is inconvenient, and query results are cumbersome. Moreover, general-purpose tool software cannot independently parse the application layer content of traffic logs. If you want to parse the application layer, or analyze the correlation between multiple indicators at the application layer, users need to spend a lot of time and energy to complete the data analysis in a custom format, which brings inconvenience to users.
[0008] Therefore, there is an urgent need to design a system and method for parsing interface traffic log messages on a network diversion device that can simplify and clarify the levels of the processing flow and effectively shorten the query time for information related to the log messages. Summary of the invention
[0009] The purpose of the present invention is to overcome the shortcomings of the prior art and provide an interface traffic log message parsing system on a network diversion device, which is used to support the configuration file to set the log message protocol layer structure, and can dynamically adjust the application layer parsing method of various messages according to the log format; the message parsing mode and data screening format can be customized according to needs, and it has excellent flexibility and scalability; it uses an in-memory database for storage, supports real-time extraction of multiple inputs, multi-channel inputs and a large number of log messages, and the method of use is simple and easy to operate.
[0010] In order to achieve the above-mentioned purpose, a system for parsing interface traffic log messages on a network diversion device is designed, including: a log configuration loading module, which includes a parsing configuration file loading module and a log configuration file parsing module. The configuration file loading module is used to load and parse a parsing configuration file in JSON format. The parsing configuration file defines the protocol layer field information of the log data packet. The parsing configuration file format allows the definition of various fields, including the field name, type and length of each protocol layer. The protocol layer includes the data link layer, network layer, transport layer and application layer. When the program is initialized or the configuration file changes, the module will read the parsing configuration file and generate a corresponding protocol layer parsing structure, which will be passed to the log data parsing module for use. The log configuration file parsing module uses the parsing structure generated by the log configuration loading module to parse the received log data packet. The log configuration file parsing module adopts a layered approach, first parsing the data link layer, For the fields of the network layer and transport layer, and the application layer fields that may change due to the log data format, the parsing module uses a dynamic parsing interface, which selects the appropriate parsing logic according to the specific log data format defined in the configuration file; the parsed log data will be provided to subsequent modules for further processing; the log message parsing module, based on the log message protocol layer parsing structure generated by the log configuration loading module, combines the fixed format of the data link layer, network layer, and transport layer field information, and uses the message layered parsing interface to parse each field in detail according to the hierarchical order of the data link layer, network layer, transport layer, and application layer, and parse out the detailed information of each field of the data link layer, network layer, transport layer, and application layer for use by the next stage of processing modules; the log data storage module is used to filter and format the parsed log data, and store the processed log data in the database; the log information query module is used to compare the stored log data according to the set comparison conditions, and output the query results.
[0011] Preferably, the system provided by the present invention also includes other technical features, wherein the parsing configuration file is used to describe the protocol layer field information of the log data message to be parsed, and a JSON format parsing configuration file is used to describe the information of each field. Each format can contain field name, type, length and other information, and supports defining various log message protocol layer fields through configuration files. Changes to the message field information can also be quickly applied to the running program by overloading the parsing configuration file.
[0012] Preferably, the system provided by the present invention also includes other technical features, including a definition of the format of the field information of the parsing configuration file, a parsing module that can preset multiple formats of log messages, or temporarily and dynamically adjust the parsing configuration file. When the program is initialized and started, the parsing configuration file will be loaded; or when the parsing configuration file is changed, the parsing configuration file will be triggered to be reloaded. At this time, the parsing field information set in the configuration file will be read according to the parsing configuration file. This information includes the parsing field name, field type and field length. Through the log configuration file parsing module, the protocol layer described by the configuration file is converted into a log message protocol layer parsing structure for use by the log message parsing module in the next stage.
[0013] Preferably, the system provided by the present invention also includes other technical features, wherein the log data storage module includes two parts, a log data screening submodule and a log data processing submodule. After removing some field information that does not need to be retained through the screening fields pre-set by the data screening submodule, the log data is formatted according to a specific data format through the log data processing submodule, and the reorganized formatted log data is written into the database.
[0014] Preferably, the system provided by the present invention also includes other technical features, wherein the log data filtering submodule sets the filtering field information in the filtering configuration file before the user needs to filter the log data, the filtering configuration file adopts the JSON format, one or more filtering field information are set, and the name used for the filtering field comes from the parsing field name set in the parsing configuration template. The two need to be consistent in order to correctly filter the log data in the log data storage module stage.
[0015] Preferably, the system provided by the present invention also includes other technical features, wherein the log information query module includes two sub-modules: log information comparison and query result output. The log information comparison module is used to check whether there are log records that meet the set conditions in the log data, and is mainly used for log analysis. The query result output module provides an interface for users to access log information in the database, provides a query interface for all field table items, and an interface for obtaining log analysis results. Users can obtain arbitrary log information and comparison results through the interface, and support exporting query results in multiple formats such as csv, xml and json.
[0016] Preferably, the system provided by the present invention also includes other technical features, wherein the log information query result output module filters the log data in pairs in the form of name / data. The user sets one or more comparison conditions and inputs them into the log information comparison submodule for comparison condition parsing. All log information is then judged based on the parsed comparison conditions, and the matched log information is automatically filtered out. The comparison results are stored in a database for users to view historical comparison data at any time.
[0017] The present invention also provides a method for parsing interface traffic log messages on a network diversion device using the system, the method steps are as follows: a) collecting log data messages of specific traffic through multiple interfaces; b) log configuration loading module: loading a parsing configuration file, the parsing configuration file is used to describe the protocol layer field information of the log data message; c) log message parsing module: according to the parsing configuration file, using a fixed parsing interface to parse the field information of the data link layer, the network layer and the transport layer; d) according to the application layer parsing structure generated by the log configuration loading module, using a custom application layer dynamic parsing interface to parse the application layer field information; e) log data storage module: pre-setting filtering field information, the filtering field information can set a combination of one or more fields; f) according to the filtering field information, removing the field information that does not need to be retained; g) formatting the filtered log data and writing it into a database; h) log information query module: setting a comparison condition, the comparison condition includes a field name and a field content configured in pairs; i) judging all log information according to the comparison condition, filtering out matching log information; j) storing the comparison result in the database and outputting the required result.
[0018] Compared with the prior art, the present invention has the following advantages: The data packets output by the traffic log interface are processed by a log parsing program, replacing the previous practice of relying on third-party general tool software for parsing. This improvement makes the log analysis process simpler and easier to manage, significantly reducing the difficulty of log analysis in the project. In addition, the present invention also provides native support for the expansion of the log parsing module, the data screening module, and the information comparison module, which not only reduces the complexity of the traffic log interface data packet analysis, but also shortens the log analysis cycle. Overall, the present invention shows excellent versatility and scalability. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 , is a flow chart of log message processing of the present invention; Figure 2 , is a flow chart of log configuration loading module processing of the present invention; Figure 3, is a processing flow chart of the log message parsing module of the present invention; Figure 4 , is a processing flow chart of the log data storage module of the present invention; Figure 5 , is a processing flow chart of the log information query module of the present invention. DETAILED DESCRIPTION
[0020] In order to make the purpose, principle and structure of the present invention more clear, it is further described below in conjunction with the drawings and specific embodiments.
[0021] The present invention improves upon the deficiencies of the prior art and provides a system and method for parsing interface traffic log messages on a network diversion device, which is used to implement content reading, log parsing, data screening, information comparison, and result output of data messages. The entire processing flow is clearly structured and the program operation is simple and clear. The new solution defines the traffic log interface, and sets the parsing configuration file for the fields to be parsed at each protocol layer. According to the parsing configuration file settings, the protocol layer data of various types of traffic log messages are dynamically parsed. At the same time, multiple log parsing information can be centrally stored in one or more database files, which can effectively shorten the query time for log message related information, and directly provide the query results, which is intuitive and convenient for users to use.
[0022] The present invention uses a screening module to perform unified screening and analysis of log content, pre-sets screening conditions through a screening configuration file, and then screens log data through the screening module, outputs the required results and stores them in a memory database for subsequent viewing and analysis by users. It also supports presetting multiple screening conditions and allows screening of single or multiple field information, providing users with better flexibility for single selection or multiple selection combinations. By using the above implementation scheme, when the interface traffic log data message format changes, the parsing configuration file is updated according to the format to adapt to the new traffic log data message format.
[0023] The present invention analyzes the field information of each level of the traffic log interface data message, defines the parsing configuration file and the screening configuration file according to the hierarchical structure for the defined field information, and then combines these configuration files with the parsing module and the screening module to design a complete traffic log data message parsing program.
[0024] By setting the log parsing configuration file globally, you can extract information more accurately and set field information such as business protocol number, business source IP address, business destination IP address, business source port, business destination port, application layer common protocol, TLV message body length, HTTP protocol field, account information, server host name, protocol identification code, etc. in the log configuration file.
[0025] After defining the log filtering configuration file in this way, the filtering module becomes simpler and more convenient. To change the filtering conditions, you only need to modify the filtering configuration file and reload it to immediately apply it to the running program. At the same time, the new solution adds a result output storage function, which stores the filtering results in an in-memory database and supports exporting result files in multiple formats through the database. Users can view the content of the filtering results or the content of the information comparison results by downloading the result file. It also supports viewing the content of historical result files and obtaining the parsing field information of historical traffic log interface data packets.
[0026] Message parsing is an important part of log analysis work, which is usually used to translate unreadable traffic log data messages into readable file formats, and can be screened and displayed according to set conditions, as an important step of preliminary log analysis. Due to the existence of multiple traffic log data messages, and the message data is huge, each log message contains multiple field information, and the field content will change according to user needs, which leads to a significant increase in the workload of traffic interface log data message analysis, and different parsing methods need to be adopted for different log data. Usually, the contents of these log message fields are different, and the screening conditions and screening combination types are diversified. Based on the prior art of frequent changes in log fields, the present invention is developed with the following purpose: the analysis system can be clearer and simpler about the analysis of log messages, and application development can be faster without re-modifying the application program.
[0027] Embodiment 1: Based on the above problems, this embodiment provides a system for parsing interface traffic log messages on a network traffic distribution device. Figure 1 This embodiment is divided into four modules according to the log processing process: a log configuration loading module, a log message parsing module, a log data storage module and a log information query module.
[0028] 1. The log configuration loading module: It mainly includes two parts: parsing configuration file loading module and log configuration file parsing module.
[0029] The parsing configuration file loading module is used to describe the protocol layer field information of the log data message to be parsed. This embodiment adopts a JSON format parsing configuration file to describe the information of each field. Each format can contain information such as field name, type, length, etc. It supports the definition of various log message protocol layer fields through configuration files, and changes to message field information can also be quickly applied to the running program by reloading the parsing configuration file.
[0030] Parsing configuration file module and protocol layer parsing structure, see Figure 2This embodiment can easily implement parsing of data at each protocol layer for any number of log messages in any format without modifying the code, and only requires updating the parsing configuration file, thereby improving the flexibility and maintainability of the program.
[0031] The parsing configuration file field information format definition can preset parsing modules for log messages in various formats, or can temporarily and dynamically adjust the parsing configuration file. You only need to define it in this format according to the actual application scenario. When the program is initialized and started, the parsing configuration file will be loaded; or when the parsing configuration file is changed, it will trigger the parsing configuration file to be reloaded. At this time, the parsing field information set in the configuration file will be read according to the parsing configuration file. This information includes the parsing field name, field type, and field length, etc. The log configuration file parsing module converts the protocol layer described by the configuration file into a log message protocol layer parsing structure for use by the log message parsing module in the next stage.
[0032] 2. The log message parsing module: Usually, for a complete log data message, the hierarchical structure and field information of the data link layer, network layer, and transport layer strictly follow the provisions of the RFC standard specification documents. These RFC documents are the basic technical standards in the Internet field. Therefore, the field information of the data link layer, network layer, and transport layer is fixed and will not change, see Table 1. For the protocol layer in the above standard format, the detailed information of each field can be extracted by parsing the data message using a fixed parsing interface. However, the application layer field information of the log data message is generally defined by the user. Its format changes with different business and application scenarios. The parsing of this part of the data needs to be customized by the user.
[0033] In this embodiment, the application layer parsing structure generated in the log configuration loading module is used to implement a custom application layer dynamic parsing interface to implement the parsing of the application layer field information of the log data message. Since the log configuration loading module has converted various application layer formats into corresponding application layer parsing structures, it is only necessary to select different application layer parsing structures according to the message format in the application layer dynamic parsing interface to complete the flexible parsing of application layer data information in log messages of different formats. Figure 3 .
[0034] According to the log message protocol layer parsing structure generated by the log configuration loading module, combined with the fixed format of the data link layer, network layer, and transport layer field information, the message layered parsing interface is used to parse each field in detail according to the hierarchical order of the data link layer, network layer, transport layer, and application layer, and parse out the detailed information of each field of the data link layer, network layer, transport layer, and application layer for use by the processing module in the next stage.
[0035] 3. The log data storage module: The log data storage module includes a log data screening submodule and a log data processing submodule. After parsing the log data message full field information in the log message parsing module, it is usually necessary to further process the parsed field information. This embodiment designs a screening module, which removes some field information that does not need to be retained through the pre-set screening fields of the data screening submodule, and then formats the log data according to a specific data format through the log data processing submodule, and writes the reorganized formatted log data into the database.
[0036] Before users need to filter log data, they only need to set the filter field information in the filter configuration file. The filter configuration file uses the JSON format. The filter field information can be set to one or more, which can be a flexible combination of any fields. The name used for the filter field comes from the parsing field name set in the parsing configuration template. The two must be consistent in order to correctly filter log data in the log message parsing module stage. For the entire log data filtering and writing process, see Figure 4 .
[0037] The significance of using a filtering module is that it provides users with flexible storage options. Fields that users do not need to pay attention to can be directly removed after parsing is completed, which can effectively save storage space and improve storage efficiency. Users can also choose not to set filtering configurations. When filtering configurations are missing, all parsed field information is stored by default. The advantage of using a log data storage module is that it can save all valid log information received, and the original log data information will not be lost. It can subsequently provide detailed analysis of different log behaviors within any time period. This solution uses an in-memory database to save data, which can meet the ability to process input log data at high speed and has good log processing performance.
[0038] 4. The log information query module: The log information query module includes two submodules: log information comparison and query result output. The log information comparison module is used to check whether there are log records that meet the set conditions in the log data, and is mainly used for log analysis. The query result output module provides an interface for users to access log information in the database, and provides a query interface for all field table items, as well as an interface for obtaining log analysis results. Users can obtain any log information and comparison results through the interface, and support exporting query results in multiple formats such as csv, xml and json.
[0039] Different from the data screening submodule in the log data storage module, the log information query module filters the log data in pairs in the form of name / data, and this field can be a flexible combination of any fields. The user sets one or more comparison conditions, inputs them into the log information comparison submodule for comparison condition parsing, and then judges all log information based on the parsed comparison conditions. The matched log information can be automatically filtered out, and the comparison results are stored in the database for users to view the historical comparison data at any time. The number of entries in the historical comparison data record is limited, and can be flexibly set according to the allocation of database storage space.
[0040] The comparison conditions in the log information comparison submodule and the field description use the same structure field: match_name1, match_data1, match_name2, match_data2... match_nameN, match_dataN, the comparison module fields are defined as follows: match_name1: name 1 of the target field to be matched.
[0041] match_data1: the content corresponding to the target field name 1 to be compared.
[0042] match_name2: name 2 of the target field to be matched.
[0043] match_data2: the content corresponding to the target field name 2 to be compared. ...... match_nameN: The name N of the target field to be compared.
[0045] match_dataN: the content corresponding to the target field name N to be compared.
[0046] For the processing flow of the log information query module, see Figure 5 .
[0047] By using the log configuration loading module, log message parsing module, log data storage module and log information query module in this embodiment, before parsing the log message, the user only needs to correctly configure the parsing configuration file and the filtering configuration file according to the specific application scenario, and the log parsing program will execute the program according to the corresponding configuration file, complete the log message analysis work, and output accurate log analysis results for the user to view and use. When the application scenario changes, it is only necessary to adjust the parsing configuration file and the filtering configuration file according to the actual situation without re-modifying the log parsing program to meet the log analysis of the specific scenario.
[0048] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent substitutions or changes according to the technical solutions and novel concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A system for parsing interface traffic log messages on a network traffic distribution device, characterized in that: include: The log configuration loading module includes two parts: a parsing configuration file loading module and a log configuration file parsing module. The configuration file loading module: Used to load and parse the parsing configuration file in JSON format, which defines the protocol layer field information of the log data packet. The parsing configuration file format allows definition of various fields, including field name, type and length for each protocol layer, including data link layer, network layer, transport layer and application layer; When the program is initialized or the configuration file changes, the module will read the parsed configuration file and generate the corresponding protocol layer parsing structure, which will be passed to the log data parsing module for use; Log configuration file parsing module: The log configuration file parsing module uses the parsing structure generated by the log configuration loading module to parse the received log data packet. The log configuration file parsing module adopts a hierarchical approach, first parsing the fields of the data link layer, network layer and transport layer according to the standard protocol specifications, For application layer fields that may change due to the log data format, the parsing module uses a dynamic parsing interface, which selects the appropriate parsing logic based on the specific log data format defined in the configuration file; The parsed log data will be provided to subsequent modules for further processing; The log message parsing module, based on the log message protocol layer parsing structure generated by the log configuration loading module, combines the fixed format of the data link layer, network layer, and transport layer field information, and uses the message layered parsing interface to parse each field in detail according to the hierarchical order of the data link layer, network layer, transport layer, and application layer, and parses out the detailed information of each field of the data link layer, network layer, transport layer, and application layer for use by the next stage of processing modules; The log data storage module is used to filter and format the parsed log data and store the processed log data in the database; The log information query module is used to perform information comparison on the stored log data according to the set comparison conditions and output the query results.
2. A system for parsing interface traffic log messages on a network traffic distribution device as claimed in claim 1, characterized in that: The parsing configuration file is used to describe the protocol layer field information of the log data message to be parsed. The JSON format parsing configuration file is used to describe the information of each field. Each format can contain field name, type, length and other information. It supports defining various log message protocol layer fields through configuration files. Changes to message field information can also be quickly applied to the running program by overloading the parsing configuration file.
3. A system for parsing interface traffic log messages on a network traffic distribution device as claimed in claim 1, characterized in that: The parsing configuration file field information format definition can preset parsing modules for log messages in various formats, or temporarily and dynamically adjust the parsing configuration file. When the program is initialized and started, the parsing configuration file will be loaded; or when the parsing configuration file is changed, it will trigger the parsing configuration file to be reloaded. At this time, the parsing field information set in the configuration file will be read according to the parsing configuration file. This information includes the parsing field name, field type and field length. Through the log configuration file parsing module, the protocol layer described by the configuration file will be converted into the log message protocol layer parsing structure for use by the log message parsing module in the next stage.
4. A system for parsing interface traffic log messages on a network traffic distribution device as claimed in claim 1, characterized in that: The log data storage module consists of two parts: a log data screening submodule and a log data processing submodule. The data screening submodule uses the pre-set screening fields to remove some field information that does not need to be retained, and then uses the log data processing submodule to format the log data according to a specific data format, and writes the reorganized formatted log data into the database.
5. A system for parsing interface traffic log messages on a network traffic distribution device as claimed in claim 4, characterized in that: Before the user needs to filter log data, the log data filtering submodule sets the filtering field information in the filtering configuration file. The filtering configuration file adopts the JSON format. One or more filtering field information are set. The name used for the filtering field comes from the parsing field name set in the parsing configuration template. The two need to be consistent in order to correctly filter the log data in the log data storage module stage.
6. The interface traffic log message parsing system on a network traffic distribution device according to claim 1, characterized in that: The log information query module includes two sub-modules: log information comparison and query result output. The log information comparison module is used to check whether there are log records that meet the set conditions in the log data. It is mainly used for log analysis. The query result output module provides an interface for users to access log information in the database, and provides a query interface for all field table items and an interface for obtaining log analysis results. Users can obtain any log information and comparison results through the interface, and support exporting query results in multiple formats such as csv, xml and json.
7. A system for parsing interface traffic log messages on a network traffic distribution device as claimed in claim 6, characterized in that: The log information query result output module filters the log data in pairs in the form of name / data. The user sets one or more comparison conditions and inputs them into the log information comparison submodule for comparison condition parsing. All log information is then judged based on the parsed comparison conditions, and the matching log information is automatically filtered out. The comparison results are stored in the database for users to view the historical comparison data at any time.
8. A method for parsing interface traffic log messages on a network shunting device using the system as described in any one of claims 1 to 7, characterized in that: The method steps are as follows: a) Collect log data packets of specific traffic through multiple interfaces; b) Log configuration loading module: loads a parsing configuration file, which is used to describe the protocol layer field information of the log data message; c) Log message parsing module: According to the parsing configuration file, a fixed parsing interface is used to parse the field information of the data link layer, network layer and transport layer; d) According to the application layer parsing structure generated by the log configuration loading module, the application layer field information is parsed using a custom application layer dynamic parsing interface; e) Log data storage module: pre-set filter field information, the filter field information can set a combination of one or more fields; f) Remove unnecessary field information based on the screening field information; g) Format the filtered log data and write it into the database; h) Log information query module: setting comparison conditions, the comparison conditions include field names and field contents configured in pairs; i) According to the comparison conditions, all log information is judged and matching log information is filtered out; j) Store the comparison results in the database and output the required results.