A multi-interface multi-channel online and offline network protocol analysis system and method
Through the multi-interface and multi-channel online offline network protocol analysis system, the problem of real-time monitoring and automated analysis in the existing technology is solved, and the persistent data storage and online output are realized, which meets the system's continuous real-time monitoring and automated analysis It also provides the ability to parse large-traffic network data and the integration capabilities of external operation and maintenance operation and control systems.
Patent Information
- Application Number
- CN202211605065.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-14
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-12-14
AI Technical Summary
The existing network protocol analysis technology cannot realize real-time monitoring and automated analysis, cannot persist and output network data online, cannot meet the system's requirements for continuous real-time monitoring and automated analysis, and cannot perform statistics on the analysis results of each dimension of the data. and in-depth analysis.
A multi-interface multi-channel online offline network protocol analysis system is designed, including the access layer, processing layer, storage layer and application layer. The multi-channel protocol analysis unit is realized through multi-process methods, and data is transmitted between layers through network connections. and processing, supports multi-output interfaces for file systems, databases and middleware releases, realizing persistent data storage and online output.
Realize the persistent storage and online output of real-time packet capture results, can perform offline data analysis, and expand back-end online data analysis, solving the problem that multi-channel parallel network protocol analysis cannot be resolved in traditional technology, and satisfying the large-stream network data The analytical requirements provide the integration capabilities of system management and external operation and control systems.
Smart Images

Figure CN116016720B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of network communication technology, and in particular to a multi-interface multi-channel online and offline network protocol parsing system and method. Background Art
[0002] In a specific application context, it is necessary to detect, parse, sort, restore and statistically analyze IP network communication data (hereinafter referred to as network data) in order to grasp the network situation and user behavior in this network. The detailed process of this service can be divided into first gathering the target network data, then performing network protocol parsing layer by layer in the hierarchical model of the network protocol stack, extracting network protocol metadata information, and then sorting the data according to certain rules based on the metadata information, and then extracting the application content in the sorted data according to specific business restoration methods, and finally combining the restored content with big data and artificial intelligence technology to count, analyze and grasp the overall network situation and user behavior. It can be seen that network protocol parsing is a key process, and its parsing quality directly determines the results of subsequent sorting, extraction and statistical analysis. At the same time, when processing real-time network data, the efficiency of network protocol parsing is also crucial. Low parsing efficiency will lead to data loss, making it impossible to relatively completely grasp the network situation and user behavior.
[0003] The existing protocol analysis technology is represented by the network protocol analysis tools of the Wireshark series of products. Its main modules include the core module, capture module, pcap loading module, protocol analysis module and user interface module. The capture module is responsible for monitoring and capturing network card data, the pcap loading module is responsible for processing pcap files saved offline, and the protocol analysis module implements a plug-in protocol analyzer through a good software architecture design. A protocol analysis plug-in corresponds to a network protocol analysis capability. The user interface module provides interactive pages, and the core module is responsible for integrating other modules and assembling them into applications. Wireshark also provides certain data analysis and processing capabilities.
[0004] Traditional technology supports online processing in the protocol parsing stage, but the parsing results are temporarily stored in memory without persistent storage and cannot be output online. The subsequent data analysis stage requires stopping real-time parsing and then performing offline data analysis based on the parsing results in memory. Although the current analysis efficiency can be guaranteed, actual needs usually require automatic, uninterrupted real-time monitoring and analysis of network data, real-time parsing of network data packets, and automatic monitoring of network status, user behavior, and abnormal event alarms. Offline data analysis neither meets the system's requirements for continuous real-time monitoring nor for automated analysis, nor does it meet the statistical, mining, and in-depth analysis of the parsing results of each dimension of the data. Summary of the invention
[0005] Based on this, it is necessary to provide a multi-interface multi-channel online and offline network protocol parsing system and method to address the above technical problems, which can perform persistent storage and online output, and can expand back-end online data analysis.
[0006] A multi-interface multi-channel online and offline network protocol analysis system, comprising: an access layer, a processing layer, a storage layer and an application layer;
[0007] The access layer and the processing layer, the processing layer and the storage layer, and the storage layer and the application layer are all connected by a network, so that the access layer inputs the data into the processing layer after completing data loading, the processing layer inputs the data into the storage layer after completing data regularization, the storage layer inputs the data into the application layer after completing data warehousing, and the application layer outputs and displays the data after completing data retrieval;
[0008] The storage layer includes a database unit, and the data input into the storage layer by the processing layer is stored in the database unit.
[0009] In one embodiment, the access layer includes: a network port capture unit, a file loading unit, a network receiving unit and a middleware subscription unit; the network port capture unit, the file loading unit, the network receiving unit and the middleware subscription unit are all connected to the access layer through a network;
[0010] The network port capture unit is used to receive the real-time data of the local network card and process it into standard network data;
[0011] The file loading unit is used to receive the local offline data and unpack and process it into standard network data;
[0012] The network receiving unit is used to receive the output data of the previous device and process it into standard network data through a private protocol;
[0013] The middleware subscription unit is used to receive real-time data from an external system and process it into standard network data through a private protocol.
[0014] In one embodiment, the processing layer includes: a plurality of protocol parsing units, each of which is connected to the processing layer via a network;
[0015] Each protocol parsing unit corresponds to a parsing service process, and performs layer-by-layer network protocol parsing on standard network data according to a layered network model to obtain protocol metadata for each layer of the network data.
[0016] In one embodiment, the processing layer further comprises: a load balancing module; the load balancing module is connected to the processing layer and each protocol processing unit via a network;
[0017] The load balancing module receives an input parsing request and accesses each protocol parsing unit.
[0018] In one embodiment, the storage layer includes: a file system unit, a database unit, a network output unit and a middleware publishing unit; the file system unit, the database unit, the network output unit and the middleware publishing unit are all connected to the storage layer through a network;
[0019] The file system unit is used to store file data;
[0020] The database unit is used to store structure data;
[0021] The network output unit is used to store private data;
[0022] The middleware publishing unit is used to store formatted data.
[0023] In one embodiment, the application layer includes: a module management unit, a system operation and maintenance unit, a monitoring and auditing unit, an external interface unit, and a human-machine interface unit; the module management unit, the system operation and maintenance unit, the monitoring and auditing unit, the external interface unit, and the human-machine interface unit are all connected to the application layer through a network;
[0024] The module management unit is used to manage the access layer, the processing layer and the storage layer, and configure the network port capture unit, the file loading unit, the network receiving unit, the middleware subscription unit, each protocol parsing unit, the file system unit, the database unit, the network output unit and the middleware publishing unit;
[0025] The system operation and maintenance unit is used to monitor host operation status information and resource recovery;
[0026] The monitoring and auditing unit is used to monitor the status of the network port capture unit, the file loading unit, the network receiving unit, the middleware subscription unit, each protocol parsing unit, the file system unit, the database unit, the network output unit and the middleware publishing unit in real time, and to audit the protocol parsing results in real time;
[0027] The external interface unit is used to provide an interface for the external operation and maintenance control system to facilitate integration;
[0028] The human-machine interface unit adopts a microservice architecture to achieve front-end and back-end separation.
[0029] In one embodiment, the microservice architecture is a B / S architecture.
[0030] A multi-interface multi-channel online and offline network protocol analysis method, using a multi-interface multi-channel online and offline network protocol analysis system, comprising:
[0031] Receive external data and initiate protocol parsing requests;
[0032] Respond to protocol parsing requests, perform network protocol parsing, and initiate storage output requests;
[0033] Respond to storage output requests and store and output parsing results according to the configuration;
[0034] Monitor, audit and display in real time, and wait to receive external control instructions.
[0035] In one embodiment, before receiving external data, the method further includes:
[0036] The system is initialized, the default system configuration is loaded, and the application layer, the access layer, the processing layer, and the storage layer execute module configuration in sequence.
[0037] In one embodiment, after receiving the external data, the method further includes:
[0038] When determining the external data type and determining non-standard network data, perform unpacking and / or private protocol processing.
[0039] Compared with the prior art which only considers single-machine processing and therefore temporarily stores the parsing results in the memory, the above-mentioned multi-interface multi-channel online and offline network protocol parsing system and method considers system-level processing. The storage output module includes multiple output interfaces for file system, database, middleware publishing and network output. A multi-channel protocol parsing unit is realized in a multi-process manner. The data input by the processing layer into the storage layer is stored in the database unit. This solves the problem that traditional technology cannot perform multi-channel parallel network protocol parsing and cannot cope with multi-access and high-traffic network data parsing scenarios. The real-time packet capture results can be persistently stored and output online, and can perform offline data analysis as well as expand the back-end online data analysis. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 It is an architecture diagram of a multi-interface multi-channel online and offline network protocol analysis system in one embodiment;
[0041] Figure 2 is an architecture diagram of an access layer in one embodiment;
[0042] Figure 3 is an architectural diagram of a processing layer in one embodiment;
[0043] Figure 4 is an architectural diagram of a storage layer in one embodiment;
[0044] Figure 5 is an architecture diagram of the application layer in one embodiment;
[0045] Figure 6 It is a flowchart of a multi-interface multi-channel online and offline network protocol parsing method in one embodiment;
[0046] Figure 7 The figure is a flowchart of a multi-interface multi-channel online and offline network protocol parsing method in another embodiment. DETAILED DESCRIPTION
[0047] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0048] It should be noted that all directional indications in the embodiments of the present application (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.
[0049] In addition, the descriptions of "first", "second", etc. in this application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "multiple groups" means at least two groups, such as two groups, three groups, etc., unless otherwise clearly and specifically defined.
[0050] In this application, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, a physical connection, or a wireless communication connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.
[0051] In addition, the technical solutions between the various embodiments of the present application can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.
[0052] This application provides a multi-interface multi-channel online and offline network protocol parsing system, such as Figure 1 As shown, in one embodiment, it includes: an access layer, a processing layer, a storage layer and an application layer, which correspond to a convergence module, a protocol parsing module, a storage output module and an operation and maintenance control module respectively.
[0053] Network connections are used between the access layer and the processing layer, between the processing layer and the storage layer, and between the storage layer and the application layer, so that the access layer inputs the data into the processing layer after completing data loading, the processing layer inputs the data into the storage layer after completing data regularization, the storage layer inputs the data into the application layer after completing data warehousing, and the application layer outputs and displays the data after completing data retrieval.
[0054] like Figure 2 As shown, the access layer includes: a network port capture unit, a file loading unit, a network receiving unit and a middleware subscription unit; the network port capture unit, the file loading unit, the network receiving unit and the middleware subscription unit are all connected to the access layer through a network. The aggregation module of the access layer has multiple interfaces and supports online-offline data processing features, and supports multiple external data aggregation methods, namely network port capture, file loading, network receiving and middleware subscription methods, corresponding to the network port capture unit, file loading unit, network receiving unit and middleware subscription unit.
[0055] The network port capture unit is used to receive the real-time data of the local network card and process it into standard network data. The network capture unit does not limit the implementation method. It can integrate tools such as tcpdump and wireshark, or it can be implemented based on libpcap. It can also be developed based on libpcap. The specific packet capture process of the network port capture unit developed based on libpcap includes: finding network devices, opening network devices, obtaining network parameters, compiling filtering policies, setting filters, using callback methods to capture data packets, and closing network devices. In short, the network port capture interface monitors the network card device and uses the data received by the local network card as the input of the aggregation module. The captured packet data is standard network data and can be directly sent to the next-level protocol parsing module through protocol requests.
[0056] The file loading unit is used to receive local offline data and unpack and process it into standard network data. Specifically, file loading supports users to load offline data stored in the local file system as input to the aggregation module. File data usually has a large amount of data and the data type is non-standard network data, such as satellite network link data or mobile network link data. At this time, it is necessary to unpack and convert it into standard network data and send it to the next level protocol parsing module through a protocol request. As for how to unpack and process non-standard network data in different formats into standard network data after receiving it, the corresponding protocol parsing should be performed according to the encapsulation protocol of the non-standard data. For example, for satellite link HDLC data, satellite link HDLC protocol parsing and IP data extraction processing are required.
[0057] The network receiving unit is used to receive the output data of the preceding device, and convert it into standard network data through link data processing or private protocol data processing. Specifically, the network receiving unit receives the output data of the preceding device, which may be applied to non-standard networks, such as satellite network link data or mobile network link data, and may also add device private data headers by itself. The protocol parsing module can only process standard network data. If the network receiving unit directly outputs the received data to the protocol parsing module, it will not be able to identify and process it. The network receiving unit integrates a variety of link processors and also supports extended private protocol processors. It supports parsing non-Ethernet links and private protocols and extracting the carried standard network data, and then sends it to the next level protocol parsing module through protocol requests. For example: for satellite link HDLC data, satellite link HDLC protocol parsing and IP data extraction processing are performed.
[0058] The middleware subscription unit is used to receive real-time data from external systems and process it into standard network data through private protocols. The middleware can be various message middleware that supports publish-subscribe communication mode, such as kafka, rocketmq, zeromq, etc. Specifically, the middleware subscription method is to receive real-time data from external systems through message middleware. External system data usually has a private protocol format defined by the system, which needs to be converted into standard network data through private protocol processing, and then sent to the next-level protocol parsing module through protocol request. Taking Kafka as an example, the aggregation module exposes a Kafka topic and subscribes to this topic. External data publishes data through this topic, and the aggregation module can receive external data through this topic. Among them, network port capture, network reception and middleware subscription are suitable for real-time parsing and processing of online data, and file loading is suitable for offline data batch processing.
[0059] like Figure 3As shown in the figure, the processing layer includes: multiple protocol parsing units and a load balancing module. The protocol parsing unit is implemented by secondary development of Wireshark. The specific method is to encapsulate the RESTful style interface of the enhanced packet parser of Wireshark, receive pcap data as a parameter, and return the protocol parsing result to the caller in the form of JSON. Each protocol parsing unit is connected to the processing layer through a network, and the load balancing module is connected to the processing layer and each protocol processing unit through a network. The basic function of the protocol parsing module of the processing layer is to be responsible for protocol parsing of standard network data. Compared with the existing protocol parsers, its typical feature is that it has multi-channel properties. The protocol parsing module is a service cluster composed of multiple parsing units.
[0060] Each protocol parsing unit corresponds to a parsing service process, and performs layer-by-layer network protocol parsing on standard network data according to a layered network model to obtain protocol metadata for each layer of the network data.
[0061] The load balancing module receives the input parsing request and accesses each protocol parsing unit. Specifically, the parsing service cluster is accessed through load balancing, the parsing request agent is responsible for processing external parsing requests, and the protocol parsing results are sent to the next level storage output module through storage requests.
[0062] like Figure 4 As shown, the storage layer includes: a file system unit, a database unit, a network output unit, and a middleware publishing unit; the file system unit, the database unit, the network output unit, and the middleware publishing unit are all connected to the storage layer through a network. The storage output module of the storage layer has the typical feature of multiple output interfaces, and supports saving or outputting protocol parsing result data in different formats in different ways. The output interfaces of the storage output module include database, file system, network output, and middleware publishing.
[0063] The file system unit is used to store file data, including raw data files, intermediate data files, result data files, parameter configuration files, system log files, etc. For example, the file system unit can be the file system NTFS, FAT32, EXT4, etc. that comes with the computer operating system.
[0064] The database unit is used to store structured data, including protocol parsing field information, module parameter configuration information, protocol parsing processing records, system log information, etc. For example, the database unit can be a structured or document-based database, including mysql, PostgreSQL, and mongodb.
[0065] The network output unit is used to store private data. Specifically, the processing results are output in real time in the manner required by the private protocol of the external device. For example, the network output unit generally uses a socket interface.
[0066] The middleware publishing unit is used to save formatted data. Specifically, the processing results are output in real time according to the formatting requirements of the external system. For example, the middleware publishing unit can be a publisher of kafka, rocketmq and rabbitmq.
[0067] It should be noted that the file system unit, database unit, network output unit and middleware publishing unit are all output to the application layer, but the network output unit and the middleware publishing unit are also output to external devices or systems.
[0068] like Figure 5 As shown in the figure, the application layer includes: module management unit, system operation and maintenance unit, monitoring and auditing unit, external interface unit and human-machine interface unit; the module management unit, system operation and maintenance unit, monitoring and auditing unit, external interface unit and human-machine interface unit are all connected to the application layer through the network. The operation and maintenance control module of the application layer is responsible for managing and controlling the entire parsing system, which specifically includes five units. Each unit is implemented in the form of microservices. Each unit is complete and realizes a single function. The unit can run independently and communicates between units through restful style interfaces.
[0069] The module management unit is used to manage the access layer, the processing layer, and the storage layer, and configure the network port capture unit, the file loading unit, the network receiving unit, the middleware subscription unit, each protocol parsing unit, the file system unit, the database unit, the network output unit, and the middleware publishing unit. Specifically, for the aggregation module, it includes configuring the aggregation data type, configuring the network card device and filtering parameters of the packet capture interface, configuring the directory and file size of the file interface, configuring the protocol type and network address of the network interface, and configuring the subject name of the middleware interface; for the protocol parsing module, it includes configuring the effective protocol, effective field, and output type of the parsing unit; for the storage output module, it includes configuring the database address and data table name of the database interface, configuring the protocol type and network address of the network interface, configuring the output directory, file size, and file retention period of the file interface, and configuring the subject name of the middleware unit.
[0070] The system operation and maintenance unit is used to monitor the host operation status information and resource recovery, including resource occupancy rate, equipment operation parameters and resource cleanup functions.
[0071] The monitoring and auditing unit is used to monitor the status of the network port capture unit, file loading unit, network receiving unit, middleware subscription unit, each protocol parsing unit, file system unit, database unit, network output unit and middleware publishing unit in real time, and to audit the protocol parsing results in real time. Specifically, the module status is monitored in real time, and the network data protocol parsing results are audited and displayed in real time; module monitoring includes the operation status of each module, parameter configuration, and real-time information of protocol parsing processing, and auditing includes extracting and generating information such as network user traffic proportion and network protocol distribution from the processing results.
[0072] The external interface unit is used to provide an interface for the external operation and maintenance control system to facilitate integration. Specifically, the interface is a POST type interface.
[0073] The human-machine interface unit adopts a microservice architecture to achieve front-end and back-end separation. Specifically, the microservice architecture is a B / S architecture.
[0074] It should be noted that the aforementioned network connection is a prior art, and specifically may be a wired Ethernet connection or a wireless network connection.
[0075] In this application, the typical feature of the convergence module is that it has multiple input interfaces, including network port capture, file loading, network reception and middleware subscription, to cope with local and external data of different input types; the typical feature of the protocol parsing module is that it has multi-channel protocol parsing capabilities, and its multi-channel capabilities are realized by the protocol parsing unit service cluster; the typical feature of the storage module is that it has multiple output interfaces, including file systems, databases, network outputs and middleware publishing; the operation and maintenance management module includes five units: module management, system operation and maintenance, monitoring and auditing, external interfaces and human-computer interfaces. The key steps of the system workflow are: data loading, data processing, data storage, data retrieval and data display. The data flow processing process is: the convergence module receives external data, the protocol parsing module performs protocol parsing processing, and the storage output module saves and outputs the processing result information. The operation and maintenance control module manages the convergence module, processing module and storage module through control flow, and provides human-computer interface, audit monitoring and external interface capabilities.
[0076] Compared with the prior art which only considers single-machine processing and therefore temporarily stores the parsing results in the memory, the above-mentioned multi-interface multi-channel online and offline network protocol parsing system considers system-level processing. The storage output module includes multiple output interfaces for file system, database, middleware publishing and network output. A multi-channel protocol parsing unit is realized in a multi-process manner. The data input by the processing layer into the storage layer is stored in the database unit. This solves the problem that traditional technology cannot perform multi-channel parallel network protocol parsing and cannot cope with multi-access and high-traffic network data parsing scenarios. The real-time packet capture results can be persistently stored and output online, and can perform offline data analysis as well as expand the back-end online data analysis.
[0077] In addition, traditional technologies usually use local network card device packet capture mode and file mode. For the network data captured by the local network card, the data structure is usually link layer-network layer-transport layer-application layer, and then it is parsed layer by layer according to the network protocol stack. When faced with the scenario where the data to be parsed is sent by an external system through the network, the data packet structure captured by the traditional parser is link layer-network layer-transport layer-network layer-transport layer-application layer. The data packets to be analyzed are all located in the last three layers. The traditional solution parses to the first transport layer and treats the next three layers as unknown data. It no longer parses and processes them, and cannot meet the data parsing needs of external users. For the file mode, the traditional technology can only identify files that encapsulate data packets in a specified format. For files that are not encapsulated in the original data packets, the traditional technology cannot unpack them, and thus cannot parse the data packets. Compared with the file and packet capture interfaces in the prior art, the aggregation module in the present application has multiple input interfaces for network port capture, file loading, network reception and middleware subscription, which solves the problem of not being able to completely parse data packets input through an external network or other interfaces, and the problem of not being able to parse original data packets that are not encapsulated in a specified format. Through the design of the aggregation module, multi-interface input support for network port capture, file loading, network reception and middleware subscription is achieved, wherein file loading is unpacked and unpacked according to the encapsulation format of the data packet in the file, and network reception and middleware subscription are subsequently processed by private protocols, which can well parse external input data and parse files whose data packets are not encapsulated in the specified format.
[0078] Traditional technology does not have the ability to capture and analyze multiple channels, and cannot process network data from multiple channels at the same time. Compared with the single-channel protocol parser in the prior art, the multi-channel protocol parsing unit of the protocol parsing module of this application solves the problem of not supporting multi-channel parallel processing. By storing the parsing results of the output module for persistent storage and real-time result output, it solves the problem that traditional technology can only perform protocol parsing online, and data analysis needs to be paused for manual offline processing or manual loading of offline files for analysis. It lays the foundation for the back-end expansion of the data analysis and processing system to achieve uninterrupted automated real-time network data processing, such as parsing result statistics, data mining and in-depth analysis.
[0079] Traditional technology is a single application with a human-machine interface. The human-machine interface and business processing are coupled and mainly implemented in C / C++. For the key part of protocol parsing, C / C++ can ensure its processing efficiency. However, the traditional technology is too complicated or not implemented for user pages, database access, external interfaces, operation and maintenance control and other functions, which is not conducive to integration with external operation and maintenance control systems. Compared with the existing technology that cannot integrate external operation and maintenance control systems, the external control unit of the operation and maintenance control module of this application; compared with the existing technology single application architecture and pure C / C++ implementation, this application adopts B / S architecture to provide external interface, realize the decoupling of pages and business, and mixes C / C++ and JAVA multi-language implementation methods to solve the problem that the single application architecture, user pages, database access, external interface, operation and maintenance control and other functions are too complicated or not implemented by C / C++, and cannot be integrated with external operation and maintenance control systems, which brings system management capabilities and external operation and maintenance control system integration capabilities that traditional technologies do not have.
[0080] The present application also provides a multi-interface multi-channel online and offline network protocol parsing method, such as Figure 6 As shown, in one embodiment, it includes:
[0081] Step 602: receiving external data and initiating a protocol analysis request;
[0082] Step 604: respond to the protocol analysis request, perform network protocol analysis, and initiate a storage output request;
[0083] Step 606: Responding to the storage output request, storing and outputting the parsing results according to the configuration;
[0084] Step 608: Real-time monitoring, auditing and display, and waiting to receive external control instructions.
[0085] Preferably, before receiving external data, it also includes: system initialization, loading default system configuration, and executing module configuration in sequence at the application layer, access layer, processing layer and storage layer.
[0086] Further preferably, after receiving the external data, the method further includes: when determining the type of the external data and determining that the data is non-standard network data, performing unpacking and / or private protocol processing, specifically, Figure 7 shown.
[0087] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0088] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A multi-interface multi-channel online and offline network protocol analysis system, characterized in that: include: Access layer, processing layer, storage layer and application layer; The access layer and the processing layer, the processing layer and the storage layer, and the storage layer and the application layer are all connected by a network, so that the access layer inputs the data into the processing layer after completing data loading, the processing layer inputs the data into the storage layer after completing data regularization, the storage layer inputs the data into the application layer after completing data warehousing, and the application layer outputs and displays the data after completing data retrieval; The storage layer includes a database unit, and the data input into the storage layer by the processing layer is stored in the database unit; The access layer includes: a network port capture unit, a file loading unit, a network receiving unit and a middleware subscription unit; the network port capture unit, the file loading unit, the network receiving unit and the middleware subscription unit are all connected to the access layer through a network; The network port capture unit is used to receive the real-time data of the local network card and process it into standard network data; The file loading unit is used to receive the local offline data and unpack and process it into standard network data; The network receiving unit is used to receive the output data of the previous device and process it into standard network data through a private protocol; The middleware subscription unit is used to receive real-time data from external systems and process it into standard network data through private protocol; The processing layer includes: a plurality of protocol parsing units, each of which is connected to the processing layer via a network; Each protocol parsing unit corresponds to a parsing service process, and performs layer-by-layer network protocol parsing on standard network data according to the layered network model to obtain protocol metadata of each layer of network data; Characterized in that the processing layer further comprises: a load balancing module; the load balancing module is connected to the processing layer and each protocol processing unit through a network; The load balancing module receives the input parsing request and accesses each protocol parsing unit; The storage layer includes: a file system unit, a database unit, a network output unit and a middleware publishing unit; the file system unit, the database unit, the network output unit and the middleware publishing unit are all connected to the storage layer through a network; The file system unit is used to store file data; The database unit is used to store structure data; The network output unit is used to store private data; The middleware publishing unit is used to store formatted data; The application layer includes: a module management unit, a system operation and maintenance unit, a monitoring and auditing unit, an external interface unit and a human-machine interface unit; the module management unit, the system operation and maintenance unit, the monitoring and auditing unit, the external interface unit and the human-machine interface unit are all connected to the application layer through a network; The module management unit is used to manage the access layer, the processing layer and the storage layer, and configure the network port capture unit, the file loading unit, the network receiving unit, the middleware subscription unit, each protocol parsing unit, the file system unit, the database unit, the network output unit and the middleware publishing unit; The system operation and maintenance unit is used to monitor host operation status information and resource recovery; The monitoring and auditing unit is used to monitor the status of the network port capture unit, the file loading unit, the network receiving unit, the middleware subscription unit, each protocol parsing unit, the file system unit, the database unit, the network output unit and the middleware publishing unit in real time, and to audit the protocol parsing results in real time; The external interface unit is used to provide an interface for the external operation and maintenance control system to facilitate integration; The human-machine interface unit adopts a microservice architecture to achieve front-end and back-end separation.
2. The multi-interface multi-channel online and offline network protocol analysis system according to claim 1, characterized in that: The microservice architecture is a B / S architecture.
3. A multi-interface multi-channel online and offline network protocol analysis method, characterized in that: The multi-interface multi-channel online and offline network protocol analysis system according to claim 1 or 2 comprises: Receive external data and initiate protocol parsing requests; Respond to protocol parsing requests, perform network protocol parsing, and initiate storage output requests; Respond to storage output requests and store and output parsing results according to the configuration; Monitor, audit and display in real time, and wait to receive external control instructions.
4. The multi-interface multi-channel online and offline network protocol analysis method according to claim 3, characterized in that: Before receiving external data, it also includes: The system is initialized, the default system configuration is loaded, and the application layer, the access layer, the processing layer, and the storage layer execute module configuration in sequence.
5. The multi-interface multi-channel online and offline network protocol analysis method according to claim 3, characterized in that: After receiving external data, it also includes: When determining the external data type and determining non-standard network data, perform unpacking and / or private protocol processing.
