Real-time network communication protocol analysis system and analysis method thereof

Abstract

The invention discloses a real-time network communication protocol analysis system and an analysis method thereof, and relates to the technical field of computer communication. The system comprises a data access module, a protocol identification module, a protocol analysis module and an output module which are connected in sequence. The data access module is used for acquiring data and transmitting the acquired data to the protocol identification module; compared with a common protocol analysis tool, the method depends on a large amount of access data of a high-speed network card, applies an intermediate processing technology of basic protocol load data, and realizes rapid support, accurate identification and correct analysis of a network protocol and real-time analysis of network large-flow data through a modularized protocol analyzer and a protocol identifier with complete types; the hardware cost required for analyzing and processing the real-time network communication data with the same bandwidth is low, consumption of associated resources in the analysis process is correspondingly reduced, and the cost performance of input and output is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer communications, and in particular to a real-time parsing network communication protocol system and a parsing method thereof. Background Art

[0002] With the rapid development of computer network communication technology, the popularization of communication networks, and the rapid expansion of network scale, people's study, work and life have been greatly facilitated; the connection between humans and computer networks has become increasingly close, and a lot of information such as images, sounds, videos, files, etc. can be packaged and encapsulated according to a certain protocol format and then transmitted through computer networks;

[0003] As the number of computer network users increases rapidly year by year, network management and network security issues are receiving more and more attention; communication protocols, as the carrier of data transmission in the network, naturally become the core of research;

[0004] Through real-time analysis of network protocols, we can have a general understanding of the composition of network traffic, know the proportion of various protocols in the total network traffic, the changing rules of protocol traffic in the network, master the characteristics of network traffic, objectively evaluate the use of various protocols, know the identity information of communication nodes and specific communication content; in some network communication abnormal scenarios, we can clarify the reasons for network traffic fluctuations and network data abnormalities; it is of great significance for locating and troubleshooting network faults, optimizing networks, improving network communication quality, and counting the amount of data carried by protocols;

[0005] Currently, the commonly used network protocol analysis software are as follows:

[0006] 1) Wireshark is an open source network packet capture and analysis tool that mainly uses port identification technology to identify protocols. Therefore, it often misidentifies protocols for non-standard ports and cannot support real-time analysis of large quantities of communication data;

[0007] 2) Sniffer Pro is often used for wide area network protocol analysis. The long-term use of the old framework system has resulted in a high system memory usage, low scalability, and slow updates of new protocol support when the software is running;

[0008] 3) There are also many protocol parsing tools developed based on specific scenarios, which can only support partial protocol parsing and are not very versatile;

[0009] In the existing methods, protocol analysis tools based on Windows platform generally use Winpcap device driver to obtain communication data from network card, and those based on Linux platform basically use Libpcap function library to obtain protocol data from network card. Network data is copied several times from network card to user space, and user space and kernel space are repeatedly called during the copy process, which greatly limits the efficiency of receiving communication data from network card, especially the processing performance of communication messages with shorter length is worse.

[0010] The protocol identification methods of common protocol analysis tools are relatively simple, and they generally identify protocols based on ports, with a high misidentification rate for non-standard port protocols. Moreover, they can only support the analysis of some common fields of the protocol, and cannot fully evaluate the operation of the network protocol;

[0011] At present, Internet technology is developing rapidly, and new protocols are emerging in an endless stream. Common protocol analysis tools are generally provided by open source communities with version support, and can only add parsing support for some new protocols at a regular and quantitative basis, which obviously cannot meet the needs of the era of network information explosion.

[0012] In view of the above problems, the inventor proposes a real-time parsing network communication protocol system and a parsing method thereof to solve the above problems. Summary of the invention

[0013] In order to solve the above problems, the purpose of the present invention is to provide a real-time parsing network communication protocol system and a parsing method thereof.

[0014] To solve the above technical problems, the present invention adopts the following technical solutions: a real-time parsing network communication protocol system, comprising a data access module, a protocol identification module, a protocol parsing module and an output module connected in sequence;

[0015] The data access module is used to acquire data and transmit the acquired data to the protocol identification module;

[0016] The protocol identification module is used to identify the communication protocol of the data transmitted by the data access module, and then transmit the data to the corresponding protocol analysis module;

[0017] The protocol analysis module is used to analyze the data protocol and transmit the analysis result to the output module;

[0018] The output module is used to output the result of the protocol parsing module.

[0019] Preferably, the data access module, protocol identification module, protocol parsing module and output module are implemented through software programs.

[0020] Preferably, the data access module uses a 10 Gigabit network card to directly obtain communication data from the network card.

[0021] Preferably, the protocol identification module can identify the type of protocol based on which the communication data network transmission is based.

[0022] The present invention also provides a method for parsing a network communication protocol system in real time, wherein the protocol parsing module parses all information carried by the communication protocol, and the specific protocol parsing method comprises the following steps:

[0023] S1, determine whether the transmission protocol is a packet splicing, if the transmission layer is not a packet splicing, call S2 based on a single packet parsing method, if the transmission layer is a packet splicing, call step S3 based on a packet parsing method;

[0024] S2. Single-packet parsing method

[0025] Different protocol types have completely different ways of organizing information. Standard protocols organize various information according to standard documents, while non-standard protocols need to use reverse methods to clarify the distribution of various information in the message;

[0026] S3, Parsing method based on sub-packetization

[0027] The OSI network structure reference model stipulates that the maximum data length that each layer protocol can carry at a time is limited. When the amount of information to be transmitted is large, a single transmission cannot be completed and multiple network communications are required to complete the information transmission. When parsing the protocol, the data transmitted in packets must be spliced ​​together first, and then the complete data can be parsed.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. Compared with common protocol parsing tools, the present invention relies on a large amount of access data from a high-speed network card, applies the intermediate processing technology of basic protocol load data, and realizes rapid support, accurate identification, correct parsing, and real-time analysis of large network traffic data through modular and complete types of protocol parsers and protocol identifiers. The hardware cost required for parsing and processing real-time network communication data of the same bandwidth is low, which correspondingly reduces the consumption of associated resources in the parsing process and improves the cost performance of input and output;

[0030] 2. Compared with traditional protocol analysis tools that can only be applied to some simple scenarios, the present invention can be applied to many scenarios, especially suitable for the analysis of real-time communication data in high-bandwidth networks;

[0031] 3. In the present invention, the data access module uses 10 Gigabit network card technology to obtain communication data, which greatly increases the efficiency of obtaining communication data from the network card; the identification module is based on a variety of identification technologies, which can basically cover the identification of most application layer protocol data in the network; the parsing module supports the parsing of all information of public protocols and the parsing of basic information of private protocols; the modularization of the identification and parsing functions can support the real-time expansion of the types of protocol identifiers and protocol parsers to meet the support of new protocols; the output module organizes the output and displays the parsing results in a more readable manner, which improves the processing effect of communication data compared to commonly used protocol parsing tools. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 4 is a diagram of an example system architecture of the method of the present invention.

[0034] Figure 2 The present invention is a flow chart of the protocol identification flow of the protocol module of the method of the present invention.

[0035] Figure 3 The present invention is a flow chart of the protocol analysis module analysis of the method of the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] Example: Figure 1-3 As shown, the present invention provides a real-time parsing network communication protocol system, including a data access module, a protocol identification module, a protocol parsing module and an output module connected in sequence;

[0038] A data access module is used to obtain data and transmit the obtained data to the protocol identification module;

[0039] The protocol identification module is used to identify the communication protocol of the data transmitted by the data access module, and then transmit the data to the corresponding protocol analysis module;

[0040] The protocol analysis module is used to analyze the data protocol and transmit the analysis results to the output module;

[0041] The output module is used to output the results of the protocol parsing module.

[0042] The data access module, protocol identification module, protocol analysis module and output module are implemented through software programs.

[0043] The data access module uses a 10 Gigabit network card to directly obtain communication data from the network card. By obtaining communication data from the network card, the data throughput rate of the data access module is greatly improved, the efficiency of obtaining communication data from the network card is increased, and the overall parsing capability of the system is indirectly increased.

[0044] The protocol identification module can identify the type of protocol based on which the communication data network transmission is based. The protocol identification module can identify most of the communication data based on the public standard protocol in the network based on the communication port; it can identify some of the communication data of the reversible private protocol in the network based on the characteristic string contained in the communication data; it can identify the communication data of the public non-standard protocol based on the standard characteristics contained in the communication data;

[0045] Relying on the above three methods, the protocol identification module can accurately identify most of the network's public standard protocols, commonly used private protocols, and a large number of public non-standard protocols, and can basically cover most of the network's communication data;

[0046] Most application layer protocols have fixed transport layer ports. The protocol identification module stores the mapping relationship between the transport layer port and the application layer protocol in the memory. The application layer protocol can be quickly confirmed through the message port. If the protocol of the port cannot be identified, the feature string matching identification method is used to confirm whether the protocol type is a private protocol.

[0047] The feature string matching identification method relies on the unique static feature string in the application layer data as the protocol sample identification feature. By searching for the identification feature in the message, the type of communication protocol can be accurately determined. If the protocol still cannot be identified, the standard feature identification method is used to confirm whether the protocol type is a public non-standard protocol.

[0048] The message format of the public non-standard protocol fully complies with the public protocol specifications. The communication data protocol type can be identified by judging whether the packet header of the data packet complies with the protocol specification type.

[0049] The protocol analysis module supports single-packet analysis and sub-packet analysis, and supports analysis of all public protocol information and basic private protocol information. Through comprehensive protocol analysis results, it is possible to observe in real time whether the communication data transmission based on the communication protocol in the network is correct, and further analyze whether the communication equipment is working normally.

[0050] The protocol identification module and protocol analysis module support modular expansion and can provide support for new protocols;

[0051] The output module organizes and displays the analysis results in a readable manner, so that users can intuitively understand whether the communication equipment running based on the communication protocol in the communication network is working normally and whether the communication data is transmitted correctly.

[0052] Computer data communication networks are generally built based on the OSI network structure reference model. Each network layer has different communication functions, and each network layer contains communication protocols with different purposes. When sending communication data, it is broken down into separate data packets and transmitted downward layer by layer. Each time a layer of data header is passed, the flag information related to the protocol of that layer is added until it is delivered to the physical layer and sent to the physical layer of the receiver. After receiving the data, the receiver peels off the identification information of the protocol of this layer layer by layer, performs corresponding processing according to the identification information, and then transmits it upward layer by layer until the application layer obtains the data sent by the sender. The process of transmitting communication data at the physical layer is the process of sending data from the source network card to the destination network card via the communication network.

[0053] The protocol analysis tool is used to analyze the protocol identification information of each layer added by the sender of the communication data to the original data header. By analyzing the identification information, it is known whether the communication equipment based on the protocol is working properly and whether the communication network can transmit data correctly.

[0054] refer to Figure 1 During the system operation, after the network communication data is sent to the machine network card, the data processing will go through the following steps:

[0055] 1) The data access module quickly obtains network communication messages based on 10G network card technology;

[0056] 2) The communication data is directly delivered to the protocol identification module, which confirms the protocol type;

[0057] 3) The communication data delivery protocol parsing module comprehensively parses the protocol information;

[0058] 4) Finally, the output module outputs the protocol analysis results in an easy-to-read and friendly manner.

[0059] refer to Figure 2 For data transmitted in computer communication networks, the protocols applied to the physical layer, data link layer, network layer, and transport layer are of the same and fixed types, while the types of application layer protocols are complex.

[0060] The protocol identification process of the protocol identification module is as follows:

[0061] A1. First identify the protocol type of the message network layer;

[0062] The data transmission of the network layer is almost based on the IP protocol. In the TCP / IP layer, the data link layer uses MTU to limit the size of the data packet that can be transmitted. When the size of the IP datagram to be sent exceeds MTU, the IP layer needs to fragment the data, otherwise the data cannot be sent successfully. The fragmented IP datagram is reassembled to generate a complete IP datagram, and then jump to A2 to identify the transport layer protocol type;

[0063] A2. Data transmission at the transport layer is basically carried on TCP / UDP. TCP is a connection-oriented protocol. A complete TCP session starts with a "three-way handshake" to establish a connection and ends with a "four-way handshake" to disconnect the connection. A session is identified by a five-tuple (source port, destination port, source IP, destination IP, and application layer protocol type). Messages with the same five-tuple information belong to the same session. The "SYN" message generated by establishing a connection identifies the start of a session, and the "FIN" message generated by disconnecting the connection identifies the end of a session. After identifying that the transport layer protocol is TCP, jump to A3 to identify the application layer protocol. If the application layer type of the message is identifiable, the messages belonging to the same session are serially identified and passed to the protocol parser.

[0064] If the application layer type of the message cannot be identified, the parser will splice and output the message for manual processing;

[0065] UDP is a non-connection-oriented protocol. Before data transmission, the source and the terminal do not establish a connection. When transmitting data, data is sent directly to the terminal, and A3 is directly jumped to identify the application layer protocol, and then delivered to the protocol parser;

[0066] A3. There are many types of application layer protocols. Network protocol identification is basically to distinguish the types of application layer protocols. The steps of application layer protocol identification are as follows:

[0067] a) First call port-based identification: Most application layer protocols register ports with the IANA Internet Assigned Numbers Authority, which is the transport layer port of the TCP / IP model;

[0068] The port mapping-based identification algorithm is simple and requires little information. The identifier is based on ports and can accurately identify most of the protocols in the network transport layer.

[0069] If the port cannot be identified, jump to b);

[0070] b) Recalling the protocol identification based on feature string matching: relying on the features in the application layer data that are different from any other protocols, that is, the field that can identify the service type of the protocol sample, as a static feature string;

[0071] The principle of finding static feature strings is to select the fields that are unique to the protocol, must appear during the interaction, and appear most frequently in the actual environment as the feature strings of the protocol;

[0072] After selecting the static feature string in the protocol, the next step is to match the protocol feature string with the captured real-time data stream bit by bit or byte by byte, which can identify some messages;

[0073] In computer networks, the four-tuple (source port, destination port, source IP, destination IP) can uniquely identify a connection;

[0074] Based on the identifiable partial messages combined with the communication quadruple, all the messages generated by a communication connection can be identified;

[0075] If the protocol cannot be identified based on the feature string, jump to c);

[0076] c) Finally, the protocol identification method based on standard features is called. The main basis is that the message format of the standard protocol fully complies with the specification. The specifications of various protocols are studied and the characteristic fields that uniquely identify the protocol are summarized as the basis for identification.

[0077] The protocol can be identified simply by analyzing the packet header.

[0078] The present invention also provides a real-time parsing method for a network communication protocol system, referring to Figure 3 ,The analysis process of the protocol analysis module ,Network communication equipment sends and receives data based on ,communication protocols. A comprehensive analysis of the protocol can accurately ,know whether the network communication data is correct during the ,transmission process and whether the communication data can be sent and received correctly under ,normal circumstances;

[0079] The protocol parsing process of the protocol parsing module is as follows:

[0080] S1, determine whether the transport layer performs packet splicing, if not, jump to step S2, if it does, jump to step S3;

[0081] S2, parsing the communication data based on a single-packet parsing method;

[0082] S3. The parsing method based on packet sub-packetization can fully parse the data content regardless of how many times the communication data is transmitted.

[0083] The present invention can be applied to many different occasions such as comprehensive protocol analysis and network communication detection, and all required information can be obtained by only filtering the analysis results output by the system.

[0084] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A real-time parsing network communication protocol system, characterized in that: It includes a data access module, a protocol identification module, a protocol analysis module and an output module which are connected in sequence; The data access module is used to acquire data and transmit the acquired data to the protocol identification module; The protocol identification module is used to identify the communication protocol of the data transmitted by the data access module, and then transmit the data to the corresponding protocol analysis module; The protocol analysis module is used to analyze the data protocol and transmit the analysis result to the output module; The output module is used to output the result of the protocol parsing module.

2. A real-time parsing network communication protocol system as claimed in claim 1, characterized in that: The data access module, protocol identification module, protocol analysis module and output module are implemented through software programs.

3. A real-time parsing network communication protocol system as claimed in claim 1, characterized in that: The data access module uses a 10 Gigabit network card to directly obtain communication data from the network card.

4. A real-time parsing network communication protocol system as claimed in claim 1, characterized in that: The protocol identification module can identify the type of protocol based on which the communication data network transmission is based.

5. A parsing method applied to the real-time parsing network communication protocol system according to any one of claims 1 to 4, characterized in that: The protocol parsing module parses all the information carried by the communication protocol. The specific protocol parsing method includes the following steps: S1, determine whether the transmission protocol is a packet splicing, if the transmission layer is not a packet splicing, call S2 based on a single packet parsing method, if the transmission layer is a packet splicing, call step S3 based on a packet parsing method; S2. Single-packet parsing method Different protocol types have completely different ways of organizing information. Standard protocols organize various information according to standard documents, while non-standard protocols need to use reverse methods to clarify the distribution of various information in the message; S3, Parsing method based on sub-packetization The OSI network structure reference model stipulates that the maximum data length that each layer protocol can carry at a time is limited. When the amount of information to be transmitted is large, a single transmission cannot be completed and multiple network communications are required to complete the information transmission. When parsing the protocol, the data transmitted in packets must be spliced ​​together first, and then the complete data can be parsed.

Images