Private protocol application identification method and device based on protocol fingerprint and storage medium

Abstract

The invention discloses a private protocol application identification method and device based on a protocol fingerprint and a storage medium, and provides the private protocol application identification method based on the protocol fingerprint, which solves the problem that the specific application of a private protocol is difficult to identify because the private protocol does not have a unified standard. And compared with other methods, protocol fingerprint identification has more advantages and is more accurate compared with a behavior identification mode. The private protocol application identification method based on the protocol fingerprint comprises the following steps: identifying a private protocol application based on the protocol fingerprint: grouping network flow, identifying and skipping a public protocol, extracting private protocol characteristics to construct a fingerprint, matching the fingerprint with a fingerprint database, and if the fingerprint is not matched, adding the fingerprint as an unknown application into the fingerprint database.

Description

Technical Field

[0001] The present invention relates to the field of computer network security-traffic analysis, and in particular to a private protocol application identification method, device and storage medium based on protocol fingerprint. Background Art

[0002] No matter what application or service on the Internet, network communication must rely on various network protocols, such as HTTP, TLS, SSH, etc. Our common public standard protocols are: HTTP, TLS, SSH, etc. However, when some application (WeChat, OICQ, etc.) or malicious program (botnet, Trojan, APT, etc.) makers implement their own network functions, the known network protocols cannot meet the needs in terms of security and scalability, so they need to design and implement private protocols. Private protocols refer to network protocols designed and produced by network applications or malicious program authors. Their communication traffic has a customized protocol format and may be protected by standard encryption algorithms or self-designed encryption algorithms. Using private protocols, data communication is safer. If hackers want to analyze the vulnerabilities of the protocol, they must first decipher the format and algorithm of the protocol, and cannot exploit the vulnerabilities of common protocols; moreover, private protocols have better scalability and can expand their own protocol formats according to their own needs and development.

[0003] Since private protocols have no unified standards and specifications, the identification of private protocol applications poses a challenge to network traffic analysis. Currently, the mainstream private protocol application identification methods are as follows.

[0004] The first is a feature matching identification method based on deep packet analysis technology (DPI). This method mainly targets known private protocols, extracts feature strings in the protocol as identification codes, and applies rule engines such as Snort, Suricata, and Yara for matching detection. After matching the features, it can be considered that the private protocol corresponding to a certain application is identified. For example: "201080065034.9 Application Protocol Identification Method and Device", "200710119852.2 Application Layer Protocol Identification Method", "201510220668.1 A Method and Device for Identifying Application Protocols", etc. However, the application that this method can identify requires that its private protocol feature code is fixed or has a certain pattern. Once the private protocol content is encrypted, it is impossible to extract a valid feature code or the feature code changes irregularly, and the detection method based on the feature code becomes invalid.

[0005] The second is the identification method based on traffic statistics, which builds a rule base or hash base by applying statistical features of traffic, such as the mean and variance of traffic behaviors such as packet length, packet sending frequency, number of packets, and port, for matching and identification. For example: "201510997641.3 A method and system for identifying network application layer protocols", "201210477026.6 A method and device for automatically identifying application protocols", etc. However, different applications of the same type in the existing network, such as video and instant messaging, may have similar traffic statistics, and this method is difficult to accurately distinguish applications of the same type.

[0006] Identification of private protocol applications: In real network environments, there are a large number of applications or malicious programs that use non-standard custom private network protocols to communicate and transmit data. Since there is no unified standard for private protocols, it is difficult to form a unified solution to identify applications with various private protocols.

[0007] In view of this, the present invention is proposed. Summary of the invention

[0008] The main purpose of the present invention is to disclose a private protocol application identification method, device and storage medium based on protocol fingerprint. The problem to be solved is to propose a private protocol application identification method based on protocol fingerprint, which can identify the traffic using private protocol communication from the real network traffic, and identify the specific application or malicious program using the private protocol.

[0009] To achieve the above object, according to one aspect of the present invention, a method for identifying private protocol applications based on protocol fingerprint is provided, and the following technical scheme is adopted:

[0010] The network traffic is grouped according to predetermined rules; the public protocol is identified for the traffic of each group to determine whether the traffic is a public protocol. If so, all the traffic of the group is skipped; if not, the execution continues downward; for each session of the group traffic that is not identified as a public protocol, the private protocol features are extracted from the payload part of each packet, and the feature attributes are recorded to construct a feature attribute table; all the feature attributes of the feature attribute table are extracted and arranged in order to form a character string, if the feature or feature attribute is missing, it is replaced by 0, and the hash value of the character string is calculated as the fingerprint of the private protocol; the fingerprint is matched with a pre-established fingerprint library, and if the same fingerprint is matched, the application name corresponding to the fingerprint is output; if the fingerprint is not matched, it is added to the fingerprint library as an unknown application.

[0011] Furthermore, the grouping of network traffic according to a predetermined rule includes: grouping TCP or UDP network traffic according to the same source IP, destination IP, and destination port. The constructing of a characteristic attribute table includes: forming a characteristic attribute table in accordance with the four aspects of characteristic, attribute, type, value, and meaning.

[0012] Furthermore, the pre-established fingerprint library includes: pre-collecting a large number of applications and malicious programs that use private protocols; obtaining the private protocol traffic during their communication through a predetermined method, extracting their protocol fingerprints through the aforementioned method, storing them in the fingerprint library according to the entry fields of the fingerprint library, gradually accumulating and expanding, and generating a fingerprint library.

[0013] According to another aspect of the present invention, a private protocol application identification device based on protocol fingerprint is provided, and the following technical solution is adopted:

[0014] A private protocol application identification device based on protocol fingerprint, characterized by comprising:

[0015] The grouping module is used to group network traffic according to predetermined rules; the identification module is used to identify the public protocol of each group of traffic and determine whether the traffic is a public protocol. If so, all the traffic of the group is skipped; if not, the execution continues downward; the extraction module is used to extract private protocol features from the payload part of each packet of each session of the group traffic that is not identified as a public protocol, and record the feature attributes to construct a feature attribute table; the calculation module is used to extract all feature attributes of the feature attribute table and arrange them in order to form a character string, if the feature or feature attribute is missing, it is replaced by 0, and the hash value of the character string is calculated as the fingerprint of the private protocol; the matching module is used to match the fingerprint with a pre-established fingerprint library, if the same fingerprint is matched, the application name corresponding to the fingerprint is output; if the fingerprint is not matched, it is added to the fingerprint library as an unknown application.

[0016] Furthermore, the extraction module includes: a feature attribute table is formed in sequence according to four aspects: feature, attribute, type, value and meaning.

[0017] Furthermore, the matching module includes: a collection module, which is used to pre-collect a large number of applications and malicious programs that use private protocols; a generation module, which is used to obtain the private protocol traffic during their communication through a predetermined method, extract their protocol fingerprints through the aforementioned method, store them in the fingerprint library according to the entry fields of the fingerprint library, gradually accumulate and expand, and generate a fingerprint library.

[0018] A storage medium includes the private protocol application identification device described above.

[0019] This invention proposes a private protocol application identification method based on protocol fingerprint, which solves the problem that private protocols have no unified standards and are difficult to identify their specific applications. Protocol fingerprint identification has more advantages than other methods: it has higher detection coverage and change adaptability than string feature matching methods, and can not only identify private protocol applications without strong string features, but also even if the string features of an application change, as long as the protocol format does not change, it can still be effectively identified; compared with behavioral identification methods, it is more accurate and more convincing, and behavioral patterns may overlap in multiple applications of the same type, such as: they are all video applications, and the protocol format is designed by each application itself, so it has better uniqueness. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0021] Figure 1 A flowchart of a method for identifying a private protocol application based on protocol fingerprint according to an embodiment of the present invention;

[0022] Figure 2 A specific logic flow chart of a private protocol application identification method based on protocol fingerprint according to an embodiment of the present invention;

[0023] Figure 3 These are the key terms and abbreviations specifically used in the method for identifying private protocol applications based on protocol fingerprints described in an embodiment of the present invention;

[0024] Figure 4 An example of a private fingerprint library for a private protocol application identification method based on protocol fingerprints according to an embodiment of the present invention;

[0025] Figure 5 This is a structural diagram of a private protocol application identification device based on protocol fingerprint according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The embodiments of the present invention are described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0027] Figure 1 The present invention is a flowchart of a method for identifying a private protocol application based on protocol fingerprint according to an embodiment of the present invention.

[0028] See also Figure 1As shown, a method for identifying a private protocol application based on a protocol fingerprint includes:

[0029] S101: Grouping network traffic according to a predetermined rule;

[0030] S102: Perform public protocol identification on the traffic of each group to determine whether the traffic is a public protocol. If so, skip all traffic of the group. If not, continue to execute below.

[0031] S103: extracting private protocol features from the payload portion of each packet of each session of the packet traffic that is not identified as a public protocol, and recording the feature attributes to construct a feature attribute table;

[0032] S104: extract all the characteristic attributes of the characteristic attribute table and arrange them in order to form a character string. If a characteristic or characteristic attribute is missing, it is replaced by 0, and a hash value of the character string is calculated as the fingerprint of the private protocol.

[0033] S105: Match the fingerprint with a pre-established fingerprint library. If the same fingerprint is matched, output the application name corresponding to the fingerprint; if the fingerprint is not matched, add it to the fingerprint library as an unknown application.

[0034] Specifically, step S101 groups TCP or UDP network traffic according to the same source IP, destination IP, and destination port (such a group of traffic usually uses the same protocol, whether it is a public protocol or a private protocol). In step S102, the traffic of each group is identified as a public protocol to determine whether the traffic is a public protocol (the common protocols defined by the RFC standard can be identified and filtered out using the feature value identification method, such as: HTTP, FTP, DNS, SSL / TLS, etc.). If it is a public protocol, the subsequent steps will not be continued, and all traffic of the group will be skipped; if it is not identified as a public protocol, the next step will be continued. In step S103, for each session in the TCP or UDP group that is not identified as a public protocol (same source IP, destination IP, source port, destination port), the private protocol features are extracted from the payload part of each packet, and the feature attributes are recorded, as shown in the following table: Figure 3 , Figure 3The data parameters of the private protocol application identification method based on protocol fingerprint of the present invention are as follows: In step S104, all the extracted feature attributes are arranged in order to form a character string. If a feature or feature attribute is missing, 0 is used to fill the place (for example: there are 6 features in total, the 5th feature is missing, and each of its attributes is filled with 0), and the hash value of the character string is calculated (hash algorithms such as MD5 or SHA-1 can be selected) as the fingerprint of the private protocol. Different applications or malicious programs use different private protocol formats. Therefore, their private protocol fingerprints are also different, and most of them are unique. In step S105, the fingerprint is matched with a pre-established fingerprint library. If the same fingerprint is matched, the application name corresponding to the fingerprint is output; if the fingerprint is not matched, and it is not known which application the private protocol traffic comes from, it is added to the fingerprint library as an unknown application. An example of a private fingerprint library is as follows: Figure 4 shown.

[0035] By collecting a large number of applications and malicious programs that use private protocols in advance, the private protocol traffic during their communication is obtained manually or by running them in a sandbox. After the aforementioned steps three and four, the protocol fingerprint is extracted and stored in the fingerprint library according to the entry fields of the fingerprint library, gradually accumulating and expanding.

[0036] This invention proposes a private protocol application identification method based on protocol fingerprint, which solves the problem that private protocols have no unified standards and are difficult to identify their specific applications. Protocol fingerprint identification has more advantages than other methods: it has higher detection coverage and change adaptability than string feature matching methods, and can not only identify private protocol applications without strong string features, but also even if the string features of an application change, as long as the protocol format does not change, it can still be effectively identified; compared with behavioral identification methods, it is more accurate and more convincing, and behavioral patterns may overlap in multiple applications of the same type, such as: they are all video applications, and the protocol format is designed by each application itself, so it has better uniqueness.

[0037] Figure 2 The present invention provides a specific logic flow chart of a method for identifying private protocol applications based on protocol fingerprints according to an embodiment of the present invention.

[0038] like Figure 2 As shown, a method for identifying private protocol applications based on protocol fingerprints specifically includes the following steps:

[0039] Step 210, input TCP or UDP traffic, grouping with the same source IP, destination IP, and destination port.

[0040] Step 220: whether it is recognized as a public protocol;

[0041] Step 221, skip all flows of the group, select the next public protocol flow, and select the next group;

[0042] Step 222, extracting private protocol features and attributes of the features;

[0043] Step 230, calculating the private protocol fingerprint;

[0044] Step 240: whether it matches the fingerprint in the fingerprint library;

[0045] Step 241, identifying the application name;

[0046] Step 242, marking as unknown application and adding to the fingerprint library;

[0047] Step 250, the detection result is output.

[0048] Specifically, step 210, input TCP or UDP stream: This is the starting point of the process, indicating that there is network traffic data that needs to be analyzed. Step 220, check whether it is a public protocol: First, the system will check whether the input network traffic belongs to a known public protocol (such as HTTP, FTP, etc.). If so, the process will jump directly to the subsequent steps. Step 221, skip all unused protocols and select the next packet: If the input network traffic does not belong to a public protocol, skip those unused protocol checks and continue to process the next network packet. This step is to reduce unnecessary processing time and improve recognition efficiency. Step 222, extract features and attributes unique to private protocols: For network traffic that does not belong to a public protocol, extract its unique features and attributes. These features and attributes may include the length of the data packet, a specific byte sequence, and a port number. Step 230 calculates the fingerprint of the private protocol, arranges all the extracted feature attributes in order to form a string, and uses 0 to replace any missing features or feature attributes (for example: there are 6 features in total, the 5th feature is missing, and each of its attributes is replaced by 0), and calculates the hash value of the string (you can choose hash algorithms such as MD5 or SHA-1) as the fingerprint of the private protocol. Step 240, calculates the features and matches them with the fingerprints in the database: the extracted features and attributes will be used to calculate a fingerprint, and then this fingerprint will be matched with the fingerprints in the database. The fingerprints in the database represent the characteristics of known applications. Step 241, identify the application name. Step 242, add the unmatched fingerprint to the fingerprint library: if the calculated fingerprint does not match any fingerprint in the database, then this fingerprint will be added to the fingerprint library so that this new application can be identified in the future. Step 250, output the detection results and identify the application name: Finally, based on the results of the fingerprint matching, the system will output the detection results and identify the application name in the network traffic.

[0049] Figure 5This is a structural diagram of an identification device based on a private protocol fingerprint according to an embodiment of the present invention.

[0050] refer to Figure 5 As shown, a recognition device based on private protocol fingerprint includes: a grouping module 51, which is used to group network traffic according to a predetermined rule; an identification module 52, which is used to perform public protocol identification on the traffic of each group, and determine whether the traffic is a public protocol. If so, skip all the traffic of the group, and if not, continue to execute downward; an extraction module 53, which is used to extract private protocol features from the payload part of each packet of each session of the group traffic that is not identified as a public protocol, and record the feature attributes to construct a feature attribute table; a calculation module 54, which is used to extract all feature attributes of the feature attribute table and arrange them in order to form a character string, if the feature or feature attribute is missing, use 0 to fill the place, and calculate the hash value of the character string as the fingerprint of the private protocol;

[0051] A storage medium provided by the present invention includes the above-mentioned private protocol application identification device based on protocol fingerprint.

[0052] The invention calculates the format features and characteristic attributes of various application private protocols into fingerprints, which not only realizes the accurate identification of private protocol applications, but also the accumulation of fingerprint libraries has the characteristics of high automation efficiency. There is no need to manually analyze the protocol format and characteristics of each application. It is only necessary to use the fingerprint calculation tool or program created by the present invention to automatically process the traffic of a certain application to generate fingerprints and add them to the library, thereby achieving the purpose of efficiently and quickly accumulating private protocol application fingerprint libraries.

[0053] The above only describes some exemplary embodiments of the present embodiment by way of illustration. It is undoubted that, for those skilled in the art, the described embodiments can be modified in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A method for identifying private protocol applications based on protocol fingerprints, characterized in that: include: Group network traffic according to pre-defined rules; Perform public protocol identification on the traffic of each group to determine whether the traffic is a public protocol. If so, skip all traffic of the group. If not, continue to execute downward. For each session of the packet traffic that is not identified as a public protocol, extract the private protocol feature from the payload part of each packet, record the feature attributes, and construct a feature attribute table; Extract all the feature attributes from the feature attribute table and arrange them in order to form a string. If a feature or feature attribute is missing, use 0 to fill the place. Calculate the hash value of the string as the fingerprint of the private protocol. The fingerprint is matched with the pre-established fingerprint library. If the same fingerprint is matched, the application name corresponding to the fingerprint is output; if the fingerprint is not matched, it is added to the fingerprint library as an unknown application.

2. The method for identifying a private protocol application according to claim 1, characterized in that: The grouping of network traffic according to a predetermined rule comprises: Group TCP or UDP network traffic into a group if the source IP, destination IP, and destination port are the same.

3. The method for identifying a private protocol application according to claim 1, characterized in that: The constructing of the feature attribute table includes: The feature attribute table is organized in order of features, attributes, types, values ​​and meanings.

4. The method for identifying a private protocol application according to claim 1, characterized in that: The pre-established fingerprint library includes: Collect a large number of applications and malicious programs using private protocols in advance; The private protocol traffic during communication is obtained in a predetermined manner, and the protocol fingerprint is extracted through the aforementioned method. It is stored in the fingerprint library according to the entry fields of the fingerprint library, and gradually accumulated and expanded to generate a fingerprint library.

5. A private protocol application identification device based on protocol fingerprint, characterized in that: include: A grouping module, used to group network traffic according to predetermined rules; The identification module is used to identify the public protocol of each group of traffic and determine whether the traffic is a public protocol. If so, all traffic of the group is skipped; if not, the execution continues; An extraction module, for extracting private protocol features from a payload portion of each packet of each session of packet traffic that is not identified as a public protocol, and recording the feature attributes to construct a feature attribute table; A calculation module is used to extract all the characteristic attributes of the characteristic attribute table and arrange them in order to form a string. If the characteristic or characteristic attribute is missing, it is replaced by 0, and the hash value of the string is calculated as the fingerprint of the private protocol. The matching module is used to match the fingerprint with the pre-established fingerprint library. If the same fingerprint is matched, the application name corresponding to the fingerprint is output; if the fingerprint is not matched, it is added to the fingerprint library as an unknown application.

6. The private protocol application identification device according to claim 5, characterized in that: The extraction module comprises: The feature attribute table is organized in order of features, attributes, types, values ​​and meanings.

7. The private protocol application identification device according to claim 5, characterized in that: The matching module comprises: A collection module is used to pre-collect a large number of applications and malicious programs that use private protocols; The generation module is used to obtain the private protocol traffic during communication through a predetermined method, extract its protocol fingerprint through the aforementioned method, store it in the fingerprint library according to the entry field of the fingerprint library, gradually accumulate and expand it, and generate a fingerprint library.

8. A storage medium, characterized in that: It comprises the private protocol application identification device as described in any one of claims 5-7.

Images