Application identification feature library construction method based on content features

Abstract

The invention discloses an application identification feature library construction method based on content features. The method comprises the steps that the content features are extracted according to a session data packet, and a temporary feature table is established; combining the repeated features, and marking the occurrence times of all the content features; performing algorithm merging on the similar features, and marking the occurrence frequency of the same similar feature; judging whether the occurrence frequency of similar features reaches a set threshold value or not, moving the similar features of which the occurrence frequency reaches the set threshold value into a formal feature table, and performing the next step on the remaining content features; detecting fuzzy features; and the features recorded in the formal feature table and the fuzzy features jointly form an application identification feature library. According to the method, the application identification feature library is constructed by ingeniously utilizing the byte part self-defined by a developer in the application protocol data packet as the content feature, and the byte feature serving as the application identification feature is automatically determined, so that the application identification feature library and the application identification feature are conveniently formed, and application identification is better realized.

Description

Technical Field

[0001] The present invention relates to the technical field of network traffic management, and in particular to a method for constructing an application identification feature library based on content features. Background Art

[0002] In the field of network traffic management and security, accurately identifying applications (APPs) in the network is crucial to maintaining network security, optimizing network performance, and enforcing compliance policies. Traditional application identification methods mainly rely on manual analysis of network traffic packets, a process that consumes a lot of human resources. Professionals must deeply analyze the characteristics of data packets, including but not limited to port numbers, protocol types, packet structures, etc., to establish application identification rules. However, this method has some limitations.

[0003] First, as applications are constantly updated and upgraded, the original identification rules may no longer apply, and professionals are required to re-analyze the traffic characteristics of the new version. This process is not only time-consuming and labor-intensive, but also increases maintenance costs. Second, it is difficult for traditional identification methods to accurately filter specific applications without affecting other applications. Since many applications may use the same port or protocol, or have similarities in data packet structure, misjudgment may occur when filtering is implemented, affecting the normal use of the network.

[0004] Moreover, existing apps usually use many third-party services, such as location services, third-party login services, message push services, third-party advertising, etc. A specific third-party service is used in various apps. For example, the WeChat login service is provided by a certain company, and this function is integrated into a large number of apps. If the network traffic generated by these third-party services is used as an identification feature, it will lead to application identification errors. What is even more difficult to deal with is that multiple apps developed by the same company have serious overlapping features because they use various services of the company. It is often difficult to accurately identify multiple apps developed by the same company. Summary of the invention

[0005] In view of the above problems in the above-mentioned prior art, the present invention provides a method for constructing an application identification feature library based on content features, which forms an application identification feature library in an automated manner and determines byte features as application identification features to facilitate accurate application identification.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A method for constructing an application identification feature library based on content features comprises the following steps:

[0008] S10, obtaining a session data packet between the client and the server, extracting a specified payload content from the session data packet as a content feature according to the session protocol type, and establishing a temporary feature table;

[0009] S20, identifying content features with completely identical byte features as duplicate features in the temporary feature table, merging the duplicate features, and marking the number of occurrences of all content features;

[0010] S30, determining whether the number of occurrences of a content feature in the temporary feature table reaches a set threshold, if so, moving the content features whose number of occurrences reaches the set threshold to the formal feature table, and proceeding to the next step for the remaining content features, if not, directly proceeding to the next step;

[0011] S40, identifying content features with the same partial byte features in the temporary feature table as similar features, merging these similar features by algorithm, and marking the number of occurrences of the same similar feature;

[0012] S50, determining whether the number of occurrences of similar features in the temporary feature table reaches a set threshold, if so, moving the similar features whose number of occurrences reaches the set threshold to the formal feature table, and proceeding to the next step for the remaining content features, if not, directly proceeding to the next step;

[0013] S60, detecting the number of content features remaining in the temporary feature table and the number of occurrences of each content feature, if the number of these content features is greater than a set threshold and the number of occurrences of each content feature is less than the set threshold, the corresponding content feature is considered to be a fuzzy feature;

[0014] S70. The features recorded in the formal feature table and the fuzzy features together constitute an application identification feature library, wherein the corresponding identical byte features in the repeated features and similar features are used as application identification features.

[0015] Specifically, in step S10, if the session protocol type is HTTP protocol, the contents of byte 0 and bytes 5 to 11 in the HTTP request packet are selected to form an eight-byte content feature; if the session protocol type is TCP / UDP protocol, the first four bytes of the first payload-containing message in each direction of the bidirectional traffic of the session between the server and the client are selected to form an eight-byte content feature.

[0016] Specifically, after the temporary feature table is established in step S10, the content features are deduplicated, and content features with completely identical byte features appearing in data packets of multiple applications are deleted.

[0017] Specifically, the repeated features in step S20 are content features belonging to the same application.

[0018] Specifically, the set threshold corresponding to the number of occurrences of the content feature in step S30 includes a first threshold and a second threshold, the first threshold being a set ratio of the total number of occurrences of all content features in the current temporary feature table; the second threshold being calculated as follows: under the condition that no content feature in the current temporary feature table satisfies the first threshold, the difference between the ratio of the highest number of occurrences to the total number of occurrences and the ratio of the second highest number of occurrences to the total number of occurrences;

[0019] When determining the relationship between the number of occurrences and the set threshold, the relationship between the number of occurrences and the first threshold is determined first. If the relationship is not satisfied, the relationship between the number of occurrences and the second threshold is determined.

[0020] Specifically, the process of algorithmically merging similar features in step S40 includes:

[0021] Traverse each content feature in the current temporary feature table, try to remove certain byte features of the content feature one by one to form a new feature, compare this new feature with other content features one by one, determine whether the new feature can cover other content features of the same application and does not contain features of other applications, merge the content features that can be covered as similar features, and calculate the total number of content features that can be covered as the number of occurrences of the merged similar features.

[0022] Specifically, in the process of algorithmically merging similar features in step S40, each attempt is regarded as a round of merging, and the set threshold judgment of step S50 is performed after each round of merging.

[0023] Specifically, in steps S20 and S40, after the features are merged, the protocol port range, the length range of the message sent by the client and the length range of the message sent by the server of the merged features are recorded simultaneously and used as application identification features together with the content features.

[0024] Specifically, the fuzzy feature in step S60 includes only one byte feature in the content feature as the application identification feature.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] (1) The present invention ingeniously utilizes the byte portion of the application protocol data packet that is customized by the developer as the content feature to construct an application identification feature library and automatically determine the byte feature as the application identification feature, which facilitates the formation of the application identification feature library and the formation of the application identification feature, thereby better realizing application identification. The present invention is ingenious in design, novel in conception, and convenient in implementation, and is suitable for application in application identification of network devices.

[0027] (2) The present invention uses repeated feature merging and similar feature merging in sequence, effectively reducing the number of content features and better determining the byte features in the content features used for application identification, thereby effectively improving the accuracy of application identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of the overall process of an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The present invention is further described below in conjunction with the accompanying drawings and examples. The embodiments of the present invention include but are not limited to the following examples.

[0030] Example

[0031] like Figure 1 As shown, the method for constructing an application identification feature library based on content features includes the following steps:

[0032] S10. Obtain the session data packet between the client and the server, extract the specified payload content from the session data packet as the content feature according to the session protocol type, and establish a temporary feature table; preferably, if the session protocol type is the HTTP protocol, the content of the 0th byte and the 5th to 11th bytes in the HTTP request packet are selected to form an eight-byte content feature; if the session protocol type is the TCP / UDP protocol, the first four bytes of the first payload-containing message in each direction of the two-way traffic between the server and the client constitute an eight-byte content feature. The use of these contents in each application is customized by the application developer and cannot be known by the user. In actual use, the specific number of bytes selected can be adjusted accordingly according to the size of the number of applications to be identified. For example, when the size of the number of applications to be identified increases, the number of bytes selected increases accordingly.

[0033] S15. To avoid misidentification, if the same content features appear in multiple applications, the content features need to be deduplicated first, that is, the content features with exactly the same eight-byte features appearing in the data packets of multiple applications need to be deleted.

[0034] S20. Content features with completely identical byte features are identified as duplicate features in the temporary feature table. Content features belonging to the same application are merged, and the number of occurrences of all content features is marked.

[0035] S30, determining whether there is a content feature in the temporary feature table whose number of occurrences reaches the set threshold, if so, moving the content features whose number of occurrences reaches the set threshold to the formal feature table, and proceeding to the next step for the remaining content features, if not, directly proceeding to the next step.

[0036] Among them, the set threshold corresponding to the number of occurrences includes a first threshold and a second threshold, the first threshold is a set ratio of the total number of occurrences of all content features in the current temporary feature table, for example, 30%; the second threshold is calculated as follows: under the condition that no content feature in the current temporary feature table meets the first threshold, the difference between the ratio of the highest number of occurrences to the total number of occurrences and the ratio of the second highest number of occurrences to the total number of occurrences, for example, 20%, that is, when the proportion of the number of occurrences of the content feature with the highest number of occurrences is significantly greater than the proportion of the number of occurrences of the content feature with the second highest number of occurrences, it can also be considered that the content feature with the highest number of occurrences is representative and meets the application identification requirements.

[0037] When determining the relationship between the number of occurrences and the set threshold, the relationship between the number of occurrences and the first threshold is determined first. If the relationship is not satisfied, the relationship between the number of occurrences and the second threshold is determined.

[0038] S40, identifying content features with the same partial byte features in the temporary feature table as similar features, merging these similar features by algorithm, and marking the number of occurrences of the same similar feature; wherein the process of merging similar features by algorithm includes:

[0039] Traverse each content feature in the current temporary feature table, try to remove certain byte features of the content feature one by one to form a new feature, compare this new feature with other content features one by one, and determine whether the new feature can cover other content features of the same application without including features of other applications. The content features that can be covered are merged as similar features, and the total number of content features that can be covered is calculated as the number of occurrences of the merged similar features.

[0040] Tables 1 to 3 show the process of algorithm merging similar features.

[0041] A. Select the first to N-1 content features respectively, remove one byte feature of the eight-byte features in turn, that is, the case where the "selection mask" is FE, FD, FB, F7, EF, DF, BF, 7F, and become seven-byte features. Compare them with other content features with larger codes, and calculate the number of features that are exactly the same as the seven-byte features. Table 1 shows the original data in the temporary feature table.

[0042]

[0043] Table 1 Some original data of temporary feature table

[0044] It can be seen from Table 1 that after removing any byte feature from the 1st to 3rd content features, the remaining byte features cannot overlap with the corresponding byte features of other content features and cannot be merged at this time; after removing the 8th byte feature from the 4th content feature, it can match the 5th and 6th content features, that is, merge the 5th and 6th content features with the 4th content feature, and change the "selection mask" of the 4th content feature to FE.

[0045] Thereafter, the content of the temporary feature table is changed to that shown in Table 2.

[0046]

[0047] Table 2 Update data of temporary feature table

[0048] B. Further remove more byte features, for example, remove the content of two byte features and transform the content features into a combination of six byte features, that is, the "selection mask" is: FC, FA, F9, F6, F5, F3, E7, EB, ED, EE, CF, AF, 9F, 6F... etc., which are not listed here one by one. If the selected byte feature is exactly the same as other content features, merge the features. Repeat this logic until only one byte feature is selected.

[0049] Among them, when the "selection mask" in the selection of four byte features is B1 (i.e. binary: 10110001, the byte corresponding to the bit of 1 is selected), the first content feature completely matches the four byte features of the second and third content, and the first, second and third content features are merged. The final temporary feature table is

[0050] As shown in Table 3.

[0051]

[0052] Table 3 Updated data after multiple merging

[0053] When merging content features with fewer occurrences, features with more occurrences will no longer be merged to maintain feature accuracy. The method is that after each round of attempts, the features that have been merged are sorted by the number of occurrences. For features with a number of occurrences greater than 30% of the total number of occurrences in the feature table, they are moved from the table to the formal feature table and no longer participate in the next round of merging attempts. If there are no features that exceed 30%, the first percentage is subtracted from the second percentage. If it is greater than 20%, it is also moved to the formal feature table.

[0054] If after selecting only one-byte features, there are still a large number of features that cannot be merged, and the number of occurrences of each feature is very small, this may be due to the application using data encryption or compression. In this case, you can choose to use only the port range and length range features and ignore the content features. This method regards features that do not select content features and only select one-byte content as fuzzy features. Fuzzy features usually cannot be used alone, but can be used in conjunction with other settings.

[0055] After the content features are merged, the protocol port range (maximum and minimum values ​​of the protocol port), the client sent message length range (maximum and minimum values), and the server sent message length range (maximum and minimum values) of the merged features are recorded at the same time and used as application identification features together with the content features.

[0056] S50, determine whether the number of occurrences of similar features in the temporary feature table reaches the set threshold. If so, move the similar features whose number of occurrences reaches the set threshold to the formal feature table, and proceed to the next step for the remaining content features. If not, proceed directly to the next step; the set threshold is the same as in step S30.

[0057] S60: Detect the number of content features remaining in the temporary feature table and the number of occurrences of each content feature. If the number of these content features is greater than a set threshold and the number of occurrences of each content feature is less than the set threshold, the corresponding content feature is considered to be a fuzzy feature.

[0058] S70. The features and fuzzy features recorded in the formal feature table, as well as the source port range, destination port range and bidirectional message length range of each feature, together constitute an application identification feature library, wherein the corresponding identical byte features in repeated features and similar features, as well as the protocol port range of each feature, the message length range generated by the client and the message length range sent by the server are used as application identification features.

[0059] Through the above process, an application identification feature library can be well established and application identification features can be determined, thereby effectively improving the accuracy of application identification.

[0060] The above embodiments are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any changes made by adopting the design principles of the present invention and performing non-creative work on this basis should fall within the protection scope of the present invention.

Claims

1. A method for constructing an application identification feature library based on content features, characterized in that: The following steps are involved: S10, obtaining a session data packet between the client and the server, extracting a specified payload content from the session data packet as a content feature according to the session protocol type, and establishing a temporary feature table; S20, identifying content features with completely identical byte features as duplicate features in the temporary feature table, merging the duplicate features, and marking the number of occurrences of all content features; S30, determining whether the number of occurrences of a content feature in the temporary feature table reaches a set threshold, if so, moving the content features whose number of occurrences reaches the set threshold to the formal feature table, and proceeding to the next step for the remaining content features, if not, directly proceeding to the next step; S40, identifying content features with the same partial byte features in the temporary feature table as similar features, merging these similar features by algorithm, and marking the number of occurrences of the same similar feature; S50, determining whether the number of occurrences of similar features in the temporary feature table reaches a set threshold, if so, moving the similar features whose number of occurrences reaches the set threshold to the formal feature table, and proceeding to the next step for the remaining content features, if not, directly proceeding to the next step; S60, detecting the number of content features remaining in the temporary feature table and the number of occurrences of each content feature, if the number of these content features is greater than a set threshold and the number of occurrences of each content feature is less than the set threshold, the corresponding content feature is considered to be a fuzzy feature; S70. The features recorded in the formal feature table and the fuzzy features together constitute an application identification feature library, wherein the corresponding identical byte features in the repeated features and similar features are used as application identification features.

2. The method for constructing an application identification feature library based on content features according to claim 1, characterized in that: In step S10, if the session protocol type is HTTP protocol, the contents of the 0th byte and the 5th to 11th bytes in the HTTP request packet are selected to form an eight-byte content feature; If the session protocol type is TCP / UDP protocol, the first four bytes of the first payload-containing message in each direction of the bidirectional traffic between the server and the client constitute an eight-byte content feature.

3. The method for constructing an application identification feature library based on content features according to claim 1, characterized in that: In the step S10, after the temporary feature table is established, the content features are deduplicated, and content features with completely identical byte features appearing in data packets of multiple applications are deleted.

4. The method for constructing an application identification feature library based on content features according to claim 1, characterized in that: The repeated features in step S20 are content features belonging to the same application.

5. The method for constructing an application identification feature library based on content features according to claim 1, characterized in that: The set threshold corresponding to the number of occurrences of the content feature in step S30 includes a first threshold and a second threshold, wherein the first threshold is a set ratio of the total number of occurrences of all content features in the current temporary feature table; the second threshold is calculated as follows: under the condition that no content feature in the current temporary feature table satisfies the first threshold, the difference between the ratio of the highest number of occurrences to the total number of occurrences and the ratio of the second highest number of occurrences to the total number of occurrences; When determining the relationship between the number of occurrences and the set threshold, the relationship between the number of occurrences and the first threshold is determined first, and if not satisfied, the relationship between the number of occurrences and the second threshold is determined.

6. The method for constructing an application identification feature library based on content features according to any one of claims 1 to 5, characterized in that: The process of algorithmically merging similar features in step S40 includes: Traverse each content feature in the current temporary feature table, try to remove certain byte features of the content feature one by one to form a new feature, compare this new feature with other content features one by one, determine whether the new feature can cover other content features of the same application and does not contain features of other applications, merge the content features that can be covered as similar features, and calculate the total number of content features that can be covered as the number of occurrences of the merged similar features.

7. The method for constructing an application identification feature library based on content features according to claim 6, characterized in that: In the process of algorithmically merging similar features in step S40, each attempt is regarded as a round of merging, and the set threshold judgment of step S50 is performed after each round of merging.

8. The method for constructing an application identification feature library based on content features according to claim 6, characterized in that: In the steps S20 and S40, after the features are merged, the protocol port range, the length range of the message sent by the client and the length range of the message sent by the server of the merged features are recorded simultaneously and used as application identification features together with the content features.

9. The method for constructing an application identification feature library based on content features according to claim 6, characterized in that: In step S60, the fuzzy feature includes only one byte feature in the content feature as the application identification feature.

Images