Method for constructing an application recognition feature library based on content features

Abstract

The application discloses a kind of application identification feature library construction methods based on content features, comprising: extracting content features according to session data packet and establishing temporary feature table;Merging is carried out to repeated features, and the appearance number of all content features is marked;Similar features are algorithmically merged, and the appearance number of the same similar feature is marked;Determine whether the appearance number of similar feature reaches the set threshold value, move the similar feature whose appearance number reaches the set threshold value to formal feature table, and the remaining content features are subjected to next step;Detect fuzzy feature;The feature recorded in formal feature table and fuzzy feature jointly constitute application identification feature library.The application ingeniously utilizes the part of byte defined by developer in application protocol data packet as content feature to construct application identification feature library and automatically determines the byte feature as application identification feature, facilitates the formation of application identification feature library and the formation of application identification feature, so that application identification is better realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network traffic management, in particular to a method for constructing an application recognition feature library based on content features. BACKGROUND

[0002] In the field of network traffic management and security, accurately identifying applications (APPs) in the network is crucial for maintaining network security, optimizing network performance, and enforcing compliance policies. Traditional methods of application recognition rely heavily on manual analysis of network traffic packets, which requires a significant amount of human resources. Professionals must delve into analyzing the features of the packets, including but not limited to port numbers, protocol types, packet structures, etc., to establish application recognition rules. However, this method has some limitations.

[0003] Firstly, as applications are constantly updated and upgraded, the original recognition rules may no longer be applicable, requiring professionals to reanalyze the traffic features of new versions. This process not only consumes time and effort but also increases maintenance costs. Secondly, traditional recognition methods are difficult to accurately filter specific applications without affecting other applications. Since many applications may use the same ports or protocols, or have similarities in packet structures, false positives may occur when implementing filtering, affecting the normal use of the network.

[0004] Moreover, existing APPs often use many third-party services, such as location services, third-party login services, message push services, and third-party advertisements. A specific third-party service may be used in various APPs, for example, WeChat login service provided by Tencent, which is integrated into a large number of APPs. If the network traffic generated by these third-party services is used as a recognition feature, it will lead to incorrect application recognition. It is even more difficult to accurately identify multiple applications developed by the same company, as they use various services provided by the company, resulting in severe feature overlap. SUMMARY

[0005] To address the above problems in the prior art, the present application provides a method for constructing an application recognition feature library based on content features, which forms an application recognition feature library in an automated manner and determines byte features as application recognition features, so as to accurately perform application recognition.

[0006] To achieve the above purpose, the technical solution adopted by the present application is as follows:

[0007] A method for constructing an application recognition feature library based on content features, comprising the following steps:

[0008] S10, obtaining session data packets between the client and the server, extracting specified payload contents as content features from the session data packets according to a session protocol type, and establishing a temporary feature table;

[0009] S20, identifying content features with the same byte features in the temporary feature table as repeated features, merging the repeated features, and marking the occurrence times of all the content features;

[0010] S30, judging whether the occurrence times of the content features in the temporary feature table reach a set threshold, if yes, moving the content features with the occurrence times reaching the set threshold to a formal feature table, and performing the next step on the remaining content features, if no, directly performing the next step;

[0011] S40, identifying content features with part of the same byte features in the temporary feature table as similar features, merging the similar features by an algorithm, and marking the occurrence times of the same similar features;

[0012] S50, judging whether the occurrence times of the similar features in the temporary feature table reach a set threshold, if yes, moving the similar features with the occurrence times reaching the set threshold to the formal feature table, and performing the next step on the remaining content features, if no, directly performing the next step;

[0013] S60, detecting the number of the remaining content features in the temporary feature table and the occurrence times of the content features, if the number of the content features is greater than a set threshold and the occurrence times of the content features are less than a set threshold, considering the corresponding content features as fuzzy features;

[0014] S70, constructing an application recognition feature library by the features recorded in the formal feature table and the fuzzy features, wherein the corresponding same byte features in the repeated features and the similar features are used as the application recognition features.

[0015] Specifically, in the step S10, if the session protocol type is HTTP protocol, the content features of eight bytes are composed of the 0th byte and the 5th to 11th bytes in the HTTP request packet; if the session protocol type is TCP / UDP protocol, the content features of eight bytes are composed of the first four bytes in the first packet containing payload in each direction of the session bidirectional traffic between the server and the client.

[0016] Specifically, in the step S10, after the temporary feature table is established, the content features are de-duplicated, and the content features with the same byte features appearing in the data packets of multiple applications are deleted.

[0017] Specifically, the repeated features in the step S20 are the content features belonging to the same application.

[0018] Specifically, the set threshold corresponding to the occurrence number of the content feature in the step S30 comprises a first threshold and a second threshold, the first threshold is a set proportion of the total number of the occurrence number of all content features in the current temporary feature table, and the second threshold is calculated in the following manner: when no content feature in the current temporary feature table meets the condition of the first threshold, the difference between the proportion of the highest occurrence number relative to the total number of the occurrence number and the proportion of the second highest occurrence number relative to the total number of the occurrence number.

[0019] When judging the relationship between the occurrence number and the set threshold, the relationship between the occurrence number and the first threshold is judged first, and then the relationship between the occurrence number and the second threshold is judged if the first relationship does not meet the condition.

[0020] Specifically, the process of algorithmic merging of similar features in the step S40 comprises:

[0021] Each content feature in the current temporary feature table is traversed, and a new feature is formed by trying to remove a byte feature from the content feature one by one, the new feature is compared with other content features one by one, it is judged whether the new feature can cover other content features of the same application and does not contain features of other applications, the content features that can be covered are merged as similar features, and the total number of the content features that can be covered is calculated as the occurrence number of the merged similar features.

[0022] Specifically, in the process of algorithmic merging of similar features in the step S40, each attempt is a round of merging, and the set threshold judgment of the step S50 is performed after each round of merging.

[0023] Specifically, in the steps S20 and S40, the protocol port range, the client sending message length range and the server sending message length range of the merged features are recorded at the same time as application recognition features together with the content features.

[0024] Specifically, the fuzzy feature in the step S60 comprises only one byte feature in the content feature as an application recognition feature.

[0025] Compared with the prior art, the application has the following beneficial effects:

[0026] (1) The application ingeniously uses the byte part defined by the developer in the application protocol data packet as the content feature to construct the application recognition feature library and automatically determines the byte feature as the application recognition feature, which facilitates the formation of the application recognition feature library and the formation of the application recognition feature, thereby better realizing application recognition. The application is designed ingeniously, conceived newly, realized conveniently and suitable for application in network equipment application recognition.

[0027] (2) The present invention uses the method of merging repeated features and merging similar features in sequence, which effectively reduces the number of content features and better determines the byte features in the content features used for application identification, thereby effectively improving the accuracy of application identification. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the overall process of an embodiment of the present invention. Detailed Implementation

[0029] The present invention will be further described below with reference to the accompanying drawings and embodiments. The embodiments of the present invention include, but are not limited to, the following embodiments.

[0030] Example

[0031] like Figure 1 As shown, the method for constructing an application recognition feature library based on content features includes the following steps:

[0032] S10. Obtain the session data packets between the client and the server. Based on the session protocol type, extract the specified payload content from the session data packets as content features and establish a temporary feature table. Preferably, if the session protocol type is HTTP, select the content of byte 0 and bytes 5 to 11 of the HTTP request packet to form an eight-byte content feature. If the session protocol type is TCP / UDP, select the first four bytes of the first payload-containing message in each direction of the bidirectional session traffic between the server and the client to form an eight-byte content feature. The purpose of this content in each application is defined by the application developer and is unknown to the user. In actual use, the specific number of bytes selected can be adjusted according to the size of the number of applications to be identified. For example, the number of bytes selected increases as the number of applications to be identified increases.

[0033] S15. To avoid misidentification, if the same content features appear in multiple applications, the content features need to be deduplicated first, that is, the content features with the same eight-byte features appearing in the data packets of multiple applications need to be deleted.

[0034] S20. In the temporary feature table, content features with identical byte characteristics are identified as duplicate features. Content features belonging to the same application are merged, and the occurrence count of all content features is marked.

[0035] S30. Determine if any content feature in the temporary feature table has reached a set threshold in the number of occurrences. If so, move the content feature that has reached the set threshold in the number of occurrences to the formal feature table, and proceed to the next step for the remaining content features. If not, proceed directly to the next step.

[0036] The set threshold corresponding to the occurrence number includes a first threshold and a second threshold. The first threshold is a set proportion of the total sum of the occurrence numbers of all content features in the current temporary feature table, for example, 30%. The second threshold is calculated in the following manner: when no content feature in the current temporary feature table meets the condition of the first threshold, the difference between the proportion of the highest occurrence number to the total number of occurrence numbers and the proportion of the second highest occurrence number to the total number of occurrence numbers, for example, 20%. That is, when the proportion of the occurrence number of the content feature with the highest occurrence number is obviously greater than the proportion of the occurrence number of the content feature with the second highest occurrence number, it can also be considered that the content feature with the highest occurrence number is representative and meets the application recognition requirement.

[0037] When judging the relationship between the occurrence number and the set threshold, the relationship between the occurrence number and the first threshold is first judged, and then the relationship between the occurrence number and the second threshold is judged if the first relationship does not meet the condition.

[0038] In S40, the content features with the same part of byte features in the temporary feature table are identified as similar features, the algorithm of the similar features is merged, and the occurrence numbers of the same similar features are marked. The process of merging the algorithm of the similar features includes the following steps.

[0039] In S41, each content feature in the current temporary feature table is traversed, and a new feature is formed by trying to remove a certain byte feature from the content feature one by one. The new feature is compared with other content features one by one to determine whether the new feature can cover other content features of the same application without including the features of other applications. The content features that can be covered are merged as similar features, and the total number of the content features that can be covered is calculated as the occurrence number of the merged similar features.

[0040] The process of merging the algorithm of the similar features is shown in Tables 1 to 3.

[0041] In A, the first to N-1 content features are selected respectively, one byte feature in the eight byte features is removed in sequence, that is, the selected mask is FE, FD, FB, F7, EF, DF, BF, and 7F, which becomes seven byte features. The seven byte features are compared with other content features with larger codes, and the number of features that are completely the same is calculated. Table 1 shows the original data in the temporary feature table.

[0042]

[0043] Table 1: Some original data of the temporary feature table

[0044] As can be seen from Table 1, after removing any one byte feature from the first to third content features, the remaining byte features cannot be covered by the corresponding byte features of other content features, and thus cannot be merged; after removing the eighth byte feature from the fourth content feature, the fourth content feature can be matched with the fifth and sixth content features, i.e., the fifth and sixth content features are merged with the fourth content feature, and the "selection mask" of the fourth content feature is changed to FE.

[0045] After that, the content of the temporary feature table is changed to that shown in Table 2.

[0046]

[0047] Table 2: update data of the temporary feature table

[0048] B, further remove more byte features, for example, remove two byte features, and convert the content features to six byte feature combinations, i.e., the "selection mask" is: FC, FA, F9, F6, F5, F3, E7, EB, ED, EE, CF, AF, 9F, 6F, and the like, which are not listed here. If the selected byte features are the same as those of other content features, the features are merged. Repeat the logic until only one byte feature is selected.

[0049] When the "selection mask" is B1 (i.e., binary: 10110001, the byte corresponding to the bit of 1 is selected) in the selection of four byte features, the first content feature is completely matched with the four byte features of the second and third content features, and the first, second and third content features are merged, and the final temporary feature table is

[0050] as shown in Table 3.

[0051]

[0052] Table 3: update data after multiple merging

[0053] In the content feature merging, the features with a small number of occurrences are not merged, and the features with a large number of occurrences are not merged to maintain the accuracy of the features. The method is to sort the features that have been merged according to the number of occurrences after each attempt, and move the features with a number of occurrences greater than 30% of the total number of occurrences in the feature table to the formal feature table, and do not participate in the next round of merging attempt. If there are no features that exceed 30%, the first share is reduced by the second share, and if it is greater than 20%, it is also moved to the formal feature table.

[0054] If there are still a large number of features that cannot be merged after selecting only one byte feature, and the occurrence number of each feature is small, it may be caused by data encryption or compression of the application. At this time, only the port range and length range features can be selected, and the content features can be ignored. The method regards the content features and the features with only one byte content as fuzzy features, which cannot be used alone and can be used in combination with other settings.

[0055] After the content features are merged, the protocol port range (maximum and minimum values of the protocol port), the client message length range (maximum and minimum values) and the server message length range (maximum and minimum values) of the merged features are recorded, which are used as application identification features together with the content features.

[0056] S50, determines whether the occurrence number of similar features in the temporary feature table reaches a set threshold value, if yes, moves the similar features with the occurrence number reaching the set threshold value to the formal feature table, and the remaining content features are subjected to the next step, if not, directly proceeds to the next step; wherein the set threshold value is the same as in step S30.

[0057] S60, detects the number of remaining content features in the temporary feature table and the occurrence number of each content feature, if the number of these content features is greater than a set threshold value and the occurrence number of each content feature is less than a set threshold value, the corresponding content feature is regarded as a fuzzy feature.

[0058] S70, the features recorded in the formal feature table and the fuzzy features, and the source port range, the destination port range and the bidirectional message length range of each feature, jointly constitute an application identification feature library, wherein the corresponding same byte features in the repeated features and the similar features, and the protocol port range, the client message length range and the server message length range of each feature are used as application identification features.

[0059] Through the above process, the application identification feature library can be well established and the application identification features can be determined, which effectively improves the accuracy of application identification.

[0060] The above embodiments are only preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Any changes made on the basis of the design principles of the present application and non-creative labor shall fall within the protection scope of the present application.

Claims

1. A method of constructing a feature library for application recognition based on content features, characterized by, The method comprises the following steps: S10, obtaining session data packets between a client and a server, extracting specified payload contents as content features from the session data packets according to a session protocol type, and establishing a temporary feature table; S20, identifying content features with the same byte features in the temporary feature table as repeated features, merging the repeated features, and marking the occurrence times of all the content features; S30, judging whether the occurrence times of the content features in the temporary feature table reach a set threshold value, if yes, moving the content features with the occurrence times reaching the set threshold value to a formal feature table, and performing the next step on the remaining content features, if no, directly performing the next step; S40, identifying content features with part of the same byte features in the temporary feature table as similar features, algorithmically merging the similar features, and marking the occurrence times of the same similar features; S50, judging whether the occurrence times of the similar features in the temporary feature table reach a set threshold value, if yes, moving the similar features with the occurrence times reaching the set threshold value to the formal feature table, and performing the next step on the remaining content features, if no, directly performing the next step; S60, detecting the number of the remaining content features in the temporary feature table and the occurrence times of the content features, if the number of the content features is greater than a set threshold value and the occurrence times of the content features are less than a set threshold value, considering that the corresponding content features are fuzzy features; S70, jointly forming an application identification feature library from the features recorded in the formal feature table and the fuzzy features, wherein the corresponding same byte features in the repeated features and the similar features are used as the application identification features.

2. The method of claim 1, wherein the content feature-based application recognition feature library construction method is characterized by, In the step S10, if the session protocol type is an HTTP protocol, the content of the 0th byte and the 5th to 11th bytes in an HTTP request packet is selected to form eight-byte content features; If the session protocol type is a TCP / UDP protocol, the first four bytes in the first message containing payload in each direction of the session bidirectional traffic between the server and the client are selected to form eight-byte content features.

3. The method of claim 1, wherein the content feature-based application recognition feature library construction method is characterized by, In the step S10, after the temporary feature table is established, the content features are de-duplicated, and the content features with the same byte features appearing in the data packets of multiple applications are deleted.

4. The method of claim 1, wherein the content feature-based application recognition feature library construction method is characterized by, The repeated features in the step S20 are the content features belonging to the same application. 5.The method of claim 1, wherein, In the step S30, the occurrence times of the content features correspond to a first threshold value and a second threshold value, the first threshold value is a set proportion of the total sum of the occurrence times of all the content features in the current temporary feature table, and the second threshold value is calculated in the following manner: when no content feature in the current temporary feature table meets the condition of the first threshold value, the difference between the proportion of the highest occurrence time relative to the total number of occurrence times and the proportion of the second highest occurrence time relative to the total number of occurrence times; When judging the relationship between the occurrence times and the set threshold value, the relationship between the occurrence times and the first threshold value is judged first, if the condition is not met, the relationship between the occurrence times and the second threshold value is then judged.

6. The method according to any one of claims 1 to 5, wherein In the step S40, the process of algorithmically merging the similar features comprises: Traverse each content feature in the current temporary feature table, try to remove a byte feature from the content feature one by one to form a new feature, compare the new feature with other content features one by one, judge whether the new feature can cover other content features of the same application and does not contain features of other applications, merge the content features that can be covered as similar features, and calculate the total number of the content features that can be covered as the occurrence frequency of the merged similar features.

7. The method according to claim 6, wherein In the process of algorithmic merging of the similar features in the step S40, each attempt is a round of merging, and the threshold setting of the step S50 is performed after each round of merging. 8.The method of claim 6, wherein the content feature-based application recognition feature library construction method is characterized by, In the steps S20 and S40, after the feature merging, the protocol port range, the client sending message length range and the server sending message length range of the merged features are recorded at the same time as the application identification features together with the content features.

9. The method of claim 6, wherein the content feature-based application recognition feature library construction method is characterized by, The fuzzy feature in the step S60 includes only one byte feature in the content feature as the application identification feature.

Images