A large-scale fine-grained classification algorithm for SNI information of mobile applications

Abstract

The present invention discloses a large-scale fine-grained classification algorithm for SNI information of mobile applications, and belongs to the field of network security technology. The problem that the traditional domain name classification method in the prior art is difficult to eliminate invalid SNI information and extract relevant characteristic SNI in large-scale SNI results is solved; the present invention deletes the second-level domain name determined as invalid information in the SNI extraction result based on statistical characteristics, and de-duplicates the characteristic character string based on the preset threshold condition according to the number of times the invalid second-level domain name appears in the APP obtained by the initial screening, obtains the first de-duplicate result and uses K-Means clustering to de-duplicate it twice, and obtains the third de-duplicate result; traverses the repeated SNI data in the third de-duplicate result, de-duplicates the SNI of the second-level domain name of the dissimilar APP, cleans the obtained result data, and obtains the final characteristic SNI result. The present invention effectively extracts the characteristic SNI of the APP, and can be applied to the identification of encrypted traffic features.

Description

Technical Field

[0001] The invention relates to a large-scale fine-grained classification algorithm for mobile application SNI information, belonging to the technical field of network security. Background Art

[0002] SNI (Server Name Indication) is a TLS (Transport Layer Security) protocol extension used to specify the server host name when establishing an encrypted connection. It plays an important role in identifying encrypted traffic. In the traditional TLS handshake process, the client sends a ClientHello message to the server. An SNI extension field can be attached to the ClientHello message, which contains the server host name requested by the client. The server can select the correct virtual host based on this host name and return the corresponding certificate and related configuration information. Through SNI, the server can identify the host name requested by the client before the connection is established and extract information about unknown traffic from it.

[0003] With the widespread use of encrypted network traffic and the requirements for related system program development, encrypted traffic accounts for the main part of current APP (mobile application) traffic. It is particularly important to identify the APP to which the encrypted traffic belongs and detect application behavior in some scenarios. The existing technology generally identifies the APP and function corresponding to the current traffic in real time based on SNI. Since SNI is sent in plain text at the beginning of the TLS handshake process, the SNI field can be read and analyzed to identify the visited website or service, and obtain the APP information corresponding to the current traffic from the original traffic. In order to realize the above functions, it is necessary to obtain the SNI information of the mobile application in order to obtain the content corresponding to the traffic.

[0004] For the acquisition of SNI information of mobile applications, the commonly used methods currently include static analysis methods and dynamic analysis methods. Static analysis is a reverse analysis method for software systems. It does not require running the software but infers the behavior of the software system by analyzing static data such as the program's source code, bytecode or binary code. Based on the static analysis method, the communication address during the program running process can be obtained relatively simply and widely; dynamic analysis generates real network traffic through methods such as automated clicks and extracts SNI from the TLS traffic of the real traffic through corresponding analysis tools after obtaining the original traffic information generated, thereby obtaining its network information characteristics; the above two analysis methods each have their own advantages and can be used in combination to extract the SNI information of the APP, but for the above two methods, there is still a lot of invalid information in the results obtained after completing the SNI extraction. The characteristic SNI corresponding to each APP only occupies a part of the extraction results. In fact, most of the extraction results are invalid content or results that are not very relevant to the APP.

[0005] There are relatively few studies on SNI classification in the prior art. The main research direction is the method of classifying URL (Uniform Resource Locator) domain names that are similar to SNI structures. URL domain name classification is the process of classifying and categorizing domain names according to their types, contents or characteristics. Classifying domain names can help understand and organize domain name resources on the Internet and provide a basis for applications such as security, network management and content filtering. Its data source and data form application scenarios are somewhat different from those of the present invention. Although the static analysis method can automatically extract a large number of APP-related SNI results in a short time, the SNI obtained by static analysis or dynamic analysis often contains some invalid results. There is still a lot of invalid information in the results extracted by the existing domain name classification tools. The invalid results of static analysis are classified as follows:

[0006] (1) Although some SNIs match the regular expression, they are actually inaccessible or have no association with the app, for example, results such as www.example.com.

[0007] (2) SNI information related to the framework used in the development process, for example, results such as www.github.com or www.h5app.com included in some code. Directly outputting such content will not help identify the characteristic SNI;

[0008] (3) Some public SNIs of larger cloud service providers, for example, domain names include Alibaba's aliyuncs, Baidu's bcebos, Huawei's hicloud, and umeng security. Although such SNIs do have communication behaviors during the operation of the APP, they cannot accurately distinguish whether they are encrypted traffic generated by a specific APP.

[0009] (4) Some commonly used public API interfaces, for example, the API interfaces corresponding to the Alipay and WeChat payment functions from other apps, and the positioning functions of Amap. Since such functions are widely present in multiple mainstream apps and are widely used, it is difficult to determine whether they are the characteristic traffic corresponding to a certain app;

[0010] (5) Some special function API interfaces that are less popular than the above types. For example, some document-related APPs use online document conversion related API interfaces. Such API interfaces are often only used by a small number of specific types of APPs, but the APP corresponding to the SNI cannot be identified based on such SNI;

[0011] (6) A string that is similar to an SNI or URL but does not have a domain name structure. It is difficult to effectively distinguish such a string from a normal SNI using common methods.

[0012] For dynamic analysis, although the results do not contain a lot of invalid information because SNI is extracted from real traffic, the results during the click process are still limited by the device network environment. The invalid results of dynamic analysis are classified as follows:

[0013] (1) Background processes running in the system. For example, in the dynamic click process of different brands of mobile phones, the background program will communicate with the SNI address related to the mobile phone manufacturer;

[0014] (2) Feature SNI corresponding to some other program function API interfaces called in the current test program;

[0015] (3) Other SNI information when the ad jumps or the APP is loaded.

[0016] At the same time, it is unrealistic to manually clean the SNI data of APPs in the case of large-scale automated analysis. This is mainly because there is a certain correlation between the features of different APPs, and it is difficult to determine whether a certain SNI also appears in another APP and whether it is the characteristic SNI of other APPs. In addition, due to the use of related functions and API interfaces, the accuracy of the results of manual analysis in this process is affected to a certain extent.

[0017] In summary, there is a need for a large-scale fine-grained classification algorithm for raw data in large-scale mobile applications, which removes all invalid SNI information from the results of static analysis and dynamic analysis and outputs actual valid feature SNI information. Summary of the invention

[0018] A brief overview of the present invention is provided below in order to provide a basic understanding of certain aspects of the present invention. It should be understood that this overview is not an exhaustive overview of the present invention. It is not intended to identify key or important parts of the present invention, nor is it intended to limit the scope of the present invention. Its purpose is merely to present certain concepts in a simplified form as a prelude to a more detailed description discussed later.

[0019] In view of this, in order to solve the problem that traditional domain name classification methods in the prior art are difficult to eliminate invalid SNI information and extract relevant feature SNI in large-scale SNI results, the present invention provides a large-scale fine-grained classification algorithm for mobile application SNI information.

[0020] The technical solution is as follows: A large-scale fine-grained classification algorithm for mobile application SNI information includes the following steps:

[0021] S1. Deleting the second-level domain names determined as invalid information in the SNI extraction results based on statistical characteristics to obtain the results of preliminary screening of invalid second-level domain names;

[0022] S2. Build an SNI list file based on the prevalence of the results of the initial screening of invalid second-level domain names and their relevance to the current APP, identify the SNI corresponding to the second-level domain name, deduplicate the feature string based on the preset threshold conditions, extract the SNI list file based on the statistical features, and obtain the first deduplication result;

[0023] S3. Based on the list file at this time, K-Means clustering is performed on the common feature values ​​of different domain names to remove duplicates from the first deduplication result, and the SNI list file is updated with the result obtained after deduplication to obtain the second deduplication result;

[0024] S4. Based on the manifest file at this time, K-Means clustering is performed on the eigenvalues ​​of different APP associations to remove duplicates from the second deduplication result, and the SNI manifest file is updated based on the result obtained after deduplication to obtain the third deduplication result;

[0025] S5. According to the current list file, the duplicate SNI data in the third deduplication result is traversed, and the APP similarity list file where the duplicate SNI data is located is constructed. Based on the similar APPs selected based on the statistical characteristics, the SNI corresponding to the secondary domain names of the remaining dissimilar APPs is deduplicated, and a new list file is constructed to obtain the characteristic SNI contained in the input APP;

[0026] S6. Perform data cleaning on the feature SNI contained in the input APP to obtain the final feature SNI result corresponding to the APP.

[0027] Furthermore, in S1, the SNI extraction results of dynamic analysis and static analysis are obtained based on the statistical characteristics of the APP, and an invalid SNI feature library is constructed. If the SNI extraction results contain feature strings in the invalid SNI feature library, they are directly deleted to obtain the result after preliminary screening of invalid secondary domain names.

[0028] Furthermore, in S2, the number of times T that each second-level domain name appears in different APPs is extracted, and the number of times S that different SNIs appear under the second-level domain name of each APP is counted. The preset threshold condition is expressed as: T>20 and S>4. A deduplication operation is performed on the SNI corresponding to the second-level domain name that is repeated more than 3 times in the same APP that meets the preset threshold condition, and a new SNI list file is constructed. Step S2 is repeated until the new SNI list file no longer changes, and the first deduplication result is obtained.

[0029] Furthermore, in S3, for the first deduplication result, considering the popularity of the domain name, the number of times S of different SNIs appearing under the same second-level domain name is subjected to the first K-Means clustering, d1=max{the minimum value in the cluster with the maximum mean, 5}, d1 is the first K-Means clustering condition, and all SNI list files with S<d1 under the current second-level domain name are read. If the number of repetitions of a certain SNI in the SNI list file is greater than t1, it is filled into the queue to be deleted, t1 is the set smaller repetition threshold, and the queue to be deleted contained in all SNI list files under the current second-level domain name is deleted and the SNI list file is updated to obtain an updated SNI list file, and step S3 is repeated until the SNI list file no longer changes, and the second deduplication result is obtained.

[0030] Furthermore, in S4, for the second deduplication result, the association degree R between the current APP and the second-level domain name is considered, the SNI corresponding to the APP corresponding to the second-level domain name is retained, and all other repeated APPs that are not related to the second-level domain name are deleted through the second K-Means clustering, d2=min{4, the number of elements of the cluster with the maximum mean}, d2 is the second K-Means clustering condition, and the screened APPs are obtained. The richness R of the screened APPs is ranked, and the top d2 APPs do not participate in the identification of the queue to be deleted, and the rest participate in the identification of the queue to be deleted. If the number of repetitions of a certain SNI in the SNI list file is greater than t2 and t2<t1, it is filled in the queue to be deleted to obtain an updated SNI list file, and step S4 is repeated many times until the list file no longer changes to obtain the third deduplication result.

[0031] Furthermore, in S5, the duplicate SNI data in the third deduplication result is traversed according to the latest SNI list file, the APPs where the duplicate SNI data are located are integrated to construct an APP similarity list file, all files with T<4 in the APP similarity list file are read and similar APPs are screened out according to the APP similarity judgment condition, and the screened out similar APPs are added to the similar APP list, and the above steps are repeated until there are no similar APPs, and the APPs that do not meet the APP similarity judgment condition are added to the queue to be deleted, and the content to be deleted under the current secondary domain name is deduplicated, and all current duplicate SNI data are deleted. A new list file is constructed, and step S5 is repeated until the new list file no longer changes, and the characteristic SNI contained in the input APP is obtained;

[0032] The APP similarity determination condition is expressed as:

[0033] (1) The same second-level domain name in the APP similarity list file appears in two APPs at the same time, and the S values ​​are equal under the same T value;

[0034] (2) If the number of second-level domain name strings ranked in the top three in terms of richness R of two APPs in the APP similarity list file is exactly the same;

[0035] (3) The package names of the APK files of the two apps in the APP similarity list file are similar, and their full phonetic names and abbreviations are similar;

[0036] Apps that meet the above three points are considered similar apps.

[0037] Furthermore, in S6, a request for the SNI is constructed and tested for a response. If the SNI does not exist or the request fails, the SNI record is deleted; if the response is normal, it is retained. Considering that the content of the constructed request may require a corresponding token or the passing of specific parameters, if the request times out or an error is given in the request method, it can be considered that the conditions are met and the current SNI is retained.

[0038] The beneficial effects of the present invention are as follows: the present invention performs fine-grained classification on the relevant SNI information extracted from static analysis or dynamic analysis by multiple deduplications, integrates the SNI feature strings in multiple APPs and the SNI features inside the current APP, extracts feature SNI for a specific APP, and subsequently realizes the identification of APP encrypted traffic features and behaviors based on the extracted feature SNI results; the present invention adopts K-Means clustering from different dimensions in the second and third deduplications, excludes the APP where the API interface is located, and only deduplicates the corresponding SNI of other APPs that call the API interface, thereby avoiding the SNI of directly providing the API interface or directly related APPs from being accidentally deleted; the present invention constructs an APP similarity list file, screens out similar APPs according to the APP similarity judgment conditions, and deduplicates the SNI corresponding to the second-level domain names of the remaining dissimilar APPs, thereby realizing the judgment of the similarity between SNI feature strings and avoiding the feature SNI of similar APPs from being repeatedly and accidentally deleted. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0040] Figure 1 This is a flowchart of a large-scale fine-grained classification algorithm for SNI information of mobile applications;

[0041] Figure 2 A flowchart for building a list file for domain name prevalence;

[0042] Figure 3 A flowchart for building a list file for domain name association;

[0043] Figure 4 A flowchart for building a new manifest file. DETAILED DESCRIPTION

[0044] In order to make the technical solutions and advantages of the embodiments of the present invention more clearly understood, the exemplary embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than an exhaustive list of all the embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0045] refer to Figure 1-4 The present embodiment is described in detail, a large-scale fine-grained classification algorithm for mobile application SNI information includes the following steps:

[0046] S1. Deleting the second-level domain names determined as invalid information in the SNI extraction results based on statistical characteristics to obtain the results of preliminary screening of invalid second-level domain names;

[0047] S2. Build an SNI list file based on the prevalence of the results of the initial screening of invalid second-level domain names and their relevance to the current APP, identify the SNI corresponding to the second-level domain name, deduplicate the feature string based on the preset threshold conditions, extract the SNI list file based on the statistical features, and obtain the first deduplication result;

[0048] S3. Based on the list file at this time, K-Means clustering is performed on the common feature values ​​of different domain names to remove duplicates from the first deduplication result, and the SNI list file is updated with the result obtained after deduplication to obtain the second deduplication result;

[0049] S4. Based on the manifest file at this time, K-Means clustering is performed on the eigenvalues ​​of different APP associations to remove duplicates from the second deduplication result, and the SNI manifest file is updated based on the result obtained after deduplication to obtain the third deduplication result;

[0050] S5. According to the current list file, the duplicate SNI data in the third deduplication result is traversed, and the APP similarity list file where the duplicate SNI data is located is constructed. Based on the similar APPs selected based on the statistical characteristics, the SNI corresponding to the secondary domain names of the remaining dissimilar APPs is deduplicated, and a new list file is constructed to obtain the characteristic SNI contained in the input APP;

[0051] S6. Clean the feature SNI contained in the input APP to obtain the final feature SNI result corresponding to the APP;

[0052] Specifically, since SNI only contains the overall host name and is different from the concepts of the second-level domain name, top-level domain name, etc. in the domain name, for the convenience of description, the present invention will introduce the relevant feature content of the domain name to describe the SNI classification algorithm;

[0053] Find all the characteristic SNIs corresponding to each APP based on the second-level domain name information, such as the example part in example.com. Build an SNI manifest file for each second-level domain name based on statistical information. The SNI manifest file contains the number of occurrences of each second-level domain name and the number of occurrences of the current second-level domain name in different apk files, i.e., different APPs. Considering the efficiency of calculation, build the SNI manifest file and evaluate SNI from two dimensions: domain name prevalence and domain name relevance. Figure 2 and Figure 3 , the specific contents of domain name prevalence and domain name relevance are as follows:

[0054] Domain name prevalence: The number of times each second-level domain name appears in different apps corresponds to the widespread use of the API. If the T value is large, it means that the corresponding API has many functions. Such second-level domain names with a high degree of repetition can be deleted, that is, their corresponding SNIs are deleted, and they are not regarded as characteristic SNIs unique to a certain app. If the T value is small, it means that the corresponding API may be a niche function, so its corresponding SNI should not be added to the queue for deletion. For example, the second-level domain name alipay corresponds to the payment function in multiple apps, and the second-level domain names qq and wechat correspond to functions such as sharing to QQ and WeChat. Such second-level domain names can be deleted on a large scale, while the second-level domain name kwaicdn only exists in Kuaishou and related applications. Although there are many of them, they are still retained in the current app.

[0055] Domain name association: Domain name association is characterized by the number of times S that different SNIs appear under each APP's second-level domain name and the degree of association R between the current APP and the second-level domain name. The R value corresponds to the amount of information that the current second-level domain name belongs only to the current APP and other APPs do not have compared to other APPs. The larger the R value, the more likely it is that the second-level domain name belongs to the current APP. For example, the corresponding relationship between the second-level domain name bytimg and Douyin and Toutiao, and the corresponding relationship between the second-level domain name alipay and Taobao, Alipay and other APPs;

[0056] The implementation of the present invention is to analyze the characteristics of each SNI in the above two dimensions under different situations, and comprehensively implement fine-grained multi-level deduplication based on the statistical results of large-scale SNI information. The purpose of multiple deduplications in the process is to identify and delete the widely called API interface information as much as possible, and delete the non-characteristic SNI of the APP as accurately as possible according to the content of the results retained at different stages and the statistical information.

[0057] Furthermore, in S1, the SNI extraction results of dynamic analysis and static analysis are obtained based on the statistical characteristics of the APP, and an invalid SNI feature library is constructed. If the SNI extraction results contain a feature string in the pre-constructed common invalid SNI feature library, it is directly deleted to obtain the result after preliminary screening of invalid secondary domain names;

[0058] Specifically, for second-level domain names that are confirmed to be invalid information, for example, second-level domain names containing strings such as github and example, step S1 can effectively reduce the workload in the subsequent deduplication process and avoid the invalid results from affecting subsequent extraction.

[0059] Furthermore, in S2, the number of times T that each second-level domain name appears in different APPs is extracted, and the number of times S that different SNIs appear under the second-level domain name of each APP is counted. The feasible preset threshold conditions are expressed as: T>20 and S>4. The SNI corresponding to the second-level domain name that is repeated more than 3 times in the same APP that meets the preset threshold conditions is deduplicated and deleted, and a new SNI list file is constructed. Step S2 is repeated until the new SNI list file no longer changes, and the first deduplication result is obtained;

[0060] Specifically, in this embodiment, based on the statistical characteristics of 6000 APPs, the SNI corresponding to the second-level domain name with T>20 and S>4 is analyzed, and the SNI with more than 3 repetitions in different APPs that meet the above conditions is deduplicated and deleted, and the SNI corresponding to the public API interface that appears on a large scale is removed. Since the T value and the S value are set large enough at this time, the deleted SNI is usually the SNI with a large usage range and called by multiple APPs. The selection of the above values ​​is based on the results of statistical analysis. When the T value and the S value are too large or too small, the results will converge too slowly or the valid results will be deleted. After the first deduplication is performed on the results, the T value and the S value of most APPs will decrease, resulting in continuous correction of the statistical results in the list file. The execution is repeated many times until the list of the file SNI correspondence relationship no longer changes, and this process ends;

[0061] The purpose of step S2 is that if large-scale deduplication is directly performed on the results after the initial screening of invalid second-level domain names, many special cases will also be directly deleted. Therefore, a coarse-grained method is selected to delete some truly invalid large-scale SNIs, such as hicloud, umeng, etc. At the same time, a lower limit of deletion is set to avoid the deletion of unique domain names, i.e., characteristic SNIs, of the widely used second-level domain name results.

[0062] Furthermore, in S3, for the first deduplication result, considering the prevalence of the domain name, the number of times S of different SNIs appearing under the same second-level domain name is subjected to the first K-Means clustering, d1=max{the minimum value in the cluster with the maximum mean, 5}, d1 is the first K-Means clustering condition, all SNI list files with S<d1 under the current second-level domain name are read, if the number of repetitions of a certain SNI in the SNI list file is greater than t1, it is filled into the queue to be deleted, t1 is the set smaller repetition threshold, the queue to be deleted contained in all SNI list files under the current second-level domain name is deleted and the SNI list file is updated to obtain the updated SNI list file, and step S3 is repeated until the SNI list file no longer changes, to obtain the second deduplication result;

[0063] Specifically, the K-Means algorithm is a commonly used clustering algorithm. Its core idea is to divide the data into k clusters so that each data point belongs to the cluster closest to it. The K-Means algorithm optimizes the clustering results by iteratively updating the distance between the data point and the cluster center;

[0064] The purpose of step S3 is to accurately identify the API interfaces called by other APPs, and use K-Means clustering to protect the SNI contained in the APP to which this type of API interface belongs from being deleted, while only extracting and deleting the public content of the application that calls the API interface. For example, the second-level domain name baidu contained in the APP developed by Baidu will have more SNIs with the second-level domain name baidu than other APPs that only call Baidu interfaces. This ensures that the characteristic SNI in the APP developed by Baidu will not be deleted.

[0065] Furthermore, in S4, for the second deduplication result, the association degree R between the current APP and the second-level domain name is considered, the SNI corresponding to the APP corresponding to the second-level domain name is retained, and all other repeated APPs that are not related to the second-level domain name are deleted through the second K-Means clustering, d2=min{4, the number of elements of the cluster with the maximum mean}, d2 is the second K-Means clustering condition, and the screened APPs are obtained. The richness R of the screened APPs is ranked, and the top d2 APPs do not participate in the identification of the queue to be deleted, and the rest participate in the identification of the queue to be deleted. If the number of repetitions of a certain SNI in the SNI list file is greater than t2 and t2<t1, it is filled in the queue to be deleted to obtain an updated SNI list file, and step S4 is repeated many times until the list file no longer changes, and the third deduplication result is obtained;

[0066] Specifically, the purpose and implementation of the second and third deduplication are similar, both of which are based on the K-Means clustering algorithm to deduplicate the SNI of the remaining API interfaces.

[0067] The purpose of step S4 is to retain only the SNIs that do not appear repeatedly in the second-level domain names of T≥4, and to ensure that the SNIs that are the same as the second-level domain names of the API interfaces in each application that provides an API interface are not deleted. For example, Baidu still stores a large number of SNIs with the second-level domain name of baidu. Although the implementation of step S4 is similar to step S3, most of the results after the second deduplication algorithm is executed will no longer meet the conditions for participating in clustering identification in step S4 and will not participate in it. This step mainly deletes the SNIs that did not meet the domain name universality deletion conditions but met the domain name association deletion conditions in the previous deduplication. The second and third deduplication results are repeated only in some cases. For the non-repeated parts, it is to delete more API interfaces that were not deleted last time. Therefore, step S4 can effectively supplement the results of the previous deduplication and repeatedly screen the results in a fine-grained manner. If step S4 is directly executed to skip the second deduplication, a lot of irrelevant data will directly participate in the second clustering operation. Some feature SNIs will be identified as SNIs of public API interfaces, and then directly deleted by fine-grained conditions, resulting in a reduction in valid data.

[0068] Furthermore, in S5, the duplicate SNI data in the third deduplication result is traversed according to the latest SNI list file, the APPs where the duplicate SNI data are located are integrated to construct an APP similarity list file, all files with T<4 in the APP similarity list file are read and similar APPs are screened out according to the APP similarity judgment condition, and the screened out similar APPs are added to the similar APP list, and the above steps are repeated until there are no similar APPs, and the APPs that do not meet the APP similarity judgment condition are added to the queue to be deleted, and the content to be deleted under the current secondary domain name is deduplicated, and all current duplicate SNI data are deleted. A new list file is constructed, and step S5 is repeated until the new list file no longer changes, and the characteristic SNI contained in the input APP is obtained;

[0069] The APP similarity determination condition is expressed as:

[0070] (1) The same second-level domain name in the APP similarity list file appears in two APPs at the same time, and the S values ​​are equal under the same T value;

[0071] (2) If the number of second-level domain name strings ranked in the top three in terms of richness R of two APPs in the APP similarity list file is exactly the same;

[0072] (3) The package names of the APK files of the two apps in the APP similarity list file are similar, and their full phonetic names and abbreviations are similar;

[0073] Apps that meet the above three points are considered similar apps;

[0074] Specifically, based on the remaining SNIs to be processed after the first three deduplication processes, step S5 is mainly to deduplicate only the SNIs used on a small scale in some apps, i.e., the duplicate results of T<4. According to the statistical results of the SNI list file, many of them are multiple versions of the same APP developed by the same service provider. For example, the second-level domain name kspay only exists in three APPs, namely, Kuaishou, Kuaishou Express Edition, and Kuaishou Concept Edition. If it is also deleted according to the first three deduplication algorithms, it will lead to the loss of many characteristic character strings. Therefore, based on multiple statistical information and the characteristics of APP development, an APP similarity judgment condition based on similarity comparison is designed;

[0075] The purpose of step S5 is to remove the remaining duplicate SNIs with fewer repetitions while retaining similar APPs, i.e., similar APK files, and retain some identical APPs, for example, Kuaishou, Kuaishou Express Edition, and Kuaishou Concept Edition, three types of APPs with the same feature extraction results, to avoid the feature SNI being accidentally deleted.

[0076] Furthermore, in S6, a request for the SNI is constructed and tested for a response. If the SNI does not exist or the request fails, the SNI record is deleted; if the response is normal, it is retained. Considering that the content of the constructed request may require a corresponding token or the passing of specific parameters, if the request times out or an error is given in the request method, it can be considered that the conditions are met and the current SNI is retained.

[0077] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is selected primarily for readability and didactic purposes, rather than for the purpose of explaining or defining the subject matter of the present invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is illustrative, not restrictive, with respect to the scope of the present invention, which is defined by the appended claims.

Claims

1. A large-scale fine-grained classification algorithm for mobile application SNI information, characterized in that: The following steps are involved: S1. Deleting the second-level domain names determined as invalid information in the SNI extraction results based on statistical characteristics to obtain the results of preliminary screening of invalid second-level domain names; S2. Build an SNI list file based on the prevalence of the results of the initial screening of invalid second-level domain names and their relevance to the current APP, identify the SNI corresponding to the second-level domain name, deduplicate the feature string based on the preset threshold conditions, extract the SNI list file based on the statistical features, and obtain the first deduplication result; S3. Based on the list file at this time, K-Means clustering is performed on the common feature values ​​of different domain names to remove duplicates from the first deduplication result, and the SNI list file is updated with the result obtained after deduplication to obtain the second deduplication result; S4. Based on the manifest file at this time, K-Means clustering is performed on the eigenvalues ​​of different APP associations to remove duplicates from the second deduplication result, and the SNI manifest file is updated based on the result obtained after deduplication to obtain the third deduplication result; S5. According to the current list file, the duplicate SNI data in the third deduplication result is traversed, and the APP similarity list file where the duplicate SNI data is located is constructed. Based on the similar APPs selected based on the statistical characteristics, the SNI corresponding to the secondary domain names of the remaining dissimilar APPs is deduplicated, and a new list file is constructed to obtain the characteristic SNI contained in the input APP; S6. Clean the feature SNI contained in the input APP to obtain the final feature SNI result corresponding to the APP; In S2, the number of times T that each second-level domain name appears in different APPs is extracted, and the number of times S that different SNIs appear under the second-level domain name of each APP is counted. The preset threshold conditions are expressed as: T>20 and S>4. The SNI corresponding to the second-level domain name that is repeated more than 3 times in the same APP that meets the preset threshold conditions is deduplicated and deleted. A new SNI list file is constructed, and step S2 is repeatedly executed until the new SNI list file no longer changes, thereby obtaining the first deduplication result; In S3, for the first deduplication result, considering the prevalence of the domain name, the number of times S of different SNIs appearing under the same second-level domain name is subjected to the first K-Means clustering, d1=max{the minimum value in the cluster with the maximum mean, 5}, d1 is the first K-Means clustering condition, all SNI list files with S<d1 under the current second-level domain name are read, if the number of repetitions of a certain SNI in the SNI list file is greater than t1, it is filled into the queue to be deleted, t1 is the set repetition threshold, the queue to be deleted contained in all SNI list files under the current second-level domain name is deleted and the SNI list file is updated to obtain the updated SNI list file, and step S3 is repeated until the SNI list file no longer changes, to obtain the second deduplication result; In S4, for the second deduplication result, the richness R between the current APP and the second-level domain name is considered, the SNI corresponding to the APP corresponding to the second-level domain name is retained, and all other repeated APPs that are not related to the second-level domain name are deleted through the second K-Means clustering, d2=min{4, the number of elements of the cluster with the maximum mean}, d2 is the second K-Means clustering condition, and the screened APPs are obtained. The richness R of the screened APPs is ranked, and the top d2 APPs do not participate in the identification of the queue to be deleted, and the rest participate in the identification of the queue to be deleted. If the number of repetitions of a certain SNI in the SNI list file is greater than t2 and t2<t1, it is filled in the queue to be deleted to obtain an updated SNI list file, and step S4 is repeated many times until the list file no longer changes to obtain the third deduplication result.

2. According to claim 1, a large-scale fine-grained classification algorithm for mobile application SNI information is characterized in that: In S1, the SNI extraction results of dynamic analysis and static analysis are obtained based on the statistical characteristics of the APP, and an invalid SNI feature library is constructed. If the SNI extraction results contain feature strings in the pre-constructed commonly used invalid SNI feature library, they are directly deleted to obtain the result after preliminary screening of invalid secondary domain names.

3. According to claim 2, a large-scale fine-grained classification algorithm for mobile application SNI information is characterized in that: In the above S5, the duplicate SNI data in the third deduplication result is traversed according to the latest SNI list file, the APPs where the duplicate SNI data are located are integrated to build an APP similarity list file, all files with T<4 in the APP similarity list file are read and similar APPs are screened out according to the APP similarity judgment condition, and the screened out similar APPs are added to the similar APP list, and the above steps are repeated until there are no similar APPs, and the APPs that do not meet the APP similarity judgment condition are added to the queue to be deleted, and the content to be deleted under the current secondary domain name is deduplicated, and all the current duplicate SNI data is deleted, and a new list file is built, and step S5 is repeated until the new list file no longer changes, and the characteristic SNI contained in the input APP is obtained; The APP similarity determination condition is expressed as: (1) The same second-level domain name in the APP similarity list file appears in two APPs at the same time, and the S values ​​are equal under the same T value; (2) The number of second-level domain name strings ranked in the top three of the richness R of the two APPs in the APP similarity list file is exactly the same; (3) The package names of the APK files of the two apps in the APP similarity list file are similar, and their full phonetic names and abbreviations are similar; Apps that meet the above three points are considered similar apps.

4. According to claim 3, a large-scale fine-grained classification algorithm for mobile application SNI information is characterized in that: In S6, a request for the SNI is constructed and tested for a response. If the SNI does not exist or the request fails, the SNI record is deleted. If the response is normal, it is retained. Considering that the content of the constructed request may require a corresponding token or the passing of specific parameters, if the request times out or an error is given in the request method, it is also considered that the conditions are met and the current SNI is retained.

Images