HTTP feature-oriented CDN identification feature set expansion method

Abstract

The application provides a CDN identification feature set expansion method for HTTP features, and the CDN identification feature set mainly matches CNAME features, and the specific steps of the method are as follows: step 1, obtaining domain name basic data, reducing the domain name basic data through a detection object reduction method; step 2, using HTTP features to make a secondary determination on the domain name basic data to be classified, expanding the CNAME feature set according to the determination result, gradually improving the pre-classification accuracy, and achieving the purpose of collecting CNAME feature keywords of new CDN manufacturers or collecting new CNAME feature keywords registered by existing CDN manufacturers. The method depends on DNS resolution and HTTP resolution, is oriented to key domain names and regional recursive DNS servers, has a small iteration time cost, and has a low requirement on machine cost, and the expansion process can be completed by a single machine.

Description

TECHNICAL FIELD

[0001] The present application relates to the CDN identification technical field, in particular to a CDN identification feature set expansion method for HTTP features. BACKGROUND

[0002] The content delivery network (CDN) is a virtual network constructed by combining a load balancing system with an edge server cluster, which can direct the requests of terminal users to the edge servers closest to them or with the best time delay performance, and has the advantages of reducing the load of the source station, resisting DDos attacks, etc.

[0003] At present, there are many concerns about security vulnerabilities, attack prevention, etc. related to CDN. For example, using CDN as a DDos attack means to paralyze the source station, using new review evasion technology of CDN to protect illegal sites, and not verifying the source server certificate, etc. In the aspects of discovering CDN attack events, vulnerability detection and repair, integrating CDN selection, CDN service ranking, etc., the identification of CDN manufacturers is indispensable.

[0004] CDN identification mainly adopts the keyword matching technology, among which, the matching of CNAME features is the main one. Therefore, it is necessary to mark the service features of newly registered CDN manufacturers or the CDN domain names newly registered by manufacturers, and to reduce the detection traffic as much as possible in the identification iteration process. SUMMARY

[0005] In view of the technical problems of collecting the CNAME feature keywords of the newly registered CDN manufacturers or the newly registered CNAME feature keywords of the CDN manufacturers, the present application proposes a CDN identification feature set expansion method for HTTP features, which relies on DNS resolution and HTTP resolution, is oriented to key domain names and regional recursive DNS servers, has a small iteration time cost, and has a low requirement for machine cost, and the expansion process can be completed by a single machine.

[0006] Therefore, the technical scheme of the present application is a CDN identification feature set expansion method for HTTP features, and the CDN identification feature set mainly matches the CNAME features. The specific steps of the method are as follows:

[0007] Step 1: Obtain domain name basic data, and reduce the domain name basic data for detection by a detection object reduction method. The specific process of the detection object reduction method is as follows:

[0008] Step 1.1 According to the CDN nearest resolution principle, taking the regional recursive DNS server as the resolution center, obtaining domain name basic data from the DNS side, using the forwarding DNS server as the request object, and using its forwarding principle to indirectly initiate a resolution request to the regional recursive DNS server;

[0009] Step 1.2 According to the service provider recommendation or other recommendation scheme, the obtained forwarding DNS servers in each region in the country are reduced, and at least one DNS server in each region is required for detection;

[0010] Step 1.3 According to the CDN load balancing function principle, taking the key domain name as the detection target;

[0011] Step 1.4 Based on HTTP related data and CNAME feature data, further reduce the detection data, use Ping as a reduction means, and reduce the key domain name;

[0012] Step 2 Using HTTP features, the secondary determination is carried out for the basic data of the domain name to be classified, the CNAME feature set is expanded according to the determination result, the pre-classification accuracy is gradually improved, and the purpose of collecting new CNAME feature keywords of CDN manufacturers or new CNAME feature keywords registered by existing CDN manufacturers is achieved;

[0013] The specific steps of step 2 are as follows:

[0014] Step 2.1 According to the results obtained in step 1, the CDN usage is pre-determined, and the domain name and domain name basic data are divided into four types: using CDN service, not using CDN service, invalid data, and data to be classified;

[0015] Step 2.2 Filter out invalid data, and train the remaining three types of data as follows:

[0016] a) Extract the data of using CDN service and not using CDN service as the training set of the model;

[0017] b) According to the training set, the feature extractor is constructed;

[0018] c) Use the feature converter to encode the training set data and the data to be classified;

[0019] d) According to the training set encoding result, a semi-supervised learning classifier is constructed;

[0020] e) Use the classifier to classify the data to be classified, and focus on the domain name basic data classified as using CDN service;

[0021] Step 2.3. For the domain name in the to-be-classified data, the CNAME feature keyword is extracted, the corresponding CDN manufacturer is determined according to the WHOIS information of the search engine or the CNAME record main domain name, and is supplemented into the CNAME keyword feature set, and a round of CNAME feature keyword expansion process is completed.

[0022] Preferably, the domain name basic data includes CNAME record, IP record, HTTP response status code and response header dictionary content.

[0023] Preferably, the specific steps of obtaining the domain name basic data are as follows:

[0024] Step S1, specifying a list of key domain names;

[0025] Step S2, using an HTTP parsing module to parse the domain name site, initiating a HEAD request to the domain name, collecting response data HTTPheader and status code Statuscode;

[0026] Step S3, using a DNS parsing module to parse the A record of the domain name, taking the domain name with HTTP parsing result in step S2 as input.

[0027] Preferably, the process of using the DNS parsing module to parse the A record of the domain name is that, for any domain name in the input domain name, all DNS servers are requested, and all canonical host name records (CNAME Record) and IP records (IP Record) appearing in the parsing process are retained.

[0028] Preferably, the WWW domain name and SLD domain name in the SecRank domain name ranking are selected as the list of key domain names.

[0029] Preferably, the specific steps of CNAME keyword expansion learning are as follows:

[0030] Step A, pre-classifying the CDN usage of the current obtained domain name basic data into four categories: invalid data, to-be-classified data, using CDN service data and not using CDN service data;

[0031] Step B, taking the data using CDN service and not using CDN service as a training set, extracting features from the HTTPHeaders data about whether to use CDN service classification;

[0032] Step C, according to the features extracted in step B, performing one-hot encoding on the training set, and using a semi-supervised machine learning method to train the model, constructing an HTTP feature classifier to automatically label the to-be-classified data obtained in step 1;

[0033] Step D, using the domain name basic data labeled in step C using CDN service as the input of CDN keyword extraction;

[0034] Preferably, the semi-supervised machine learning method adopts a TSVM method.

[0035] Preferably, the CDN keyword extraction extracts the main domain name of the input CNAME record, and then manually labels the extraction result, determines the service provider name of the current main domain name by using a search engine or using WHOIS information, and supplements the CDN keyword database to improve the CDN recognition accuracy.

[0036] The method of the present application has the advantages that the method relies on DNS resolution and HTTP resolution, is oriented to key domain names and regional recursive DNS servers, has small iteration time cost, and has low requirements for machine cost, and the expansion process can be completed by a single machine. In the present application, HTTP features are used in each iteration to make a secondary determination on the related data of the domain name to be classified, the CNAME feature set is expanded according to the determination result, the pre-classification accuracy is gradually improved, and the purpose of collecting CNAME feature keywords of new CDN manufacturers or new CNAME feature keywords registered by existing CDN manufacturers is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0037] Fig. 1 is the overall flowchart of the CNAME keyword expansion in the present application;

[0038] Fig. 2 is the detailed flowchart of the CDN recognition scheme in the present application;

[0039] Fig. 3 is the flowchart of the domain name using CDN situation pre-classification rule in the present application. DETAILED DESCRIPTION

[0040] The present application will be further described below in combination with embodiments.

[0041] Figs. 1-3 is an embodiment of the CDN recognition feature set expansion method oriented to HTTP features in the present application, and the CDN recognition feature set mainly matches CNAME features, and the specific steps of the method are as follows:

[0042] Step 1: Obtain domain name basic data, and the domain name basic data includes CNAME records, IP records, HTTP response status codes and response header dictionary contents. The domain name basic data is reduced by a probe object reduction method, and the specific process of the probe object reduction method is as follows:

[0043] Step 1.1 According to the CDN nearest resolution principle, the regional recursive DNS server is used as a resolution center, the domain name basic data is obtained from the DNS side, the forwarding DNS server is used as a request object, and the resolution request is indirectly initiated to the regional recursive DNS server by using the forwarding principle.

[0044] Since the regional recursive DNS server generally only provides resolution services to the local area, and the forwarding DNS server (hereinafter referred to as the DNS server) has a clear corresponding relationship with the recursive DNS server, the application uses the forwarding DNS server as a request object, and indirectly initiates a resolution request to the regional recursive DNS server by using the forwarding principle.

[0045] Step 1.2 According to the service provider recommendation or other recommendation scheme, the forwarding DNS servers obtained in each region of the country are reduced, and the detection requirement that each region has at least one DNS server is met.

[0046] Step 1.3 According to the CDN load balancing function principle, the key domain name is used as a detection target. Because the domain name using CDN is most likely a high-traffic access domain name, or a key domain name.

[0047] Step 1.4 Based on HTTP related data and CNAME feature data, the detection data is further reduced, and the ping reachable is used as a reduction means to reduce the key domain name. Since HTTP is based on TCP, TCPPing results can also be used, or HTTP request results can be directly used for reduction.

[0048] By using the recommended forwarding DNS server in each region to request the key domain name, the number of DNS request packets in the real network environment can be significantly reduced, and the detection efficiency of each iteration process can be improved. At the same time, the ping or TCPPing method is used to further pre-screen the key domain name and reduce the request amount of subsequent multi-region detection.

[0049] Further, the specific steps of obtaining the domain name basic data are as follows:

[0050] Step S1 Specify the key domain name list. When the domain name site is important (or the traffic is large), CDN technology is usually used to meet service needs, reduce costs, or resist DDOS attacks. Therefore, reasonable selection of the key domain name list can obtain more information of CDN service providers. In this embodiment, the SecRank domain name ranking published by Qianxin is selected as a key reference, and the WWW domain name and SLD domain name are selected as the key domain name list because they are most likely websites.

[0051] Step S2 uses the HTTP parsing module to parse the domain name site, initiates a HEAD request to the domain name, collects response data HTTP header and status code Statuscode. Among them, if multiple level information is returned, only the last response data and the first received status code are retained.

[0052] Step S3 uses the DNS parsing module to parse the domain name as the input of the HTTP parsing result in step S2. For any domain name in the input domain name, all DNS servers are requested, and all canonical host name records (CNAME Record) and IP records (IP Record) that appear in the parsing process are retained. The DNS server needs to cover as many geographical locations as possible, such as selecting a DNS server in each province and city, and each operator.

[0053] It should be noted that if a distributed system is used to directly request the parsing of the system node corresponding to the regional recursive server, the EDNS extension option needs to be closed. The principle of CDN proximal parsing: without opening EDNS, CDN will return a better IP record according to the DNS recursive server location, so as to achieve the need to obtain a more comprehensive IP record set. The purpose is the same as using the indirect request between the forwarding DNS server.

[0054] Step 2 provides a method for expanding the CNAME feature set based on HTTP. Using HTTP features, the related data of the domain name to be classified is determined again, the CNAME feature set is expanded according to the determination result, the pre-classification accuracy is gradually improved, and the purpose of collecting CNAME feature keywords of new registered CDN manufacturers or new CNAME feature keywords registered by existing CDN manufacturers is achieved.

[0055] For the site using CDN service, the HTTP response delivered by CDN to the terminal user usually contains the header information dictionary of the manufacturer keyword information, the hit situation and other special fields. Therefore, the method of semi-supervised machine learning is used to determine whether the domain name HTTP information meets the use of CDN features:

[0056] Step 2.1, according to the result obtained in step 1, pre-determines the use of CDN, divides the domain name and domain name related data into four types (domain name related data includes CNAME record, IP record, HTTP response status code and response header dictionary content): using CDN service, not using CDN service, invalid data and data to be classified.

[0057] Step 2.2 filters out invalid data, and trains the remaining three types of data models as follows:

[0058] a) Extracting data using CDN services and not using CDN services as a training set for the model

[0059] b) Feature extraction according to the training set, building a feature converter

[0060] c) Using the feature converter to encode the training set data and the data to be classified

[0061] d) Building a semi-supervised learning classifier according to the training set encoding results

[0062] e) Using the classifier to classify the data to be classified, focusing on the classification results of the domain name related data using CDN services

[0063] Step 2.3 For the domain name identified in the data to be classified, extract its CNAME feature keywords, determine the keywords corresponding to the CDN vendor according to the WHOIS information of the search engine or CNAME record main domain name, and supplement to the CNAME keyword feature set, complete a round of keyword expansion process.

[0064] The traditional CDN recognition method based on matching rules mostly uses CNAME keywords, WHOIS information, IP attribution, HTTP, etc. for recognition, among which the CNAME keyword feature is the attribute with higher recognition efficiency and smaller parsing cost. However, the current research on the collection method of CNAME keywords is less, therefore, the present application proposes a learning method of expanding the CNAME keyword feature set using HTTP features, which aims to continuously iterate the CNAME keyword feature set with low execution cost, supplement the newly appeared CNAME keyword features in the market, and improve the CDN recognition classification precision (in the present application, it refers to the pre-classification process to identify the domain name using CDN service).

[0065] Further, the specific steps of CNAME keyword expansion learning are as follows:

[0066] Step A Classify the current obtained domain name basic data according to the CDN usage, and the classification process is as shown in Fig. 3 The data can be roughly divided into four categories: (1) invalid data, (2) data to be classified, (3) data using CDN services, and (4) data not using CDN services.

[0067] The classification is relatively conservative, and its theoretical basis is that the HTTP response status code (Status code) is an error code (4XX, 5XX) or a parsing failure data, and the domain name without CNAME record is invalid data filtering (CNAME expansion is based on HTTP information and CNAME information, therefore, the data not containing the two types of information is filtered out).

[0068] Secondly, the data with successful keyword matching of CNAME record are used as the data using CDN service. The data with failed CNAME matching, with normal HTTP response status code and same IP record in multi-site resolution, are used as the data not using CDN service. The remaining data are classified as the data to be classified.

[0069] Step B: The data using CDN service and the data not using CDN service are used as the training set, and the feature extraction is performed on the HTTPHeaders data for the classification of whether to use CDN service.

[0070] The keywords are statistically sorted by using the TF-IDF method, and the features obviously helping the classification effect are extracted (the principal component analysis method such as PCA and LDA can also be used for feature extraction, which is not described here).

[0071] Step C: According to the features extracted in step B, the training set is one-hot encoded, and the model training is performed by using the semi-supervised machine learning method, to construct the HTTP feature classifier, and the data to be classified obtained in step 1 are automatically labeled. The TSVM method is used for semi-supervised learning, which has good effect on binary classification problem, and can efficiently complete the limited set labeling work.

[0072] Step D: The domain name basic data using CDN service labeled in step C are used as the input of CDN keyword extraction. The CDN keyword extraction extracts the main domain name from the input CNAME record, and the extracted results are manually labeled in the subsequent step, to determine the service provider name of the current main domain name by using the search engine or WHOIS information, and supplement to the CDN keyword database, to improve the CDN recognition accuracy.

[0073] The application provides a CDN recognition feature set expansion method, mainly relying on DNS resolution and HTTP resolution, facing key domain names and regional recursive DNS servers, with small iteration time cost, and low requirement on machine cost, and the expansion process can be completed by using a single machine. In the application, the HTTP feature is used for secondary determination on the data related to the domain name to be classified in each iteration, the CNAME feature set is expanded according to the determination result, the pre-classification accuracy is gradually improved, and the purpose of collecting the CNAME feature keywords of new CDN manufacturers or the new CNAME feature keywords registered by the existing CDN manufacturers is achieved.

[0074] The above is only a specific embodiment of the application, and cannot limit the scope of the application. Therefore, the replacement of equivalent components or equivalent changes and modifications made within the scope of the patent protection of the application should still be within the scope of the claims of the application.

Claims

1. A method for identifying a feature set of a CDN oriented to HTTP features, characterized in that, The CDN identifies the matching of the CNAME feature as the main feature set, and the specific steps of the method are as follows: Step 1: Obtain domain name basic data, reduce the domain name basic data by a detection object reduction method, and the specific process of the detection object reduction method is as follows: Step 1.1: According to the CDN nearest resolution principle, take the regional recursive DNS server as the resolution center, and obtain the domain name basic data from the DNS side, use the forwarding DNS server as the request object, and use its forwarding principle to indirectly initiate the resolution request to the regional recursive DNS server; Step 1.2: According to the service provider's recommendation scheme, reduce the forwarding DNS servers obtained in each region of the country, and achieve the detection requirement that there is at least one DNS server in each region; Step 1.3: According to the CDN load balancing function principle, take the key domain name as the detection target; Step 1.4: Based on HTTP related data and CNAME feature data, further reduce the detection data, and use Ping as the reduction means to reduce the key domain name; Step 2: Use HTTP features to make a second determination on the domain name basic data to be classified, expand the CNAME feature set according to the determination result, gradually improve the pre-classification accuracy, and achieve the purpose of collecting new CNAME feature keywords of CDN manufacturers or new CNAME feature keywords registered by existing CDN manufacturers; The specific steps of step 2 are as follows: Step 2.1: According to the results obtained in step 1, pre-determine the CDN usage, and divide the domain name and domain name basic data into four categories: using CDN service, not using CDN service, invalid data, and data to be classified; Step 2.2: Filter out invalid data, and train the remaining three types of data as follows: a) Extract the data of using CDN service and not using CDN service as the training set of the model; b) Perform feature extraction based on the training set to construct a feature converter; c) Use the feature converter to encode the training set data and the data to be classified; d) Construct a semi-supervised learning classifier according to the encoding results of the training set; e) Use the classifier to classify the data to be classified, and focus on the domain name basic data whose classification result is using CDN service; Step 2.3: For the domain name whose identification result is using CDN service in the data to be classified, extract its CNAME feature keywords, determine the corresponding CDN manufacturer according to the WHOIS information of the search engine or CNAME record main domain name, and supplement to the CNAME keyword feature set to complete a round of CNAME feature keyword expansion process.

2. The method of claim 1, wherein the set of HTTP feature-oriented CDN identifying features is augmented by, The domain name basic data includes CNAME record, IP record, HTTP response status code, and response header dictionary content.

3. The method of claim 1, wherein the HTTP feature-oriented CDN identification feature set augmentation method is characterized by, The specific steps of obtaining domain name basic data are as follows: Step S1: Specify the key domain name list; Step S2: Use the HTTP resolution module to resolve the domain name site, initiate a HEAD request to the domain name, collect the response data HTTPheader and status code Statuscode; Step S3 uses the DNS resolution module to perform A record resolution on the domain name with the HTTP resolution result in step S2.

4. The HTTP feature-oriented CDN identification feature set augmentation method of claim 3, wherein, The process of using the DNS resolution module to perform A record resolution on the domain name is that, for any domain name in the input domain name, all DNS servers are requested, and all canonical host name records (CNAME Record) and IP records (IP Record) that appear in the resolution process are retained.

5. The method of claim 3, wherein the HTTP feature-oriented CDN identification feature set augmentation method is characterized by, The WWW domain name and SLD domain name in the SecRank domain name ranking are selected as the key domain name list.

6. The HTTP feature oriented CDN identification feature set augmentation method of claim 1, wherein, The specific steps of the CNAME keyword expansion learning are as follows: Step A: The CDN usage condition of the current obtained domain name basic data is pre-classified into four categories: invalid data, data to be classified, CDN service, and CDN service data not used; Step B: The data of the CDN service and the CDN service data not used are used as the training set, the HTTPHeaders data is classified according to whether the CDN service is used, and the features are extracted; Step C: According to the features extracted in step B, the training set is one-hot encoded, a semi-supervised machine learning method is used for model training, an HTTP feature classifier is constructed, and the data to be classified obtained in step 1 is automatically labeled; Step D: The domain name basic data of the CDN service labeled in step C is used as the input of the CDN keyword extraction.

7. The HTTP feature-oriented CDN identification feature set augmentation method of claim 6, wherein, The semi-supervised machine learning method uses the TSVM method.

8. The HTTP feature oriented CDN identification feature set augmentation method of claim 6, wherein, The CDN keyword extraction extracts the main domain name from the input CNAME record, and the extracted result is manually labeled in the subsequent step. The service provider name of the current main domain name is determined by using a search engine or using WHOIS information, and is supplemented into the CDN keyword database to improve the CDN recognition accuracy.

Images