Method, device and medium for distributing traffic application identification statistics

Abstract

The present disclosure provides a method and device for distributing traffic application identification statistics and a medium, and relates to the technical field of network analysis. The method comprises: obtaining DNS logs and NETFLOW logs of different applications distributing traffic through the same service IP address in a first period; dividing the first period into multiple second periods, establishing a first association between each application in the DNS logs and each flow in the NETFLOW logs in each second period, and statistically obtaining a first proportion of each flow corresponding to each application in the DNS logs in the NETFLOW logs in the first period according to the first association; and obtaining the traffic distributed by the different applications through the same service IP address in a third period according to the first proportion. By establishing the segmented association between the DNS logs and the NETFLOW logs, the present disclosure can obtain the traffic of different applications through the same service IP address, and can obtain the traffic of different applications in the same service IP close to the real situation.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of network analysis, and particularly relates to a method for identifying and counting distributed traffic applications, a device for identifying and counting distributed traffic applications, and a computer readable storage medium. BACKGROUND

[0002] In the current IDC (Internet Data Center) room environment, there is a phenomenon that multiple applications provide services to the outside through the same service IP (Internet Protocol) address, that is, the same service IP address distributes traffic for multiple different applications.

[0003] Currently, there are two methods to estimate how much each application occupies distributed traffic: estimating the proportion by the number of requests in the DNS log, but the traffic of each application request is not the same, so the traffic proportion estimated according to the number of times cannot reflect the real situation; all traffic in a period of time is attributed to the traffic of an application, which obviously cannot reflect the real traffic proportion. SUMMARY

[0004] The technical problem to be solved by the present disclosure is to provide a method for identifying and counting distributed traffic applications, a device for identifying and counting distributed traffic applications, and a computer readable storage medium to solve the problem of how to accurately estimate how much traffic is distributed by multiple applications through the same service IP address.

[0005] In a first aspect, the present disclosure provides a method for identifying and counting distributed traffic applications, the method comprising:

[0006] The method comprises:

[0007] obtaining DNS logs and NETFLOW logs of traffic distributed by different applications through the same service IP address in a first period of time;

[0008] dividing the first period of time into multiple second periods of time, establishing a first association between each application in the DNS logs and each traffic in the NETFLOW logs in each second period of time, and counting a first proportion of each application in the DNS logs corresponding to each traffic in the NETFLOW logs in the first period of time according to the first association;

[0009] obtaining traffic distributed by the different applications through the same service IP address in a third period of time according to the first proportion;

[0010] wherein IP is Internet Protocol, DNS is Domain Name System, and NETFLOW is network traffic.

[0011] Further, the DNS log and the NETFLOW log of the different applications distributing traffic through the same service IP address in a first period are acquired, and the first period includes the following steps:

[0012] The service IP address of a CDN server carrying traffic of the plurality of applications is acquired from an IDC, and the service IP address is used as the same service IP address.

[0013] A first period in which the CDN traffic of the plurality of applications has typical distribution characteristics in a third period is acquired.

[0014] All DNS logs and NETFLOW logs in the first period pointing to the same service IP address are acquired.

[0015] The IDC is an Internet data center, and the CDN is a content distribution network.

[0016] Further, the third period is a day, the first period includes at least six fourth periods, the at least six fourth periods are discontinuous, and the fourth period is at least one hour.

[0017] The same service IP address corresponds to a plurality of domain names, one or several domain names correspond to one application, and the plurality of domain names correspond to a plurality of applications.

[0018] The first period is divided into a plurality of second periods, and the first association between each application in the DNS log and each traffic in the NETFLOW log is established in each second period.

[0019] The first proportion of each application in the DNS log corresponding to each traffic in the NETFLOW log in the first period is calculated according to the first association, and the first period includes the following steps:

[0020] The request IP, the first log time, and the application of each DNS log in the first period are acquired, and the destination IP, the second log time, and the traffic of each NETFLOW log in the first period are acquired.

[0021] The first period is divided into a plurality of second periods using a first statistical step.

[0022] In each second period, each DNS log and each NETFLOW log are traversed, and the first association between the application corresponding to the traffic of a certain NETFLOW log recorded in a certain DNS log is acquired according to the same request IP and destination IP, and the difference between the second log time and the first log time is less than the first time difference value.

[0023] The second proportion of the different applications distributing traffic through the same service IP address in each second period is acquired according to the first association.

[0024] The average of the second percentages of the multiple second time periods is obtained as the first percentage.

[0025] Furthermore, obtain the requesting IP, first log time, and application for each DNS log entry within the first time period, and obtain the destination IP, second log time, and traffic for each NEFLOW log entry within the first time period, specifically including:

[0026] Obtain the request IP, first log time, and request domain name directly from each DNS log entry, and then retrieve the application based on the request domain name;

[0027] The destination IP, NEFLOW log generation time, and traffic calculation parameters are directly obtained from each NEFLOW log entry. The NEFLOW log generation time is used as the second log time, and the traffic is obtained based on the traffic calculation parameters.

[0028] Furthermore, obtaining the first association of the application that records traffic from a certain NEFLOW log entry to a certain DNS log entry specifically includes:

[0029] For each NEFLOW log entry, a first data record in a two-layer KEY-VALUE format is created. The outer KEY of the first data record stores the destination IP of the NEFLOW log, the inner KEY stores the second log time of the NEFLOW log, and the inner VALUE stores the traffic of the NEFLOW log.

[0030] For each second time period, a second data record in a two-layer KEY-VALUE format is created. The outer KEY of the second data record stores the tag for each second time period, the inner KEY stores the tag of the application obtained from the DNS log, and the inner VALUE stores the traffic of the corresponding application's DNS log associated with the NEFLOW log.

[0031] Furthermore, the method also includes:

[0032] Set multiple second statistical step sizes and multiple second time differences, and design a double loop operation:

[0033] In the first loop, the first time period is divided into multiple fifth time periods using each of the second statistical step sizes.

[0034] In the second loop, a second time difference is selected sequentially.

[0035] Within each fifth time period, in response to the requesting IP and destination IP being the same, and the difference between the second log time and the first log time being less than the selected second time difference, the second association corresponding to the application that records the traffic of a certain NEFLOW log to a certain DNS log is obtained.

[0036] According to the second association, a third proportion of traffic respectively distributed by the different applications through the same service IP address in each fifth time period is obtained,

[0037] RMSE of the third proportions of the plurality of fifth time periods and respective average values thereof is obtained;

[0038] The second statistical step and the second time difference value corresponding to the round with the minimum RMSE in all cycles are obtained as the first statistical step and the first time difference value.

[0039] Wherein, RMSE is the root mean square error.

[0040] Further, according to the first proportion, the traffic respectively distributed by the different applications through the same service IP address in the third time period is obtained, specifically comprising:

[0041] The total traffic distributed through the same service IP address in the third time period is obtained, and the total traffic is calculated as the traffic respectively distributed by the different applications through the same service IP address according to the first proportion.

[0042] In a second aspect, the present disclosure provides a distributed traffic application identification and statistical device, the device comprising:

[0043] A log acquisition module is configured to acquire DNS logs and NETFLOW logs of traffic distributed by different applications through the same service IP address in a first time period;

[0044] A log association module is connected with the log acquisition module and is configured to divide the first time period into a plurality of second time periods, establish a first association between each application in the DNS logs and each traffic in the NETFLOW logs in each second time period, and statistically obtain a first proportion of each application in the DNS logs corresponding to each traffic in the NETFLOW logs in the first time period according to the first association;

[0045] A traffic statistical module is connected with the log association module and is configured to obtain the traffic respectively distributed by the different applications through the same service IP address in the third time period according to the first proportion;

[0046] Wherein, IP is the Internet Protocol, DNS is the Domain Name System, and NETFLOW is the network flow.

[0047] In a third aspect, the present disclosure provides a computer readable storage medium, the computer readable storage medium storing a computer program, when the computer program is run by a processor, the distributed traffic application identification and statistical method is realized.

[0048] The disclosure provides a distribution traffic application identification statistics method, a distribution traffic application identification statistics device and a computer readable storage medium. By establishing a segmented association of DNS logs and NETFLOW logs, the distribution traffic of different applications through the same service IP address is obtained, and the distribution traffic of different applications in the same service IP is close to the real distribution traffic. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 is a flowchart of a distribution traffic application identification statistics method according to an embodiment of the disclosure;

[0050] Figure 2 is a structural schematic diagram of a distribution traffic application identification statistics device according to an embodiment of the disclosure. DETAILED DESCRIPTION

[0051] To make the skilled in the art better understand the technical solutions of the disclosure, the embodiments of the disclosure will be further described in detail below with reference to the drawings.

[0052] It can be understood that the specific embodiments and drawings described herein are only used to explain the disclosure, but not to limit the disclosure.

[0053] It can be understood that the embodiments in the disclosure and the features in the embodiments can be combined with each other without conflict.

[0054] It can be understood that, for the convenience of description, only parts related to the disclosure are shown in the drawings of the disclosure, and parts unrelated to the disclosure are not shown in the drawings.

[0055] It can be understood that each module and unit involved in the embodiments of the disclosure can only correspond to one entity structure, or can be composed of multiple entity structures, or multiple modules and units can be integrated into one entity structure.

[0056] It can be understood that the functions and steps marked in the flowchart and block diagram of the disclosure can occur in an order different from that marked in the drawings without conflict.

[0057] It can be understood that in the flowchart and block diagram of the disclosure, the system, device, equipment, method according to the embodiments of the disclosure are shown as possible implementation architecture, function and operation. Each block in the flowchart or block diagram can represent a module, unit, program segment, code, which contains executable instructions for realizing the specified function. Moreover, each block or combination of blocks in the block diagram and flowchart can be realized by a hardware-based device for realizing the specified function, or by a combination of hardware and computer instructions.

[0058] It can be understood that the modules and units involved in the embodiments of the present disclosure can be implemented in the form of software or hardware, for example, the modules and units can be located in a processor.

[0059] Embodiment 1:

[0060] As shown in Figure 1 The present disclosure provides a method for identifying and counting application distribution traffic, which comprises:

[0061] S1, obtaining DNS logs and NETFLOW logs of different applications distributing traffic through the same service IP address in a first period;

[0062] S2, dividing the first period into multiple second periods, establishing a first association between each application in the DNS logs and each flow in the NETFLOW logs in each second period, and counting the first proportion of each flow corresponding to each application in the DNS logs in the first period according to the first association;

[0063] S3, obtaining the traffic distributed by the different applications through the same service IP address in a third period according to the first proportion;

[0064] Wherein, IP is Internet Protocol, DNS is Domain Name System, and NETFLOW is network traffic.

[0065] In this embodiment, by establishing the segmented association of DNS logs and NETFLOW logs, the traffic distributed by different applications through the same service IP address is obtained, and the traffic of different applications in the same service IP can be obtained close to the real traffic.

[0066] Specifically, multiple applications provide services to the outside through the same service IP address, which is achieved by scheduling through the CNAME (Canonical Name) record of DNS (Domain Name System). A complete DNS domain name resolution may have a CNAME resolution process, which will be recorded in the same log.

[0067] The reason why the disclosure needs to estimate the proportion of the same service IP output traffic application by using a statistical method is that: first, the request domain name is recorded in the record of the DNS log, and the application that sends the request can be corresponded according to the request domain name, but the amount of traffic sent for the corresponding request is not recorded; the NETFLOW (network flow, a network monitoring function) log can count the IP traffic, but can only count the IP traffic and cannot distinguish which application the traffic belongs to. If the two kinds of logs are associated, there is a problem that the records of the two kinds of logs are uncertain, that is, not every request has a DNS log record, and not every traffic has a NETFLOW log record, so it is difficult to accurately associate the two.

[0068] Therefore, the disclosure proposes a method of establishing association based on statistics, and the principle is explained as follows:

[0069] According to the law of large numbers in probability theory, when a large number of repeated random experiments are performed, the frequency of the experimental results will tend to a certain constant, that is, its theoretical probability. This law shows that, under a sufficient number of test times, the sample average of a random event will tend to its true expected value.

[0070] This means that in a large number of repeated random experiments (which can be understood as DNS requests and NETFLOW log records), the sample average will tend to the true expected value. Even if there is uncertainty in DNS requests and NETFLOW log records, the actual situation of network traffic can still be accurately estimated through a sufficient number of data samples.

[0071] Due to the existence of DNS cache, DNS requests have uncertainty, that is, not every access to the domain name will request the DNS server. According to the law of large numbers, this uncertainty will tend to be average in front of a large amount of data, so that the overall trend can be predicted.

[0072] NETFLOW is based on sampling ratio to obtain traffic data, and the existence of each log also depends on the sampling ratio, and there is a certain probability. Through the law of large numbers, even if sampling is performed, the traffic statistics close to the true situation can be obtained under a sufficient amount of data.

[0073] The disclosure associates the two kinds of logs, and realizes repeated statistics by dividing the data into multiple segments, the DNS logs and the NETFLOW logs in the first period and the second period meet the requirements of the law of large numbers for data samples, each segment of data is equivalent to a data sample, and multiple data samples are repeated to obtain the traffic proportion of different applications in the same service IP.

[0074] In an embodiment, S1, acquiring DNS logs and NETFLOW logs of different applications distributing traffic through the same service IP address in a first period, specifically includes:

[0075] obtaining a service IP address of a certain CDN server carrying multiple application traffic distribution as the same service IP address through an IDC;

[0076] obtaining a first time period in which the multiple applications use CDN traffic to have typical distribution characteristics in a third time period;

[0077] obtaining all DNS logs and NETFLOW logs in the first time period pointing to the same service IP address;

[0078] Wherein, the IDC is an Internet Data Center, and the CDN is a Content Delivery Network.

[0079] In the embodiment, a statistical method for CDN (Content Delivery Network) distribution traffic application identification is specifically provided, and the following two DNS log segments show that multiple applications provide services through the same service IP address:

[0080] Log 1:

[0081] 202.102.227.68|53|42.234.29.128|9780|19942|aliqcmmov.a.etoot e.com|A|(CNAME)_aliqcmmov.a.etoote.com.w.alikunlun.com;

[0082] (A)_222.141.5.211|20240821194244.324|0|r

[0083] Log 2:

[0084] 202.102.227.68|53|182.118.238.184|27808|9147|pull-tsl-l3.douyincdn.com|A|(CNAME)_pull-tsl-l3.douyincdn.com.w.alikunlun.net;(A)_222.141.5.211|20240821194244.271|0|r

[0085] In the two logs, the domain name of the request and the IP address resolved are shown respectively. Specifically, the domain name in the first log is aliqcmmov.a.etoote.com, and the resolved IP is 222.141.5.211; while the domain name in the second log is pull-tsl-l3.douyincdn.com, and the resolved IP is also 222.141.5.211. Although the domain names are different, the resolved IP addresses are the same. It can be seen through the CNAME record of DNS that different domain names are distinguished by specific keywords to point to the servers of respective applications, which are generally implemented through content distribution servers, and can be understood as gateways, which are scheduled through CNAME.

[0086] This phenomenon raises a question: in an IDC, how to determine the proportion of each different application in the traffic output by the IP address 222.141.5.211 (how to distinguish the traffic output by the IP belongs to which application). At present, the industry generally uses NETFLOW (a network monitoring function) technology to count IP traffic (only IP traffic can be counted, but it cannot be distinguished which application the traffic belongs to). In order to distinguish the traffic of different applications, the resolved domain name recorded in the DNS log is usually referred to. There are two specific methods: one is to allocate the traffic proportion according to the request times of the domain name; the other is to directly identify the IP traffic as the traffic corresponding to the domain name with the most resolution times.

[0087] In the statistics of the traffic generated by different applications corresponding to the same IP, the main disadvantage of the above method is that the resolution times of DNS cannot completely and accurately reflect the situation of the traffic output to the outside, such as one application is a video stream and the other application is an HTTP web page. Then in one traffic output to the outside, it cannot be guaranteed that the video stream output traffic is greater than, less than or equal to the HTTP web page output traffic. Therefore, relying solely on DNS resolution times cannot completely and accurately reflect the proportion of the application in the traffic output by the same service IP.

[0088] The embodiment realizes a method for associating DNS and NETFLOW, which comprises: collecting and associating DNS logs and NETFLOW logs within a certain time range (not less than 6 hours, that is, a first period, the first period has a typical distribution characteristic of application traffic usage, which can be determined according to experience), according to the association of the two kinds of logs, the proportion of application traffic is counted, and the distribution of application traffic throughout the day is analyzed according to the proportion. The first period of the embodiment is within the third period. It should be noted that the present disclosure is also applicable to the case where the third period is equal to or less than the first period or outside the first period.

[0089] In an embodiment, wherein:

[0090] The third period is a certain day, the first period includes at least six fourth periods, the at least six fourth periods are discontinuous with each other, and the fourth period is at least one hour;

[0091] The same service IP address corresponds to multiple domain names, one or several domain names correspond to one application, and the multiple domain names correspond to multiple applications.

[0092] In the embodiment, to implement the method, first, the first step 1) of data preparation is performed, specifically as follows:

[0093] 1.1) Obtain the DNS log and NETFLOW log to be learned: for example, obtain the DNS log and NETFLOW log of 0-1 o'clock, 3-4 o'clock, 8-9 o'clock, 12-1 o'clock, 16-17 o'clock, and 20-21 o'clock of a certain day, respectively. The above acquisition time requirement is that the first period includes at least six fourth periods, and the at least six fourth periods are discontinuous with each other.

[0094] 1.2) Determine the service IP to be analyzed: through the IP address segment of the IDC and the DNS log, obtain the IP in the IDC as the resolution IP of the DNS, and the domain name exists in the case of multiple applications. Take 222.141.5.211 as an example, query 148 secondary domain names of the service through the DNS log (can be obtained through the DNS log, and multiple domain names may belong to one application). Some examples are as follows: douyincdn.com, amap.com, etoote.com, sina.com.cn, douyinvod.com, vivo.com.cn, snssdk.com, amemv.com, douyinliving.com, and sinaimg.cn.

[0095] 1.3) Domain name to application: because one application will hang multiple domain names, before identifying the application traffic, it is necessary to convert the domain name to the application. This process is generally obtained through the accumulated knowledge base or through the domain name record information. Some examples are as follows: douyincdn.com|Douyin, amap.com|Gaode Map, etoote.com|Kuaishou, sina.com.cn|Sina, douyinvod.com|Douyin, vivo.com.cn|VIVO, snssdk.com|Today's Headlines, amemv.com|Douyin, douyinliving.com|Douyin, and sinaimg.cn|Sina.

[0096] In an embodiment, S2 divides the first time period into a plurality of second time periods, establishes a first association between each application in the DNS log and each flow in the NETFLOW log in each second time period, and statistically calculates a first proportion of each application in the DNS log to each flow in the NETFLOW log in the first time period according to the first association, specifically comprising:

[0097] obtaining the request IP, the first log time and the application of each DNS log in the first time period, and obtaining the destination IP, the second log time and the flow of each NETFLOW log in the first time period;

[0098] dividing the first time period into a plurality of second time periods using a first statistical step;

[0099] In each second time period, each DNS log and each NETFLOW log is traversed, and a first association of an application corresponding to a flow of a certain NETFLOW log to a certain DNS log is obtained according to the same request IP and destination IP and a second log time difference less than a first time difference value from a first log time;

[0100] According to the first association, a second proportion of the different applications respectively distributing flows through the same service IP address in each second time period is obtained;

[0101] The average value of the second proportions of the plurality of second time periods is obtained as the first proportion.

[0102] In this embodiment, one of the keys to implementing the method is step 2): the association algorithm of the DNS log and the NETFLOW log, which includes:

[0103] 2.1) Obtain the necessary information for associating the two kinds of logs, which mainly includes the request IP, the first log time and the application of the DNS log, and the destination IP, the second log time and the flow of the NETFLOW log.

[0104] 2.2) Associate the two kinds of logs with a reasonable time difference value, and output the proportion of the application associated flow with a reasonable statistical time length (statistical step). The statistical time length corresponds to the length of the second time period, which requires that there is flow of each application in each second time period, and the flow distribution of each application in the plurality of second time periods is minimized. The specific value can be obtained according to experience or algorithm optimization.

[0105] In an embodiment, the request IP, the first log time and the application of each DNS log in the first time period are obtained, and the destination IP, the second log time and the flow of each NETFLOW log in the first time period are obtained, specifically comprising:

[0106] The request IP, the first log time and the request domain name are directly obtained from each DNS log, and the application is obtained according to the request domain name.

[0107] Directly obtain the destination IP, NETFLOW log generation time and traffic calculation parameters from each NETFLOW log, take the NETFLOW log generation time as the second log time, and obtain the traffic according to the traffic calculation parameters.

[0108] In this embodiment, the necessary information can be obtained based on the formats of the DNS and NETFLOW logs, and the formats of the two logs are as follows:

[0109] DNS response log format: DNS server IP | server port | request IP | request port | random port | request domain name | request record | resolution result | log time | resolution type.

[0110] NETFLOW log format: log generation time | log collection time | duration | protocol type | source address + port | destination address + port | TCP flag | TOS service type | packet number | byte number | packet number per second | bit number per second | byte number per packet | flow number | next hop address | router IP | source autonomous domain | destination autonomous domain | flow-in interface | flow-out interface | source mask | destination mask | direction.

[0111] In an embodiment, a first association corresponding to an application of a traffic record of a certain NETFLOW log to a certain DNS log is obtained, specifically including:

[0112] For each NETFLOW log, a first data record in a double-layer KEY-VALUE format is established, the outer KEY of the first data record stores the destination IP of the NETFLOW log, the inner KEY stores the second log time of the NETFLOW log, and the inner VALUE stores the traffic of the NETFLOW log;

[0113] For each second time period, a second data record in a double-layer KEY-VALUE format is established, the outer KEY of the second data record stores the label of each second time period, the inner KEY stores the label of the application obtained according to the DNS log, and the inner VALUE stores the traffic of the DNS log associated with the NETFLOW log corresponding to the application.

[0114] In this embodiment, 2.2) associates the two logs with a reasonable time difference, and outputs the proportion of application-associated traffic with a reasonable statistical time length (statistical step), including:

[0115] 2.2.1) process the NETFLOW log into the following data storage type:

[0116] Map<String,Map<Long,Long>>

[0117] Where the KEY of the outer layer MAP is the destination IP of the 222.141.5.211 outflow, and the VALUE is the timestamp and traffic (Bytes). <Stirng, <>> This is the outer layer, and the key of the String is the traffic of 222.141.5.211 outflow to a certain IP at a certain time.

[0118] Assume using:

[0119] EXFLOWKEY represents the KEY of the outer layer MAP of the storage type.

[0120] EXFLOWVAL represents the VALUE of the outer layer MAP of the storage type.

[0121] INFLOWKEY represents the KEY of EXFLOWVAL.

[0122] INFLOWVAL represents the VALUE value of EXFLOWVAL.

[0123] For example: <”123.5.178.69”, <1724240>400, 1454>>

[0124] 2.2.2) Process the DNS log into the following data storage type:

[0125] Map<Integer, Map<String, Long>>

[0126] Where the KEY of the outer layer MAP is the number of minutes from the DNS log time to 0 o'clock divided by the statistical time length. For example: 2024-08-12 01:22:30, which is 82 minutes away from 2024-08-12 00:00:00, and assuming the statistical time length is 5 minutes, then the KEY value is 16. The KEY value of the inner layer VALUE is the application name, and the VALUE is the traffic.

[0127] Assume using:

[0128] EXDNSKEY represents the KEY of the outer layer MAP of the storage type.

[0129] EXDNSVAL represents the VALUE of the outer layer MAP of the storage type.

[0130] INDNSKEY represents the KEY of EXDNSVAL.

[0131] INDNSVAL represents the VALUE value of EXDNSVAL.

[0132] For example: <”16”, <”douyin”, 1454>>

[0133] 2.2.3) The method for obtaining each value in the DNS log processing data is as follows:

[0134] 2.2.3.1) All DNS logs are traversed to obtain the request IP, log time (first log time), and request domain name.

[0135] 2.2.3.2) T1 (first log time): If the DNS request IP and the EXFLOWKEY value in the FLOW (a piece of NETFLOW log) are consistent, the log time of the request IP is T1, that is, EXDNSKEY = T1.

[0136] 2.2.3.3) T2 (second log time): T1 of EXDNSKEY is not empty, and the time of the outer INFLOWKEY in the FLOW is traversed.

[0137] 2.2.3.4) If T2-T1 is less than the time difference value, the association is successful. The KEY of the outer VLAUE is the application converted from the request domain name. If the application already exists as INDNSKEY, then INDNSVAL = INDNSVAL + INFLOWVALVALUE, otherwise, INDNSVAL = INFLOWVALVALUE.

[0138] In an embodiment, the method further comprises:

[0139] a plurality of second statistical step lengths and a plurality of second time difference values are set, and a double-loop operation is designed:

[0140] In the first loop, each second statistical step length is used in turn to divide the first time period into a plurality of fifth time periods,

[0141] In the second loop, a second time difference value is selected in turn,

[0142] In each fifth time period, in response to the request IP being consistent with the destination IP and the difference between the second log time and the first log time being less than the selected second time difference value, a second association corresponding to the application of recording the traffic of a piece of NETFLOW log to a piece of DNS log is obtained,

[0143] According to the second association, a third proportion that each different application respectively distributes traffic through the same service IP address in each fifth time period is obtained,

[0144] The RMSE of the third proportion of the plurality of fifth time periods and the average value thereof is obtained;

[0145] The second statistical step length and the second time difference value corresponding to the round with the minimum RMSE in all loops are obtained as the first statistical step length and the first time difference value;

[0146] wherein RMSE is the root mean square error.

[0147] In the embodiment, two logs are associated, and the optimal application traffic proportion is obtained through learning of two parameters of time difference value and statistical time length, so as to truly reflect the traffic proportion of different applications in the same service IP. The main steps are as follows:

[0148] I. Collect and associate DNS logs and NETFLOW logs in a certain time range (not less than 6 hours).

[0149] II. Formulate a reasonable time difference value. The time difference value refers to that the DNS request time is earlier than the NETFLOW log device generation time in the request process of the same application. The two logs are associated with time difference values of 1s, 2s, 3s, 4s and 5s.

[0150] III. Formulate a reasonable statistical time length. After the association of the DNS log and the FLOW log, the proportion of each application can be output according to the time statistical time length. The statistical time length is 1 minute, 2 minutes and up to 10 minutes. The statistical time length is commonly referred to as the time period of statistics. For example, whether to statistics 1-minute logs or 2-minute logs, the statistical time length that is too short may result in no one accessing a certain application during the period, and the statistical time length that is too long may result in the lengthening of the log time of the learning sample, thereby increasing the learning cost. The purpose of the statistical time length is to obtain a suitable time length, and the application traffic proportion behind the IP in each time length is basically the same.

[0151] IV. Obtain the optimal application distribution proportion through RMSE (root mean square error), and then analyze the application traffic of the service IP throughout the day according to the optimal statistical time length and time difference value. The application traffic carried by CDN or cloud (IP) can be accurately calculated, for example, the traffic of Taobao, Tmall, Tiktok and Kuaishou behind Wocloud can be accurately identified.

[0152] That is, the method also includes step 3) optimal solution learning, the process of which is as follows:

[0153] 3.1) The learning target is two key parameters, which are statistical step length and time difference value. The statistical step length range is set to [5min, 10min, 15min, 20min, 30min], and the time difference value range is set to [1s, 2s, 3s, 4s, 5s].

[0154] 3.2) The statistical step length and time difference value are brought into the association algorithm in step 2) as double loops. Assuming that the application of the IP service has 5, then in the step length of 5 minutes, the 6-hour log will produce 6*60 / 5=72 5-dimensional vectors. The vector value is the percentage of the traffic in a certain statistical step length. As follows:

[0155] 01 -- [59, 21, 12, 5, 3]

[0156] 02 -- [52, 23, 14, 6, 5]

[0157] …

[0158] 72 -- [54, 20, 12, 10, 4]

[0159] 3.3) Using RMSE (root mean square error) to obtain the optimal statistical step size, time difference, RMSE refers to the difference between a group of vector values and the average value of the group of vector values, then square the difference, then divide by the total number of vectors, then take the square root, which round of RMSE is the smallest, it is considered to be the optimal solution. That is, take the average of each round of vector group to obtain the average vector AVG[A1, A2, A3, A4, A5], then obtain the RMSE value of the vector group and the AVG vector, and the round with the smallest RMSE value is the optimal statistical step size and time difference.

[0160]

[0161] Where x is the 72 5-dimensional vectors as described above, y is the average vector of x, m = 72, y = AVG[A1, A2, A3, A4, A5] = [(59+52+…+54) / 72, (21+23+…+20) / 72, (12+14+…+12) / 72, (5+6+…+10) / 72, (3+5+…+4) / 72].

[0162] In an embodiment, S3, according to the first proportion, obtains the traffic respectively distributed by the different applications through the same service IP address in the third period of time, specifically comprising:

[0163] Obtain the total traffic distributed through the same service IP address in the third period of time, and calculate the total traffic as the traffic respectively distributed by the different applications through the same service IP address according to the first proportion.

[0164] In this embodiment, the method is implemented in step 4) : the case of obtaining the whole day IP output application traffic can be obtained, including:

[0165] 4.1) After calculating to obtain the optimal statistical step size and time difference, obtain the AVG[A1, A2, A3, A4, A5] distribution of the optimal step size and time difference, that is, the optimal proportion;

[0166] 4.2) Calculate the total traffic output by the IP all day, and combine the optimal proportion of the application, so as to obtain the traffic of each application.

[0167] The above results have certain estimation, but when the algorithm output reaches the optimum, the data correlation estimation in the two logs is also the most accurate. A specific implementation effect demonstration is as follows:

[0168] As shown in Table 1, taking IP as an example 222.141.5.211, the application traffic proportion calculated by using the algorithm and the application traffic proportion calculated by the CDN manufacturer are basically consistent, which can be used as a feasible method for distribution traffic calculation. The CDN manufacturer can count the distribution traffic data, and the method of the embodiment can help the operator side to master this data, so that the operator obtains accurate data support for traffic settlement.

[0169] Table 1 Comparison of traffic proportion calculated by the embodiment and actual traffic proportion

[0170] Algorithm evaluation of traffic proportion Traffic proportion monitored by CDN TikTok 46.51％ 51.09％ Gaode map 20.65％ 19.65％ Kuaishou 19.18％ 19.18％ Sina 5.30％ 3.30％ VIVO 1.54％ 1.24％ Today's headlines 1.45％ 1.15％ Alibaba 0.85％ 0.66％ Volcano engine 0.82％ 0.92％ NetEase 0.45％ 0.65％ Unicom 0.43％ 0.40％ Others 2.82％ 1.76％ Flow rate 667 Mb / s 653 Mb / s

[0171] The embodiment realizes an algorithm for identifying application traffic accurately when a service IP carries multiple applications based on DNS\NETFLOW, and proposes a correlation algorithm for distribution traffic identification, and a method for obtaining optimal statistical step and time difference value, which can effectively solve the differentiation and identification of different application traffic when multiple applications provide services through the same service IP.

[0172] Embodiment 2:

[0173] As Figure 2 shown, the disclosure provides a distribution traffic application identification and statistical device, which comprises:

[0174] A log acquisition module 1 is configured to acquire DNS logs and NETFLOW logs of distribution traffic of different applications through the same service IP address in a first period;

[0175] A log correlation module 2 is connected with the log acquisition module 1 and is configured to divide the first period into multiple second periods, establish a first correlation between each application in the DNS logs and each flow in the NETFLOW logs in each second period, and statistically calculate a first proportion of each application in the DNS logs corresponding to each flow in the NETFLOW logs in the first period according to the first correlation;

[0176] A traffic statistical module 3 is connected with the log correlation module 2 and is configured to acquire distribution traffic of the different applications through the same service IP address in a third period according to the first proportion.

[0177] Wherein, IP is Internet Protocol, DNS is Domain Name System, and NETFLOW is network traffic.

[0178] In an embodiment, the log acquisition module 1 specifically comprises:

[0179] The service address obtaining unit is configured to obtain a service IP address of a CDN server carrying traffic of multiple applications through an IDC, as the same service IP address.

[0180] The characteristic time period obtaining unit is configured to obtain a first time period in which the multiple applications have a typical distribution characteristic of CDN traffic in a third time period.

[0181] The log obtaining unit is connected with the service address obtaining unit and the characteristic time period obtaining unit, and is configured to obtain all DNS logs and NETFLOW logs directed to the same service IP address in the first time period.

[0182] The IDC is an Internet Data Center, and the CDN is a Content Delivery Network.

[0183] In an embodiment, wherein:

[0184] The third time period is a day, the first time period includes at least six fourth time periods, the at least six fourth time periods are discontinuous with each other, and each fourth time period is at least one hour.

[0185] The same service IP address corresponds to multiple domain names, one or several domain names correspond to one application, and the multiple domain names correspond to multiple applications.

[0186] In an embodiment, the log association module 2 specifically includes:

[0187] The content obtaining unit is configured to obtain a request IP, a first log time and an application of each DNS log in the first time period, and obtain a destination IP, a second log time and traffic of each NETFLOW log in the first time period.

[0188] The time period dividing unit is configured to divide the first time period into multiple second time periods using a first statistical step.

[0189] The traversal association unit is connected with the content obtaining unit and the time period dividing unit, and is configured to, in each second time period, traverse each DNS log and each NETFLOW log, and obtain a first association of an application corresponding to a certain NETFLOW log recording traffic to a certain DNS log according to the request IP being consistent with the destination IP and the second log time being less than the first log time by a first time difference value.

[0190] The proportion calculation unit is connected with the traversal association unit, and is configured to obtain a second proportion of the different applications respectively distributing traffic through the same service IP address in each second time period according to the first association.

[0191] The average calculation unit is connected with the proportion calculation unit, and is configured to obtain an average value of the second proportions of the multiple second time periods, as a first proportion.

[0192] In an embodiment, the content acquisition unit specifically comprises:

[0193] The DNS content acquisition unit is configured to directly acquire the request IP, the first log time and the request domain name from each DNS log, and acquire the application according to the request domain name.

[0194] The NETFLOW content acquisition unit is configured to directly acquire the destination IP, the NETFLOW log generation time and the flow calculation parameter from each NETFLOW log, take the NETFLOW log generation time as the second log time, and acquire the flow according to the flow calculation parameter.

[0195] In an embodiment, the traversal association unit specifically comprises:

[0196] The first recording unit is configured to establish a first data record in a double-layer KEY-VALUE format for each NETFLOW log, the outer layer KEY of the first data record stores the destination IP of the NETFLOW log, the inner layer KEY stores the second log time of the NETFLOW log, and the inner layer VALUE stores the flow of the NETFLOW log.

[0197] The second recording unit is connected with the first recording unit and is configured to establish a second data record in a double-layer KEY-VALUE format for each second time period, the outer layer KEY of the second data record stores the label of each second time period, the inner layer KEY stores the label of the application acquired according to the DNS log, and the inner layer VALUE stores the flow of the DNS log associated with the NETFLOW log of the corresponding application.

[0198] In an embodiment, the device further comprises an optimization learning module, which specifically comprises:

[0199] The loop operation unit is configured to set a plurality of second statistical step lengths and a plurality of second time difference values, and design double-loop operation.

[0200] In the first loop, the period division unit is called to divide the first time period into a plurality of fifth time periods by using each second statistical step length in turn,

[0201] In the second loop, the traversal association unit is called to select one second time difference value in turn,

[0202] In each fifth time period, in response to the request IP being consistent with the destination IP and the difference between the second log time and the first log time being less than the selected second time difference value, the second association of the application corresponding to the flow record of a certain NETFLOW log to a certain DNS log is acquired.

[0203] The proportion calculation unit is called to obtain, according to the second association, a third proportion of the traffic respectively distributed by the different applications through the same service IP address in each fifth time period,

[0204] The average calculation unit is called to obtain the RMSE of the third proportions of the plurality of fifth time periods and the average values thereof;

[0205] The optimal output unit is connected with the loop operation unit and is configured to obtain the second statistical step length and the second time difference value corresponding to the round corresponding to the minimum RMSE in all loops as the first statistical step length and the first time difference value.

[0206] The RMSE is a root mean square error.

[0207] In an embodiment, the traffic statistics module 3 specifically includes:

[0208] The total traffic unit is configured to obtain the total traffic distributed through the same service IP address in the third time period.

[0209] The distributed traffic unit is connected with the total traffic unit and is configured to calculate the total traffic as the traffic respectively distributed by the different applications through the same service IP address according to the first proportion.

[0210] Embodiment 3

[0211] Embodiment 3 of the present disclosure provides a computer readable storage medium, wherein a computer program is stored in the computer readable storage medium, and when the computer program is run by a processor, a distributed traffic application identification statistics method as described in embodiment 1 is implemented, or a distributed traffic application identification statistics device as described in embodiment 2 is implemented.

[0212] The computer readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information, such as computer readable instructions, data structures, computer program units or other data. The computer readable storage medium includes but is not limited to RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable read only memory), flash memory or other memory technology, CD-ROM (Compact Disc Read-Only Memory), digital versatile disc (DVD) or other optical disc storage, magnetic cassette, magnetic tape, magnetic disc storage or other magnetic storage device, or any other medium that can be used to store desired information and can be accessed by a computer.

[0213] In addition, the present disclosure can also provide a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the distribution traffic application identification statistics method as described in Embodiment 1. The computer device can be the distribution traffic application identification statistics device as described in Embodiment 2.

[0214] The memory is connected with the processor, and the memory can be a flash memory or a read-only memory or other memories, and the processor can be a central processing unit or a single chip microcomputer.

[0215] Embodiments 1-3 of the present disclosure provide a distribution traffic application identification statistics method, a distribution traffic application identification statistics device and a computer readable storage medium, by establishing the segment association of DNS logs and NETFLOW logs, the traffic of different applications distributed through the same service IP address is obtained by statistics, and the traffic of different applications in the same service IP close to the real can be obtained.

[0216] It can be understood that the above embodiments are only exemplary embodiments adopted for illustrating the principles of the present disclosure, and the present disclosure is not limited thereto. Various modifications and improvements can be made by those skilled in the art without departing from the spirit and essence of the present disclosure, and these modifications and improvements are also considered as the protection scope of the present disclosure.

Claims

1. A method for distributing traffic application recognition statistics, the method comprising: The method comprises: obtaining DNS logs and NETFLOW logs of different applications distributing traffic through the same service IP address in a first period, specifically comprising: obtaining the service IP address of a CDN server carrying the traffic distribution of multiple applications through an IDC, as the same service IP address, obtaining a first period in which the CDN traffic used by the multiple applications has typical distribution characteristics in a third period, obtaining all DNS logs and NETFLOW logs pointing to the same service IP address in the first period; dividing the first period into multiple second periods, establishing a first association between each application in the DNS logs and each traffic in the NETFLOW logs in each second period, and statistically analyzing the first proportion of each application in the DNS logs corresponding to each traffic in the NETFLOW logs in the first period according to the first association, specifically comprising: obtaining the request IP, first log time and application of each DNS log in the first period, and obtaining the destination IP, second log time and traffic of each NETFLOW log in the first period, dividing the first period into multiple second periods using a first statistical step, in each second period, traversing each DNS log and each NETFLOW log, and obtaining the first association of the application corresponding to the traffic record of a certain NETFLOW log to a certain DNS log according to the same request IP and destination IP, and the difference between the second log time and the first log time is less than the first time difference value, obtaining the second proportion of the different applications respectively distributing traffic through the same service IP address in each second period according to the first association, obtaining the average value of the second proportion of the multiple second periods as the first proportion; obtaining the traffic distributed by the different applications through the same service IP address in the third period according to the first proportion; wherein IP is the Internet Protocol, DNS is the Domain Name System, NETFLOW is network traffic, IDC is the Internet Data Center, and CDN is the Content Delivery Network.

2. The method of claim 1, wherein, Wherein: The third period is a day, the first period includes at least six fourth periods, and the at least six fourth periods are discontinuous, and the fourth period is at least one hour; The same service IP address corresponds to multiple domain names, one or several domain names correspond to one application, and the multiple domain names correspond to multiple applications.

3. The method according to any of claims 1-2, characterized in that, Obtaining the request IP, first log time and application of each DNS log in the first period, and obtaining the destination IP, second log time and traffic of each NETFLOW log in the first period, specifically comprising: directly obtaining the request IP, first log time and request domain name from each DNS log, and obtaining the application according to the request domain name; directly obtaining the destination IP, NETFLOW log generation time and traffic calculation parameter from each NETFLOW log, taking the NETFLOW log generation time as the second log time, and obtaining the traffic according to the traffic calculation parameter.

4. The method according to any one of claims 1-2, characterized in that, obtaining the first association of the application corresponding to the traffic record of a certain NETFLOW log to a certain DNS log, specifically comprising: A first data record in a double-layer KEY-VALUE format is established for each NETFLOW log, the outer-layer KEY of the first data record stores the destination IP of the NETFLOW log, the inner-layer KEY stores the second log time of the NETFLOW log, and the inner-layer VALUE stores the traffic of the NETFLOW log; A second data record in a double-layer KEY-VALUE format is established for each second time period, the outer-layer KEY of the second data record stores the label of each second time period, the inner-layer KEY stores the label of the application obtained according to the DNS log, and the inner-layer VALUE stores the traffic of the DNS log associated with the NETFLOW log corresponding to the application.

5. The method according to any of claims 1-2, characterized by, The method further comprises: a plurality of second statistical step lengths and a plurality of second time difference values are set, and a double-loop operation is designed: in the first loop, each second statistical step length is used in turn to divide the first time period into a plurality of fifth time periods, in the second loop, a second time difference value is selected in turn, in each fifth time period, in response to the request IP being consistent with the destination IP and the difference between the second log time and the first log time being less than the selected second time difference value, a second association of the application corresponding to the traffic record of a certain NETFLOW log to a certain DNS log is obtained, a third proportion of the different applications respectively distributing traffic through the same service IP address in each fifth time period is obtained according to the second association, the RMSE of the third proportions of the plurality of fifth time periods and their respective average values is obtained; the second statistical step length and the second time difference value corresponding to the round with the minimum RMSE in all loops are obtained as the first statistical step length and the first time difference value; wherein, the RMSE is the root mean square error.

6. The method according to any one of claims 1-2, characterized in that, According to the first proportion, the traffic distributed by the different applications through the same service IP address in the third time period is obtained, specifically including: the total traffic distributed through the same service IP address in the third time period is obtained, and the total traffic is calculated as the traffic respectively distributed by the different applications through the same service IP address according to the first proportion.

7. A distribution traffic application identification and statistics device, characterized in that, The device comprises: a log obtaining module for obtaining DNS logs and NETFLOW logs of different applications distributing traffic through the same service IP address in a first time period, specifically including: obtaining the service IP address of a certain CDN server carrying the traffic distribution of a plurality of applications through an IDC as the same service IP address, obtaining a first time period in which the CDN traffic used by the plurality of applications has typical distribution characteristics in a third time period, obtaining all DNS logs and NETFLOW logs pointing to the same service IP address in the first time period; a log association module connected with the log obtaining module, for dividing the first time period into a plurality of second time periods, establishing a first association of each application in the DNS log and each traffic in the NETFLOW log in each second time period, and statistically obtaining a first proportion of each application in the DNS log corresponding to each traffic in the NETFLOW log in the first time period according to the first association, specifically including: obtaining a request IP, a first log time and an application of each DNS log in a first period, obtaining a destination IP, a second log time and a flow of each NETFLOW log in the first period, dividing the first period into a plurality of second periods using a first statistical step, in each second period, traversing each DNS log and each NETFLOW log, obtaining a first association of an application corresponding to a record of a flow of a certain NETFLOW log to a certain DNS log according to that the request IP is consistent with the destination IP and that the difference between the second log time and the first log time is less than a first time difference value, obtaining a second proportion of each second period in which the different applications respectively distribute the flow through the same service IP address according to the first association, obtaining an average value of the second proportions of the plurality of second periods as a first proportion; a flow statistics module, connected with the log association module, for obtaining the flow distributed by the different applications through the same service IP address in a third period according to the first proportion; wherein IP is Internet Protocol, DNS is Domain Name System, NETFLOW is network flow, IDC is Internet Data Center, and CDN is Content Delivery Network.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and when the computer program is run by the processor, the distribution flow application identification and statistical method of any one of claims 1-6 is realized.

Images