Distribution flow application identification statistical method and device and medium

Abstract

The invention provides a flow distribution application identification statistical method and device and a medium, and relates to the technical field of network analysis, and the method comprises the steps: obtaining a DNS log and an NETFLOW log of flow distribution of different applications through the same service IP address in a first time period; dividing the first time period into a plurality of second time periods, establishing a first association between each application in the DNS log and each flow in the NETFLOW log in each second time period, and counting a first proportion of each flow in the NETFLOW log corresponding to each application in the DNS log in the first time period according to the first association; and obtaining the traffic distributed by the different applications through the same service IP address in a third time period according to the first proportion. According to the method and the device, the segmented association of the DNS log and the NETFLOW log is established, the flow distributed by different applications through the same service IP address is obtained through statistics, and the flow, close to the reality, of different applications in the same service IP can be obtained.

Description

Technical Field

[0001] The present disclosure relates at least to the field of network analysis technology, and in particular to a distribution traffic application identification and statistics method, a distribution traffic application identification and statistics device, and a computer-readable storage medium. Background Art

[0002] In the current IDC (Internet Data Center) computer room environment, there is a phenomenon: multiple applications provide services to the outside world through the same service IP (Internet Protocol) address, that is, the same service IP address distributes traffic to multiple different applications.

[0003] Currently, there are two ways to estimate the distribution traffic occupied by each application in the above situation: estimating the proportion based on the number of requests in the DNS log. However, the amount of traffic each application requests varies, so the traffic proportion estimated based on the number of requests cannot reflect the actual situation; attributing all traffic within a period of time to the traffic of a certain application obviously cannot reflect the actual traffic proportion. Summary of the Invention

[0004] The technical problem to be solved by the present disclosure is to address the above-mentioned deficiencies and provide a distribution traffic application identification and statistics method, a distribution traffic application identification and statistics device, and a computer-readable storage medium to solve the problem of how to accurately estimate the amount of traffic distributed by multiple applications through the same service IP address.

[0005] In the first aspect, the present disclosure provides a method for identifying and counting distribution traffic applications.

[0006] The methods include:

[0007] Obtain DNS logs and NETFLOW logs for traffic distribution by different applications through the same service IP address during the first period;

[0008] Divide the first time period into multiple second time periods, establish a first association between each application in the DNS log and each flow in the NETFLOW log in each second time period, and calculate a first proportion of each application in the DNS log corresponding to each flow in the NETFLOW log in the first time period based on the first association;

[0009] Obtaining, according to the first proportion, the traffic distributed by the different applications through the same service IP address in a third time period;

[0010] Among them, IP is the Internet Protocol, DNS is the Domain Name System, and NETFLOW is network traffic.

[0011] Furthermore, obtain the DNS logs and NETFLOW logs of traffic distributed by different applications through the same service IP address during the first period, specifically including:

[0012] Obtaining the service IP address of a CDN server that carries multiple application traffic distributions through IDC as the same service IP address;

[0013] Obtaining a first time period in which the CDN traffic used by the multiple applications has a typical distribution characteristic within a third time period;

[0014] Obtain all DNS logs and NETFLOW logs pointing to the same service IP address within a first period;

[0015] Among them, IDC is Internet Data Center and CDN is Content Distribution Network.

[0016] Further, wherein:

[0017] The third time period is a certain day, the first time period includes at least 6 fourth time periods, the at least 6 fourth time periods are not consecutive to each other, and the fourth time period is at least one hour;

[0018] The same service IP address corresponds to multiple domain names, one or several domain names correspond to one application, and the multiple domain names correspond to multiple applications.

[0019] Furthermore, the first time period is divided into multiple second time periods, and a first association is established between each application in the DNS log and each flow in the NETFLOW log in each second time period. Based on the first association, a first proportion of each application in the DNS log corresponding to each flow in the NETFLOW log in the first time period is calculated, specifically including:

[0020] Get the request IP, first log time, and application of each DNS log in the first period, and get the destination IP, second log time, and traffic of each NETFLOW log in the first period;

[0021] Divide the first time period into a plurality of second time periods using a first statistical step length;

[0022] In each second time period, traverse each DNS log and each NETFLOW log, and obtain a first association corresponding to an application that records the traffic of a NETFLOW log to a DNS log based on whether the request IP is consistent with the destination IP and the difference between the second log time and the first log time is less than the first time difference;

[0023] Obtaining, according to the first association, second proportions of traffic distributed by the different applications through the same service IP address in each second time period;

[0024] An average value of the second proportions of the plurality of second time periods is obtained as the first proportion.

[0025] Furthermore, the request IP, first log time, and application of each DNS log in the first period are obtained, and the destination IP, second log time, and traffic of each NETFLOW log in the first period are obtained, specifically including:

[0026] Directly obtain the request IP, first log time, and request domain name from each DNS log, and obtain the application based on the request domain name;

[0027] Directly obtain the destination IP, NETFLOW log generation time, and traffic calculation parameters from each NETFLOW log. Use the NETFLOW log generation time as the second log time and obtain the traffic based on the traffic calculation parameters.

[0028] Furthermore, obtaining a first association corresponding to an application that records the traffic of a certain NETFLOW log to a certain DNS log specifically includes:

[0029] For each NETFLOW log, a first data record in a double-layer KEY-VALUE format is created. The outer KEY of the first data record stores the destination IP address of the NETFLOW log, the inner KEY stores the second log time of the NETFLOW log, and the inner VALUE stores the traffic volume of the NETFLOW log.

[0030] A second data record in a double-layer KEY-VALUE format is established for each second time period. The outer KEY of the second data record stores the tag of each second time period, the inner KEY stores the tag of the application obtained according to the DNS log, and the inner VALUE stores the traffic of the NETFLOW log associated with the DNS log of the corresponding application.

[0031] Furthermore, the method further comprises:

[0032] Set multiple second statistical step sizes and multiple second time differences, and design a double-loop operation:

[0033] In the first cycle, each second statistical step is used in turn to divide the first time period into a plurality of fifth time periods.

[0034] In the second cycle, a second time difference is selected in turn,

[0035] In each fifth time period, in response to the request IP being consistent with the destination IP and the difference between the second log time and the first log time being less than the selected second time difference, obtaining a second association corresponding to an application that records traffic from a certain NETFLOW log to a certain DNS log,

[0036] According to the second association, a third proportion of traffic distributed by the different applications through the same service IP address in each fifth time period is obtained,

[0037] Obtaining the RMSE of the third proportions of the plurality of fifth time periods and their respective average values;

[0038] Obtain the second statistical step length and the second time difference corresponding to the round with the smallest RMSE in all cycles as the first statistical step length and the first time difference;

[0039] Here, RMSE is the root mean square error.

[0040] Furthermore, obtaining the traffic distributed by the different applications through the same service IP address in the third time period according to the first proportion specifically includes:

[0041] The total traffic distributed through the same service IP address in the third time period is obtained, and the total traffic is calculated as the traffic distributed by the different applications respectively through the same service IP address according to the first proportion.

[0042] In a second aspect, the present disclosure provides a distribution traffic application identification and statistics device, the device comprising:

[0043] A log acquisition module, used to obtain DNS logs and NETFLOW logs of traffic distributed by different applications through the same service IP address during the first period;

[0044] a log correlation module, connected to the log acquisition module, configured to divide the first time period into a plurality of second time periods, establish a first correlation between each application in the DNS log and each flow in the NETFLOW log in each second time period, and calculate a first proportion of each application in the DNS log corresponding to each flow in the NETFLOW log in the first time period based on the first correlation;

[0045] a traffic statistics module, connected to the log correlation module, configured to obtain, according to the first proportion, traffic distributed by the different applications through the same service IP address during a third time period;

[0046] Among them, IP is the Internet Protocol, DNS is the Domain Name System, and NETFLOW is network traffic.

[0047] In a third aspect, the present disclosure provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the distribution traffic application identification and statistical method as described above is implemented.

[0048] The present disclosure provides a distribution traffic application identification and statistics method, a distribution traffic application identification and statistics device, and a computer-readable storage medium. By establishing a segmented association between DNS logs and NETFLOW logs, statistics are obtained to obtain the traffic distributed by different applications through the same service IP address, which is close to the actual traffic of different applications in the same service IP. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flow chart of a method for identifying and counting distributed traffic applications according to an embodiment of the present disclosure;

[0050] Figure 2 It is a structural diagram of a distribution traffic application identification and statistics device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0051] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the embodiments of the present disclosure will be described in further detail below with reference to the accompanying drawings.

[0052] It should be understood that the specific embodiments and drawings described herein are only used to explain the present disclosure rather than to limit the present disclosure.

[0053] It can be understood that, in the absence of conflict, the various embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0054] It will be understood that, for the convenience of description, the drawings of the present disclosure only show parts related to the present disclosure, while parts irrelevant to the present disclosure are not shown in the drawings.

[0055] It can be understood that each module and unit involved in the embodiments of the present disclosure may correspond to only one physical structure, or may be composed of multiple physical structures, or multiple modules and units may be integrated into one physical structure.

[0056] It will be understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of the present disclosure may occur in an order different from that marked in the drawings.

[0057] It is understood that the flowcharts and block diagrams of the present disclosure illustrate the possible architectures, functions, and operations of the systems, devices, equipment, and methods according to the various embodiments of the present disclosure. Each box in the flowchart or block diagram may represent a module, unit, program segment, or code, which contains executable instructions for implementing the specified functions. Moreover, each box or combination of boxes in the block diagram and flowchart may be implemented by a hardware-based device that implements the specified functions, or by a combination of hardware and computer instructions.

[0058] It can be understood that the modules and units involved in the embodiments of the present disclosure can be implemented by software or hardware, for example, the modules and units can be located in a processor.

[0059] Example 1:

[0060] like Figure 1 As shown, the present disclosure provides a method for identifying and counting distribution traffic applications, the method comprising:

[0061] S1. Obtain the DNS logs and NETFLOW logs of traffic distributed by different applications through the same service IP address during the first period;

[0062] S2. Divide the first time period into multiple second time periods, establish a first association between each application in the DNS log and each flow in the NETFLOW log in each second time period, and calculate a first proportion of each application in the DNS log corresponding to each flow in the NETFLOW log in the first time period based on the first association;

[0063] S3. Obtaining, according to the first proportion, the traffic of the different applications distributed through the same service IP address in a third time period;

[0064] Among them, IP is the Internet Protocol, DNS is the Domain Name System, and NETFLOW is network traffic.

[0065] In this embodiment, by establishing segmented association between DNS logs and NETFLOW logs, statistics are obtained on the traffic distributed by different applications through the same service IP address, and traffic of different applications in the same service IP address that is close to the real situation can be obtained.

[0066] Specifically, multiple applications provide services through the same service IP address, which is achieved through scheduling using CNAME (Canonical Name) records in the DNS (Domain Name System). A complete DNS domain name resolution may include the CNAME resolution process, which is recorded in the same log.

[0067] The reason why this disclosure needs to use a statistical method to estimate the application ratio of the same service IP output traffic is that: first, the DNS log records the request domain name, which can be mapped to the application that issued the request based on the request domain name, but does not record the amount of traffic sent for the corresponding request; NETFLOW (network traffic, a network monitoring function) log can count IP traffic, but it can only count IP traffic and cannot distinguish which application the traffic belongs to. If the two logs are correlated, there is another problem. The records of both logs are uncertain, that is, not every request is recorded in the DNS log, and not every traffic is recorded in the NETFLOW log, so it is difficult to establish an accurate correlation between the two.

[0068] In view of this, the present disclosure proposes a method for establishing association based on statistics, the principle of which is explained as follows:

[0069] Based on the law of large numbers in probability theory, when a large number of random experiments are repeated, the frequency of the experimental results will approach a certain constant, namely its theoretical probability. This law states that with a sufficient number of trials, the sample mean of a random event will approach its true expected value.

[0070] This means that in a large number of repeated random experiments (which can be understood as DNS requests and NETFLOW log records), the sample average will approach the true expected value. Even if there is uncertainty in DNS requests and NETFLOW log records, with enough data samples, it is still possible to accurately estimate the actual network traffic.

[0071] Due to the existence of DNS caching, DNS requests are subject to uncertainty. This means that not every domain name access will be requested on the DNS server. According to the law of large numbers, this uncertainty will average out over large amounts of data, making the overall trend more predictable.

[0072] NETFLOW collects traffic data based on a sampling ratio. The presence or absence of each log entry also depends on the sampling ratio, which is subject to a certain degree of probability. However, the law of large numbers allows us to obtain traffic statistics that are close to the actual situation, even with sufficient data volume, even with sampling.

[0073] This paper associates the two logs and implements repeated statistics by dividing the data into multiple segments. The DNS logs and NETFLOW logs of the first and second time periods meet the requirements of the law of large numbers for data samples. Each segment of data is equivalent to a data sample. Multiple data samples are repeated to obtain a true reflection of the traffic proportion of different applications in the same service IP.

[0074] In one embodiment, S1, obtaining DNS logs and NETFLOW logs of traffic distributed by different applications through the same service IP address during a first period, specifically including:

[0075] Obtaining the service IP address of a CDN server that carries multiple application traffic distributions through IDC as the same service IP address;

[0076] Obtaining a first time period in which the CDN traffic used by the multiple applications has a typical distribution characteristic within a third time period;

[0077] Obtain all DNS logs and NETFLOW logs pointing to the same service IP address within a first period;

[0078] Among them, IDC is Internet Data Center and CDN is Content Distribution Network.

[0079] In this embodiment, a statistical method for identifying applications in CDN (Content Delivery Network) distributed traffic is specifically provided. The following two DNS log snippets show that multiple applications provide external services through the same service IP address:

[0080] Log 1:

[0081] 202.102.227.68|53|42.234.29.128|9780|19942|aliqcmmov.a.etoot e.com|A|(CNAME)_aliqcmmov.a.etoote.com.w.alikunlun.com;

[0082] (A)_222.141.5.211|20240821194244.324|0|r

[0083] Log 2:

[0084] 202.102.227.68|53|182.118.238.184|27808|9147|pull-tsl-l3.douyincdn.com|A|(CNAME)_ pull-tsl-l3.douyincdn.com.w.alikunlun.net;(A)_222.141.5.211|20240821194244.271|0|r

[0085] In these two logs, the requested domain name and the resolved IP address are displayed respectively. Specifically, the domain name in the first log is aliqcmmov.a.etoote.com, and the resolved IP is 222.141.5.211; and the domain name in the second log is pull-tsl-l3.douyincdn.com, and the resolved IP is also 222.141.5.211. Although the domain names are different, the resolved IP addresses are the same. Through the DNS CNAME record, we can see that different domain names are distinguished by specific keywords, thus pointing to the servers of their respective applications. This is generally achieved through a content distribution server, which can be understood as a gateway and is dispatched through CNAME.

[0086] This phenomenon raises a question: In an IDC, how can we determine the proportion of traffic from the IP address 222.141.5.211 that belongs to each application (how can we distinguish which application the traffic from the IP address belongs to?). Currently, the industry generally uses NETFLOW (a network monitoring function) to count IP traffic (which can only count IP traffic but cannot distinguish which application the traffic belongs to). To distinguish traffic from different applications, the resolved domain names recorded in the DNS log are usually referenced. There are two specific methods: one is to allocate traffic proportions based on the number of domain name requests; the other is to directly identify the IP traffic as the traffic corresponding to the domain name with the most resolutions.

[0087] When counting traffic generated by different applications associated with the same IP address, the main drawback of the above method is that DNS resolution times cannot fully and accurately reflect the outbound traffic. For example, if one application is streaming video and the other is HTTP web pages, there is no guarantee that the video stream's outbound traffic will be greater, less than, or equal to the HTTP web page's. Therefore, relying solely on DNS resolution times cannot fully and accurately reflect the application's contribution to the outbound traffic of the same service IP address.

[0088] This embodiment implements a method for associating DNS and NETFLOW, including: collecting and associating DNS logs and NETFLOW logs within a certain time range (no less than 6 hours, that is, a first time period, which has typical distribution characteristics of application usage traffic and can be determined based on experience), calculating the proportion of application traffic based on the correlation of the two logs, and analyzing the distribution of application traffic throughout the day based on the proportion. In this embodiment, the first time period is within the third time period. It should be noted that this disclosure is also applicable to situations where the third time period is equal to or less than the first time period or is outside the first time period.

[0089] In one embodiment, wherein:

[0090] The third time period is a certain day, the first time period includes at least 6 fourth time periods, the at least 6 fourth time periods are not consecutive to each other, and the fourth time period is at least one hour;

[0091] The same service IP address corresponds to multiple domain names, one or several domain names correspond to one application, and the multiple domain names correspond to multiple applications.

[0092] In this embodiment, to implement the method, first perform step 1): data preparation, which is as follows:

[0093] 1.1) Obtain the DNS and NETFLOW logs to be studied: For example, obtain the DNS and NETFLOW logs for six hours on a particular day: 00:00-10:00, 3:00-4:00, 8:00-9:00, 12:00-1:00, 16:00-17:00, and 20:00-21:00. The above acquisition time requirement means that the first period includes at least six fourth periods, and these at least six fourth periods are not consecutive.

[0094] 1.2) Determine the service IP to be analyzed: Use the IDC's IP address segment and DNS logs to obtain the IP address in the IDC as the DNS resolution IP address. If the requested domain name exists in multiple applications, take 222.141.5.211 as an example. Query the service's 148 second-level domain names through the DNS log (this can be obtained through the DNS log; multiple domain names may belong to the same application). Some examples are as follows: douyincdn.com, amap.com, etoote.com, sina.com.cn, douyinvod.com, vivo.com.cn, snssdk.com, amemv.com, douyinliving.com, sinaimg.cn.

[0095] 1.3) Domain name conversion to application: Because an application may have multiple domain names attached to it, the domain name must be converted to an application before identifying application traffic. This process is generally obtained through a cumulative knowledge base or through domain name registration information. Some examples are as follows: douyincdn.com | Douyin, amap.com | AutoNavi Map, etoote.com | Kuaishou, sina.com.cn | Sina, douyinvod.com | Douyin, vivo.com.cn | VIVO, snssdk.com | Toutiao, amemv.com | Douyin, douyinliving.com | Douyin, sinaimg.cn | Sina.

[0096] In one embodiment, S2, dividing the first time period into a plurality of second time periods, establishing a first association between each application in the DNS log and each flow in the NETFLOW log in each second time period, and calculating a first proportion of each application in the DNS log corresponding to each flow in the NETFLOW log in the first time period based on the first association, specifically including:

[0097] Get the request IP, first log time, and application of each DNS log in the first period, and get the destination IP, second log time, and traffic of each NETFLOW log in the first period;

[0098] Divide the first time period into a plurality of second time periods using a first statistical step length;

[0099] In each second time period, traverse each DNS log and each NETFLOW log, and obtain a first association corresponding to an application that records the traffic of a NETFLOW log to a DNS log based on whether the request IP is consistent with the destination IP and the difference between the second log time and the first log time is less than the first time difference;

[0100] Obtaining, according to the first association, second proportions of traffic distributed by the different applications through the same service IP address in each second time period;

[0101] An average value of the second proportions of the plurality of second time periods is obtained as the first proportion.

[0102] In this embodiment, one of the keys to implementing the method is step 2): establishing an association algorithm between DNS logs and NETFLOW logs, including:

[0103] 2.1) Obtain the necessary information to associate the two logs. This information mainly includes the request IP, first log time, and application for the DNS log, and the destination IP, second log time, and traffic for the NETFLOW log.

[0104] 2.2) Correlate the two logs using a reasonable time difference, and output the percentage of application-associated traffic using a reasonable statistical duration (statistical step). The statistical duration corresponds to the length of the second period. The requirement is to ensure that each second period contains traffic from each application, and that the difference in traffic distribution for each application across multiple second periods is minimal. This can be achieved through experience or algorithmic optimization.

[0105] In one embodiment, obtaining the request IP, first log time, and application of each DNS log in the first period, and obtaining the destination IP, second log time, and traffic of each NETFLOW log in the first period specifically include:

[0106] Directly obtain the request IP, first log time, and request domain name from each DNS log, and obtain the application based on the request domain name;

[0107] Directly obtain the destination IP, NETFLOW log generation time, and traffic calculation parameters from each NETFLOW log. Use the NETFLOW log generation time as the second log time and obtain the traffic based on the traffic calculation parameters.

[0108] In this embodiment, necessary information can be obtained based on the formats of DNS and NETFLOW logs. The formats of the two logs are described as follows:

[0109] DNS response log format: DNS server IP | server port | request IP | request port | random port | request domain name | request record | resolution result | log time | resolution type.

[0110] NETFLOW log format: Log generation time | Log collection time | Duration | Protocol type | Source address + port | Destination address + port | TCP flags | TOS service type | Number of packets | Number of bytes | Number of packets per second | Number of bits per second | Number of bytes per packet | Number of flows | Next hop address | Router IP address | Source autonomous domain | Destination autonomous domain | Inbound interface | Outbound interface | Source mask | Destination mask | Direction.

[0111] In one embodiment, obtaining a first association corresponding to an application that records traffic from a NETFLOW log to a DNS log specifically includes:

[0112] For each NETFLOW log, a first data record in a double-layer KEY-VALUE format is created. The outer KEY of the first data record stores the destination IP address of the NETFLOW log, the inner KEY stores the second log time of the NETFLOW log, and the inner VALUE stores the traffic volume of the NETFLOW log.

[0113] A second data record in a double-layer KEY-VALUE format is established for each second time period. The outer KEY of the second data record stores the tag of each second time period, the inner KEY stores the tag of the application obtained according to the DNS log, and the inner VALUE stores the traffic of the NETFLOW log associated with the DNS log of the corresponding application.

[0114] In this embodiment, 2.2) correlating the two logs with a reasonable time difference and outputting the proportion of application-associated traffic with a reasonable statistical duration (statistical step) includes:

[0115] 2.2.1) Process NETFLOW logs into the following data storage types:

[0116] Map <String,Map<Long,Long> >

[0117] Among them, the KEY of the outer MAP is the destination IP flowing out from 222.141.5.211, and the VALUE is the timestamp and traffic (Bytes). <Stirng,<>> This is the outer layer. The key of the String is the traffic of 222.141.5.211 flowing out to a certain IP at a certain moment.

[0118] Assume the use of:

[0119] EXFLOWKEY represents the KEY of the outer MAP of the storage type.

[0120] EXFLOWVAL represents the VALUE of the outer MAP of the storage type.

[0121] INFLOWKEY represents the KEY of EXFLOWVAL.

[0122] INFLOWVAL represents the VALUE value of EXFLOWVAL.

[0123] For example: <”123.5.178.69”,<1724240>400,1454>>

[0124] 2.2.2) Process the DNS log into the following data storage type:

[0125] Map<Integer,Map<String,Long>>

[0126] Among them, the KEY of the outer MAP is the number of minutes from the DNS log time to 0 o'clock divided by the statistical duration. For example: 2024-08-12 01:22:30, it is 82 minutes from 2024-08-12 00:00:00. Assuming the statistical duration is 5 minutes, the KEY value is 16. The KEY value of the inner VALUE is the application name, and the VALUE is the traffic.

[0127] Assume the use of:

[0128] EXDNSKEY represents the KEY of the outer MAP of the storage type.

[0129] EXDNSVAL represents the VALUE of the outer MAP of the storage type.

[0130] INDNSKEY represents the KEY of EXDNSVAL.

[0131] INDNSVAL represents the VALUE value of EXDNSVAL.

[0132] For example: <”16”,<”douyin”,1454>>

[0133] 2.2.3) The method for obtaining each value in the DNS log processing data is as follows:

[0134] 2.2.3.1) Traverse all DNS logs and obtain the request IP, log time (first log time), and request domain name.

[0135] 2.2.3.2) T1 (first log time): If the DNS request IP and the EXFLOWKEY value in FLOW (a NETFLOW log) are consistent, the log time of the request IP is T1, that is, EXDNSKEY = T1.

[0136] 2.2.3.3) T2 (second log time): The time when T1 of EXDNSKEY is not empty and the outer INFLOWKEY in FLOW is traversed.

[0137] 2.2.3.4) If T2-T1 is less than the time difference, the association is successful. The KEY of the outer VLAUE is the application after the request domain name is converted. If the application already exists as INDNSKEY, then INDNSVAL = INDNSVAL + the flow in INFLOWVALVALUE. Otherwise, INDNSVAL = INFLOWVALVALUE.

[0138] In one embodiment, the method further comprises:

[0139] Set multiple second statistical step sizes and multiple second time differences, and design a double-loop operation:

[0140] In the first cycle, each second statistical step is used in turn to divide the first time period into a plurality of fifth time periods.

[0141] In the second cycle, a second time difference is selected in turn,

[0142] In each fifth time period, in response to the request IP being consistent with the destination IP and the difference between the second log time and the first log time being less than the selected second time difference, obtaining a second association corresponding to an application that records traffic from a certain NETFLOW log to a certain DNS log,

[0143] According to the second association, a third proportion of traffic distributed by the different applications through the same service IP address in each fifth time period is obtained,

[0144] Obtaining the RMSE of the third proportions of the plurality of fifth time periods and their respective average values;

[0145] Obtain the second statistical step length and the second time difference corresponding to the round with the smallest RMSE in all cycles as the first statistical step length and the first time difference;

[0146] Here, RMSE is the root mean square error.

[0147] In this embodiment, the two logs are correlated. By learning the two parameters of time difference and statistical duration, the optimal application traffic ratio is obtained, thereby truly reflecting the traffic ratio of different applications in the same service IP. The main steps are as follows:

[0148] 1. Collect and correlate DNS logs and NETFLOW logs within a certain time range (no less than 6 hours).

[0149] 2. Establish a reasonable time difference. The time difference here means that during the request process of the same application, the DNS request time must be earlier than the time when the NETFLOW log device is generated. The two logs are associated with a time difference of 1s, 2s, 3s, 4s, and 5s respectively.

[0150] 3. Establish a reasonable statistical duration. After linking DNS logs with FLOW logs, you can output the proportion of each application based on the statistical duration. The statistical duration ranges from 1 minute, 2 minutes, and up to 10 minutes. The statistical duration is generally referred to as the statistical time period, such as whether to count 1 minute of logs or 2 minutes of logs. If the statistical duration is too short, an application may not be accessed during this period. If the statistical duration is too long, the logging time of the learning sample will be extended, resulting in increased learning costs. The purpose of the statistical duration is to obtain an appropriate duration, so that the application traffic behind the IP account for a roughly equal proportion in each duration.

[0151] Fourth, we use RMSE (root mean square error) to determine the optimal application distribution ratio. We then analyze the application traffic for the service IP throughout the day using the optimal statistical duration and time difference. This allows us to accurately calculate application traffic carried by the CDN or cloud (IP), for example, accurately identifying traffic from Taobao, Tmall, Douyin, and Kuaishou behind Wo Cloud.

[0152] That is, the implementation of the method further includes step 3): optimal solution learning, the process of which is as follows:

[0153] 3.1) The learning objective consists of two key parameters: statistical step size and time difference. The statistical step size range is set to [5min, 10min, 15min, 20min, 30min], and the time difference range is set to [1s, 2s, 3s, 4s, 5s].

[0154] 3.2) Use the statistical step size and time difference as a double loop to feed the correlation algorithm in step 2). Assuming there are five IP service applications, and a step size of 5 minutes, 6 hours of logs will generate 6 * 60 / 5 = 72 5-dimensional vectors. The vector value is the percentage of traffic in a certain statistical step size. This is as follows:

[0155] 01--[59,21,12,5,3]

[0156] 02--[52,23,14,6,5]

[0157] ………

[0158] 72--[54,20,12,10,4]

[0159] 3.3) Use RMSE (root mean square error) to determine the optimal statistical step size and time difference. RMSE is the square root of the difference between a set of vector values ​​and their average, divided by the total number of vectors, and then taken. The round with the smallest RMSE is considered the optimal solution. This involves taking the average of the vector groups in each round to obtain the mean vector AVG[A1,A2,A3,A4,A5]. The RMSE value is then calculated by combining the vector group and the AVG vector. The round with the smallest RMSE value has the optimal statistical step size and time difference.

[0160]

[0161] Where x is the 72 5-dimensional vectors mentioned above, y is the mean vector of x, m = 72, y = AVG[A1, A2, A3, A4, A5] = [(59+52+…+54) / 72, (21+23+…+20) / 72, (12+14+…+12) / 72, (5+6+…+10) / 72, (3+5+…+4) / 72].

[0162] In one embodiment, S3, obtaining, according to the first proportion, traffic distributed by the different applications through the same service IP address in a third time period, specifically includes:

[0163] The total traffic distributed through the same service IP address in the third time period is obtained, and the total traffic is calculated as the traffic distributed by the different applications respectively through the same service IP address according to the first proportion.

[0164] In this embodiment, the method of implementing step 4) can obtain the situation of IP output application traffic throughout the day, including:

[0165] 4.1) After calculating the optimal statistical step length and time difference, obtain the AVG[A1, A2, A3, A4, A5] distribution of the optimal step length and time difference, that is, the optimal proportion;

[0166] 4.2) Calculate the total traffic output by the IP throughout the day. Combined with the optimal application ratio, you can obtain the traffic of each application.

[0167] The above results are somewhat estimated, but when the algorithm output reaches the optimal value, the data association between the two logs is also estimated most accurately. A specific implementation effect is demonstrated as follows:

[0168] As shown in Table 1 below, using the IP address 222.141.5.211 as an example, the application traffic percentage calculated using this algorithm is essentially consistent with that calculated by the CDN manufacturer, making it a viable method for calculating distribution traffic. CDN manufacturers can collect and distribute distribution traffic data, and this embodiment's method can help operators understand this data, enabling them to obtain accurate data-based traffic settlement.

[0169] Table 1 Comparison between the calculated traffic ratio and the actual traffic ratio in this embodiment

[0170] Algorithm evaluation traffic ratio Traffic proportion monitored by CDN Tik Tok 46.51％ 51.09％ Amap 20.65％ 19.65％ quick worker 19.18％ 19.18％ Sina 5.30％ 3.30％ VIVO 1.54％ 1.24％ Toutiao 1.45％ 1.15％ Alibaba 0.85％ 0.66％ Volcano Engine 0.82％ 0.92％ NetEase 0.45％ 0.65％ China Unicom 0.43％ 0.40％ other 2.82％ 1.76％ flow rate 667Mb / s 653Mb / s

[0171] This embodiment implements an algorithm based on DNS\NETFLOW calculation to accurately identify application traffic when a service IP carries multiple applications, and proposes an association algorithm for distributed traffic identification and a method for obtaining the optimal statistical step and time difference. It can effectively solve the problem of distinguishing and identifying different application traffic when multiple applications provide services externally through the same service IP.

[0172] Example 2:

[0173] like Figure 2 As shown, the present disclosure provides a distribution traffic application identification and statistics device, the device comprising:

[0174] Log acquisition module 1, used to obtain DNS logs and NETFLOW logs of traffic distributed by different applications through the same service IP address during a first period;

[0175] The log correlation module 2 is connected to the log acquisition module 1 and is used to divide the first time period into multiple second time periods, establish a first correlation between each application in the DNS log and each flow in the NETFLOW log in each second time period, and calculate a first proportion of each application in the DNS log corresponding to each flow in the NETFLOW log in the first time period based on the first correlation;

[0176] The traffic statistics module 3 is connected to the log correlation module 2 and is used to obtain the traffic distributed by the different applications through the same service IP address in the third time period according to the first proportion;

[0177] Among them, IP is the Internet Protocol, DNS is the Domain Name System, and NETFLOW is network traffic.

[0178] In one embodiment, the log acquisition module 1 specifically includes:

[0179] A service address acquisition unit, configured to acquire, through the IDC, a service IP address of a CDN server that carries multiple application traffic distributions, as the same service IP address;

[0180] a characteristic period acquisition unit, configured to acquire a first period having a typical distribution characteristic of CDN traffic used by the multiple applications within a third period;

[0181] A log acquisition unit, connected to the service address acquisition unit and the characteristic period acquisition unit, configured to acquire all DNS logs and NETFLOW logs pointing to the same service IP address within a first period;

[0182] Among them, IDC is Internet Data Center and CDN is Content Distribution Network.

[0183] In one embodiment, wherein:

[0184] The third time period is a certain day, the first time period includes at least 6 fourth time periods, the at least 6 fourth time periods are not consecutive to each other, and the fourth time period is at least one hour;

[0185] The same service IP address corresponds to multiple domain names, one or several domain names correspond to one application, and the multiple domain names correspond to multiple applications.

[0186] In one embodiment, the log correlation module 2 specifically includes:

[0187] A content acquisition unit is used to obtain the request IP, first log time and application of each DNS log in the first period, and obtain the destination IP, second log time and flow of each NETFLOW log in the first period;

[0188] a time period division unit, configured to divide the first time period into a plurality of second time periods using a first statistical step length;

[0189] A traversal association unit is connected to the content acquisition unit and the time period division unit, and is used to traverse each DNS log and each NETFLOW log in each second time period, and obtain a first association corresponding to an application that records the traffic of a NETFLOW log to a DNS log based on whether the request IP is consistent with the destination IP and the difference between the second log time and the first log time is less than the first time difference;

[0190] a proportion calculation unit, connected to the traversal association unit, configured to obtain, according to the first association, a second proportion of traffic distributed by the different applications through the same service IP address in each second time period;

[0191] The average calculation unit is connected to the proportion calculation unit and is used to obtain an average value of the second proportions of the multiple second time periods as the first proportion.

[0192] In one embodiment, the content acquisition unit specifically includes:

[0193] The DNS content acquisition unit is used to directly obtain the request IP, first log time and request domain name from each DNS log, and obtain the application based on the request domain name;

[0194] The NETFLOW content acquisition unit is used to directly obtain the destination IP, NETFLOW log generation time and traffic calculation parameters from each NETFLOW log, use the NETFLOW log generation time as the second log time, and obtain traffic according to the traffic calculation parameters.

[0195] In one embodiment, traversing the association unit specifically includes:

[0196] The first recording unit is used to create a first data record in a double-layer KEY-VALUE format for each NETFLOW log, wherein the outer KEY of the first data record stores the destination IP address of the NETFLOW log, the inner KEY stores the second log time of the NETFLOW log, and the inner VALUE stores the traffic volume of the NETFLOW log;

[0197] The second recording unit is connected to the first recording unit and is used to establish a second data record in a double-layer KEY-VALUE format for each second time period. The outer KEY of the second data record stores the mark of each second time period, the inner KEY stores the mark of the application obtained according to the DNS log, and the inner VALUE stores the traffic of the NETFLOW log associated with the DNS log of the corresponding application.

[0198] In one embodiment, the device further includes an optimization learning module, specifically including:

[0199] The loop operation unit is used to set multiple second statistical step sizes and multiple second time differences, and design a double loop operation:

[0200] In the first loop, the time period division unit is called, and each second statistical step is used in turn to divide the first time period into a plurality of fifth time periods.

[0201] In the second loop, call the traversal associated unit and select a second time difference in turn.

[0202] In each fifth time period, in response to the request IP being consistent with the destination IP and the difference between the second log time and the first log time being less than the selected second time difference, obtaining a second association corresponding to an application that records traffic from a certain NETFLOW log to a certain DNS log,

[0203] The proportion calculation unit is called to obtain, according to the second association, a third proportion of the traffic distributed by the different applications through the same service IP address in each fifth time period,

[0204] Calling an average calculation unit to obtain the RMSE of the third proportions of the plurality of fifth time periods and their respective average values;

[0205] The optimal output unit is connected to the loop operation unit and is used to obtain the second statistical step and the second time difference corresponding to the round with the smallest RMSE in all cycles as the first statistical step and the first time difference;

[0206] Here, RMSE is the root mean square error.

[0207] In one embodiment, the traffic statistics module 3 specifically includes:

[0208] a total traffic unit, configured to obtain the total traffic distributed through the same service IP address during a third period;

[0209] The traffic distribution unit is connected to the total traffic unit and is used to calculate the total traffic as the traffic distributed by the different applications respectively through the same service IP address according to the first proportion.

[0210] Example 3:

[0211] Embodiment 3 of the present disclosure provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the distribution traffic application identification and statistics method as described in embodiment 1 is implemented, or the distribution traffic application identification and statistics device as described in embodiment 2 is implemented.

[0212] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program elements or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable read only memory), flash memory or other memory technology, CD-ROM (Compact Disc Read-Only Memory), digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer.

[0213] In addition, the present disclosure may also provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the distribution traffic application identification and statistics method as described in Example 1. The computer device may be the distribution traffic application identification and statistics device as described in Example 2.

[0214] The memory is connected to the processor, the memory may be a flash memory, a read-only memory or other memory, and the processor may be a central processing unit or a single-chip microcomputer.

[0215] Embodiments 1-3 of the present disclosure provide a method for identifying and counting distributed traffic applications, a device for identifying and counting distributed traffic applications, and a computer-readable storage medium. By establishing a segmented association between DNS logs and NETFLOW logs, statistics are obtained to obtain traffic distributed by different applications through the same service IP address, and traffic of different applications in the same service IP address that is close to the real situation can be obtained.

[0216] It is understood that the above embodiments are merely exemplary embodiments for illustrating the principles of the present disclosure, and the present disclosure is not limited thereto. Those skilled in the art may make various modifications and improvements without departing from the spirit and substance of the present disclosure, and such modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. A method for identifying and counting distribution traffic applications, characterized in that: The method comprises: Obtain the DNS logs and NETFLOW logs of traffic distributed by different applications through the same service IP address during the first period; Divide the first time period into a plurality of second time periods, establish a first association between each application in the DNS log and each flow in the NETFLOW log in each second time period, and count a first proportion of each flow in the NETFLOW log corresponding to each application in the DNS log in the first time period according to the first association; Obtaining, according to the first proportion, the traffic distributed by the different applications respectively through the same service IP address in a third time period; Among them, IP is the Internet Protocol, DNS is the Domain Name System, and NETFLOW is network traffic.

2. The method according to claim 1, characterized in that Obtain the DNS logs and NETFLOW logs of different applications distributing traffic through the same service IP address during the first period, including: Obtaining, through IDC, a service IP address of a CDN server that carries multiple application traffic distributions as the same service IP address; Obtaining a first time period in which the CDN traffic used by the multiple applications has a typical distribution characteristic within a third time period; Obtain all DNS logs and NETFLOW logs pointing to the same service IP address within a first period of time; Among them, IDC is Internet Data Center and CDN is Content Distribution Network.

3. The method according to claim 2, characterized in that in: The third time period is a certain day, the first time period includes at least 6 fourth time periods, the at least 6 fourth time periods are not continuous with each other, and the fourth time period is at least one hour; The same service IP address corresponds to multiple domain names, one or several domain names correspond to one application, and the multiple domain names correspond to multiple applications.

4. The method according to any one of claims 1 to 3, characterized in that: The first time period is divided into a plurality of second time periods, and a first association is established between each application in the DNS log and each flow in the NETFLOW log in each second time period. According to the first association, a first proportion of each flow in the NETFLOW log corresponding to each application in the DNS log in the first time period is calculated, specifically including: Get the request IP, first log time and application of each DNS log in the first period, and get the destination IP, second log time and flow of each NETFLOW log in the first period; Using a first statistical step length to divide the first time period into a plurality of second time periods; In each second time period, traverse each DNS log and each NETFLOW log, and obtain a first association corresponding to an application that records the flow of a NETFLOW log to a DNS log based on the consistency between the request IP and the destination IP and the difference between the second log time and the first log time being less than the first time difference; Obtaining, according to the first association, a second proportion of traffic distributed by the different applications through the same service IP address in each second time period; An average value of the second proportions of the plurality of second time periods is obtained as the first proportion.

5. The method according to claim 4, characterized in that Get the request IP, first log time and application of each DNS log in the first period, and get the destination IP, second log time and traffic of each NETFLOW log in the first period, including: Directly obtain the request IP, first log time, and request domain name from each DNS log, and obtain the application based on the request domain name; Directly obtain the destination IP, NETFLOW log generation time and traffic calculation parameters from each NETFLOW log, use the NETFLOW log generation time as the second log time, and obtain the traffic based on the traffic calculation parameters.

6. The method according to claim 4, characterized in that Obtaining a first association corresponding to an application that records a flow of a NETFLOW log to a DNS log, specifically including: For each NETFLOW log, a first data record in a double-layer KEY-VALUE format is created. The outer KEY of the first data record stores the destination IP of the NETFLOW log, the inner KEY stores the second log time of the NETFLOW log, and the inner VALUE stores the traffic of the NETFLOW log. A second data record in a double-layer KEY-VALUE format is established for each second time period. The outer KEY of the second data record stores the mark of each second time period, the inner KEY stores the mark of the application obtained according to the DNS log, and the inner VALUE stores the traffic of the NETFLOW log associated with the DNS log of the corresponding application.

7. The method according to claim 4, characterized in that The method further comprises: Set multiple second statistical steps and multiple second time differences, and design a double loop operation: In the first cycle, each second statistical step is used in turn to divide the first time period into a plurality of fifth time periods. In the second cycle, a second time difference is selected in turn, In each fifth time period, in response to the request IP being consistent with the destination IP and the difference between the second log time and the first log time being less than the selected second time difference value, obtaining a second association corresponding to an application that records a flow of a NETFLOW log to a DNS log, According to the second association, a third proportion of the traffic distributed by the different applications through the same service IP address in each fifth time period is obtained, Obtaining the RMSE of the third proportions of the plurality of fifth time periods and their respective average values; Obtain the second statistical step length and the second time difference corresponding to the round with the smallest RMSE in all cycles as the first statistical step length and the first time difference; Here, RMSE is the root mean square error.

8. The method according to any one of claims 1 to 3, characterized in that: Obtaining, according to the first proportion, the traffic distributed by the different applications through the same service IP address in the third time period, specifically includes: The total traffic distributed through the same service IP address in the third time period is obtained, and the total traffic is calculated as the traffic respectively distributed by the different applications through the same service IP address according to the first proportion.

9. A distribution flow application identification and statistics device, characterized in that: The device comprises: A log acquisition module, used to acquire DNS logs and NETFLOW logs of traffic distributed by different applications through the same service IP address during a first period; A log association module, connected to the log acquisition module, is used to divide the first time period into a plurality of second time periods, establish a first association between each application in the DNS log and each flow in the NETFLOW log in each second time period, and count a first proportion of each application in the DNS log corresponding to each flow in the NETFLOW log in the first time period according to the first association; A traffic statistics module, connected to the log association module, for obtaining the traffic distributed by the different applications through the same service IP address in a third time period according to the first proportion; Among them, IP is the Internet Protocol, DNS is the Domain Name System, and NETFLOW is network traffic.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the distribution traffic application identification and statistical method as described in any one of claims 1 to 8 is implemented.

Images