Abnormal traffic detection method and device and computer device

Abstract

The application discloses an abnormal traffic detection method and device and a computer device. The method comprises the following steps: collecting traffic data to be detected according to a preset collection period; adopting multiple clustering dimensions to cluster the traffic data to be detected, thereby obtaining multiple traffic clusters, wherein the clustering dimensions at least include a ratio of total uplink traffic of a target area to total downlink traffic of the target area, a ratio of uplink traffic outside the target area to total uplink traffic, a ratio of downlink traffic inside the target area to total downlink traffic, and a ratio of total traffic outside the target area to total traffic inside the target area; multiple detection strategies corresponding to the multiple traffic clusters are determined respectively; and the traffic in different traffic clusters is detected by adopting the detection strategy corresponding to each traffic cluster, thereby obtaining detection results.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to an abnormal traffic detection method, apparatus, and computer equipment. Background Technology

[0002] In communication networks, network traffic monitoring and anomaly detection are critical tasks for ensuring network stability and security. With the rapid development of the internet and various communication technologies, the complexity and scale of network traffic are constantly increasing, making traditional traffic monitoring methods ineffective in addressing inter-provincial traffic anomalies. In particular, abnormal fluctuations in uplink and downlink traffic can lead to network congestion, decreased service quality, and even cybersecurity incidents.

[0003] Key anomalies include cross-regional traffic pulling and PCDN (Personal Content Delivery Network) services. Cross-regional traffic pulling can lead to resource consumption and waste, affecting other users' normal services, increasing manpower and technical costs, and complicating network management. PCDN services increase bandwidth consumption and may involve unauthorized content distribution, increasing the risk of content leakage and infringement. Traditional methods for detecting anomalies rely heavily on manual determination of traffic baselines and logical judgments, resulting in low detection rates and a high likelihood of false positives and false negatives when dealing with complex and large-scale network traffic. Furthermore, they lack real-time monitoring and analysis capabilities, exhibiting a lag; they are also less adaptable to the dynamic and complex network environment and diverse anomalies of operators, failing to fully leverage the rich information in network traffic logs. Summary of the Invention

[0004] This application provides an abnormal traffic detection method, apparatus, and computer equipment to at least solve the technical problem of low accuracy in detecting abnormal traffic behavior in related technologies.

[0005] According to one aspect of the embodiments of this application, an abnormal traffic detection method is provided, comprising: collecting traffic data to be detected according to a preset collection period; clustering the traffic data to be detected using multiple clustering dimensions to obtain multiple traffic clusters, wherein the clustering dimensions include at least the ratio of total uplink traffic to total downlink traffic in the target area, the ratio of uplink traffic outside the target area to total uplink traffic, the ratio of downlink traffic within the target area to total downlink traffic, and the ratio of total traffic outside the target area to total traffic within the target area, wherein the total uplink traffic includes the sum of uplink traffic within the target area and uplink traffic outside the target area, and the total downlink traffic includes the sum of downlink traffic within the target area and downlink traffic outside the target area; determining multiple detection strategies corresponding to the multiple traffic clusters respectively; and detecting traffic in different traffic clusters using the detection strategy corresponding to each traffic cluster respectively to obtain detection results.

[0006] Optionally, the traffic data to be detected is clustered using multiple clustering dimensions, including: obtaining the value of each data point in the traffic data to be detected, the mean of all data points in the traffic data to be detected, and the standard deviation of each data point in the traffic data to be detected; determining the score of each data point in the traffic data to be detected based on the value of each data point in the traffic data to be detected, the mean of all data points in the traffic data to be detected, and the standard deviation of each data point in the traffic data to be detected; identifying data points whose absolute scores are greater than a preset threshold as abnormal data; deleting missing values ​​in the traffic data to be detected, and re-collecting traffic data from a preset period adjacent to the abnormal data to replace the abnormal data, thereby obtaining processed traffic data; selecting target traffic data from the processed traffic data; and clustering the target traffic data using multiple clustering dimensions to obtain the multiple traffic clusters.

[0007] Optionally, selecting target traffic data from the processed traffic data includes: dividing the processed traffic data into traffic data within the target area and traffic data outside the target area; obtaining the uplink traffic rate and downlink traffic rate of each traffic data in the processed traffic data; determining a preset number of traffic data with the highest uplink traffic rate and a preset number of traffic data with the highest downlink traffic rate in the traffic data within the target area as the target traffic data; and determining a preset number of traffic data with the highest uplink traffic rate and a preset number of traffic data with the highest downlink traffic rate in the traffic data outside the target area as the target traffic data.

[0008] Optionally, clustering the target traffic data using multiple clustering dimensions to obtain the multiple traffic clusters includes: obtaining normalized feature values ​​of the target traffic data in the multiple clustering dimensions; randomly obtaining multiple cluster centers, and assigning each data point in the target traffic data to the nearest cluster center according to the normalized feature values ​​to obtain multiple initial clusters; determining the mean of all data points in the multiple initial clusters as the new cluster centers, and redistributing the data points in each initial cluster until the cluster centers no longer change, to obtain the multiple traffic clusters.

[0009] Optionally, the detection strategy corresponding to each traffic cluster is used to detect the traffic in different traffic clusters to obtain the detection results, including: obtaining the first network address associated with the traffic data of the first cluster among the multiple traffic clusters, wherein the first cluster represents a cluster composed of traffic data whose ratio of total uplink traffic to total downlink traffic is greater than a first proportion; obtaining the network type of the network address associated with the traffic data of the first cluster, and if the network type is a metropolitan area network, obtaining the first number of uplink ports with the largest traffic among the uplink ports of the first network address; and marking the first network address as a Personal Content Delivery Network (PCDN) if the ratio of the uplink traffic of the first number of uplink ports to the total uplink traffic of the first network address exceeds a first threshold, the ratio of the target end of the uplink traffic of the first network address to the network address in the target area exceeds a second threshold, and the ratio of non-public ports among the uplink ports of the first network address exceeds a third threshold.

[0010] Optionally, the detection strategy corresponding to each traffic cluster is used to detect the traffic in different traffic clusters to obtain the detection results, including: obtaining the second network address associated with the traffic data of the second cluster in the plurality of traffic clusters, wherein the second cluster represents a cluster composed of traffic data in which the uplink traffic outside the target area is higher than the downlink traffic outside the target area; obtaining the network type of the network address associated with the traffic data of the second cluster, and if the network type is a metropolitan area network, counting the traffic data of the second network address within a specified period; using a pre-trained seasonality prediction method based on the traffic data within the specified period to predict the traffic data within the prediction period, and obtaining the prediction error; if the prediction error is greater than a preset error threshold and the proportion of the number of network addresses interacting within the specified period to the total number of network addresses interacting is greater than a fourth threshold, determining that the second network address is marked.

[0011] Optionally, the detection strategy corresponding to each traffic cluster is used to detect the traffic in different traffic clusters to obtain the detection results, including: obtaining the third network address associated with the traffic data of the second cluster among the multiple traffic clusters; obtaining the network type of the network address associated with the traffic data of the second cluster, and if the network type is an Internet Data Center network, statistically analyzing the traffic data of the third network address within a specified period; evaluating the traffic data within the specified period using an autocorrelation function, and marking the third network address if the traffic data within the specified period shows periodic changes.

[0012] Optionally, the method further includes: identifying the network address to be marked as the abnormal traffic address in the detection result.

[0013] According to another aspect of the embodiments of this application, an abnormal traffic detection device is also provided, comprising: a collection module, configured to collect traffic data to be detected according to a preset collection period; a clustering module, configured to cluster the traffic data to be detected using multiple clustering dimensions to obtain multiple traffic clusters, wherein the clustering dimensions include at least the ratio of total uplink traffic to total downlink traffic in the target area, the ratio of uplink traffic outside the target area to total uplink traffic, the ratio of downlink traffic within the target area to total downlink traffic, and the ratio of total traffic outside the target area to total traffic within the target area, wherein the total uplink traffic includes the sum of uplink traffic within the target area and uplink traffic outside the target area, and the total downlink traffic includes the sum of downlink traffic within the target area and downlink traffic outside the target area; a determination module, configured to determine multiple detection strategies corresponding to the multiple traffic clusters respectively; and a detection module, configured to detect the traffic in different traffic clusters using the detection strategy corresponding to each traffic cluster respectively, and obtain detection results.

[0014] According to another aspect of the embodiments of this application, a computer device is also provided, including: a memory and a processor, wherein the memory is used to store program instructions; and the processor, connected to the memory, is used to execute the above-described abnormal traffic detection method.

[0015] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-described abnormal traffic detection method by running the computer program.

[0016] According to another aspect of the embodiments of this application, a computer program product is also provided, including computer instructions that, when executed by a processor, implement the above-described abnormal traffic detection method.

[0017] In this embodiment, traffic data to be detected is collected according to a preset collection period; the traffic data to be detected is clustered using multiple clustering dimensions to obtain multiple traffic clusters, wherein the clustering dimensions include at least the ratio of total uplink traffic to total downlink traffic in the target area, the ratio of uplink traffic outside the target area to total uplink traffic, the ratio of downlink traffic within the target area to total downlink traffic, and the ratio of total traffic outside the target area to total traffic within the target area. The total uplink traffic includes the sum of uplink traffic within the target area and uplink traffic outside the target area, and the total downlink traffic includes the sum of downlink traffic within the target area and downlink traffic outside the target area. Multiple detection strategies corresponding to the multiple traffic clusters are determined respectively; the detection strategy corresponding to each traffic cluster is used to detect the traffic in different traffic clusters to obtain detection results. By analyzing the traffic data collected over multiple periods using multiple clustering dimensions, the purpose of accurately identifying abnormal traffic behavior is achieved, thereby improving the technical effect of detecting abnormal traffic behavior and solving the technical problem of low accuracy in detecting abnormal traffic behavior in related technologies. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This is a hardware structure block diagram of a computer terminal for implementing an abnormal traffic detection method according to an embodiment of this application;

[0020] Figure 2 This is a flowchart of an abnormal traffic detection method according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of a traffic data prediction time series according to an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of traffic data autocorrelation detection results according to an embodiment of this application;

[0023] Figure 5 This is a flowchart of an abnormal traffic behavior detection method according to an embodiment of this application;

[0024] Figure 6 This is a structural diagram of an abnormal flow detection device according to an embodiment of this application. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] The information collected in this application embodiment is information and data authorized by the user or fully authorized by all parties. The collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant regions, and necessary confidentiality measures have been taken. It does not violate public order and good morals, and provides corresponding operation entry points for users to choose to authorize or reject the automated decision results. If the user chooses to reject, the process will proceed to the expert decision-making process.

[0028] To address the problems existing in related technologies, this application provides an abnormal traffic detection method, which can be run on... Figure 1 The computer terminal shown is explained below.

[0029] The abnormal traffic detection method provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal for implementing an abnormal traffic detection method is shown. Figure 1As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions connected via wired and / or wireless networks. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0030] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be implemented wholly or partially as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be wholly or partially integrated into any other element in the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as processor control (e.g., selection of a variable resistor termination path connected to an interface).

[0031] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the abnormal traffic detection method in this embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned abnormal traffic detection method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0032] The transmission module 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission module 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission module 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0033] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.

[0034] It should be noted here that, in some optional embodiments, the above... Figure 1 The computer terminal shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 1 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computer terminal.

[0035] In the above operating environment, this application provides an embodiment of an abnormal traffic detection method. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0036] Figure 2 This is a flowchart of an abnormal traffic detection method according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0037] Step S202: Collect the flow data to be detected according to the preset collection cycle;

[0038] In step S202, the preset acquisition period includes, but is not limited to: 5 minutes, 1 hour, and 24 hours.

[0039] To detect abnormal behavior more quickly, a 5-minute interval can be used.

[0040] Step S204: Cluster the traffic data to be detected using multiple clustering dimensions to obtain multiple traffic clusters. The clustering dimensions include at least the ratio of the total uplink traffic to the total downlink traffic in the target area, the ratio of the uplink traffic outside the target area to the total uplink traffic, the ratio of the downlink traffic within the target area to the total downlink traffic, and the ratio of the total traffic outside the target area to the total traffic within the target area. The total uplink traffic includes the sum of the uplink traffic within the target area and the uplink traffic outside the target area. The total downlink traffic includes the sum of the downlink traffic within the target area and the downlink traffic outside the target area.

[0041] It should be noted that the target area can be set according to actual needs, such as: provincial administrative region, target area (within the province), target area (outside the province).

[0042] Furthermore, the data table contains multiple dimensions of features, including source city code, destination city code, source province code, destination province code, traffic type, out-of-province uplink traffic rate, out-of-province downlink traffic rate, intra-province uplink traffic rate, intra-province downlink traffic rate, total uplink traffic rate, and total downlink traffic rate. Correspondingly, source and destination refer to the start and end points of traffic, respectively; traffic type includes IDC traffic and metropolitan area network traffic. IDC (Internet Data Center) traffic refers to inter-provincial traffic exchange between operator IDCs, while metropolitan area network traffic refers to intra-provincial uplink and downlink traffic within the operator's metropolitan area network; the unit of traffic rate is Gbps.

[0043] Step S206: Determine multiple detection strategies corresponding to the multiple traffic clusters respectively;

[0044] Step S208: Use the detection strategy corresponding to each traffic cluster to detect the traffic in different traffic clusters and obtain the detection results.

[0045] Through steps S202 to S208, traffic data to be detected is collected according to a preset collection cycle. Multiple clustering dimensions are used to cluster the traffic data to be detected, resulting in multiple traffic clusters. Each clustering dimension includes at least the ratio of total uplink traffic to total downlink traffic in the target area, the ratio of uplink traffic outside the target area to total uplink traffic, the ratio of downlink traffic within the target area to total downlink traffic, and the ratio of total traffic outside the target area to total traffic within the target area. The total uplink traffic includes the sum of uplink traffic within the target area and uplink traffic outside the target area. The total downlink traffic includes the sum of downlink traffic within the target area and downlink traffic outside the target area. Multiple detection strategies corresponding to the multiple traffic clusters are determined. The detection strategy corresponding to each traffic cluster is used to detect traffic in different traffic clusters, obtaining detection results. By analyzing the collected traffic data from multiple cycles using multiple clustering dimensions, the aim of accurately identifying abnormal traffic behavior is achieved, thereby improving the technical effect of detecting abnormal traffic behavior and solving the technical problem of low accuracy in detecting abnormal traffic behavior in related technologies. The following is a detailed explanation.

[0046] In some embodiments of this application, the specific steps for clustering the traffic data to be detected using multiple clustering dimensions are as follows: obtaining the value of each data point in the traffic data to be detected, the mean of all data points in the traffic data to be detected, and the standard deviation of each data point in the traffic data to be detected; determining the score of each data point in the traffic data to be detected based on the value of each data point in the traffic data to be detected, the mean of all data points in the traffic data to be detected, and the standard deviation of each data point in the traffic data to be detected; identifying data points whose absolute scores are greater than a preset threshold as abnormal data; deleting missing values ​​in the traffic data to be detected, and re-collecting traffic data from a preset period adjacent to the abnormal data to replace the abnormal data, thereby obtaining processed traffic data; selecting target traffic data from the processed traffic data; and clustering the target traffic data using multiple clustering dimensions to obtain the multiple traffic clusters.

[0047] The specific steps for selecting target traffic data from the processed traffic data are as follows: Dividing the processed traffic data into traffic data within the target area and traffic data outside the target area; obtaining the uplink and downlink traffic rates of each traffic data item in the processed traffic data; determining a preset number of traffic data items with the highest uplink traffic rate and a preset number of traffic data items with the highest downlink traffic rate within the target area as the target traffic data; and determining a preset number of traffic data items with the highest uplink traffic rate and a preset number of traffic data items with the highest downlink traffic rate within the traffic data outside the target area as the target traffic data.

[0048] Specifically, the traffic data contains outliers or missing values. For missing values, they are directly deleted because the traffic acquisition system receives 5-minute average data, indicating that no data was collected for 5 consecutive minutes, and therefore will not be analyzed at this time. For outliers, the Z-Score method is used to identify and repair them. The formula is as follows: Where z is the standard score, x is the traffic data point, μ is the mean of the traffic dataset, and σ is the standard deviation of the traffic dataset. If |z|>k, typically k is set to 3, it is considered an outlier. Traffic data from the most recent hour is selected, and data identified as outliers are deleted. Then, the traffic clustering results for the most recent 5 minutes are obtained again.

[0049] The preprocessed data is classified into intra-provincial traffic and inter-provincial traffic, and sorted according to uplink and downlink traffic rates respectively. The top N records with the highest uplink and downlink traffic are selected as data for subsequent analysis, where N is 1000.

[0050] data topn =Retrieve the top 1000 records with the highest uplink and downlink traffic respectively

[0051] Furthermore, 1,000 upstream and downstream traffic records within the province and 1,000 upstream and downstream traffic records outside the province were selected, for a total of 4,000 data records, which were then merged for unified analysis.

[0052] In some embodiments of this application, the specific steps of clustering the target traffic data using multiple clustering dimensions to obtain the multiple traffic clusters include: obtaining the normalized feature values ​​of the target traffic data in the multiple clustering dimensions; randomly obtaining multiple cluster centers, and assigning each data point in the target traffic data to the nearest cluster center according to the normalized feature values ​​to obtain multiple initial clusters; determining the mean of all data points in the multiple initial clusters as the new cluster center, and redistributing the data points in each initial cluster until the cluster centers no longer change, to obtain the multiple traffic clusters.

[0053] Taking the target region as a provincial-level administrative region as an example, the clustering dimensions include: total uplink traffic of the target region / total downlink traffic of the target region, uplink traffic outside the province / total uplink traffic, downlink traffic within the province / total downlink traffic, and total traffic outside the province / total traffic within the province.

[0054] Alternatively, unsupervised learning algorithms can be used to cluster traffic data, dividing the dataset into K clusters, such that the similarity between data points within a cluster is as high as possible, while the similarity between different clusters is as low as possible.

[0055] The goal of the K-means algorithm is to minimize the sum of squared Euclidean distances from data points within a cluster to the cluster center. The objective function is as follows:

[0056]

[0057] Where J represents the objective function, k represents the k-th cluster center, K represents the target number of cluster centers, and x i Let C represent the i-th data point. k Let x represent all data points assigned to the k-th cluster center, j represent the j-th feature of the data point, and m represent the data point x. i Total number of features, x ij Represents data point x i The j-th eigenvalue, u kj This represents the j-th feature value of the k-th cluster center.

[0058] The clustering process is as follows:

[0059] The feature values ​​of each data point (traffic data) in multiple dimensions are normalized to obtain normalized feature values.

[0060] Randomly select multiple cluster centers, for example: 4.

[0061] Each data point is assigned to the nearest cluster center, forming multiple clusters, as shown in the following formula:

[0062] C j ={xi :‖x i -u j || 2 ≤‖x i -u k || 2 for all k}

[0063] Among them, c j Denotes the set of clusters, x i Let u represent the i-th data point. j Let u represent the j-th cluster center. k This indicates that there are k cluster centers, and the cluster with the smallest distance among the k cluster centers is selected as the cluster of data points.

[0064] Calculate the mean of all data points within each cluster as the new cluster center:

[0065]

[0066] Among them, u j It is the new cluster center, C j Denotes the set of clusters, x i Represents a data point.

[0067] Repeat the above steps until the cluster centers stop changing or the change reaches a threshold, at which point the iteration stops.

[0068] Taking a clustering result of 5 clusters as an example, specifically:

[0069] Cluster 1 (First Cluster): Represents traffic data where the ratio of total uplink traffic to total downlink traffic is greater than the first proportion, indicating that there is a lot of outbound traffic in this area.

[0070] Cluster 2 (Second Cluster): Represents a cluster where the outbound uplink flow from outside the province is higher than the outbound downlink flow from outside the province.

[0071] Cluster 3: This represents traffic data where the ratio of outbound traffic to total outbound traffic is greater than the second ratio, indicating that the outbound traffic in this region mainly flows to other provinces.

[0072] Cluster 4: This represents traffic data where the ratio of intra-provincial downlink traffic to total downlink traffic is greater than the third ratio, indicating that the downlink traffic in this region is mainly concentrated within the province.

[0073] Cluster 5: Represents the flow behavior of high total flow outside the province / total flow within the province, indicating that the total flow in this region mainly flows to other provinces.

[0074] By analyzing the clustering results, we can further identify abnormal behaviors within each cluster. By combining periodic prediction algorithms and time-series prediction algorithms, we can identify and provide early warnings of abnormal traffic conditions, thereby ensuring network stability and security.

[0075] The specific detection method is as follows: Obtain the first network address associated with the traffic data of the first cluster among the multiple traffic clusters, wherein the first cluster represents a cluster composed of traffic data whose ratio of total uplink traffic to total downlink traffic is greater than a first proportion; Obtain the network type of the network address associated with the traffic data of the first cluster; if the network type is a metropolitan area network, obtain the first number of uplink ports with the largest traffic among the uplink ports of the first network address; if the ratio of the uplink traffic of the first number of uplink ports to the total uplink traffic of the first network address exceeds a first threshold, and the ratio of the target end of the uplink traffic of the first network address to the network address within the target area exceeds a second threshold, and the ratio of non-public ports among the uplink ports of the first network address exceeds a third threshold, mark the first network address as a Personal Content Delivery Network (PCDN).

[0076] Specifically, for user IPs (first network addresses) belonging to the first cluster within the metropolitan area network, the following judgment is made:

[0077] If the top 10 user IP addresses (first threshold) account for more than 80% of the total uplink traffic, and the target IP addresses are within the province, accounting for more than 70% of all target IP addresses (second threshold), and the user's uplink service port is a non-public port, accounting for more than 50% of all uplink ports (third threshold), then the user IP is determined to be a PCDN and marked accordingly.

[0078] In another scenario, the second network address associated with the traffic data of the second cluster among the plurality of traffic clusters is obtained, wherein the second cluster represents a cluster of traffic data where the uplink traffic outside the target area is higher than the downlink traffic outside the target area; the network type of the network address associated with the traffic data of the second cluster is obtained, and if the network type is a metropolitan area network, the traffic data of the second network address within a specified period is statistically analyzed; a pre-trained seasonal prediction method is used to predict the traffic data within the prediction period based on the traffic data within the specified period to obtain a prediction error; if the prediction error is greater than a preset error threshold and the proportion of the number of network addresses interacting within the specified period to the total number of network addresses interacting is greater than a fourth threshold, the second network address is determined to be marked.

[0079] Specifically, for user IPs (second network addresses) belonging to the second cluster in the metropolitan area network, traffic data of the user IPs over the past 7 days (specified period) is collected, and the ARIMA algorithm is used to detect whether there are any abnormal surges in traffic.

[0080] The ARIMA model takes the following form:

[0081]

[0082] Among them, y t It represents the value of the time series at time t, where c is a constant term. It is the autoregressive coefficient, θ i It is the moving average coefficient, ∈ t It is a white noise error term.

[0083] Based on the properties of the time series, the difference order is determined to be 1. The number of autoregressive terms p and moving average terms q are determined using the autocorrelation function (ACF) and partial autocorrelation function (PACF) plots. Here, both p and q are set to 2.

[0084] Furthermore, the trained ARIMA model is used to predict future traffic, such as... Figure 3 As shown, blue represents actual traffic data (true), and red represents predicted traffic data (predict). The prediction error is calculated, and if the prediction error is greater than 3 times the standardized residual, the data within the specified statistical period is considered to be bursty data.

[0085] If a user's IP address shows a sudden increase in the past 7 days, and the number of IP addresses interacting with the user within a specified period also shows a sudden increase within that period, and the proportion of IP addresses interacting with the user within that period to the total number of IP addresses interacting with the user is greater than 70% (fourth threshold), and the number of IP addresses interacting with the user is relatively frequent, then the user's IP address is considered to be being targeted and will be marked as a priority.

[0086] In other embodiments of this application, a third network address associated with the traffic data of the second cluster among the plurality of traffic clusters is obtained; the network type of the network address associated with the traffic data of the second cluster is obtained; if the network type is an Internet Data Center network, the traffic data of the third network address within a specified period is statistically analyzed; the traffic data within the specified period is evaluated using an autocorrelation function; if the traffic data within the specified period shows periodic changes, the third network address is marked.

[0087] Specifically, the user IP address (third network address) belonging to the second cluster of the IDC;

[0088] To determine whether traffic data from the past 7 days exhibits periodicity, an autocorrelation function (ACF) is used. Specifically:

[0089]

[0090] Where, ρ k It is the autocorrelation coefficient with a lag period of k, x t It is the t-th sample in the time series. is the mean, and N is the total number of samples in the time series.

[0091] If the autocorrelation coefficient is significantly higher in certain lag periods, such as 24 hours, than in other lag periods, it indicates that the time series has periodicity. If the traffic data does not show obvious periodicity in the past 7 days and the traffic data remains at a relatively high level, then the user may be being manipulated by traffic brokers; this should be marked for focused analysis. Figure 4 As shown, the self-periodic detection results of several types of traffic data are presented.

[0092] The network address that is marked is finally identified as the abnormal traffic address in the detection results. In practical application scenarios, the marked IP can be manually verified to finally determine the abnormal traffic.

[0093] Figure 5 Another method for detecting abnormal traffic behavior is shown, such as Figure 5 As shown, Step 1: Connect to the NetFlow (a traffic acquisition platform) traffic acquisition platform to obtain traffic clustering data from the 5-minute clustering table; Step 2: Further process the acquired basic data, including Top-N filtering, data cleaning and preprocessing, etc.; Step 3: Cluster the TOP data, with clustering dimensions including total uplink traffic / total downlink traffic, out-of-province uplink traffic / total uplink traffic, in-province downlink traffic / total downlink traffic, and out-of-province total traffic / in-province total traffic; Step 4: Use different discrimination methods to judge the user IPs in different clusters and identify abnormal traffic.

[0094] Figure 6 An abnormal traffic detection device according to an embodiment of this application includes:

[0095] The acquisition module 60 is used to acquire the traffic data to be detected according to a preset acquisition cycle;

[0096] Clustering module 62 is used to cluster the traffic data to be detected using multiple clustering dimensions to obtain multiple traffic clusters. The clustering dimensions include at least the ratio of the total uplink traffic to the total downlink traffic in the target area, the ratio of the uplink traffic outside the target area to the total uplink traffic, the ratio of the downlink traffic within the target area to the total downlink traffic, and the ratio of the total traffic outside the target area to the total traffic within the target area. The total uplink traffic includes the sum of the uplink traffic within the target area and the uplink traffic outside the target area. The total downlink traffic includes the sum of the downlink traffic within the target area and the downlink traffic outside the target area.

[0097] The determination module 64 is used to determine multiple detection strategies corresponding to the multiple traffic clusters respectively;

[0098] The detection module 66 is used to detect the traffic in different traffic clusters using the detection strategy corresponding to each traffic cluster, and obtain the detection results.

[0099] The aforementioned abnormal traffic detection device collects traffic data to be detected according to a preset collection cycle. Multiple clustering dimensions are used to cluster the traffic data to be detected, resulting in multiple traffic clusters. Each clustering dimension includes at least the ratio of total uplink traffic to total downlink traffic in the target area, the ratio of uplink traffic outside the target area to total uplink traffic, the ratio of downlink traffic within the target area to total downlink traffic, and the ratio of total traffic outside the target area to total traffic within the target area. The total uplink traffic includes the sum of uplink traffic within and outside the target area, and the total downlink traffic includes the sum of downlink traffic within and outside the target area. Multiple detection strategies corresponding to the multiple traffic clusters are determined. The detection strategy corresponding to each traffic cluster is used to detect traffic in different traffic clusters, obtaining detection results. By analyzing the collected traffic data from multiple cycles using multiple clustering dimensions, the aim of accurately identifying abnormal traffic behavior is achieved, thereby improving the technical effect of detecting abnormal traffic behavior and solving the technical problem of low accuracy in detecting abnormal traffic behavior in related technologies.

[0100] The clustering module 62 includes a preprocessing submodule, used to cluster the traffic data to be detected using multiple clustering dimensions, including: obtaining the value of each data point in the traffic data to be detected, the mean of all data points in the traffic data to be detected, and the standard deviation of each data point in the traffic data to be detected; determining the score of each data point in the traffic data to be detected based on the value of each data point in the traffic data to be detected, the mean of all data points in the traffic data to be detected, and the standard deviation of each data point in the traffic data to be detected; identifying data points whose absolute value of the score is greater than a preset threshold as abnormal data; deleting missing values ​​in the traffic data to be detected, and re-collecting traffic data of a preset period adjacent to the abnormal data to replace the abnormal data, thereby obtaining processed traffic data; selecting target traffic data from the processed traffic data; and clustering the target traffic data using multiple clustering dimensions to obtain the multiple traffic clusters.

[0101] The preprocessing submodule includes a preprocessing unit for selecting target traffic data from the processed traffic data, including: dividing the processed traffic data into traffic data within the target area and traffic data outside the target area; obtaining the uplink traffic rate and downlink traffic rate of each traffic data in the processed traffic data; determining a preset number of traffic data with the highest uplink traffic rate and a preset number of traffic data with the highest downlink traffic rate in the traffic data within the target area as the target traffic data; and determining a preset number of traffic data with the highest uplink traffic rate and a preset number of traffic data with the highest downlink traffic rate in the traffic data outside the target area as the target traffic data.

[0102] The clustering module 62 further includes a clustering submodule, used to cluster the target traffic data using multiple clustering dimensions to obtain the multiple traffic clusters, including: obtaining the normalized feature values ​​of the target traffic data in the multiple clustering dimensions; randomly obtaining multiple cluster centers, and assigning each data point in the target traffic data to the nearest cluster center according to the normalized feature values ​​to obtain multiple initial clusters; determining the mean of all data points in the multiple initial clusters as the new cluster center, and redistributing the data points in each initial cluster until the cluster centers no longer change, to obtain the multiple traffic clusters.

[0103] The detection module 66 includes a first detection submodule, a second detection submodule, and a third detection submodule. The first detection submodule is used to detect traffic in different traffic clusters using a detection strategy corresponding to each traffic cluster, and obtain detection results. This includes: obtaining the first network address associated with the traffic data of the first cluster among the multiple traffic clusters, where the first cluster represents a cluster of traffic data whose ratio of total uplink traffic to total downlink traffic is greater than a first proportion; obtaining the network type of the network address associated with the traffic data of the first cluster; if the network type is a metropolitan area network (MAN), obtaining the first number of uplink ports with the largest traffic among the uplink ports of the first network address; and marking the first network address as a Personal Content Delivery Network (PCDN) if the ratio of the uplink traffic of the first number of uplink ports to the total uplink traffic of the first network address exceeds a first threshold, the ratio of the target end of the uplink traffic of the first network address to network addresses within the target area exceeds a second threshold, and the ratio of non-public ports among the uplink ports of the first network address exceeds a third threshold.

[0104] The second detection submodule is used to detect traffic in different traffic clusters using the detection strategy corresponding to each traffic cluster, and obtain detection results. This includes: obtaining the second network address associated with the traffic data of the second cluster among the multiple traffic clusters, wherein the second cluster represents a cluster of traffic data where uplink traffic outside the target area is higher than downlink traffic outside the target area; obtaining the network type of the network address associated with the traffic data of the second cluster; if the network type is a metropolitan area network, statistically analyzing the traffic data of the second network address within a specified period; using pre-trained seasonality prediction based on the traffic data within the specified period to predict the traffic data within a prediction period, obtaining a prediction error; and determining that the second network address is marked if the prediction error is greater than a preset error threshold and the proportion of the number of interacting network addresses within the specified period to the total number of interacting network addresses is greater than a fourth threshold.

[0105] The third detection submodule is used to detect traffic in different traffic clusters using the detection strategy corresponding to each traffic cluster, and obtain detection results, including: obtaining the third network address associated with the traffic data of the second cluster among the multiple traffic clusters; obtaining the network type of the network address associated with the traffic data of the second cluster, and if the network type is an Internet Data Center network, statistically analyzing the traffic data of the third network address within a specified period; evaluating the traffic data within the specified period using an autocorrelation function, and marking the third network address if the traffic data within the specified period shows periodic changes.

[0106] The third detection submodule includes a determination unit, used to determine the marked network address as the abnormal traffic address in the detection result.

[0107] It should be noted that, Figure 6 The abnormal flow detection device shown is used to perform Figure 2 The abnormal flow detection method shown above also applies to this abnormal flow detection device, and will not be repeated here.

[0108] This application also provides a computer device, including: a memory and a processor, wherein the memory is used to store program instructions; and the processor, connected to the memory, is used to execute the above-described abnormal traffic detection method.

[0109] This application also provides a non-volatile storage medium, which includes a stored computer program, wherein the device containing the non-volatile storage medium executes the above-described abnormal traffic detection method by running the computer program.

[0110] This application also provides a computer program product, including computer instructions that, when executed by a processor, implement the steps of the abnormal traffic detection method in this application.

[0111] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0112] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0114] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0115] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0116] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0117] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An abnormal flow detection method, characterized in that, include: Collect the flow data to be detected according to the preset collection cycle; The traffic data to be detected is clustered using multiple clustering dimensions to obtain multiple traffic clusters. The clustering dimensions include at least the ratio of total uplink traffic to total downlink traffic in the target area, the ratio of uplink traffic outside the target area to total uplink traffic, the ratio of downlink traffic within the target area to total downlink traffic, and the ratio of total traffic outside the target area to total traffic within the target area. The total uplink traffic includes the sum of uplink traffic within the target area and uplink traffic outside the target area. The total downlink traffic includes the sum of downlink traffic within the target area and downlink traffic outside the target area. Each of the multiple traffic clusters is assigned a separate detection strategy. The detection strategy corresponding to each traffic cluster is used to detect the traffic in different traffic clusters, and the detection results are obtained. The detection strategy corresponding to each traffic cluster is used to detect the traffic in different traffic clusters, and the detection results are obtained, including: Obtain the first network address associated with the traffic data of the first cluster among the plurality of traffic clusters, wherein the first cluster represents a cluster of traffic data in which the ratio of the total uplink traffic to the total downlink traffic is greater than a first proportion; Obtain the network type of the network address associated with the traffic data of the first cluster. If the network type is a metropolitan area network, obtain the first number of uplink ports with the largest traffic among the uplink ports of the first network address. If the ratio of the uplink traffic of the first number of uplink ports to the total uplink traffic of the first network address exceeds a first threshold, and the ratio of the target end of the uplink traffic of the first network address to the network address in the target area exceeds a second threshold, and the ratio of non-public ports among the uplink ports of the first network address exceeds a third threshold, then the first network address is marked as a Personal Content Delivery Network (PCDN).

2. The method according to claim 1, characterized in that, The traffic data to be detected is clustered using multiple clustering dimensions, including: Obtain the value of each data point in the traffic data to be detected, the mean of all data points in the traffic data to be detected, and the standard deviation of each data point in the traffic data to be detected; The score of each data point in the traffic data to be detected is determined based on the value of each data point in the traffic data to be detected, the mean of all data points in the traffic data to be detected, and the standard deviation of each data point in the traffic data to be detected. Data points whose absolute scores exceed a preset threshold are identified as outliers. The missing values ​​in the traffic data to be detected are deleted, and traffic data from a preset period adjacent to the abnormal data are collected to replace the abnormal data, thus obtaining the processed traffic data. Select target traffic data from the processed traffic data; The target traffic data is clustered using multiple clustering dimensions to obtain the multiple traffic clusters.

3. The method according to claim 2, characterized in that, Selecting target traffic data from the processed traffic data includes: The processed traffic data is divided into traffic data within the target area and traffic data outside the target area; Obtain the uplink and downlink flow rates of each data point in the processed traffic data; The target traffic data is determined by identifying the preset number of traffic data with the highest uplink traffic rate and the preset number of traffic data with the highest downlink traffic rate in the traffic data within the target area. The target traffic data is determined by selecting the preset number of traffic data with the highest uplink traffic rate and the preset number of traffic data with the highest downlink traffic rate from the traffic data outside the target area.

4. The method according to claim 2, characterized in that, The target traffic data is clustered using multiple clustering dimensions to obtain the multiple traffic clusters, including: Obtain the normalized feature values ​​of the target traffic data across the multiple clustering dimensions; Multiple cluster centers are randomly selected, and each data point in the target traffic data is assigned to the nearest cluster center according to the normalized feature value to obtain multiple initial clusters; The mean of all data points in the multiple initial clusters is determined as the new cluster center, and the data points in each initial cluster are redistributed until the cluster center no longer changes, thus obtaining the multiple traffic clusters.

5. The method according to claim 1, characterized in that, The detection strategy corresponding to each traffic cluster is used to detect the traffic in different traffic clusters, and the detection results are obtained, including: Obtain the second network address associated with the traffic data of the second cluster among the plurality of traffic clusters, wherein the second cluster represents a cluster composed of traffic data in which the uplink traffic outside the target area is higher than the downlink traffic outside the target area; Obtain the network type of the network address associated with the traffic data of the second cluster. If the network type is a metropolitan area network, count the traffic data of the second network address within a specified period. The prediction error is obtained by using pre-trained seasonal data based on traffic data within the specified period to predict traffic data within the prediction period. If the prediction error is greater than a preset error threshold and the proportion of the number of network addresses interacting within the specified period to the total number of network addresses interacting is greater than a fourth threshold, then the second network address is determined to be marked.

6. The method according to claim 5, characterized in that, The detection strategy corresponding to each traffic cluster is used to detect the traffic in different traffic clusters, and the detection results are obtained, including: Obtain the third network address associated with the traffic data of the second cluster among the plurality of traffic clusters; Obtain the network type of the network address associated with the traffic data of the second cluster; if the network type is an Internet Data Center network, count the traffic data of the third network address within a specified period. The traffic data within the specified period is evaluated using an autocorrelation function, and the third network address is marked if the traffic data within the specified period shows periodic changes.

7. The method according to claim 6, characterized in that, The method further includes: The network address that is marked is identified as the abnormal traffic address in the detection results.

8. An abnormal flow detection device, characterized in that, include: The data acquisition module is used to collect traffic data to be detected according to a preset acquisition cycle; The clustering module is used to cluster the traffic data to be detected using multiple clustering dimensions to obtain multiple traffic clusters. The clustering dimensions include at least the ratio of the total uplink traffic to the total downlink traffic in the target area, the ratio of the uplink traffic outside the target area to the total uplink traffic, the ratio of the downlink traffic within the target area to the total downlink traffic, and the ratio of the total traffic outside the target area to the total traffic within the target area. The total uplink traffic includes the sum of the uplink traffic within the target area and the uplink traffic outside the target area. The total downlink traffic includes the sum of the downlink traffic within the target area and the downlink traffic outside the target area. The determination module is used to determine multiple detection strategies corresponding to the multiple traffic clusters respectively; The detection module is used to detect the traffic in different traffic clusters using the detection strategy corresponding to each traffic cluster, and obtain the detection results. The detection module is further configured to obtain a first network address associated with the traffic data of a first cluster among the plurality of traffic clusters, wherein the first cluster represents a cluster of traffic data in which the ratio of the total uplink traffic to the total downlink traffic is greater than a first proportion; obtain the network type of the network address associated with the traffic data of the first cluster; and, if the network type is a metropolitan area network, obtain a first number of uplink ports with the largest traffic among the uplink ports of the first network address; and mark the first network address as a Personal Content Delivery Network (PCDN) if the ratio of the uplink traffic of the first number of uplink ports to the total uplink traffic of the first network address exceeds a first threshold, the ratio of the target end of the uplink traffic of the first network address to the network address within the target area exceeds a second threshold, and the ratio of non-public ports among the uplink ports of the first network address exceeds a third threshold.

9. A computer device, characterized in that, include: A memory and a processor, wherein the memory is used to store program instructions; The processor, connected to the memory, is used to execute the abnormal traffic detection method according to any one of claims 1 to 7.

10. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the abnormal traffic detection method according to any one of claims 1 to 7.

Images