Abnormal traffic detection method and device and computer equipment

Abstract

The invention discloses an abnormal traffic detection method and device and computer equipment. The method comprises the following steps: collecting to-be-detected flow data according to a preset collection period; clustering the traffic data to be detected by adopting a plurality of clustering dimensions to obtain a plurality of traffic clusters, the clustering dimension at least comprises a ratio of the total uplink flow of the target area to the total downlink flow of the target area, a ratio of the uplink flow outside the target area to the total uplink flow, a ratio of the downlink flow inside the target area to the total downlink flow, and a ratio of the total flow outside the target area to the total flow inside the target area; respectively determining a plurality of detection strategies corresponding to the plurality of traffic clusters; and detecting the traffic in different traffic clusters by adopting the detection strategy corresponding to each traffic cluster to obtain a detection result.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to an abnormal traffic detection method, device and computer equipment. Background Art

[0002] In communication networks, network traffic monitoring and anomaly detection are key tasks to ensure network stability and security. With the rapid development of the Internet and various communication technologies, the complexity and scale of network traffic are increasing. Traditional traffic monitoring methods are difficult to effectively deal with inter-provincial traffic anomalies. In particular, abnormal fluctuations in upstream and downstream traffic may lead to network congestion, reduced service quality, and even network security incidents.

[0003] The main abnormal behaviors include traffic being pulled between different regions and PCDN (Personal Content Delivery Network) services. Traffic being pulled between different regions may lead to resource occupation and waste of network resources, affect the normal services of other users, increase manpower and technical costs, and increase the complexity of network management. PCDN services will lead to increased bandwidth consumption and involve unauthorized content distribution, increasing the risk of content leakage and infringement. Traditional traffic abnormal behavior detection methods mostly rely on manual determination of traffic baselines and logical judgments. When faced with complex and large-scale network traffic, the detection rate is low and it is easy to false positives and omissions. At the same time, it is impossible to monitor and analyze network traffic in real time and has a lag; the adaptability to the complex network environment and diversified abnormal behaviors of operators is weak, and the rich information of network traffic logs cannot be fully mined and utilized. Summary of the invention

[0004] The embodiments of the present application provide a method, apparatus and computer device for detecting abnormal traffic, so as to at least solve the technical problem of low accuracy in detecting abnormal traffic behavior in the related art.

[0005] According to one aspect of an embodiment of the present application, there is provided an abnormal traffic detection method, comprising: collecting traffic data to be detected according to a preset collection period; clustering the traffic data to be detected using multiple clustering dimensions to obtain multiple traffic clusters, wherein the clustering dimensions include at least the ratio of the total upstream traffic in a target area to the total downstream traffic in the target area, the ratio of the upstream traffic outside the target area to the total upstream traffic, the ratio of the downlink traffic in the target area to the total downlink traffic, and the ratio of the total traffic outside the target area to the total traffic in the target area, the total upstream traffic includes the sum of the upstream traffic in the target area and the upstream traffic outside the target area, and the total downlink traffic includes: the sum of the downlink traffic in the target area and the downlink traffic outside the target area; respectively determining multiple detection strategies corresponding to the multiple traffic clusters; respectively using the detection strategy corresponding to each traffic cluster to detect the traffic in different traffic clusters to obtain detection results.

[0006] Optionally, multiple clustering dimensions are used to cluster the traffic data to be detected, including: obtaining the value of each data point in the traffic data to be detected, the mean of all data points in the traffic data to be detected, and the standard deviation of each data point in the traffic data to be detected; determining the score of each data point in the traffic data to be detected according to the value of each data point in the traffic data to be detected, the mean of all data points in the traffic data to be detected, and the standard deviation of each data point in the traffic data to be detected; determining the data point whose absolute value of the score is greater than a preset threshold as abnormal data; deleting the missing values ​​in the traffic data to be detected, and re-collecting the traffic data of a preset period adjacent to the abnormal data to replace the abnormal data to obtain processed traffic data; selecting target traffic data from the processed traffic data; clustering the target traffic data using multiple clustering dimensions to obtain the multiple traffic clusters.

[0007] Optionally, selecting target traffic data from the processed traffic data includes: dividing the processed traffic data into traffic data within the target area and traffic data outside the target area; obtaining the uplink traffic rate and the downlink traffic rate of each traffic data in the processed traffic data; respectively determining a preset number of traffic data pieces with the highest uplink traffic rate and a preset number of traffic data pieces with the highest downlink traffic rate in the traffic data within the target area as the target traffic data; respectively determining a preset number of traffic data pieces with the highest uplink traffic rate and a preset number of traffic data pieces with the highest downlink traffic rate in the traffic data outside the target area as the target traffic data.

[0008] Optionally, the target traffic data is clustered using multiple clustering dimensions to obtain the multiple traffic clusters, including: obtaining normalized eigenvalues ​​of the target traffic data in the multiple clustering dimensions; randomly obtaining multiple cluster centers, and assigning each data point in the target traffic data to the nearest cluster center according to the normalized eigenvalues ​​to obtain multiple initial clusters; determining the mean of all data points in the multiple initial clusters as a new cluster center, and redistributing the data points in each initial cluster until the cluster center no longer changes, to obtain the multiple traffic clusters.

[0009] Optionally, the detection strategies corresponding to each traffic cluster are respectively used to detect the traffic in different traffic clusters to obtain detection results, including: obtaining a first network address associated with the traffic data of a first cluster among the multiple traffic clusters, wherein the first cluster represents a cluster composed of traffic data whose ratio of the total upstream traffic to the total downstream traffic is greater than a first ratio; obtaining the network type of the network address associated with the traffic data of the first cluster, and when the network type is a metropolitan area network, obtaining a first number of upstream ports with the largest traffic among the upstream ports of the first network address; marking the first network address as a personal content distribution network PCDN when the proportion of the upstream traffic of the first number of upstream ports to the total upstream traffic of the first network address exceeds a first threshold and the proportion of the target end of the upstream traffic of the first network address to the network address in the target area exceeds a second threshold and the proportion of non-public ports among the upstream ports of the first network address exceeds a third threshold.

[0010] Optionally, the detection strategies corresponding to each traffic cluster are respectively used to detect the traffic in different traffic clusters to obtain detection results, including: obtaining a second network address associated with the traffic data of a second cluster among the multiple traffic clusters, wherein the second cluster represents a cluster composed of traffic data in which the upstream traffic outside the target area is higher than the downstream traffic outside the target area; obtaining the network type of the network address associated with the traffic data of the second cluster, and when the network type is a metropolitan area network, counting the traffic data of the second network address within a specified period; using pre-trained seasonality to predict the traffic data within a prediction period based on the traffic data within the specified period to obtain a prediction error; and determining that the second network address is marked when the prediction error is greater than a preset error threshold and the proportion of the number of network addresses interacting within the specified period to the total number of interacting network addresses is greater than a fourth threshold.

[0011] Optionally, the detection strategies corresponding to each traffic cluster are respectively used to detect the traffic in different traffic clusters to obtain detection results, including: obtaining a third network address associated with the traffic data of the second cluster among the multiple traffic clusters; obtaining the network type of the network address associated with the traffic data of the second cluster, and when the network type is an Internet data center network, counting the traffic data of the third network address within a specified period; using an autocorrelation function to evaluate the traffic data within the specified period, and when there is periodic change in the traffic data within the specified period, marking the third network address.

[0012] Optionally, the method further includes: determining the marked network address as an abnormal traffic address in the detection result.

[0013] According to another aspect of an embodiment of the present application, an abnormal traffic detection device is also provided, including: a collection module, used to collect traffic data to be detected according to a preset collection period; a clustering module, used to cluster the traffic data to be detected using multiple clustering dimensions to obtain multiple traffic clusters, wherein the clustering dimensions at least include the ratio of the total upstream traffic in the target area to the total downstream traffic in the target area, the ratio of the upstream traffic outside the target area to the total upstream traffic, the ratio of the downlink traffic in the target area to the total downlink traffic, and the ratio of the total traffic outside the target area to the total traffic in the target area, the total upstream traffic includes the sum of the upstream traffic in the target area and the upstream traffic outside the target area, and the total downlink traffic includes: the sum of the downlink traffic in the target area and the downlink traffic outside the target area; a determination module, used to respectively determine multiple detection strategies corresponding to the multiple traffic clusters; a detection module, used to respectively use the detection strategy corresponding to each traffic cluster to detect the traffic in different traffic clusters to obtain detection results.

[0014] According to another aspect of the embodiment of the present application, a computer device is provided, including: a memory and a processor, wherein the memory is used to store program instructions; and the processor is connected to the memory and is used to execute the above-mentioned abnormal traffic detection method.

[0015] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned abnormal traffic detection method by running the computer program.

[0016] According to another aspect of the embodiments of the present application, a computer program product is provided, including computer instructions, which implement the above-mentioned abnormal traffic detection method when executed by a processor.

[0017] In an embodiment of the present application, traffic data to be detected is collected according to a preset collection period; the traffic data to be detected is clustered using multiple clustering dimensions to obtain multiple traffic clusters, wherein the clustering dimensions at least include the ratio of the total upstream traffic in the target area to the total downstream traffic in the target area, the ratio of the upstream traffic outside the target area to the total upstream traffic, the ratio of the downlink traffic in the target area to the total downlink traffic, and the ratio of the total traffic outside the target area to the total traffic in the target area, the total upstream traffic includes the sum of the upstream traffic in the target area and the upstream traffic outside the target area, and the total downlink traffic includes: the sum of the downlink traffic in the target area and the downlink traffic outside the target area; multiple detection strategies corresponding to the multiple traffic clusters are determined respectively; the detection strategies corresponding to each traffic cluster are respectively used to detect the traffic in different traffic clusters to obtain detection results, and the traffic data of multiple periods collected are analyzed through multiple clustering dimensions, so as to achieve the purpose of accurately identifying abnormal traffic behavior, thereby achieving the technical effect of improving the detection of abnormal traffic behavior, and thus solving the technical problem of low accuracy in detecting abnormal traffic behavior in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 It is a hardware structure block diagram of a computer terminal for implementing an abnormal traffic detection method according to an embodiment of the present application;

[0020] Figure 2 is a flow chart of an abnormal traffic detection method according to an embodiment of the present application;

[0021] Figure 3 is a flow data prediction timing diagram according to an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of a flow data autocorrelation detection result according to an embodiment of the present application;

[0023] Figure 5 is a flow chart of a method for detecting abnormal traffic behavior according to an embodiment of the present application;

[0024] Figure 6 It is a structural diagram of an abnormal flow detection device according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] The information collected in the embodiments of the present application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or reject automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered.

[0028] In order to solve the problems existing in the related art, the embodiment of the present application provides a method for detecting abnormal traffic flow, which can be run on Figure 1 In the computer terminal shown, the computer terminal is explained below.

[0029] The abnormal traffic detection method embodiment provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing an abnormal traffic detection method. Figure 1As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art can understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0030] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0031] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the abnormal flow detection method in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned abnormal flow detection method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0032] The transmission module 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0033] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0034] It should be noted that, in some optional embodiments, the above Figure 1 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0035] In the above-mentioned operating environment, an embodiment of the present application provides an embodiment of an abnormal traffic detection method. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0036] Figure 2 is a flow chart of an abnormal traffic detection method according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0037] Step S202, collecting the flow data to be detected according to a preset collection period;

[0038] In step S202, the preset collection period includes but is not limited to: 5 minutes, 1 hour and 24 hours.

[0039] To detect abnormal behavior more quickly, 5 minutes may be used.

[0040] Step S204, clustering the traffic data to be detected using multiple clustering dimensions to obtain multiple traffic clusters, wherein the clustering dimensions at least include the ratio of the total uplink traffic in the target area to the total downlink traffic in the target area, the ratio of the uplink traffic outside the target area to the total uplink traffic, the ratio of the downlink traffic in the target area to the total downlink traffic, and the ratio of the total traffic outside the target area to the total traffic in the target area, the total uplink traffic includes the sum of the uplink traffic in the target area and the uplink traffic outside the target area, and the total downlink traffic includes: the sum of the downlink traffic in the target area and the downlink traffic outside the target area;

[0041] It should be noted that the target area can be set according to actual needs, for example: provincial administrative area, within the target area (within the province), outside the target area (outside the province).

[0042] Furthermore, there are multiple dimensional features in the data table, including source city code, target city code, source province code, target province code, traffic type, out-of-province upstream traffic rate, out-of-province downstream traffic rate, intra-province upstream traffic rate, intra-province downstream traffic rate, total upstream traffic rate, total downstream traffic rate, etc. Correspondingly, the source and target refer to the start and end points of the traffic respectively; traffic types include IDC traffic and metropolitan area network traffic, IDC (Internet Data Center) traffic refers to the inter-provincial interactive traffic of the operator's IDC, and metropolitan area network traffic refers to the uplink and downlink traffic of the operator's intra-province metropolitan area network; the traffic rate unit is Gbps.

[0043] Step S206, determining a plurality of detection strategies corresponding to the plurality of traffic clusters respectively;

[0044] Step S208, respectively use the detection strategy corresponding to each traffic cluster to detect the traffic in different traffic clusters to obtain detection results.

[0045] Through the above steps S202 to S208, the traffic data to be detected is collected according to the preset collection period; the traffic data to be detected is clustered using multiple clustering dimensions to obtain multiple traffic clusters, wherein the clustering dimensions at least include the ratio of the total upstream traffic in the target area to the total downstream traffic in the target area, the ratio of the upstream traffic outside the target area to the total upstream traffic, the ratio of the downstream traffic in the target area to the total downstream traffic, and the ratio of the total traffic outside the target area to the total traffic in the target area, the total upstream traffic includes the sum of the upstream traffic in the target area and the upstream traffic outside the target area, and the total downstream traffic includes: the sum of the downlink traffic in the target area and the downlink traffic outside the target area; multiple detection strategies corresponding to the multiple traffic clusters are determined respectively; the detection strategies corresponding to each traffic cluster are respectively used to detect the traffic in different traffic clusters to obtain detection results, and the traffic data of multiple periods collected are analyzed through multiple clustering dimensions, so as to achieve the purpose of accurately identifying abnormal traffic behavior, thereby achieving the technical effect of improving the detection of abnormal traffic behavior, and thus solving the technical problem of low accuracy of abnormal traffic behavior detection in related technologies. The following details are given.

[0046] In some embodiments of the present application, the specific steps of clustering the traffic data to be detected using multiple clustering dimensions are as follows: obtaining the value of each data point in the traffic data to be detected, the mean of all data points in the traffic data to be detected, and the standard deviation of each data point in the traffic data to be detected; determining the score of each data point in the traffic data to be detected according to the value of each data point in the traffic data to be detected, the mean of all data points in the traffic data to be detected, and the standard deviation of each data point in the traffic data to be detected; determining the data point whose absolute value of the score is greater than a preset threshold as abnormal data; deleting the missing values ​​in the traffic data to be detected, and re-collecting the traffic data of a preset period adjacent to the abnormal data to replace the abnormal data, so as to obtain processed traffic data; selecting target traffic data from the processed traffic data; clustering the target traffic data using multiple clustering dimensions to obtain the multiple traffic clusters.

[0047] Among them, the specific steps of selecting target flow data from the processed flow data are as follows: dividing the processed flow data into flow data within the target area and flow data outside the target area; obtaining the uplink flow rate and the downlink flow rate of each flow data in the processed flow data; respectively determining a preset number of flow data with the highest uplink flow rate and a preset number of flow data with the highest downlink flow rate in the flow data within the target area as the target flow data; respectively determining a preset number of flow data with the highest uplink flow rate and a preset number of flow data with the highest downlink flow rate in the flow data outside the target area as the target flow data.

[0048] Specifically, there are abnormal values ​​or missing values ​​in the traffic data. For missing values, choose to delete them directly, because the data obtained by the traffic collection system is the average data of 5 minutes, which means that no data has been collected for 5 consecutive minutes and will not be analyzed for the time being; for abnormal values, use the Z-Score method to identify and repair abnormal values. The formula is Where z is the standard score, x is the traffic data point, μ is the mean of the traffic data set, and σ is the standard deviation of the traffic data set. If |z|>k, usually k is set to 3, it is considered an outlier. Select the traffic data of the last hour, delete the data that is considered an outlier, and re-obtain the traffic clustering results of the last 5 minutes.

[0049] The preprocessed data is classified into intra-provincial traffic and extra-provincial traffic, and sorted according to the upstream traffic rate and downstream traffic rate respectively. The top N records of upstream traffic and downstream traffic are selected as subsequent analysis data, specifically, N is 1000.

[0050] data topn = Get the top 1000 records with the highest upstream and downstream traffic respectively

[0051] Furthermore, 1,000 records of upstream and downstream traffic within the province were screened, and 1,000 records of upstream and downstream traffic outside the province were screened. A total of 4,000 data records were merged and subsequently analyzed in a unified manner.

[0052] In some embodiments of the present application, the target traffic data is clustered using multiple clustering dimensions, and the specific steps for obtaining the multiple traffic clusters include: obtaining normalized eigenvalues ​​of the target traffic data in the multiple clustering dimensions; randomly obtaining multiple cluster centers, and assigning each data point in the target traffic data to the nearest cluster center according to the normalized eigenvalues, to obtain multiple initial clusters; determining the mean of all data points in the multiple initial clusters as a new cluster center, and redistributing the data points in each initial cluster until the cluster center no longer changes, to obtain the multiple traffic clusters.

[0053] Taking the target area as a provincial administrative region as an example, the clustering dimensions include: total upstream traffic in the target area / total downstream traffic in the target area, upstream traffic outside the province / total upstream traffic, downstream traffic within the province / total downstream traffic, and total traffic outside the province / total traffic within the province.

[0054] Optionally, an unsupervised learning algorithm may be used to cluster the traffic data, and the data set may be divided into K clusters, so that the similarity between data points within a cluster is as high as possible, while the similarity between different clusters is as low as possible.

[0055] The goal of the K-means algorithm is to minimize the sum of squared Euclidean distances from data points within a cluster to the cluster center. The objective function is as follows:

[0056]

[0057] Among them, J represents the objective function, k represents the kth cluster center, K represents the number of target cluster centers, and x i represents the i-th data point, C k represents all data points assigned to the kth cluster center, j represents the jth feature of the data point, and m represents the data point x i The total number of features, x ij Represents data point x i The jth eigenvalue of kj Represents the j-th eigenvalue of the k-th cluster center.

[0058] The clustering process is as follows:

[0059] Normalize the characteristic values ​​of multiple dimensions of each data point (traffic data) to obtain normalized characteristic values;

[0060] Randomly select multiple cluster centers, for example: 4.

[0061] Assign each data point to the nearest cluster center to form multiple clusters, as shown in the following formula:

[0062] C j ={xi :‖x i -u j ‖ 2 ≤‖x i -u k ‖ 2 for all k}

[0063] Among them, c j represents the cluster set, x i represents the i-th data point, u j represents the jth cluster center, u k There are k cluster centers in total, and the cluster with the smallest distance is selected from the k cluster centers as the cluster of data points.

[0064] Calculate the mean of all data points in each cluster as the new cluster center:

[0065]

[0066] Among them, u j is the new cluster center, C j represents the cluster set, x i Represents a data point.

[0067] Repeat the above steps until the cluster center no longer changes or the change reaches a threshold and then stop the iteration.

[0068] Take the clustering result with 5 clusters as an example, specifically:

[0069] Cluster 1 (the first cluster): represents traffic data in which the ratio of total upstream traffic to total downstream traffic is greater than the first ratio, indicating that the outbound traffic in this area is large.

[0070] Cluster 2 (second cluster): represents a cluster in which the out-of-province upstream traffic is higher than the out-of-province downstream traffic.

[0071] Cluster 3: represents the traffic data where the ratio of the out-of-province uplink traffic to the total uplink traffic is greater than the second ratio, indicating that the uplink traffic in this area mainly flows to other provinces.

[0072] Cluster 4: represents traffic data where the ratio of the downstream traffic within the province to the total downstream traffic is greater than the third ratio, indicating that the downstream traffic in this area is mainly concentrated within the province.

[0073] Cluster 5: represents the traffic behavior of high out-of-province total traffic / in-province total traffic, indicating that the total traffic in this area mainly flows to other provinces.

[0074] The clustering results can be used to further determine abnormal behaviors in each cluster, and combined with the period prediction algorithm and timing prediction algorithm, traffic anomalies can be identified and warned to ensure network stability and security.

[0075] The specific detection method is as follows: obtaining a first network address associated with the traffic data of a first cluster among the multiple traffic clusters, wherein the first cluster represents a cluster composed of traffic data whose ratio of the total upstream traffic to the total downstream traffic is greater than a first ratio; obtaining a network type of the network address associated with the traffic data of the first cluster, and when the network type is a metropolitan area network, obtaining a first number of upstream ports with the largest traffic among the upstream ports of the first network address; when the proportion of the upstream traffic of the first number of upstream ports to the total upstream traffic of the first network address exceeds a first threshold and the proportion of the target end of the upstream traffic of the first network address to the network address in the target area exceeds a second threshold and the proportion of non-public ports among the upstream ports of the first network address exceeds a third threshold, marking the first network address as a personal content distribution network PCDN.

[0076] Specifically, for a user IP (first network address) in a metropolitan area network and belonging to the first cluster, the following judgment is performed, specifically:

[0077] If the uplink port traffic of the top 10 user IP statistics (first quantity) accounts for more than 80% of the total uplink traffic (first threshold), and the proportion of the target IP address within the province accounts for more than 70% of all target IP addresses (second threshold), and the proportion of the user's uplink service port that is a non-public port accounts for more than 50% of all uplink ports (third threshold), then the user IP is judged to be PCDN and marked as important;

[0078] In another case, a second network address associated with the traffic data of a second cluster among the multiple traffic clusters is obtained, wherein the second cluster represents a cluster composed of traffic data in which the upstream traffic outside the target area is higher than the downstream traffic outside the target area; the network type of the network address associated with the traffic data of the second cluster is obtained, and when the network type is a metropolitan area network, the traffic data of the second network address within a specified period is counted; the traffic data within a prediction period is predicted based on the traffic data within the specified period using a pre-trained seasonality to obtain a prediction error; and when the prediction error is greater than a preset error threshold and the proportion of the number of network addresses interacting within the specified period to the total number of interacting network addresses is greater than a fourth threshold, it is determined that the second network address is marked.

[0079] Specifically, for the user IP (second network address) in the metropolitan area network and belonging to the second cluster, the traffic data of the user IP in the past 7 days (specified period) is counted, and the ARIMA algorithm is used to detect whether there is an abnormal surge in traffic.

[0080] The ARIMA model has the form:

[0081]

[0082] Among them, y t is the value of the time series at time t, c is a constant term, is the autoregressive coefficient, θ i is the moving average coefficient, ∈ t is the white noise error term.

[0083] According to the nature of the time series, the number of differences is determined to be 1, and the autocorrelation function (ACF) and partial autocorrelation function (PACF) graphs are used to determine the number of autoregressive terms p and moving average terms q, where p and q are both 2.

[0084] Furthermore, the trained ARIMA model is used to predict future traffic, such as Figure 3 As shown, blue is the actual traffic data (true), red is the predicted traffic data (predict), and the prediction error is calculated. If the prediction error is greater than 3 times the standardized residual, the data within the specified statistical period is considered to be a sudden increase in data.

[0085] If the user IP has a sudden increase in the past 7 days, and the number of IPs interacting within the specified period has a sudden increase in the specified period, and the proportion of the number of IPs interacting within the specified period to the total number of interacting IPs is greater than 70% (the fourth threshold), and the number of interacting IPs is relatively frequent, then the user IP is considered to be pulled and marked as such.

[0086] In other embodiments of the present application, a third network address associated with the traffic data of a second cluster among the multiple traffic clusters is obtained; a network type of the network address associated with the traffic data of the second cluster is obtained, and when the network type is an Internet data center network, traffic data of the third network address within a specified period is counted; an autocorrelation function is used to evaluate the traffic data within the specified period, and when there is periodic change in the traffic data within the specified period, the third network address is marked.

[0087] Specifically, the user IP (third network address) of the IDC and belonging to the second cluster;

[0088] To determine whether the traffic data of the past 7 days is periodic, we use the autocorrelation function (ACF). Specifically:

[0089]

[0090] Among them, ρ k is the autocorrelation coefficient with lag period k, x t is the tth sample of the time series, is the mean, and N is the total number of samples in the time series.

[0091] If the autocorrelation coefficient is significantly higher in some lag periods, such as 24 hours, than in other lag periods, it indicates that the time series has periodicity. If the traffic data has no obvious periodicity in the past 7 days and the traffic data has remained at a large level, there is a possibility that the user has been pulled. Mark it for key analysis, such as Figure 4 As shown, the self-period detection results of several types of traffic data are shown.

[0092] The marked network address is finally determined as the abnormal traffic address in the detection result. In actual application scenarios, the marked IP can be manually verified to finally determine the abnormal traffic.

[0093] Figure 5 Another abnormal traffic behavior detection method is shown, such as Figure 5 As shown, step 1: connect to the NetFlow (a traffic collection platform) traffic collection platform to obtain traffic clustering data from the 5-minute clustering table; step 2: further process the basic data obtained, including Top-N screening, data cleaning preprocessing, etc.; step 3: cluster the TOP data, and the clustering dimensions include total upstream traffic / total downstream traffic, upstream traffic outside the province / total upstream traffic, downstream traffic within the province / total downstream traffic, total traffic outside the province / total traffic within the province; step 4 uses different discrimination methods to judge the user IPs in different clusters and identify abnormal traffic.

[0094] Figure 6 An abnormal flow detection device according to an embodiment of the present application includes:

[0095] The collection module 60 is used to collect the flow data to be detected according to a preset collection period;

[0096] A clustering module 62 is used to cluster the traffic data to be detected using multiple clustering dimensions to obtain multiple traffic clusters, wherein the clustering dimensions at least include the ratio of the total uplink traffic in the target area to the total downlink traffic in the target area, the ratio of the uplink traffic outside the target area to the total uplink traffic, the ratio of the downlink traffic in the target area to the total downlink traffic, and the ratio of the total traffic outside the target area to the total traffic in the target area, the total uplink traffic includes the sum of the uplink traffic in the target area and the uplink traffic outside the target area, and the total downlink traffic includes: the sum of the downlink traffic in the target area and the downlink traffic outside the target area;

[0097] A determination module 64, configured to respectively determine a plurality of detection strategies corresponding to the plurality of traffic clusters;

[0098] The detection module 66 is used to detect the traffic in different traffic clusters using the detection strategy corresponding to each traffic cluster to obtain the detection result.

[0099] Through the above-mentioned abnormal traffic detection device, the traffic data to be detected is collected according to the preset collection period; the traffic data to be detected is clustered by using multiple clustering dimensions to obtain multiple traffic clusters, wherein the clustering dimensions at least include the ratio of the total upstream traffic of the target area to the total downstream traffic of the target area, the ratio of the upstream traffic outside the target area to the total upstream traffic, the ratio of the downstream traffic in the target area to the total downstream traffic, and the ratio of the total traffic outside the target area to the total traffic in the target area, the total upstream traffic includes the sum of the upstream traffic in the target area and the upstream traffic outside the target area, and the total downstream traffic includes: the sum of the downlink traffic in the target area and the downlink traffic outside the target area; multiple detection strategies corresponding to the multiple traffic clusters are determined respectively; the detection strategies corresponding to each traffic cluster are respectively used to detect the traffic in different traffic clusters to obtain detection results, and the traffic data of multiple periods collected are analyzed by using multiple clustering dimensions, so as to achieve the purpose of accurately identifying abnormal traffic behavior, thereby achieving the technical effect of improving the detection of abnormal traffic behavior, and thus solving the technical problem of low accuracy of abnormal traffic behavior detection in related technologies.

[0100] The clustering module 62 includes: a preprocessing submodule, which is used to cluster the traffic data to be detected using multiple clustering dimensions, including: obtaining the value of each data point in the traffic data to be detected, the mean of all data points in the traffic data to be detected, and the standard deviation of each data point in the traffic data to be detected; determining the score of each data point in the traffic data to be detected according to the value of each data point in the traffic data to be detected, the mean of all data points in the traffic data to be detected, and the standard deviation of each data point in the traffic data to be detected; determining the data point whose absolute value of the score is greater than a preset threshold as abnormal data; deleting the missing values ​​in the traffic data to be detected, and re-collecting the traffic data of the preset period adjacent to the abnormal data to replace the abnormal data to obtain the processed traffic data; selecting the target traffic data from the processed traffic data; clustering the target traffic data using multiple clustering dimensions to obtain the multiple traffic clusters.

[0101] The preprocessing submodule includes: a preprocessing unit, which is used to select target flow data from the processed flow data, including: dividing the processed flow data into flow data within the target area and flow data outside the target area; obtaining the uplink flow rate and the downlink flow rate of each flow data in the processed flow data; respectively determining a preset number of flow data with the highest uplink flow rate and a preset number of flow data with the highest downlink flow rate in the flow data within the target area as the target flow data; respectively determining a preset number of flow data with the highest uplink flow rate and a preset number of flow data with the highest downlink flow rate in the flow data outside the target area as the target flow data.

[0102] The clustering module 62 also includes: a clustering submodule, which is used to cluster the target traffic data using multiple clustering dimensions to obtain the multiple traffic clusters, including: obtaining the normalized characteristic values ​​of the target traffic data in the multiple clustering dimensions; randomly obtaining multiple cluster centers, and assigning each data point in the target traffic data to the nearest cluster center according to the normalized characteristic values ​​to obtain multiple initial clusters; determining the mean of all data points in the multiple initial clusters as a new cluster center, and reallocating the data points in each initial cluster until the cluster center no longer changes, to obtain the multiple traffic clusters.

[0103] The detection module 66 includes: a first detection submodule, a second detection submodule and a third detection submodule, wherein the first detection submodule is used to respectively detect the traffic in different traffic clusters by using the detection strategy corresponding to each traffic cluster to obtain the detection result, including: obtaining a first network address associated with the traffic data of a first cluster among the multiple traffic clusters, wherein the first cluster represents a cluster composed of traffic data whose ratio of the total upstream traffic to the total downstream traffic is greater than a first ratio; obtaining the network type of the network address associated with the traffic data of the first cluster, and when the network type is a metropolitan area network, obtaining a first number of upstream ports with the largest traffic among the upstream ports of the first network address; marking the first network address as a personal content distribution network PCDN when the proportion of the upstream traffic of the first number of upstream ports to the total upstream traffic of the first network address exceeds a first threshold and the proportion of the target end of the upstream traffic of the first network address to the network address in the target area exceeds a second threshold and the proportion of non-public ports among the upstream ports of the first network address exceeds a third threshold.

[0104] The second detection submodule is used to detect the traffic in different traffic clusters by respectively adopting the detection strategy corresponding to each traffic cluster to obtain the detection result, including: obtaining the second network address associated with the traffic data of the second cluster in the multiple traffic clusters, wherein the second cluster represents a cluster composed of traffic data in which the upstream traffic outside the target area is higher than the downstream traffic outside the target area; obtaining the network type of the network address associated with the traffic data of the second cluster, and when the network type is a metropolitan area network, counting the traffic data of the second network address within a specified period; using pre-trained seasonality to predict the traffic data within a prediction period based on the traffic data within the specified period to obtain a prediction error; and determining that the second network address is marked when the prediction error is greater than a preset error threshold and the proportion of the number of network addresses interacting within the specified period to the total number of network addresses interacting is greater than a fourth threshold.

[0105] The third detection submodule is used to detect the traffic in different traffic clusters by respectively adopting the detection strategy corresponding to each traffic cluster to obtain the detection result, including: obtaining the third network address associated with the traffic data of the second cluster among the multiple traffic clusters; obtaining the network type of the network address associated with the traffic data of the second cluster, and when the network type is an Internet data center network, counting the traffic data of the third network address within a specified period; using an autocorrelation function to evaluate the traffic data within the specified period, and when there is periodic change in the traffic data within the specified period, marking the third network address.

[0106] The third detection submodule includes: a determination unit, configured to determine the marked network address as the abnormal traffic address in the detection result.

[0107] It should be noted that Figure 6 The abnormal flow detection device shown is used to perform Figure 2 The abnormal flow detection method shown in the figure, therefore the relevant explanations in the above abnormal flow detection method are also applicable to the abnormal flow detection device, and will not be repeated here.

[0108] An embodiment of the present application also provides a computer device, including: a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the above-mentioned abnormal traffic detection method.

[0109] An embodiment of the present application further provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned abnormal traffic detection method by running the computer program.

[0110] An embodiment of the present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the abnormal traffic detection method in the present application.

[0111] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0112] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0114] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0115] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0116] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0117] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for detecting abnormal traffic, characterized in that: include: Collect the flow data to be tested according to the preset collection period; Clustering the traffic data to be detected using multiple clustering dimensions to obtain multiple traffic clusters, wherein the clustering dimensions at least include the ratio of the total uplink traffic in the target area to the total downlink traffic in the target area, the ratio of the uplink traffic outside the target area to the total uplink traffic, the ratio of the downlink traffic in the target area to the total downlink traffic, and the ratio of the total traffic outside the target area to the total traffic in the target area, the total uplink traffic includes the sum of the uplink traffic in the target area and the uplink traffic outside the target area, and the total downlink traffic includes: the sum of the downlink traffic in the target area and the downlink traffic outside the target area; Determining a plurality of detection strategies corresponding to the plurality of traffic clusters respectively; The detection strategy corresponding to each traffic cluster is used to detect the traffic in different traffic clusters to obtain the detection results.

2. The method according to claim 1, characterized in that The traffic data to be detected is clustered using multiple clustering dimensions, including: Obtaining the value of each data point in the flow data to be detected, the mean of all data points in the flow data to be detected, and the standard deviation of each data point in the flow data to be detected; Determine a score for each data point in the traffic data to be detected according to the value of each data point in the traffic data to be detected, the mean of all data points in the traffic data to be detected, and the standard deviation of each data point in the traffic data to be detected; The data points whose absolute values ​​of scores are greater than a preset threshold are determined as abnormal data; Deleting missing values ​​in the flow data to be detected, and recollecting flow data of a preset period adjacent to the abnormal data to replace the abnormal data, so as to obtain processed flow data; Selecting target flow data from the processed flow data; The target traffic data is clustered using multiple clustering dimensions to obtain the multiple traffic clusters.

3. The method according to claim 2, characterized in that Selecting target flow data from the processed flow data includes: dividing the processed flow data into flow data within the target area and flow data outside the target area; Obtaining an uplink flow rate and a downlink flow rate of each flow data in the processed flow data; Respectively determining a preset number of pieces of traffic data with the highest uplink traffic rate and a preset number of pieces of traffic data with the highest downlink traffic rate among the traffic data in the target area as the target traffic data; A preset number of pieces of traffic data with the highest uplink traffic rate and a preset number of pieces of traffic data with the highest downlink traffic rate in the traffic data outside the target area are respectively determined as the target traffic data.

4. The method according to claim 2, characterized in that: Clustering the target traffic data using multiple clustering dimensions to obtain the multiple traffic clusters includes: Obtaining normalized characteristic values ​​of the target traffic data in the multiple clustering dimensions; Randomly obtain multiple cluster centers, and assign each data point in the target traffic data to the cluster center with the closest distance according to the normalized characteristic value, so as to obtain multiple initial clusters; The mean of all data points in the multiple initial clusters is determined as a new cluster center, and the data points in each initial cluster are redistributed until the cluster center no longer changes, thereby obtaining the multiple traffic clusters.

5. The method according to claim 1, characterized in that Use the detection strategy corresponding to each traffic cluster to detect the traffic in different traffic clusters and obtain the detection results, including: Obtaining a first network address associated with traffic data of a first cluster among the multiple traffic clusters, wherein the first cluster represents a cluster consisting of traffic data whose ratio of the total upstream traffic to the total downstream traffic is greater than a first ratio; Acquire a network type of a network address associated with the traffic data of the first cluster, and when the network type is a metropolitan area network, acquire a first number of uplink ports with the largest traffic among the uplink ports of the first network address; When the proportion of the upstream traffic of the first number of upstream ports to the total upstream traffic of the first network address exceeds a first threshold, the proportion of the target end of the upstream traffic of the first network address to the network address in the target area exceeds a second threshold, and the proportion of non-public ports in the upstream ports of the first network address exceeds a third threshold, the first network address is marked as a personal content distribution network PCDN.

6. The method according to claim 1, characterized in that Use the detection strategy corresponding to each traffic cluster to detect the traffic in different traffic clusters and obtain the detection results, including: Acquire a second network address associated with traffic data of a second cluster among the multiple traffic clusters, wherein the second cluster represents a cluster consisting of traffic data in which the uplink traffic outside the target area is higher than the downlink traffic outside the target area; Acquire a network type of a network address associated with the traffic data of the second cluster, and when the network type is a metropolitan area network, count the traffic data of the second network address within a specified period; Predicting the flow data within the prediction period using the pre-trained seasonality based on the flow data within the specified period to obtain a prediction error; When the prediction error is greater than a preset error threshold and the proportion of the number of interactive network addresses in the specified period to the total number of interactive network addresses is greater than a fourth threshold, it is determined that the second network address is marked.

7. The method according to claim 1, characterized in that Use the detection strategy corresponding to each traffic cluster to detect the traffic in different traffic clusters and obtain the detection results, including: Obtaining a third network address associated with traffic data of a second cluster among the multiple traffic clusters; Obtaining a network type of a network address associated with the traffic data of the second cluster, and when the network type is an Internet data center network, counting traffic data of the third network address within a specified period; The flow data within the specified period is evaluated using an autocorrelation function, and when the flow data within the specified period has periodic changes, the third network address is marked.

8. The method according to claim 7, characterized in that The method further comprises: The marked network address is determined as the abnormal traffic address in the detection result.

9. An abnormal flow detection device, characterized in that: include: A collection module, used to collect the flow data to be detected according to a preset collection period; A clustering module, used for clustering the traffic data to be detected using multiple clustering dimensions to obtain multiple traffic clusters, wherein the clustering dimensions at least include the ratio of the total uplink traffic in the target area to the total downlink traffic in the target area, the ratio of the uplink traffic outside the target area to the total uplink traffic, the ratio of the downlink traffic in the target area to the total downlink traffic, and the ratio of the total traffic outside the target area to the total traffic in the target area, the total uplink traffic includes the sum of the uplink traffic in the target area and the uplink traffic outside the target area, and the total downlink traffic includes: the sum of the downlink traffic in the target area and the downlink traffic outside the target area; A determination module, used to respectively determine a plurality of detection strategies corresponding to the plurality of traffic clusters; The detection module is used to detect the traffic in different traffic clusters using the detection strategy corresponding to each traffic cluster to obtain the detection result.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory is used to store program instructions; The processor is connected to the memory and is used to execute the abnormal traffic detection method described in any one of claims 1 to 8.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the abnormal traffic detection method described in any one of claims 1 to 8 is implemented.

Images