Sequential network abnormal behavior detection method for abnormal decision

Abstract

The invention provides a time sequence network abnormal behavior detection method based on abnormal decision. The method comprises the following steps: firstly, defining an abnormal detection research process of time sequence data of an overall network behavior in a computer network environment; secondly, network anomalies generated for different flow characteristics of the security events are observed and researched, the network anomalies are classified into two types, namely flow load type anomalies and communication mode type anomalies, the two types correspond to network flow characteristics and flow diagram characteristics respectively, and the overall network behavior contour is described; finally, an abnormal accumulation algorithm is adopted, and a historical time point taking method suitable for network flow time sequence data is defined; absolute change measurement is adopted on a horizontal time axis, relative change and trend change measurement is adopted on a vertical time axis, and then a time sequence accumulation deviation degree anomaly detection algorithm of multi-dimensional features is defined. According to the technical scheme, abnormal behaviors can be screened out more quickly, the possibility of missing report and false report is reduced, and therefore the safety of a network environment is effectively protected in time.

Description

Technical Field

[0001] The present invention relates to the technical field of network behavior analysis, and in particular to a method for detecting abnormal behavior of a time-series network with abnormal decision-making. Background Art

[0002] With the in-depth development of the Internet, the scale and complexity of network interactions are increasing. Network time series data contains rich behavioral patterns and information, and time series network abnormal behavior detection plays an irreplaceable role in maintaining network security and improving network service quality. Traditional rule-based and threshold-based anomaly detection methods have become inadequate in the face of complex and changing network environments. Therefore, it is imperative to explore more intelligent and efficient abnormal behavior detection methods.

[0003] In the process of overall network behavior analysis and anomaly detection, the normal behavior profile of the overall network behavior is usually described by extracting behavioral features through traffic attribute measurement methods. If it deviates far from the normal behavior profile, an anomaly has occurred. The research idea of ​​the present invention is how to select a measurement method and construct a suitable set of behavioral features. A large number of features and measurement methods have appeared in existing research work, and network traffic load-type features are one of the most widely used ones. However, there are still problems such as high false alarm rate and low efficiency when they are actually applied to the network environment. However, different types of network anomalies have different impacts on the overall network behavior, and different types of features correspond to different perceived network anomalies. Existing research still has the following shortcomings.

[0004] 1) The data volume of network traffic has increased dramatically. In a high-speed network environment, it is difficult to extract and apply packet-level features, the network flow feature information is small, and the traditional network flow load features cannot effectively measure the changes in communication modes. For the structural properties of the traffic graph constructed by network traffic data, existing research lacks in-depth research on the characteristics and laws of time series data changes.

[0005] 2) Diversity of abnormal network behavior. With the continuous development of computer network technology, new network attacks and security incidents have had various impacts on the network environment, with delayed and difficult-to-detect damage. The detection method using traditional traffic load attribute characteristics cannot detect abnormalities with small fluctuations in traffic characteristics, such as network traffic bandwidth, upstream and downstream traffic, and data packet data. These attributes are called low-density and low-degree traffic attributes. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a method for detecting abnormal behavior in a time-series network with abnormal decision-making, which can screen out abnormal behavior more quickly and reduce the possibility of missed reports and false alarms, thereby protecting the security of the network environment in a timely and effective manner.

[0007] To achieve the above-mentioned purpose, the present invention adopts the following technical scheme: a method for detecting abnormal behavior of a time series network with abnormal decision-making, first defining the abnormal detection research process of the time series data of the overall network behavior in a computer network environment; then by observing and studying the network anomalies caused by different traffic characteristics of security events, the network anomalies are summarized into two categories, namely traffic load type anomalies and communication mode type anomalies, which correspond to network flow characteristics and traffic graph characteristics respectively, and describe the overall network behavior profile; finally, an abnormal accumulation algorithm is adopted to define a historical time point method suitable for network traffic time series data; absolute change measurement is adopted on the horizontal time axis, and relative change and trend change measurement are adopted on the vertical time axis, thereby defining a time series cumulative deviation anomaly detection algorithm with multi-dimensional characteristics.

[0008] In a preferred embodiment, the anomaly detection research process of defining the time series data of the overall network behavior in a computer network environment includes the following steps:

[0009] Step A1: Detect and use the flow data of network traffic as input for subsequent data processing;

[0010] Step A2: Convert the flow data into graph data to build a flow graph model, providing a data basis for extracting graph model attributes in the feature extraction step;

[0011] Step A3: Count the features of time series data in a fixed time window, including flow data and graph data; define static measurement and dynamic measurement methods, extract features from the above two measurement spaces respectively, describe the overall network behavior of the current time window, and the overall network behavior changes of adjacent time windows, so as to form a feature vector describing the time series behavior profile of the overall network behavior;

[0012] Step A4: Taking the multidimensional feature set of the overall network behavior as input, combining the defined historical time 0 point method to select historical data, combining the detection ideas of single point analysis and subsequence analysis of time series data, defining the angles for quantifying changes, relative changes and trend changes, thereby constructing a quantitative method for the deviation of time series data, and then measuring the time series changes of the multidimensional features of the overall network behavior;

[0013] Step A5: Define an anomaly accumulation method, by calculating the time series cumulative value of multi-dimensional feature behavior and the threshold set by long-term observation experience, to make a decision on whether the current state of the overall network behavior is abnormal or not.

[0014] In a preferred embodiment, the step A2 includes a static flow graph and a dynamic flow graph with time attributes;

[0015] Static graphs are the basis of dynamic graphs. Static flow graphs are defined as a graph G = (V, E), where V represents the set of communication nodes and E is the set of edges. This means that a pair of nodes e ij =(v i ,v j ), |V| represents the number of communication nodes, and |E| represents the number of edges; node v i With node v j Communication means that there is a sending or receiving behavior of network data packets between two nodes, that is, there is network communication between the two nodes;

[0016] The dynamic flow graph represents a flow graph sequence G = {G(tn), G(tn-1), ..., G(t-1), G(t)} formed over time tn<tn-1<…<t-1<t, where G(t) represents a snapshot of the static flow graph of G at time t. The vertex set, edge set, number of vertices, and number of edges (V, E, |V|, |E|) of the graph will change over time, so we can see that Gi≠Gi+1

[0017] The graph analysis method is used to extract feature values ​​for static flow graphs and dynamic flow graphs respectively; first, the static features of the graph model attributes are described; second, the dynamic features of the graph model that change over time are described. The relevant definitions are introduced as follows:

[0018] Features of static traffic graph: Features obtained from static traffic graph are called static features of the graph, which are statistical summary information of static traffic graph snapshots and are used to describe and distinguish graphs;

[0019] Static features focus on the relevant properties of the graph model, including statistical features such as node degree, degree distribution, maximum degree, degree, average shortest distance, clustering coefficient, degree correlation coefficient, and degree-degree correlation;

[0020] Features of dynamic flow graphs; Features obtained from dynamic flow graphs are called dynamic features of graphs, which are the differences in the overall network behavior dynamic graph over time. They describe the changes of two adjacent graphs from the graph model attributes and are dynamic flow graphs that measure two continuous time series.

[0021] Dynamic graph metrics compare the similarity of graphs in two adjacent time windows. The dynamic features of the graph at the time of calculation involve two graph snapshots, namely, the graph data of G(t-1) and G(t);

[0022] In a graph sequence, the graph edit distance method and the maximum common subgraph method are used to measure the changes of two consecutive graphs. The graph edit distance measurement method describes the absolute changes of two adjacent graphs, while the maximum common subgraph method describes their relative changes.

[0023] Graph absolute change distance; Graph edit distance GED is the absolute change between two consecutive graph snapshots in a time series graph;

[0024] Relative change distance of graph; The maximum common subgraph method is used to measure the graph distance, as shown in formula (4), which describes the relative degree of change and uses the concept of maximum common subgraph MCS;

[0025] ged(G i ,G j )=|V i |+|V j |-2|V i ∩V j |+|E i |+|E j |-2|E i ∩E j | (4)

[0026] The maximum common subgraph means that if G is a common subgraph of G1 and G2, then there is no other common subgraph with more points than G. The distance between two non-empty graphs G1 and G2 is defined as shown in formula (5), where |G| represents the number of points in the graph and also the number of edges in the graph, where 0≤d(G1,G2)≤1. When the distance is 0, it means that the two graphs are isomorphic.

[0027] d(G1,G2)=1-|mcs(G1,G2)| / max(|G1|,|G2|) (5)

[0028] Inspired by the distance calculation method of the graph model, we also define the measurement methods of dynamic graph edge change distance and dynamic graph node change @ to quantify the changes of nodes and edges of adjacent time-series graphs;

[0029] Dynamic graph edge distance; Dynamic graph edge distance is the absolute change of edges between two consecutive graph snapshots in a time series graph;

[0030] de(G i ,G j )=2|E i ∩E j | / (|E i |+|E j |) (6)

[0031] Dynamic graph node distance; Dynamic graph node distance is the absolute change of nodes between two consecutive graph snapshots in a time series graph;

[0032] dn(G i ,G j )=2|V i ∩V j | / (|V i |+|V j|) (7)

[0033] Different graph attributes are used for quantification according to the different network behavior subject objects in network traffic analysis; the characteristics of graph nodes are used for network individual behavior analysis; the characteristics of graph evolution and evolution events are used for host group behavior analysis.

[0034] In a preferred embodiment, definition 1: traffic load type anomaly; when the impact of a security incident causes an abnormal change in traffic load, the network uplink or downlink traffic will show a situation that does not conform to the daily fluctuation range; the traffic characteristics clearly show that the time series curve of the number of bytes and data packets increases or decreases to an unexpected level, which is caused by many factors; external sudden attacks, hot news, and course selection lead to an increase in network traffic; a popular video, report, new software, or worm virus outbreak is released on a website or host; hosts infect each other and connect to remote control hosts, and drag database behaviors cause sudden changes in traffic; through these event scenarios, sudden changes in abnormal traffic load type characteristics can be observed;

[0035] Definition 2: Communication pattern anomaly; when the impact of a security incident causes the communication pattern to change abnormally, the graph model constructed by the traffic graph will change accordingly;

[0036] The static and dynamic methods for extracting network traffic features are defined, and combined with traffic load intensity features and communication mode features to detect low-intensity abnormal behaviors and traditional traffic anomalies; the methods and steps of static and dynamic measurements are described as follows:

[0037] Definition 3: Static measurement method; The static measurement method is used to associate only a data set of a time window when performing feature extraction, and only extract the required features from the data set of the current time window;

[0038] Definition 4: Dynamic measurement method; The dynamic measurement method is used to associate only two data sets of adjacent time windows during feature extraction, including the data set of the current data window and the data set of the adjacent historical time window, and calculate the dynamic changes of the required features in the adjacent windows.

[0039] In a preferred embodiment, the definition of the historical time point method is described, and the anomaly of the time series data is detected through historical data and the computing cost and storage cost are reduced; the historical time point method includes two important definitions: the vertical time axis and the horizontal time axis;

[0040] Definition 5: Vertical time axis; the vertical time axis is in days and days are of the same time type;

[0041] Definition 6: Horizontal time axis; the horizontal time axis is the data within a fixed day, and the time unit needs to be set;

[0042] In the network traffic time series data, the traffic characteristics show differences between weekdays and weekends. The historical time method also distinguishes the network flow time series data on weekdays and weekends.

[0043] In a preferred embodiment, the deviation of the overall network behavior is further measured by absolute change, relative change and trend change:

[0044] Definition 7: Absolute change; calculate the overall network behavior time series profile at the current time, obtain the overall network behavior time series profile historical data, and calculate the absolute value of the adjacent time of the i-th feature |f i (t)-f i (t′)|,w i The weight of eigenvalue i, ∑w i =1(i=1,i≤m); Here, multiple data points i are selected from adjacent data, and the cumulative value of the absolute change of the overall network behavior time series profile is shown in formula (8):

[0045]

[0046] Definition 8: Relative change; calculate the overall network behavior time series profile at the current time t, obtain the overall network behavior time series profile historical data, extract the periodicity related data of the i-th feature at the current time t, and calculate the adjacent previous K-th week time. The ratio of the maximum values ​​of the data points, As shown in formula (9):

[0047]

[0048] Definition 9: Trend change; The characteristic value of the overall network behavior time series profile establishes the current shape subsequence C′ at the current time t, obtains the historical data of the overall network behavior time series profile, and calculates the Euclidean distance of the subsequence; the shape change value of the overall network behavior time series profile mainly focuses on the characteristic shape trend of the overall network behavior time series profile in this time period, rather than the specific characteristic value size.

[0049] In a preferred embodiment, the subsequence of the N most recent data points before the detection time is obtained. It is known that the time sequence of the N adjacent data points at the current time is U′={U′1,U′2,…,U′ N}, the key steps of subsequence comparison using SAX are as follows;

[0050] 1) Normalization: Normalize the time series U′ into a standard series U={U1,U2,…,U N}, as shown in formula (10):

[0051]

[0052] 2) PAA dimensionality reduction: Perform PAA representation on U to obtain Where N is the length of the PAA representation of the time series, describing a time series of length n, as shown in formula (11):

[0053]

[0054] 3) Symbolization; Select the base size a = 10, and find the split point β of the interval according to the letter base size in the Gaussian distribution table as shown below i ; alpha i represents the i-th letter in the alphabet, for example, alpha2 = b, alpha3 = c, through Map the value represented by PAA to the corresponding letter in the alphabet and finally discretize it into a string

[0055] Next, use formula (12) to calculate the distance between each letter in base 10;

[0056] 4) Similarity measurement: Compare and accumulate the SAX sequences of each feature value at the current moment; Formula (13) is used to calculate the string sequence and The minimum distance;

[0057]

[0058] Based on the above introduction of the key points of timing algorithm and parameter selection, is the string form of A, constructed through time series data; where w is the weight value of the feature, δ is the weight of the time distance between the subsequence and the current subsequence, and the calculation of the cumulative value of the time series trend is shown in formula (14):

[0059]

[0060] In summary, according to formula (14), the formula for the cumulative deviation of the overall network behavior time series profile based on the time series analysis method is shown in formula (15), where

[0061] EA=θ1EA (1) +θ2EA (1) +θ3EA (1) (15).

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] 1) In view of the problems that strong packet-level feature extraction is difficult to apply in high-speed environments and the lack of network flow feature information, the present invention adopts a flow graph to describe the structural attributes of the communication mode, enriches the flow characteristics of the overall network behavior, and improves the anomaly detection capability; defines static measurement and dynamic measurement methods, and at the same time, studies the problem that the abnormal communication mode does not cause the fluctuation of the flow load intensity characteristics and cannot be calibrated, and adopts the global attributes of the flow graph to describe the changes in the node interaction structure of the communication mode. In the work of the present invention, a feature extraction method based on flow and flow graph is defined, which can detect changes in network communication modes and fluctuations in flow load intensity.

[0064] 2) The present invention constructs a multi-dimensional feature to construct the overall network behavior profile, and defines a historical time point method based on the periodicity, stability and regularity of network traffic to address the shortcomings of high computational and spatial costs of the multi-dimensional feature time series data anomaly detection algorithm.

[0065] 3) The present invention defines a time series data anomaly detection algorithm based on cumulative deviation. Using the proposed historical time point method, absolute change measurement is used for horizontal time, and relative change and trend change measurement are used for vertical time, and a cumulative deviation anomaly detection method is defined. By comparing with the detection algorithms of time series data: dynamic ARIMA algorithm and static ARIMA algorithm, the calculation cost can be effectively reduced, and the performance advantage is obvious. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A schematic diagram of the research idea of ​​overall network behavior anomaly detection in a preferred embodiment of the present invention;

[0067] Figure 2 A data schematic diagram of static characteristics of a preferred embodiment of the present invention;

[0068] Figure 3 A data schematic diagram of the dynamic characteristics of a preferred embodiment of the present invention;

[0069] Figure 4 A schematic diagram of a flow diagram of a preferred embodiment of the present invention;

[0070] Figure 5 It is a characteristic schematic diagram of a static flow diagram of a preferred embodiment of the present invention;

[0071] Figure 6 It is a characteristic schematic diagram of a dynamic flow graph of a preferred embodiment of the present invention;

[0072] Figure 7 Schematic diagram of historical time points of the time axis of the preferred embodiment of the present invention, wherein (a) is the time axis for working days, and (b) is the time axis for weekends;

[0073] Figure 8 A schematic diagram of the shape change of time series data in a preferred embodiment of the present invention;

[0074] Fig. 9 A schematic diagram showing a comparison of detection rates in a preferred embodiment of the present invention;

[0075] Fig.10 This is a schematic diagram of comparing the accuracy of a preferred embodiment of the present invention;

[0076] Fig.11 Schematic diagram of false alarm rate comparison of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0077] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0078] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0079] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0080] A method for detecting abnormal behavior in temporal networks with abnormal decision making, such as Figure 1 As shown, the present invention defines the research idea of ​​anomaly detection of time series data of overall network behavior in a computer network environment, which mainly includes several key research levels and analysis steps of "data source-data transformation-feature extraction-deviation measurement-abnormal decision". The data input is the network traffic time series data with timestamps. The feature set describing the overall network behavior profile is extracted from the data, and then the time series changes of the overall network behavior are quantified through the time series analysis technology. Finally, the abnormal deviation of the overall network behavior is calculated by the abnormal accumulation method, and the abnormal decision of the overall network behavior is realized through the threshold. The key steps of the research idea of ​​anomaly detection of overall network behavior are introduced as follows.

[0081] 1) Data source. The detection uses the streaming data of network traffic as the input for subsequent data processing. Most actual network environments and public data sets have PCAP files, but due to the large amount of data, streaming data is provided, or directly converted into streaming data when collecting data. The technologies and tools used in this part will be introduced later. When collecting network traffic, it is more suitable for network environments with high security requirements. The time series data of network traffic can be used for further time series analysis and detection. At the same time, these data are stored in databases (such as MySQL\MongoDB) or distributed file systems HDFS, which can be used for offline data analysis.

[0082] Data transformation: Convert flow data into graph data to build a flow graph model, which provides a data basis for extracting graph model attributes in the feature extraction step.

[0083] Feature extraction. The features of time series data are counted in fixed time windows, including flow data and graph data. Static and dynamic measurement methods are also defined to extract features from the above two measurement spaces, respectively, to describe the overall network behavior of the current time window and the overall network behavior changes of adjacent time windows, thereby forming a feature vector that describes the time series behavior profile of the overall network behavior.

[0084] Deviation measurement. Taking the multidimensional feature set of the overall network behavior as input, combining the historical time 0 point method defined to select historical data, combining the detection ideas of single point analysis and subsequence analysis of time series data, defining the angle of quantification of change, relative change and trend change, thus constructing a quantitative method for the deviation of time series data, and then measuring the time series changes of the multidimensional characteristics of the overall network behavior.

[0085] Abnormal decision-making. The abnormal accumulation method is defined. By calculating the time series cumulative value of multi-dimensional feature behavior and the threshold set by long-term observation experience, the decision on whether the current state of the overall network behavior is abnormal is made on the abnormality of the time series data.

[0086] Network behavior construction method and definition:

[0087] By observing and studying the network anomalies caused by different traffic characteristics of security events, the network anomalies are summarized into two categories, namely traffic load anomalies and communication mode anomalies, which correspond to network flow characteristics and traffic graph characteristics respectively, and describe the overall network behavior profile.

[0088] Definition 1: Traffic load anomaly. When the impact of a security incident causes an abnormal change in traffic load, the network uplink or downlink traffic will show a situation that does not conform to the daily fluctuation range. The traffic characteristics clearly show that the time series curve of the number of bytes and data packets increases or decreases to an unexpected level, which is caused by many factors. External sudden attacks, hot news, and course selection lead to an increase in network traffic; a popular video, report, new software, or worm virus outbreak is released on a website or host; hosts infect each other and connect to remote control hosts, and drag databases to cause sudden changes in traffic; through these event scenarios, sudden changes in abnormal traffic load characteristics can be observed.

[0089] Definition 2: Communication pattern anomaly. When the impact of a security incident causes an abnormal change in the communication pattern, the graph model constructed by the traffic graph will change accordingly. However, it is often found that the traffic load characteristics, such as the number of bytes and the number of packets, do not fluctuate abnormally enough to attract the attention of network security administrators, which does not mean that the network is normal. There are also a large number of low-frequency, low-intensity, and low-density network attack behaviors in network attacks, or unusual host-network interaction behaviors suddenly appear in computer network environments. In addition, another goal is to understand the changes in the communication patterns of network nodes, such as the functional failure of network devices, abnormal network operations, network device interruptions, and unusual changes in communication patterns due to network configuration, adding new devices, migrating devices, changing device network addresses, etc. Many networks need to use network communication behaviors to detect device failures.

[0090] The present invention will define static and dynamic methods for network traffic feature extraction, and combine traffic load intensity features and communication mode features to detect low-intensity abnormal behaviors and traditional traffic anomalies. The methods and steps of static and dynamic measurements are described as follows.

[0091] Definition 3: Static measurement method. The static measurement method is used to associate only one time window data set when performing feature extraction, and only extract the required features from the data set of the current time window.

[0092] Definition 4: Dynamic measurement method. The dynamic measurement method is used to associate only two data sets of adjacent time windows during feature extraction, including the data set of the current data window and the data set of the adjacent historical time window, and calculate the dynamic changes of the required features in the adjacent windows.

[0093] According to the above definition, four feature sets are summarized, as shown in Table 1, including network flow static feature set, network flow dynamic feature set, traffic graph static feature set and traffic graph dynamic feature set.

[0094] Table 1. Classification of characteristics of overall network behavior

[0095]

[0096] Formal representation of streaming data

[0097] The main ways to obtain data are public data sets and production environments. Set up network flow collection points in the actual environment, capture full flow and sampled IP packets, and use NetFlow generation tools to generate flow data. Network flow timing data detection can be performed, real-time analysis, or storage first and then offline detection. In the public data set, at least one data format file will be provided for use. Considering that PCAP files are large, in addition to providing PCAP files of original data packets, more and more research institutions also provide flow data generated by PCAP files to meet the needs of different studies.

[0098] The five-tuple of an IP data packet includes the source IP address, source port number, destination IP address, destination port number, and transport layer protocol. Network flows can be obtained by parsing data packets, aggregating data packets with the same attribute five-tuple to generate a network flow record.

[0099] TCP network flow. TCP network flow refers to the collection of all IP data packets from the SYN packet transmitted through TCP between the source IP address and the destination IP address until the first FIN packet is sent. The collection of IP data packets in a TCP network flow has the same five-tuple. When collecting data, in view of the occurrence of packet loss or delay in the network environment, the TCP network flow will set an expiration time. If no data packet is received within a period of time, it means that the communication of this TCP network flow has ended.

[0100] UDP network flow. UDP network flow is all data packets transmitted between the source IP address and the destination IP address through UDP, and they have the same five-tuple. Also, in view of the occurrence of packet loss or delay in the network environment, the UDP network flow will set an expiration time. If no data packet is received within a period of time, it means that the communication of this UDP network flow has ended.

[0101] TCP and UDP are indispensable. Support for other protocols can be selected according to needs, and data of other protocols can be further generated on this basis. The recording format and content of network flows depend on the network equipment. Products from different manufacturers correspond to different network flow collection protocols, such as the commonly used Cisco NetFlow and Huawei NetStream, and other manufacturers' CFlowd, sFlow, IPFIX and other protocols are more common. In addition, there are a large number of open source and production-level network flow generators, such as netmate-flowcalc, Tranalyzer, etc. Different data acquisition methods have their own advantages and disadvantages, which can be selected during the research and development process. The comparison of advantages and disadvantages is shown in Table 2.

[0102] Table 2 Comparison of network flow generation methods

[0103]

[0104] Generate a network flow data set from the IP data packets of the computer network environment, which can be used for further data processing and anomaly detection research. The traditional time series data detection of the overall network behavior focuses on the characteristic value of the current time window. Based on this, the present invention also defines a dynamic measurement method for time series data. This method focuses on the change value of the network time series characteristics of the two windows, the current time window and the adjacent time window. In order to introduce the dynamic measurement method, two time-related data sets are defined below: a continuous connection data set and an active connection data set. Among them, slwn represents the current time window, slwh represents the previous time window (or the next time window) adjacent to the current time window, cicf represents flow data, and cicf slwn It represents the flow data of the current time window, cicf slwh It represents the streaming data of the historical time window.

[0105] Continuously connected dataset C cief

[0106] Continuous connection data set, which means that there is network flow data cicf where the IP address pair appears in the previous historical time window slwh slwh At the same time, these IP address pairs also exist in the network flow data cicf of the current time window slwn swln In, it is expressed as:

[0107] C cicf =cicf slwn ∩cicf slwh (1)

[0108] Active Connection Dataset A cicf

[0109] Active connection dataset, meaning the network flow cicf of the IP pair that did not appear in the previous historical time window slwh slwh These IP address pairs only exist in the network flow data cicf of the current time window slwn slwh In, it is expressed as:

[0110] A cicf =cicf slwn -cicf slwh (2)

[0111] Network flow static features. Network flow static features are statistical feature sets extracted from network flow data using static measurement methods, using only data from a time window, such as the number of packets, packet size, communication duration, port number, IP address, number of transport layer protocols, etc. Figure 2 shown.

[0112] Network flow dynamic features. Network flow dynamic features are statistical feature sets extracted from flow data using dynamic measurement methods, such as the number of bytes, number of packets, and ratio of deduplicated IP addresses in adjacent windows. Data from adjacent time windows are required, such as Figure 3 shown.

[0113] When analyzing the overall network behavior, we mainly focus on the characteristic values ​​of the current time window. Data packets generate network flows. We can also consider the dynamic properties of time series data. The data in adjacent time windows are similar and will not mutate. The dynamic measurement method is to first use the dataset cicf slwh and dataset cicf slwh According to the definition, the continuous connection dataset C is separated cicf and active connection dataset A cicf , and then calculate the proportion of each feature Where f={f1,f2,…,f n}, f i Represents the i-th eigenvalue of the data set. Considering A cicf The eigenvalue of the data set may be zero. In order to avoid the situation where the divisor is zero, the e exponent is used to solve it. The mathematical calculation method is shown in formula (3).

[0114]

[0115] Formal representation of traffic graph:

[0116] In order to describe the interactive communication behavior of network nodes, graph theory analysis methods are used in traffic network behavior. Based on the data application scenario of network traffic, it is defined as "the source address and destination address in a network flow record represent the nodes in the graph, and a network flow record indicates that there is an edge between the nodes in the graph corresponding to the source address and the destination address, which means that a network communication has taken place", which are collectively referred to as traffic graphs in the following description.

[0117] At present, in the related research work at home and abroad that constructs network flow data into a graph model, there is no unified name for this method, which is either called a traffic scatter diagram, a traffic graph (TrafficActivityGraph, TAG) or a graph-based traffic analysis. The common point of these research works is that they all convert network traffic into a description method of a graph model in order to analyze which communication nodes in a computer network have network communications with whom and the interactions between nodes, and then apply graph analysis methods to network traffic analysis (such as traffic classification, anomaly detection, network security monitoring and other related research).

[0118] The following introduces the static flow graph and the dynamic flow graph with time attributes, as well as the relevant definitions and measurement methods of feature extraction of the static flow graph and the dynamic flow graph.

[0119] Static flow graph. Static graph is the basis of dynamic graph. Static flow graph can be defined as a graph G = (V, E), where V represents the set of communication nodes and E is the set of edges. This means that a pair of nodes e ij =(v i ,v j ), |V| represents the number of communication nodes, and |E| represents the number of edges. i With node v j Communication means that there is a sending or receiving behavior of network data packets between two nodes, that is, there is network communication between the two nodes.

[0120] like Figure 4 As shown, both node v5 and node v6 have network communications with three nodes, nodes with two node behaviors include node v1, node v2, and node v4, and nodes that have communication behaviors with only one node include node v3 and node v7.

[0121] Dynamic flow graph. A dynamic flow graph represents a flow graph sequence G = {G(tn), G(tn-1), ..., G(t-1), G(t)} formed over time tn<tn-1<...<t-1<t ， Where G(t) represents a snapshot of the static traffic graph of G at time t. The vertex set, edge set, number of vertices, and number of edges (V, E, |V|, |E|) of the graph will change over time. i ≠G i+1 .

[0122] Therefore, the graph analysis method can be used to extract feature values ​​for static flow graphs and dynamic flow graphs. First, the static features that describe the attributes of the graph model; second, the dynamic features that describe the changes of the graph model over time. The relevant definitions are introduced as follows.

[0123] Features of static traffic graphs. Features obtained from static traffic graphs are called static features of graphs. They are statistical summary information of static traffic graph snapshots and are used to describe and distinguish graphs. Figure 5 shown.

[0124] Static features focus on the relevant properties of the graph model, mainly including statistical features such as node degree, degree distribution, maximum degree, average shortest distance, clustering coefficient, degree correlation coefficient and degree-degree correlation.

[0125] Features of dynamic flow graphs. Features obtained from dynamic flow graphs are called dynamic features of graphs. They are the differences in the overall network behavior dynamic graph over time. They describe the changes of two adjacent graphs from the graph model attributes, and are usually dynamic flow graphs that measure two continuous time series. For example, the edit distance between two adjacent time series graphs, the edge distance of a dynamic graph, the node distance of a dynamic graph, etc.

[0126] Dynamic graph metrics can compare the similarity of graphs in two adjacent time windows. Calculating the dynamic features of graphs at time involves two graph snapshots, namely, the graph data of G(t-1) and G(t), such as Figure 6 shown.

[0127] In the graph sequence, the graph edit distance method and the maximum common subgraph method are used to measure the changes of two consecutive graphs. Among them, the graph edit distance measurement method describes the absolute changes of two adjacent graphs, and the maximum common subgraph method describes their relative changes. The absolute change measurement method is based on the scale of the graph, which is more suitable for

[0128] It is used to detect a large number of reduced or increased graph elements and can describe dynamic network behavior. The specific method is as follows.

[0129] Graph Edit Distance (GED) is the absolute change between two consecutive graph snapshots in a time series graph. This absolute change metric is suitable for detecting anomalies in graph size or graph model.

[0130] Relative change distance of graph. The maximum common subgraph method is used to measure the graph distance, as shown in formula (4), which describes the relative degree of change and uses the concept of maximum common subgraph (MCS).

[0131] ged(G i ,G j )=|V i |+|V j |-2|V i ∩V j |+|E i |+|E j |-2|Ei ∩E j | (4)

[0132] The maximum common subgraph means that if G is a common subgraph of G1 and G2, then there is no other common subgraph with more points than G. The distance between two non-empty graphs G1 and G2 is defined as shown in formula (5), where |G| can represent the number of graph points or the number of graph edges, where 0≤d(G1,G2)≤1. When the distance is 0, it means that the two graphs are isomorphic.

[0133] d(G1,G2)=1-|mcs(G1,G2)| / max(G1|,|G2|) (5)

[0134] Inspired by the distance calculation method of the graph model, we also define the measurement methods of dynamic graph edge change distance and dynamic graph node change @ to quantify the changes of nodes and edges of adjacent time-series graphs.

[0135] Dynamic graph edge distance. The dynamic graph edge distance is the absolute change in the edge between two consecutive graph snapshots in a time series graph.

[0136] de(G i ,G j )=2|E i ∩E j | / (|E i |+|E j |) (6)

[0137] Dynamic graph node distance. Dynamic graph node distance is the absolute change of a node between two consecutive graph snapshots in a time series graph.

[0138] dn(G i ,G j )=2|V i ∩V j | / (|V i |+|V j |) (7)

[0139] According to the different network behavior subjects of network traffic analysis, different graph attributes can be used for quantification. For example, the graph features used in the overall network behavior measurement are intended to describe the overall structure and properties of the graph, such as graph edit distance, mean value of graph node degree, etc.; the characteristics of graph nodes are used for network traffic analysis of individual network behaviors. For example, the degree of graph nodes, the degree of neighbor nodes of nodes, etc.; the characteristics of graph evolution and evolution events are used for host group behavior analysis, such as the number of host groups, the number of host group members, the number of evolution events, etc. The relevant definitions of the quantity graph will not be repeated later.

[0140] Network behavior collection and extraction algorithm:

[0141] The study found that when analyzing network traffic, the results of network feature selection directly affect the detection results of network abnormal behavior. Appropriate network anomaly classification helps to extract a common feature set for abnormal characterization. Abnormal behaviors triggered by different network security events will lead to different changes in network traffic characteristics. Based on the existing research on overall network behavior characteristics, the feature set is effectively integrated and examples of representative features of overall network behavior are summarized, as shown in Table 3.

[0142] Table 3 Examples of representative characteristics of overall network behavior

[0143]

[0144]

[0145] Here, an algorithm for extracting features of overall network behavior is defined. The input data of the algorithm is network flow data, and the output is a feature set that describes the overall network behavior profile. The network flow data set includes timestamps. In order to process offline PCAP files and NetFlow files, sliding window technology can be used to extract fixed timestamp record sets. The following introduces the construction algorithm for overall network behavior features. The input data of the algorithm is flow data, and the output is a feature value set, which is used as the data input for the next step of storage and network security visualization analysis. The Spark algorithm is described as follows.

[0146] Algorithm 1: Overall network behavior feature extraction algorithm

[0147] Input: Streaming data

[0148] Output: feature vector

[0149] 1SparkContext initialization;

[0150] 2 Get data KafkaStream = KafkaU tils.createStream();

[0151] 3 Get time window data datawin = KafkaStream.window();

[0152] 4for each rdd in DStream do

[0153] 5slice now =Get the previous time window data datawin.slice(0);

[0154] 6slice now =Get the current time window data datawin.slice(1);

[0155] 7 Parsing data rddslice his = clicehis.map().cache();

[0156] 8 Parsing data rddslice his = clicehis.map().cache();

[0157] 9ifrddslice his ≠null then

[0158] 10ftv s =Calculate the static characteristics of network flows;

[0159] / / Initialize edge data;

[0160] 11edgeRDD=rddslicenow.map(new Edge(_.srcip,_.dstip,1L);

[0161] / / Build the graph;

[0162] 12rddgraph=Graph.fromEdges(edgeRDD,None);

[0163] 13ftag s =Calculate the static characteristics of the flow diagram;

[0164] 14end

[0165] 15ifrddslicehis≠null andrddslicenow≠null then

[0166] / / Calculate the continuous connection data set rddcommon;

[0167] 16rddcommon=rddslice his .intersection(rddslicenow);

[0168] / / Calculate the active connection dataset rddactive;

[0169] 17rddactive=rddslicenow.subtract(rddcommon);

[0170] 18ftv d =Calculate network flow dynamic characteristics;

[0171] 19ftag d =Calculate the dynamic characteristics of the flow diagram;

[0172] 20end

[0173] 21f=ftv s U ftag s U ftv d U ftag d

[0174] / / Data storage and processing;

[0175] The 22f feature vector is used as the data input for the anomaly detection algorithm;

[0176] 23end

[0177] 24 Start StreamingContext;

[0178] 25 until StreamingContext terminates;

[0179] Time series refers to the characteristic sequence of data that changes over time. In order to solve the problems of high storage cost and space cost of multi-dimensional feature time series anomaly detection algorithm, an anomaly accumulation algorithm is adopted, and a historical time point method suitable for network traffic time series data is defined; absolute change measurement is used on the horizontal time axis, and relative change and trend change measurement are used on the vertical time axis, thereby defining a multi-dimensional feature time series cumulative deviation anomaly detection algorithm.

[0180] The following introduces the mathematical description and calculation method of the detection algorithm, including the symbolic aggregation approximation method of subsequence anomalies and the historical time point method.

[0181] The historical time point selection method of time series data:

[0182] In order to define the detection algorithm for abnormal accumulation of network flow time series data, the definition of the historical time point method is first described, which detects time series data anomalies through historical data and reduces computing and storage costs. The historical time point method includes two important definitions: vertical time axis and horizontal time axis.

[0183] Definition 5: Vertical time axis. The vertical time axis is based on days and days are of the same time type, such as weekdays or weekends, to construct vertical distances.

[0184] Definition 6: Horizontal time axis. The horizontal time axis is the data within a fixed day, and the time unit can be set, such as hours or minutes.

[0185] In the network traffic time series data, the traffic characteristics show differences between weekdays and weekends. The historical time method also distinguishes the network flow time series data on weekdays and weekends. The schematic diagram is as follows: Figure 7 shown.

[0186] according to Figure 7 The schematic diagram of the historical time point method is shown in FIG. 4 . The extracted data set is used as the historical data of the cumulative anomaly detection method. The data set and symbol description are shown in Table 4 .

[0187] Table 4 Datasets and symbols of historical data point method

[0188]

[0189] Timing anomaly detection method:

[0190] Data selection is performed based on the historical time point method, and the deviation of the overall network behavior is further measured through absolute change, relative change and trend change. The advantage of this algorithm is that it retains the true overall network behavior profile driven by user network behavior in the computer network environment, has advantages such as multi-dimensional characteristics and low computational cost, improves the accuracy of overall network behavior time series data monitoring, can reduce false alarms, and has higher practical application value. The following introduces the meanings of these three methods and the problems they solve.

[0191] Absolute change. The absolute change of network flow time series data mainly focuses on the changes in adjacent data, aiming to discover the sudden increase or decrease of the multi-dimensional characteristics of the overall network behavior time series profile, so as to solve the problem of sudden changes in the overall network behavior time series profile.

[0192] Relative change. The relative change of network flow time series data mainly focuses on the changes in periodic related data. For example, when a server provides services for a long time, considering the provision of scheduled and periodic network services, user access behaviors have concentrated access situations. The purpose is to reduce false alarms, discover service interruptions, etc., to solve the false alarm problems caused by scheduled, periodic, and concentrated user access behaviors.

[0193] Trend changes. The trend changes of network flow time series data mainly focus on the changes in trend and periodic data. Considering that network attack behaviors tend to be low-frequency, low-intensity, and slow, it aims to discover situations where the overall network behavior time series profile does not conform to the time evolution trend.

[0194] The overall network behavior time series profile anomaly detection is divided into two types: single point anomaly detection and subsequence anomaly detection. To solve the problem of sudden changes in the overall network behavior time series profile, time series single point anomaly detection is used; to solve the problem of fluctuation trend changes in the overall network behavior time series profile, subsequence anomaly detection is used. Based on the above description, the mathematical calculation method and definition of deviation degree are as follows.

[0195] Definition 7: Absolute change. Calculate the overall network behavior time series profile at the current time, obtain the overall network behavior time series profile historical data, and calculate the absolute value of the adjacent time of the i-th feature |f i (t)-f i(t′)|,w i The weight of eigenvalue i, ∑w i =1(i=1,i≤m). Here, multiple data points i are selected from adjacent data, and the cumulative value of the absolute change of the overall network behavior time series profile is shown in formula (8):

[0196]

[0197] Definition 8: Relative change. Calculate the overall network behavior time series profile at the current time t, obtain the historical data of the overall network behavior time series profile, extract the periodicity-related data of the i-th feature at the current time t, and calculate the adjacent previous K-th week. The ratio of the maximum values ​​of the data points, As shown in formula (9):

[0198]

[0199] Definition 9: Trend change. The overall network behavior time series profile feature value establishes the current shape subsequence C′ at the current time t, obtains the overall network behavior time series profile historical data, and calculates the Euclidean distance of the subsequence. The shape change value of the overall network behavior time series profile mainly focuses on the overall network behavior time series profile feature shape trend in this time period, rather than the specific feature value size.

[0200] like Figure 8 As shown, the time axis depicts three periodic data. Considering only the data distribution, the values ​​of t1 and t2 in the timing diagram are the same and are not considered abnormal timing points. However, from the perspective of the entire periodic data distribution, the value of time point t2 is an abnormal value, and the data trend is different from the timing curve shape of the normal period.

[0201] Based on the Piecewise Aggregate Approximation (PAA) method, Jesica et al. proposed the Symbolic Aggregate Approximation (SAX) method for time series. The SAX method was used in the study to study the approximation of time series data. In order to describe the key points of the anomaly detection algorithm for the overall network behavior time series profile, the symbols to be used and their meanings are summarized below, as shown in Table 5.

[0202] Table 5 Symbols and meanings defined

[0203]

[0204]

[0205] Get the subsequence of the most recent N data points before the detection time. It is known that the time sequence of the current N adjacent data points is U′={U′1,U′2,…,U′ N}, the key steps of subsequence comparison using SAX representation are as follows.

[0206] 1) Normalization. Normalize the time series U′ into a standard series U={U1,U2,…,U N}, as shown in formula (10):

[0207]

[0208] 2) PAA dimensionality reduction. Perform PAA on U to obtain Where N is the length of the PAA representation of the time series, which can describe a time series of length n, as shown in formula (11):

[0209]

[0210] 3) Symbolization. Select the base size a=10, and find the split point β of the interval according to the letter base size in the Gaussian distribution table as shown below. i alpha i represents the i-th letter in the alphabet, for example, alpha2 = b, alpha3 = c, through Map the value represented by PAA to the corresponding letter in the alphabet and finally discretize it into a string To this end, the split points of 3 to 10 to be used are first calculated, as shown in Table 6.

[0211] Table 6 Splitting points from base 3 to 10

[0212]

[0213]

[0214] Next, the distance values ​​between the letters in base 10 are calculated using formula (12), as shown in Table 7. From the table, we can see that dist(a, b) = 0 and dist(a, d) = 0.76.

[0215]

[0216] Table 7 Letter distance table with base a=10

[0217]

[0218] 4) Similarity measurement. Compare and accumulate the SAX sequences of each feature value at the current moment. Formula (13) is used to calculate the string sequence and The minimum distance.

[0219]

[0220] Based on the above introduction of the key points of timing algorithm and parameter selection, is the string form of A, constructed through time series data. Among them, w is the weight value of the feature, δ is the weight of the time distance between the subsequence and the current subsequence, and the calculation of the cumulative value of the time series trend is shown in formula (14):

[0221]

[0222] In summary, according to formula (14), the formula for the cumulative deviation of the overall network behavior time series profile based on the time series analysis method is shown in formula (15), where

[0223] EA=θ1EA (1) +θ2EA (1) +θ3EA (1) (15)

[0224] Anomaly Detection Algorithms

[0225] First, obtain the detection time window feature set f, clean and standardize the data, construct data that meets the input requirements of the next step, and then calculate the outlier value of the detection time window to identify normal or abnormal. The specific Spark algorithm is described as follows.

[0226] Algorithm 2: Overall network behavior anomaly detection algorithm

[0227] Input: Feature vector f(t) = (f1, f2, ..., f m )

[0228] Output: Outliers

[0229] 1. Get the time type tc of the detection time window;

[0230] 2. Get historical data R and calculate absolute changes based on time type tc;

[0231] 3. Get historical data Z and calculate relative changes based on time type tc;

[0232] 4. Obtain historical data Q and calculate trend changes based on time type tc;

[0233] 5. Clear the anomalies marked by R, Z and Q in the historical data;

[0234] 6. Construct the feature vector singledata(t)=f(t) of the detection time window; / / Construct the subsequence seqdata;

[0235]

[0236] 8for each f i in singledata(t)do / / Calculate absolute change;

[0237] 9 / / Calculate relative change;

[0238] 10

[0239] 11end

[0240] 12for each fin segdata(t)do / / Calculate trend changes;

[0241] 13Get i-th feature subsequence history data

[0242] 14Get i-th feature subsequence current data

[0243] 15

[0244] 16end

[0245] 17

[0246] 18alarm and save EAto MySQL;

[0247] 19end

[0248] The present invention first defines the research ideas of anomaly detection of time series data of overall network behavior in a computer network environment, which mainly includes several key research levels and analysis steps of "data source-data transformation-feature extraction-deviation measurement-abnormal decision". Then, by observing and studying the network anomalies generated by different traffic characteristics of security events, the network anomalies are summarized into two categories, namely traffic load type anomalies and communication mode type anomalies, which correspond to network flow characteristics and traffic graph characteristics respectively, and describe the overall network behavior profile. Finally, in order to solve the problems of high storage cost and space cost of multi-dimensional feature time series anomaly detection algorithm, an anomaly accumulation algorithm is adopted, and a historical time point method suitable for network traffic time series data is defined; absolute change measurement is used on the horizontal time axis, and relative change and trend change measurement are used on the vertical time axis, thereby defining a multi-dimensional feature time series cumulative deviation anomaly detection algorithm. Finally, the algorithm is designed.

[0249] In order to detect the advantages of the user abnormal behavior analysis algorithm, the present invention considers three indicators: detection rate, accuracy and false alarm rate. A detection algorithm with high detection rate, high accuracy and low false alarm rate is considered to be a better detection algorithm.

[0250] Detection rate = number of detected attack samples / total number of attack samples × 100%;

[0251] Accuracy = total number of detected abnormal samples / number of abnormal samples × 100%;

[0252] False alarm rate = the number of normal samples that are falsely reported as abnormal / the number of normal samples × 100%.

[0253] The present invention is demonstrated by the following experiments.

[0254] 1 Dataset

[0255] The experimental data set is KDD CUP99. This data set comes from the experimental data set of network anomaly detection published by the Massachusetts Institute of Technology. The data set includes 58 typical attack methods in five categories, including Probe, DoS, R2L, U2R and Data, and is used to simulate various types of users, attack methods and network traffic. Each record in the data set contains 42 attributes to identify normal or specific attack behaviors.

[0256] In order to detect the performance of the user abnormal behavior analysis algorithm, the three indicators of detection rate, accuracy and false alarm rate are considered. A detection algorithm with high detection rate, high accuracy and low false alarm rate is considered to be a good detection algorithm. Detection rate = number of detected attack samples / total number of attack samples × 100%;

[0257] Accuracy = total number of detected abnormal samples / number of abnormal samples × 100%;

[0258] False alarm rate = the number of normal samples that are falsely reported as abnormal / the number of normal samples × 100%.

[0259] 2 Data processing

[0260] 2.1 Digital Coding

[0261] The second dimension (protocol type), the third dimension (service type), and the fourth dimension (state identifier) ​​of the KDD CUP99 data set are all non-numeric types. In order to use the user abnormal behavior detection method proposed in the present invention for identification, they need to be digitized.

[0262] 1) Protocol Type

[0263] There are three types of protocols in the dataset, and their codes are shown in Table 8.

[0264] Table 8 Protocol type coding

[0265]

[0266] 2) Service Type

[0267] The 70 service types in the data set are encoded in the following process: a 70-bit binary code is used to represent each service type, and the binary bits of the corresponding digits are assigned 1 in ascending order.

[0268] 3) Status indicator

[0269] There are a total of 11 status identifiers in the data set, and their codes are shown in Table 9.

[0270] Table 9 Status identification code

[0271]

[0272]

[0273] 4) Normalize the behavior data attribute values

[0274] The normalization and normalization of data attribute values ​​can not only facilitate subsequent data processing, but also speed up the convergence of the program when it is running. Assuming that there are n pieces of behavior data, the attribute value of each piece of behavior data X is normalized to [0,1], as shown in formula (6):

[0275]

[0276] Among them, X′ is the normalized data; X min is the minimum value of a specific attribute in n pieces of behavior data; X max It is the maximum value of a specific attribute in n pieces of behavior data.

[0277] Abnormal traffic feature selection

[0278] The final 600,000 valid data were selected from the KDD CUP99 dataset as learning samples. The KDDCUP99 dataset not only has a large number of samples, but also has many characteristic attributes for each sample. Each connection sample in the dataset is described as 41 different features, including 9 basic features of TCP connections, 13 content features of TCP connections, 9 time-based network traffic statistics features, and 10 host-based network traffic statistics features. Too many data features will inevitably increase the computational complexity and lead to a long training time. Therefore, when selecting features, it is necessary to simplify the 41 different features. Here, the network behavior construction method is used to simplify the attributes, and finally 6 basic connection features that have a greater impact on the abnormal behavior type are selected for experiments. The detailed features are shown in Table 10.

[0279] Table 10 Basic characteristics table

[0280]

[0281]

[0282] Experimental Results

[0283] The KDD CUP99 dataset used has a total of 4,898,431 data points. If the dataset is modeled directly, a lot of resources will be consumed. Therefore, 600,000 data points are randomly selected from the dataset as experimental data. In the test set, 20,000 data points are selected to verify the performance of the algorithm, of which 1,000 are initialized, and then the data flow environment is simulated, and the sliding window is used to continuously obtain other data.

[0284] refer to Figure 9-11 , an algorithm with a higher detection rate can analyze abnormal behavior more accurately, so that the attack behavior can be interrupted in time and the user's personal behavior data can be effectively protected. When the number of test samples is very small, the detection rate of each type of algorithm can reach 100%. When the number of samples is between 2000 and 4000, an abnormal attack occurs, but the user abnormal behavior detection method of K-means clustering mistakenly judges the behavior as normal behavior, causing the detection rate to drop rapidly; at this time, the user abnormal behavior analysis methods such as the FC clustering algorithm and the algorithm of the present invention can accurately identify the attack behavior, so a stable detection rate is maintained. With the increase in the number of test samples, compared with the traditional FC clustering algorithm, the algorithm of the present invention adds a selection step, reduces the interference of inferior cluster members on the fusion results, improves the clustering quality, and the detection rate is relatively high and relatively stable.

[0285] Depend on Figure 5It can be seen that the accuracy of the three user anomaly analysis algorithms can reach 100% when the number of detection samples is very small. Since abnormal attacks occurred when the number of samples was 2000-4000, the accuracy of the user abnormal behavior detection method based on K-means clustering dropped rapidly, while the other two detection methods could detect abnormal attacks, making the accuracy relatively stable. As the number of samples increases, the accuracy of the algorithm of the present invention is higher.

[0286] For the three detection methods, the false alarm rate gradually increases with the increase in the number of samples. However, compared with the other two algorithms, the false alarm rate of the user abnormal behavior analysis method proposed in the present invention is relatively low, indicating that the algorithm has a good ability to recognize abnormal user behavior.

[0287] A patent for a method for detecting abnormal behavior in a time-series network based on abnormal decision-making has at least the following three advantages compared with the existing technology:

[0288] 1. Efficiently identify unknown and emerging network attack behaviors: This method can effectively identify abnormal behaviors that do not conform to regular behavior patterns by acquiring the time behavior data to be tested and combining it with the pre-configured time behavior security baseline. This capability is crucial for timely detection and response to new network attacks because it does not rely on known attack signatures or patterns, but discovers anomalies by analyzing the temporal characteristics of the behavior.

[0289] 2. Improve detection efficiency and accuracy: By optimizing the data processing process, such as introducing structures such as the time behavior security baseline gradient tree, this method can improve detection efficiency while ensuring detection accuracy. This means that when facing large-scale network data, this method can more quickly screen out abnormal behaviors, reduce the possibility of missed reports and false positives, and thus protect the security of the network environment in a timely and effective manner.

[0290] In summary, the temporal network abnormal behavior detection method based on abnormal decision-making proposed in the present invention shows significant advantages in identifying unknown attacks, combining expert knowledge, and improving detection efficiency and accuracy, providing a new and effective technical means for the field of network security.

Claims

1. A method for detecting abnormal behavior of a time series network with abnormal decision-making, characterized in that: First, the anomaly detection research process of the time series data of the overall network behavior in the computer network environment is defined; then, by observing and studying the network anomalies caused by different traffic characteristics of security events, the network anomalies are classified into two categories, namely traffic load type anomalies and communication mode type anomalies, which correspond to network flow characteristics and traffic graph characteristics respectively, and describe the overall network behavior profile; finally, the anomaly accumulation algorithm is used to define the historical time point method applicable to network traffic time series data; absolute change measurement is used on the horizontal time axis, and relative change and trend change measurement are used on the vertical time axis, and then the time series cumulative deviation anomaly detection algorithm of multi-dimensional features is defined.

2. According to the method for detecting abnormal behavior of a time series network with abnormal decision-making, the method is characterized in that: The anomaly detection research process for time series data that defines the overall network behavior in a computer network environment includes the following steps: Step A1: Detect and use the flow data of network traffic as input for subsequent data processing; Step A2: Convert the flow data into graph data to build a flow graph model, providing a data basis for extracting graph model attributes in the feature extraction step; Step A3: Count the features of time series data in a fixed time window, including flow data and graph data; define static measurement and dynamic measurement methods, extract features from the above two measurement spaces respectively, describe the overall network behavior of the current time window, and the overall network behavior changes of adjacent time windows, so as to form a feature vector describing the time series behavior profile of the overall network behavior; Step A4: Taking the multidimensional feature set of the overall network behavior as input, combining the defined historical time 0 point method to select historical data, combining the detection ideas of single point analysis and subsequence analysis of time series data, defining the angles for quantifying changes, relative changes and trend changes, thereby constructing a quantitative method for the deviation of time series data, and then measuring the time series changes of the multidimensional features of the overall network behavior; Step A5: Define an anomaly accumulation method, by calculating the time series cumulative value of multi-dimensional feature behavior and the threshold set by long-term observation experience, to make a decision on whether the current state of the overall network behavior is abnormal or not.

3. The method for detecting abnormal behavior of a time series network with abnormal decision-making according to claim 2 is characterized in that: The step A2 includes a static flow graph and a dynamic flow graph with time attributes; Static graphs are the basis of dynamic graphs. Static flow graphs are defined as a graph G = (V, E), where V represents the set of communication nodes and E is the set of edges. This means that a pair of nodes e ij =(v i ,v j ), |V| represents the number of communication nodes, and |E| represents the number of edges; node v i With node v j Communication means that there is a sending or receiving of network data packets between two nodes, that is, there is network communication between the two nodes; The dynamic flow graph represents a flow graph sequence G = {G(tn), G(tn-1), ..., G(t-1), G(t)} formed over time tn<tn-1<…<t-1<t, where G(t) represents a snapshot of the static flow graph of G at time t. The vertex set, edge set, number of vertices, and number of edges (V, E, |V|, |E|) of the graph will change over time. It can be seen that G i ≠G i+1 The graph analysis method is used to extract characteristic values ​​from the static flow graph and the dynamic flow graph respectively; First, static features describing the properties of the graph model; Second, describe the dynamic characteristics of the graph model over time. The relevant definitions are introduced as follows: Features of static traffic graph: Features obtained from static traffic graph are called static features of the graph, which are statistical summary information of static traffic graph snapshots and are used to describe and distinguish graphs; Static features focus on the relevant properties of the graph model, including statistical features such as node degree, degree distribution, maximum degree, degree, average shortest distance, clustering coefficient, degree correlation coefficient, and degree-degree correlation; Features of dynamic flow graphs; Features obtained from dynamic flow graphs are called dynamic features of graphs, which are the differences in the overall network behavior dynamic graph over time. They describe the changes of two adjacent graphs from the graph model attributes and are dynamic flow graphs that measure two continuous time series. Dynamic graph metrics compare the similarity of graphs in two adjacent time windows. The dynamic features of the graph at the time of calculation involve two graph snapshots, namely, the graph data of G(t-1) and G(t); In a graph sequence, the graph edit distance method and the maximum common subgraph method are used to measure the changes of two consecutive graphs. The graph edit distance measurement method describes the absolute changes of two adjacent graphs, while the maximum common subgraph method describes their relative changes. Graph absolute change distance; Graph edit distance GED is the absolute change between two consecutive graph snapshots in a time series graph; Relative change distance of graph; The maximum common subgraph method is used to measure the graph distance, as shown in formula (4), which describes the relative degree of change and uses the concept of maximum common subgraph MCS; ged(G i ,G j )=|V i |+|V j |-2|V i ∩V j |+|E i |+|E j |-2|E i ∩E j | (4) The maximum common subgraph means that if G is a common subgraph of G1 and G2, then there is no other common subgraph with more points than G. The distance between two non-empty graphs G1 and G2 is defined as shown in formula (5), where |G| represents the number of points in the graph and also the number of edges in the graph, where 0≤d(G1,G2)≤1. When the distance is 0, it means that the two graphs are isomorphic. d(G1,G2)=1-mcs(G1,G2)| / max(G1|,|G2|) (5) Inspired by the distance calculation method of the graph model, we also define the measurement methods of dynamic graph edge change distance and dynamic graph node change @ to quantify the changes of nodes and edges of adjacent time-series graphs; Dynamic graph edge distance; Dynamic graph edge distance is the absolute change of edges between two consecutive graph snapshots in a time series graph; of(G i ,G j )=2E i ∩E j | / (E i |+E j ) (6) Dynamic graph node distance; Dynamic graph node distance is the absolute change of nodes between two consecutive graph snapshots in a time series graph; dn(G i ,G j )=2|V i ∩V j | / (|V i |+|V j |) (7) Different graph attributes are used for quantification according to the different network behavior subject objects in network traffic analysis; the characteristics of graph nodes are used for network individual behavior analysis; the characteristics of graph evolution and evolution events are used for host group behavior analysis.

4. According to the method for detecting abnormal behavior of a time series network with abnormal decision-making, the characteristic is that: 1: Traffic load anomaly; when the impact of a security incident causes an abnormal change in traffic load, the network uplink or downlink traffic will show a situation that does not conform to the daily fluctuation range; the traffic characteristics clearly show that the time series curve of the number of bytes and data packets increases or decreases to an unexpected level, which is caused by many factors; external sudden attacks, hot news, and course selection lead to an increase in network traffic; a popular video, report, new software, or worm virus outbreak is released on a website or host; hosts infect each other and connect to remote control hosts, and drag databases cause sudden changes in traffic; through these event scenarios, sudden changes in abnormal traffic load characteristics can be observed; Definition 2: Communication pattern anomaly; when the impact of a security incident causes the communication pattern to change abnormally, the graph model constructed by the traffic graph will change accordingly; The static and dynamic methods for extracting network traffic features are defined, and combined with traffic load intensity features and communication mode features, low-intensity abnormal behaviors and traditional traffic anomalies can be detected; the methods and steps of static and dynamic measurements are described as follows: Definition 3: Static measurement method; The static measurement method is used to associate only a data set of a time window when performing feature extraction, and only extract the required features from the data set of the current time window; Definition 4: Dynamic measurement method; The dynamic measurement method is used to associate only two data sets of adjacent time windows during feature extraction, including the data set of the current data window and the data set of the adjacent historical time window, and calculate the dynamic changes of the required features in the adjacent windows.

5. The method for detecting abnormal behavior of a time series network with abnormal decision-making according to claim 1 is characterized in that: Describe the definition of the historical time point method, which detects time series data anomalies through historical data and reduces computing and storage costs; the historical time point method includes two important definitions: vertical time axis and horizontal time axis; Definition 5: Vertical time axis; the vertical time axis is in days and days are of the same time type; Definition 6: Horizontal time axis; the horizontal time axis is the data within a fixed day, and the time unit needs to be set; In the network traffic time series data, the traffic characteristics show differences between weekdays and weekends. The historical time method also distinguishes the network flow time series data on weekdays and weekends.

6. The method for detecting abnormal behavior of a time series network with abnormal decision-making according to claim 1 is characterized in that: The deviation of overall network behavior is further measured by absolute change, relative change, and trend change: Definition 7: Absolute change; calculate the overall network behavior time series profile at the current time, obtain the overall network behavior time series profile historical data, and calculate the absolute value of the adjacent time of the i-th feature |f i (t)-f i (t′)|,w i The weight of eigenvalue i, Σw i =1(i=1,i≤m); Here, multiple data points i are selected from adjacent data, and the cumulative value of the absolute change of the overall network behavior time series profile is shown in formula (8): Definition 8: Relative change; calculate the overall network behavior time series profile at the current time t, obtain the overall network behavior time series profile historical data, extract the periodicity related data of the i-th feature at the current time t, and calculate the adjacent previous K-th week time. The ratio of the maximum values ​​of the data points, As shown in formula (9): Definition 9: Trend change; The characteristic value of the overall network behavior time series profile establishes the current shape subsequence C′ at the current time t, obtains the historical data of the overall network behavior time series profile, and calculates the Euclidean distance of the subsequence; the shape change value of the overall network behavior time series profile mainly focuses on the characteristic shape trend of the overall network behavior time series profile in this time period, rather than the specific characteristic value size.

7. The method for detecting abnormal behavior of a time series network with abnormal decision-making according to claim 6 is characterized in that: Get the subsequence of the most recent N data points before the detection time. It is known that the time sequence of the current N adjacent data points is U′={U′1,U′2,…,U′ N }, the key steps of subsequence comparison using SAX are as follows; 1) Normalization: Normalize the time series U′ into a standard series U={U1,U2,…,U N }, as shown in formula (10): 2) PAA dimensionality reduction: Perform PAA representation on U to obtain 1≤N≤n; where N is the length of the PAA representation of the time series, describing a time series of length n, as shown in formula (11): 3) Symbolization; Select the base size a = 10, and find the split point β of the interval according to the letter base size in the Gaussian distribution table as shown below i ; alpha i Represents the i-th letter in the alphabet, for example, alpha2 = b, alpha3 = c, through Map the value represented by PAA to the corresponding letter in the alphabet and finally discretize it into a string Next, use formula (12) to calculate the distance between each letter in base 10; 4) Similarity measurement: Compare and accumulate the SAX sequences of each feature value at the current moment; Formula (13) is used to calculate the string sequence and The minimum distance; Based on the above introduction of the key points of timing algorithm and parameter selection, is the string form of A, constructed through time series data; where w is the weight value of the feature, δ is the weight of the time distance between the subsequence and the current subsequence, and the calculation of the cumulative value of the time series trend is shown in formula (14): In summary, according to formula (14), the formula for the cumulative deviation of the overall network behavior time series profile based on the time series analysis method is shown in formula (15), where EA=θ1EA (1) +θ2EA (1) +θ3EA (1) (15)。

Images