A method and system for detecting network traffic anomalies based on big data

Abstract

This invention belongs to the field of network security and communication technology. It discloses a method and system for detecting network traffic anomalies based on big data. The method includes collecting and analyzing historical network traffic data, extracting cross-protocol dependencies and traffic cycle patterns, and generating a historical steady-state invariant baseline. The historical steady-state invariant baseline is compared with a real-time steady-state invariant baseline generated from real-time traffic data. In short-term burst traffic windows, sudden increases in traffic rate and deviations in protocol distribution are identified. In long-term periodic traffic windows, slow scanning and covert outward transmission behaviors are identified, generating abnormal traffic events. These abnormal traffic events are mapped into node and edge relationships of different dimensions. In a cloud computing environment, abnormal behavior of virtual machines accessing the database is linked with VPN account login records at the graph level to form a dynamic network behavior graph. This improves the capabilities of network traffic anomaly identification, attack tracing, and security protection.

Description

Technical Field

[0001] This invention relates to the field of network security and communication technology, and more specifically, to a method and system for detecting network traffic anomalies based on big data. Background Technology

[0002] Existing methods and systems for detecting network traffic anomalies mainly suffer from the following problems:

[0003] With the widespread application of cloud computing and big data technologies, enterprise and institutional networks are constantly expanding in scale, significantly increasing the complexity of network traffic and the diversity of threats. Traditional methods for detecting network traffic anomalies mainly rely on single ports or simple statistical indicators, making it difficult to quantify the diversity and randomness of source entity access ports, and easily missing attacks disguised as fixed ports. In addition, existing technologies can usually only analyze abnormal events such as virtual machine access to databases or VPN login events independently, and cannot determine whether there is a correlation between the two, making it difficult to identify the causal relationship of attack behaviors.

[0004] Existing technologies primarily present anomalies in the form of logs or single records, lacking the ability to systematically analyze the propagation paths and impact ranges between anomaly events, thus limiting advanced threat detection and attribution capabilities. In cross-subnet or cross-host scenarios, traditional methods typically monitor only within a single subnet or host, making it difficult to correlate and analyze abnormal access behavior of the same entity across different subnets, and failing to effectively identify potential lateral movement attacks—that is, attackers gradually scanning or infiltrating other subnets from one subnet. Traditional anomaly detection methods mostly rely on instantaneous traffic thresholds or single-event judgments, lacking analysis of the temporal regularity of abnormal behavior of the same entity, making it difficult to detect low-speed scanning, periodic covert access, or long-term persistent potential attack behaviors. For potential abnormal control behaviors of internal network hosts accessing external network servers, existing technologies cannot comprehensively judge through multi-dimensional features, easily leading to missed or false alarms.

[0005] In summary, existing technologies mostly rely on single-dimensional or single-subnet analysis methods, lacking cross-domain and cross-subnet collaborative analysis methods. This makes it difficult to form a complete abnormal propagation link or causal path, limiting the improvement of the ability to identify and defend against abnormal behavior in complex network environments.

[0006] In view of this, the present invention proposes a network traffic anomaly detection method based on big data to solve the above problems. Summary of the Invention

[0007] To overcome the aforementioned deficiencies of the prior art and to achieve the above objectives, the present invention provides the following technical solution:

[0008] A method for detecting network traffic anomalies based on big data includes:

[0009] S1. Collect and analyze historical network traffic data, extract cross-protocol dependencies and traffic cycle patterns, and generate historical steady-state invariant baselines;

[0010] S2. Compare the historical steady-state invariant baseline with the real-time steady-state invariant baseline generated from real-time traffic data. In short-term burst traffic windows, identify sudden increases in traffic rate and deviations in protocol distribution. In long-term periodic traffic windows, identify slow scanning and covert transmission behaviors and generate abnormal traffic events.

[0011] S3. Map abnormal traffic events into nodes and edge relationships of different dimensions, and in the cloud computing environment, link the abnormal behavior of virtual machines accessing the database with VPN account login records at the graph level to form a dynamic network behavior graph.

[0012] S4. In the dynamic network behavior graph, the abnormal trigger points are connected with the corresponding propagation and impact paths to form a causal chain of abnormal evidence; the minimum cut evidence extraction method is used to extract the necessary abnormal evidence sub-chains, and the necessary abnormal evidence sub-chains are solidified by timestamps and hash chain digests.

[0013] S5. Based on the solidified necessary abnormal evidence subchain, cross-domain abnormal synthesis is performed between different subnets. If the same IP periodically scans between different subnets, it is determined to be a potential lateral movement attack. If an internal network host abnormally accesses an external network server within a preset time window, it is determined to be an abnormal control command, thereby realizing the collaborative perception of abnormal network traffic.

[0014] Specifically, the method for generating the historical steady-state invariant baseline includes:

[0015] Historical network traffic data is collected, including five-tuple information, transport layer message data, and application layer protocol interaction data. The historical network traffic data is then cleaned, standardized, and timestamp aligned to form a standardized traffic record set.

[0016] Based on a normalized traffic record set, different session traffic is mapped according to protocol level. Association rule mining and frequent subsequence analysis methods are used to identify causal dependencies between protocols. Conditional probability and mutual information are used to measure the stability of protocol dependencies and obtain stable cross-protocol dependency chains.

[0017] The normalized traffic record set is constructed as a time series, and the periodic components of the traffic are extracted using the sliding window averaging method. The stability of the periodic pattern is tested by combining the autocorrelation function and the partial autocorrelation function to obtain the periodic pattern under different time scales. The cross-protocol stable dependency chain is integrated with the periodic pattern to form a set of steady-state features in cross-protocol and periodic dimensions, and solidified into a historical steady-state invariant baseline in the form of graph structure and time series.

[0018] Specifically, the method for identifying sudden increases in traffic rate and deviations in protocol distribution includes:

[0019] Collect real-time traffic data, extract cross-protocol dependencies and traffic cycle patterns, and generate a real-time steady-state invariant baseline; divide network traffic into short-term burst traffic windows at the minute level; compare the statistical parameters of the real-time steady-state invariant baseline with those of historical steady-state invariant baselines; the statistical parameters include traffic rate, protocol distribution, port distribution, session creation rate, and session duration;

[0020] When the real-time traffic rate exceeds the fluctuation range of the sum of the historical average and the preset multiple standard deviation, and remains above this range in consecutive different windows, it is determined to be a sudden increase in traffic rate. For protocol distribution, the chi-square test is used to compare differences and calculate the degree of difference between the real-time protocol distribution and the historical protocol distribution. When the degree of difference exceeds the preset degree of difference threshold, it is determined to be a deviation of the protocol distribution.

[0021] Specifically, the method for generating abnormal traffic events includes:

[0022] Network traffic is divided into long-term periodic traffic windows on a weekly basis. Within each periodic window, the frequency of source IP activity, the frequency of destination IP activity, the number of port scans, the session establishment and termination coefficient, the packet size, and the transmission rate change trends are summarized.

[0023] The behavior of the same source IP accessing different target IPs within a long-term traffic window is analyzed. When the access covers different targets and the duration of each connection is greater than a preset duration threshold, the connection interval is greater than a preset time interval, and the behavior deviates from the historical steady-state invariant baseline, it is judged as slow scanning behavior.

[0024] Analyze the data transmission patterns of internal network hosts to external servers. When data packets are continuously sent to a preset external IP, and the transmission rate, traffic volume, and periodicity all deviate from the historical steady-state invariant baseline, it is determined to be a hidden transmission behavior, and an abnormal traffic event is generated.

[0025] Specifically, the method for mapping abnormal traffic events into node-edge relationships of different dimensions includes:

[0026] The key entities in the abnormal traffic event are mapped as graph nodes. The key entities include host nodes corresponding to the source IP and destination IP, protocol nodes and port nodes involved in the abnormal access, session nodes corresponding to the abnormal connection session, and event nodes that identify the abnormal type and occurrence time window.

[0027] Graph edges are established based on the relationships between entities in abnormal traffic events. The graph edges include host-to-host communication edges between source IP and destination IP, host-to-protocol edges between hosts and protocol nodes, host-to-port edges between hosts and port nodes, session-to-host edges between session nodes and involved host nodes, and event-to-entity edges between event nodes and their corresponding hosts, ports, and protocol nodes.

[0028] Specifically, the method for forming a dynamic network behavior graph includes:

[0029] In a cloud computing environment, for abnormal behavior of virtual machines accessing the database, the preset access port set and access port distribution are statistically analyzed, and the port conditional entropy is calculated. The port conditional entropy is normalized to distinguish between random port scanning and abnormal access behavior disguised as fixed ports.

[0030] Abnormal events involving virtual machine access to the database are defined as VM-DB abnormal events, and VPN login operations are defined as VPN login events. A matching set is established for the two types of events. The matching condition is that the events have the same unique identifier and the time interval between the two events is within a preset time window threshold. When this matching condition is met, it is considered that the VM abnormal access and the VPN login event are related.

[0031] The VPN account node is connected to the corresponding virtual machine node, and then the virtual machine node is connected to the database node to form a path relationship from the VPN account to the corresponding virtual machine, and then from the virtual machine to the database. The exception type, event timestamp and port conditional entropy attribute are marked on the path; automatic cascading across events is realized, and a single VM-DB exception event is associated with the corresponding VPN login event to form a causal path, and finally a dynamic network behavior graph is constructed.

[0032] Specifically, the method for obtaining the causal chain-like abnormal evidence chain includes:

[0033] The abnormal triggering nodes in the dynamic network behavior graph are identified. The abnormal triggering nodes include abnormal traffic sources, abnormal virtual machine access events, abnormal VPN login events, and abnormal database operation events. The abnormal type, event time, and related attribute information of each abnormal triggering node are recorded.

[0034] The propagation path of the abnormal triggering node is traced along the edge relationships between nodes in the dynamic network behavior graph. The propagation path includes access, session establishment or data transmission behavior from the source node to the target node. The abnormal triggering node and the nodes traced along the propagation path are connected in sequence to form a complete causal chain of abnormal evidence.

[0035] Specifically, the method for solidifying the necessary abnormal evidence sub-chain includes:

[0036] By using the minimum cut algorithm, we find the minimum set of edge weights that can separate the source node from the terminal node in the complete causal chain, identify the key event chain segments in the abnormal propagation process, and extract the necessary abnormal evidence sub-chains. For the extracted necessary abnormal evidence sub-chains, we generate hash digests according to the time sequence of the key events and mark them with timestamps to solidify the integrity and immutability of the sub-chains.

[0037] Specifically, the method for collaboratively sensing network traffic anomalies includes:

[0038] Collect abnormal event information of each subnet involved in the solidified necessary abnormal evidence subchain. The abnormal event information includes source IP, destination IP, event time, access port and abnormal event type. Summarize the abnormal events of each subnet by subject entity and count the number of abnormal accesses, access ports and activity time of each subject in different subnets.

[0039] For abnormal access behavior of the same entity in different subnets, calculate the autocorrelation function of the access sequence and obtain the maximum periodic score within a preset time window; if the periodic score of the same entity exceeds the preset periodic score threshold, and the number of abnormal events in the subnet exceeds the corresponding preset periodic score threshold in at least m subnets, then the entity is determined to have a potential lateral movement attack.

[0040] For internal network hosts accessing external network servers, the frequency, volume, and port distribution of access within a preset time window are statistically analyzed. When the access behavior deviates from the historical steady-state invariant baseline and conforms to a preset abnormal access pattern, it is determined as a potential abnormal control command, thus achieving collaborative perception of network traffic anomalies.

[0041] A network traffic anomaly detection system based on big data includes:

[0042] The steady-state pattern analysis module collects and analyzes historical network traffic data, extracts cross-protocol dependencies and traffic cycle patterns, and generates historical steady-state invariant baselines.

[0043] The abnormal traffic identification module compares the historical steady-state invariant baseline with the real-time steady-state invariant baseline generated from real-time traffic data. In short-term burst traffic windows, it identifies sudden increases in traffic rate and deviations in protocol distribution; in long-term periodic traffic windows, it identifies slow scanning and covert transmission behavior and generates abnormal traffic events.

[0044] The behavior graph construction module maps abnormal traffic events into nodes and edge relationships of different dimensions, and in the cloud computing environment, it links the abnormal behavior of virtual machines accessing the database with VPN account login records at the graph level to form a dynamic network behavior graph.

[0045] The verifiable solidification module connects the abnormal trigger points with the corresponding propagation and impact paths in the dynamic network behavior graph to form a causal chain of abnormal evidence; it uses the minimum cut evidence extraction method to extract necessary abnormal evidence sub-chains and solidifies the necessary abnormal evidence sub-chains through timestamps and hash chain digests;

[0046] The collaborative threat assessment module, based on the solidified necessary abnormal evidence sub-chain, performs cross-domain anomaly synthesis between different subnets. If the same IP periodically scans between different subnets, it is identified as a potential lateral movement attack; if an internal network host abnormally accesses an external network server within a preset time window, it is identified as an abnormal control command, thus achieving collaborative perception of network traffic anomalies.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] This invention utilizes port diversity measurement normalized by port conditional entropy to effectively identify random port scans and spoofed port access, reducing false negatives and false positives. It establishes causal paths by key matching and time window association between VM-DB anomaly events and VPN login events, enabling cross-event and cross-domain anomaly analysis and enhancing network security situational awareness. A dynamic network behavior graph is constructed, linking the anomaly behaviors of VPN accounts, virtual machines, and databases to visually display attack paths, impact ranges, and propagation chains, providing a basis for security response, attribution analysis, and defense strategies. By constructing a dynamic graph in a cloud computing environment, combined with port entropy and time window mechanisms, it achieves real-time perception and analysis of anomaly behaviors in highly dynamic and complex networks, suitable for multi-tenant and multi-virtualization environments.

[0049] By collecting anomalous event information from various subnets within the necessary anomaly evidence subchain, and summarizing it by subject entity, the system statistically analyzes the anomalous access frequency, access port, and activity time of each subject across different subnets to analyze the cross-subnet anomalous behavior of the same subject. Using an autocorrelation function to calculate the temporal periodicity of the access sequence and calculating the maximum periodicity score within a preset time window, the system combines the temporal regularity with the spatial subnet distribution to identify potential lateral movement attacks. By combining the maximum periodicity score with a subnet anomalous event number threshold, the system quantifies the periodic scanning behavior of the attacking subject and scientifically judges the attack behavior, avoiding false positives from single anomalous events. By combining the lateral movement judgment function with temporal periodicity characteristics and spatial subnet distribution characteristics, the system achieves accurate detection of potential cross-domain scanning or attack behaviors. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of a network traffic anomaly detection method based on big data according to the present invention.

[0051] Figure 2This is a schematic diagram of the structure of a network traffic anomaly perception system based on big data according to the present invention.

[0052] Figure 3 This is a schematic diagram of the method for forming a dynamic network behavior graph provided by the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Example 1

[0055] Please see Figure 1 and Figure 3 As shown, this embodiment provides a method for detecting network traffic anomalies based on big data, specifically including the following steps:

[0056] S1. Collect and analyze historical network traffic data, extract cross-protocol dependencies and traffic cycle patterns, and generate historical steady-state invariant baselines;

[0057] S2. Compare the historical steady-state invariant baseline with the real-time steady-state invariant baseline generated from real-time traffic data. In short-term burst traffic windows, identify sudden increases in traffic rate and deviations in protocol distribution. In long-term periodic traffic windows, identify slow scanning and covert transmission behaviors and generate abnormal traffic events.

[0058] S3. Map abnormal traffic events into nodes and edge relationships of different dimensions, and in the cloud computing environment, link the abnormal behavior of virtual machines accessing the database with VPN account login records at the graph level to form a dynamic network behavior graph.

[0059] S4. In the dynamic network behavior graph, the abnormal trigger points are connected with the corresponding propagation and impact paths to form a causal chain of abnormal evidence; the minimum cut evidence extraction method is used to extract the necessary abnormal evidence sub-chains, and the necessary abnormal evidence sub-chains are solidified by timestamps and hash chain digests.

[0060] S5. Based on the solidified necessary abnormal evidence subchain, cross-domain abnormal synthesis is performed between different subnets. If the same IP periodically scans between different subnets, it is determined to be a potential lateral movement attack. If an internal network host abnormally accesses an external network server within a preset time window, it is determined to be an abnormal control command, thereby realizing the collaborative perception of abnormal network traffic.

[0061] Methods for generating historical steady-state invariant baselines include:

[0062] Historical network traffic data is collected, including five-tuple information, transport layer message data, and application layer protocol interaction data. The historical network traffic data is then cleaned, standardized, and timestamp aligned to form a standardized traffic record set.

[0063] Based on a normalized traffic record set, different session traffic is mapped according to protocol level. Association rule mining and frequent subsequence analysis methods are used to identify causal dependencies between protocols. Conditional probability and mutual information are used to measure the stability of protocol dependencies and obtain stable cross-protocol dependency chains.

[0064] The normalized traffic record set is constructed as a time series, and the periodic components of the traffic are extracted using the sliding window averaging method. The stability of the periodic pattern is tested by combining the autocorrelation function and the partial autocorrelation function to obtain the periodic pattern under different time scales. The cross-protocol stable dependency chain is integrated with the periodic pattern to form a set of steady-state features in cross-protocol and periodic dimensions, and solidified into a historical steady-state invariant baseline in the form of graph structure and time series.

[0065] Methods for identifying sudden increases in traffic rates and deviations in protocol distribution include:

[0066] Collect real-time traffic data, extract cross-protocol dependencies and traffic cycle patterns, and generate a real-time steady-state invariant baseline; divide network traffic into short-term burst traffic windows at the minute level; compare the statistical parameters of the real-time steady-state invariant baseline with those of historical steady-state invariant baselines; the statistical parameters include traffic rate, protocol distribution, port distribution, session creation rate, and session duration;

[0067] When the real-time traffic rate exceeds the fluctuation range of the sum of the historical average and the preset multiple standard deviation, and remains above this range in consecutive different windows, it is determined to be a sudden increase in traffic rate. For protocol distribution, the chi-square test is used to compare differences and calculate the degree of difference between the real-time protocol distribution and the historical protocol distribution. When the degree of difference exceeds the preset degree of difference threshold, it is determined to be a deviation of the protocol distribution.

[0068] Methods for generating abnormal traffic events include:

[0069] Network traffic is divided into long-term periodic traffic windows on a weekly basis. Within each periodic window, the frequency of source IP activity, the frequency of destination IP activity, the number of port scans, the session establishment and termination coefficient, the packet size, and the transmission rate change trends are summarized.

[0070] The behavior of the same source IP accessing different target IPs within a long-term traffic window is analyzed. When the access covers different targets and the duration of each connection is greater than a preset duration threshold, the connection interval is greater than a preset time interval, and the behavior deviates from the historical steady-state invariant baseline, it is judged as slow scanning behavior.

[0071] Analyze the data transmission patterns of internal network hosts to external servers. When data packets are continuously sent to a preset external IP, and the transmission rate, traffic volume, and periodicity all deviate from the historical steady-state invariant baseline, it is determined to be a hidden transmission behavior, and an abnormal traffic event is generated.

[0072] Methods for mapping abnormal traffic events into node-edge relationships of different dimensions include:

[0073] The key entities in the abnormal traffic event are mapped as graph nodes. The key entities include host nodes corresponding to the source IP and destination IP, protocol nodes and port nodes involved in the abnormal access, session nodes corresponding to the abnormal connection session, and event nodes that identify the abnormal type and occurrence time window.

[0074] Graph edges are established based on the relationships between entities in abnormal traffic events. The graph edges include host-to-host communication edges between source IP and destination IP, host-to-protocol edges between hosts and protocol nodes, host-to-port edges between hosts and port nodes, session-to-host edges between session nodes and involved host nodes, and event-to-entity edges between event nodes and their corresponding hosts, ports, and protocol nodes.

[0075] Methods for generating dynamic network behavior graphs include:

[0076] In a cloud computing environment, for abnormal behavior of virtual machines accessing the database, the preset access port set and access port distribution are statistically analyzed, and the port conditional entropy is calculated. The port conditional entropy is normalized to distinguish between random port scanning and abnormal access behavior disguised as fixed ports.

[0077] The normalized port diversity metric is: ;in, This represents the normalized port diversity metric, used to measure the source entity. Port access diversity; Represents the port conditional entropy; This represents the total size of the port set, i.e., the total number of all ports that can be accessed in the network; The logarithm of the total size of the ports is used as the maximum possible value of the entropy for normalization. Indicates the default port;

[0078] Abnormal events involving virtual machine access to the database are defined as VM-DB abnormal events, and VPN login operations are defined as VPN login events. A matching set is established for the two types of events. The matching condition is that the events have the same unique identifier and the time interval between the two events is within a preset time window threshold. When this matching condition is met, it is considered that the VM abnormal access and the VPN login event are related.

[0079] The set of matches that meet the matching conditions is: ;in, This represents all elements that satisfy the matching condition. The set of matching pairs; This refers to an abnormal event in which a virtual machine (VM) accesses a database (DB). It can be understood as an event object that records the abnormal access situation of a virtual machine (VM) to the database (DB) at a certain time. This refers to VPN login records, which are another event object that records information such as the login time, source IP, and target VM for a specific VPN account. Indicates an abnormal event The key is used to identify the event subject, such as a VPN account, VM ID, or a combination of account and host. Its purpose is to match whether VPN login and abnormal VM-DB access belong to the same account or the same subject. Indicates an abnormal event The key is used to match the subject of VPN login records; Indicates an abnormal event The timestamp of the event; Indicates an abnormal event The timestamp of the event; Indicates the preset time window threshold;

[0080] The VPN account node is connected to the corresponding virtual machine node, and then the virtual machine node is connected to the database node to form a path relationship from the VPN account to the corresponding virtual machine, and then from the virtual machine to the database. The exception type, event timestamp and port conditional entropy attribute are marked on the path; automatic cascading across events is realized, and a single VM-DB exception event is associated with the corresponding VPN login event to form a causal path, and finally a dynamic network behavior graph is constructed.

[0081] Methods for obtaining causal chain-like abnormal evidence chains include:

[0082] The abnormal triggering nodes in the dynamic network behavior graph are identified. The abnormal triggering nodes include abnormal traffic sources, abnormal virtual machine access events, abnormal VPN login events, and abnormal database operation events. The abnormal type, event time, and related attribute information of each abnormal triggering node are recorded.

[0083] The propagation path of the abnormal triggering node is traced along the edge relationships between nodes in the dynamic network behavior graph. The propagation path includes access, session establishment or data transmission behavior from the source node to the target node. The abnormal triggering node and the nodes traced along the propagation path are connected in sequence to form a complete causal chain of abnormal evidence.

[0084] Methods for solidifying necessary sub-chains of anomalous evidence include:

[0085] Using the minimum cut algorithm, we find the minimum set of edge weights that can separate the source node from the terminal node in a complete causal chain, and identify the key event chain segments in the anomaly propagation process. The key event chain segments are chain-like structures consisting of anomaly triggering nodes, node attribute information, and propagation paths between nodes. We extract necessary anomaly evidence sub-chains. For the extracted necessary anomaly evidence sub-chains, we generate hash digests according to the time sequence of the key events and mark them with timestamps to solidify the integrity and immutability of the sub-chains.

[0086] It should be noted that the necessary anomaly evidence subchain is the smallest event chain extracted from the complete anomaly propagation chain or key event segment, which fully reflects the causal relationship of anomaly propagation and removes redundant events. Specifically, the necessary anomaly evidence subchain has the following characteristics:

[0087] Causal integrity means that each event node and path contained in the subchain has a direct or indirect causal effect on the formation of the anomaly propagation, and can fully describe the key process of the anomaly from triggering to spread.

[0088] Minimal redundancy: Subchains do not contain event nodes or paths that do not actually contribute to anomaly propagation analysis or tracing; only necessary nodes and edges in key event chain segments are retained, making the chain segments concise and efficient.

[0089] It can be used for forensics and analysis. Events in the subchain can be used for anomaly tracing, propagation analysis, security response, and auditing, ensuring that each event is verifiable and has analytical value.

[0090] The necessary anomaly evidence sub-chain can be solidified using timestamps and hash chain digests, ensuring immutability and integrity, and facilitating subsequent security audits and legal evidence collection. In summary, the necessary anomaly evidence sub-chain is the smallest, core event chain with causal explanation capabilities selected from the complete anomaly propagation chain; it is the foundational chain segment used for anomaly analysis, tracing, and solidifying evidence.

[0091] Methods for achieving collaborative awareness of network traffic anomalies include:

[0092] Collect abnormal event information of each subnet involved in the solidified necessary abnormal evidence subchain. The abnormal event information includes source IP, destination IP, event time, access port and abnormal event type. Summarize the abnormal events of each subnet by subject entity and count the number of abnormal accesses, access ports and activity time of each subject in different subnets.

[0093] For abnormal access behavior of the same entity across different subnets, calculate the autocorrelation function (ACF) of the access sequence within a preset time window. [{{\tau}_{min},{\tau}_{max}}] Find the maximum periodicity score within the range; the periodicity score reflects the temporal regularity of abnormal access by the same entity between different subnets and is used to identify potential lateral movement behavior. This represents the minimum time lag value for periodic analysis; This represents the maximum time lag value in periodic analysis;

[0094] The maximum periodicity score is: ;in, Representing the same subject The periodic score is taken as the maximum value of the autocorrelation function within the lag interval, which is used to quantify the periodic scanning behavior over time. Representing the same subject Abnormal event sequence with time lag value The autocorrelation function value reflects the temporal regularity of abnormal events; Indicates the time delay lag value; This indicates that the same entity (e.g., the same IP address, the same user, or the same device) is used as the object of lateral movement and anomalous activity analysis.

[0095] If the periodic score of the same entity exceeds the preset periodic score threshold, and the number of abnormal events in the subnet exceeds the corresponding preset periodic score threshold in at least m subnets, then the entity is determined to have a potential lateral movement attack. The lateral movement determination function combines the periodic features of the time dimension with the subnet distribution features of the spatial dimension to identify cross-domain scanning or potential attack behaviors.

[0096] Based on temporal periodicity and spatial distribution, potential lateral movement attacks are identified: ;in, Indicates the potential lateral movement determination result, when Time indicates the same subject If a potential lateral movement attack is detected, the value is 0; otherwise, the value is 0. This indicates an indicator function that returns 1 if the condition within the parentheses is true, and 0 otherwise. This indicates a preset periodic scoring threshold; This indicates a subnet identifier, representing different network subnets involved in the same entity's activities; The subnet set represents all the subnets in the network to be analyzed. This indicates the number of anomalous events occurring within the same entity in the subnet; Indicates the minimum number of subnets;

[0097] For internal network hosts accessing external network servers, the frequency, volume, and port distribution of access within a preset time window are statistically analyzed. When the access behavior deviates from the historical steady-state invariant baseline and conforms to a preset abnormal access pattern, it is determined as a potential abnormal control command, thus achieving collaborative perception of network traffic anomalies.

[0098] It should be noted that the preset abnormal access pattern refers to a set of rules or features that are defined or set in advance to identify potential abnormal access behavior. It describes the behavior pattern that network traffic may exhibit under abnormal conditions.

[0099] Preset abnormal access patterns include those based on statistical characteristics, such as abnormal ranges in metrics like access frequency, traffic volume, and session duration; those based on behavioral patterns, such as periodic scanning, access during abnormal time periods, and distribution characteristics across subnets or targets; and those based on protocol or port characteristics, such as abnormally concentrated or dispersed access ports and cross-protocol access deviating from historical patterns. Combined conditions are also used; abnormal access patterns are usually not based on a single indicator but rather on a combination of multiple indicators and characteristics, forming rules or threshold logic used to determine whether a certain access behavior is abnormal.

[0100] By associating the results of lateral movement determination and anomaly control command determination with a solidified chain of necessary anomaly evidence, a collaborative sensing result across subnets and domains is formed. Through this collaborative sensing mechanism, comprehensive identification and correlation analysis of complex network traffic anomalies can be achieved by integrating temporal, spatial, and behavioral characteristics, providing a basis for security defense and response decisions.

[0101] The preset difference threshold is set by staff based on the results of historical data analysis. This historical analysis process includes the system collecting multiple differences and calculating their average value as a reference to obtain the preset difference threshold. Similarly, the preset duration threshold, preset time window threshold, and preset periodic scoring threshold are also set by staff based on the system's historical operating data and the specific application scenario requirements.

[0102] This embodiment effectively identifies random port scanning and spoofed port access by using port conditional entropy normalization for port diversity measurement, reducing false negatives and false positives. It performs key matching and time window association between VM-DB anomaly events and VPN login events to form causal paths, enabling cross-event and cross-domain anomaly analysis and enhancing network security situational awareness. A dynamic network behavior graph is constructed, linking the abnormal behaviors of VPN accounts, virtual machines, and databases to visually display attack paths, impact ranges, and propagation chains, providing a basis for security response, source tracing analysis, and defense strategies. By constructing a dynamic graph in a cloud computing environment, combined with port entropy and time window mechanisms, it achieves real-time perception and analysis of anomaly behavior in highly dynamic and complex networks, suitable for multi-tenant and multi-virtualization environments.

[0103] By collecting anomalous event information from various subnets within the necessary anomaly evidence subchain, and summarizing it by subject entity, the system statistically analyzes the anomalous access frequency, access port, and activity time of each subject across different subnets to analyze the cross-subnet anomalous behavior of the same subject. Using an autocorrelation function to calculate the temporal periodicity of the access sequence and calculating the maximum periodicity score within a preset time window, the system combines the temporal regularity with the spatial subnet distribution to identify potential lateral movement attacks. By combining the maximum periodicity score with a subnet anomalous event number threshold, the system quantifies the periodic scanning behavior of the attacking subject and scientifically judges the attack behavior, avoiding false positives from single anomalous events. By combining the lateral movement judgment function with temporal periodicity characteristics and spatial subnet distribution characteristics, the system achieves accurate detection of potential cross-domain scanning or attack behaviors.

[0104] This system tracks and analyzes the access frequency, traffic volume, and port distribution of internal network hosts to external network servers. By combining this data with historical steady-state invariant baselines and preset abnormal access patterns, it identifies potential abnormal control commands. It can capture long-term, low-speed, and covert external control behaviors, enhancing the deep perception capabilities of network defense. Through cross-subnet lateral movement analysis and internal / external network abnormal access analysis, it integrates multi-dimensional features for collaborative perception, achieving global monitoring of network traffic anomalies and causal chain anomaly analysis. This improves the accuracy and completeness of anomaly detection, supporting comprehensive analysis of attack paths, attackers, and potential threats.

[0105] Example 2

[0106] Please see Figure 2 As shown, parts not described in detail in this embodiment are described in Embodiment 1. A network traffic anomaly detection system based on big data is provided, including:

[0107] The steady-state pattern analysis module collects and analyzes historical network traffic data, extracts cross-protocol dependencies and traffic cycle patterns, and generates historical steady-state invariant baselines.

[0108] The abnormal traffic identification module compares the historical steady-state invariant baseline with the real-time steady-state invariant baseline generated from real-time traffic data. In short-term burst traffic windows, it identifies sudden increases in traffic rate and deviations in protocol distribution; in long-term periodic traffic windows, it identifies slow scanning and covert transmission behavior and generates abnormal traffic events.

[0109] The behavior graph construction module maps abnormal traffic events into nodes and edge relationships of different dimensions, and in the cloud computing environment, it links the abnormal behavior of virtual machines accessing the database with VPN account login records at the graph level to form a dynamic network behavior graph.

[0110] The verifiable solidification module connects the abnormal trigger points with the corresponding propagation and impact paths in the dynamic network behavior graph to form a causal chain of abnormal evidence; it uses the minimum cut evidence extraction method to extract necessary abnormal evidence sub-chains and solidifies the necessary abnormal evidence sub-chains through timestamps and hash chain digests;

[0111] The collaborative threat assessment module, based on the solidified necessary abnormal evidence sub-chain, performs cross-domain anomaly synthesis between different subnets. If the same IP periodically scans between different subnets, it is identified as a potential lateral movement attack; if an internal network host abnormally accesses an external network server within a preset time window, it is identified as an abnormal control command, thus achieving collaborative perception of network traffic anomalies.

[0112] Since the electronic device described in this embodiment is the one used to implement the network traffic anomaly detection method and system based on big data in this application embodiment, those skilled in the art can understand the specific implementation methods and various variations of the electronic device in this embodiment based on the network traffic anomaly detection method and system based on big data in this application embodiment. Therefore, how the electronic device implements the method in this application embodiment will not be described in detail here. Any electronic device used by those skilled in the art to implement the network traffic anomaly detection method and system based on big data in this application embodiment falls within the scope of protection of this application.

[0113] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0114] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for network traffic anomaly perception based on big data, characterized in that, The application relates to a network traffic anomaly detection method based on stable state invariant baseline. The method comprises the following steps: S1, collecting and analyzing historical network traffic data to extract cross-protocol dependencies and traffic periodic patterns and generate a historical stable state invariant baseline; The method for generating the historical stable state invariant baseline comprises: collecting historical network traffic data, which comprises quintuple information, transport layer message data and application layer protocol interaction data, and performing cleaning, standardization and timestamp alignment processing on the historical network traffic data to form a standardized traffic record set; based on the standardized traffic record set, mapping different session traffic according to protocol levels, identifying the causal dependency relationship between protocols by using an association rule mining and frequent subsequence analysis method, and adopting conditional probability and mutual information to measure the stability of the protocol dependency relationship to obtain a cross-protocol stable dependency chain; the standardized traffic record set is constructed into a time sequence, the periodic component of the traffic is extracted by using a sliding window average method, and the stability of the periodic law is tested by using an autocorrelation function and a partial autocorrelation function to obtain periodic patterns at different time scales; the cross-protocol stable dependency chain and the periodic patterns are fused to form a stable state characteristic set in the cross-protocol dimension and the periodic dimension, and are solidified into a historical stable state invariant baseline in the form of a graph structure and a time sequence; S2, comparing the historical stable state invariant baseline with a real-time stable state invariant baseline generated by real-time traffic data, identifying traffic rate surges and protocol distribution deviations in a short-time burst traffic window, identifying slow scanning and hidden external transmission behaviors in a long-time periodic traffic window, and generating abnormal traffic events; S3, mapping the abnormal traffic events into node and edge relationships in different dimensions, and in a cloud computing environment, performing graph cascade actions on abnormal behaviors of virtual machine access databases and VPN account login records to form a dynamic network behavior graph; S4, in the dynamic network behavior graph, connecting abnormal trigger points and corresponding propagation influence paths to form a causal chain evidence chain; necessary abnormal evidence subchains are extracted by using a minimum cut evidence extraction method, and the necessary abnormal evidence subchains are solidified by using a timestamp and a hash chain digest; The necessary abnormal evidence subchain is a minimum event chain extracted from a complete abnormal propagation chain or a key event segment, which can fully reflect the abnormal propagation causal relationship and remove redundant events; 2. The method of claim 1, wherein, S5, based on the solidified necessary abnormal evidence subchain, performing cross-domain abnormal synthesis between different subnets, if the same IP periodically scans in different subnets, the potential horizontal movement attack is determined; if an internal host abnormally accesses an external server within a preset time window, the abnormal control instruction is determined, and the network traffic anomaly is cooperatively perceived. The method for identifying traffic rate surges and protocol distribution deviations comprises: collecting real-time traffic data, extracting cross-protocol dependencies and traffic periodic patterns, and generating a real-time stable state invariant baseline; dividing network traffic into short-time burst traffic windows according to minute levels; comparing statistical parameters of the real-time stable state invariant baseline and the historical stable state invariant baseline; the statistical parameters comprise traffic rate, protocol distribution, port distribution, session new creation rate and session duration. When the real-time traffic rate exceeds the fluctuation interval of the sum of the historical average and the preset multiple standard deviation, and remains above in continuous different windows, it is determined that the traffic rate has increased rapidly; for protocol distribution, the chi-square test is used to compare the differences, and the difference degree between the real-time protocol distribution and the historical protocol distribution is calculated, and when the difference degree exceeds the preset difference degree threshold, it is determined that the protocol distribution deviates. 3.The network traffic anomaly perception method based on big data according to claim 2, characterized in that, The method for generating abnormal traffic events comprises: Divide network traffic into long-period traffic windows by week level, and aggregate source IP activity frequency, destination IP activity frequency, port scanning times, session establishment and termination coefficient, packet size and transmission rate change trend in each period window; Analyze the behavior of the same source IP accessing different target IPs within the long-period traffic window, and when the access covers different targets and each connection duration is greater than a preset duration threshold, the connection interval is greater than a preset time interval, and deviates from the historical steady-state invariant baseline, it is determined that it is a slow scanning behavior; Analyze the data transmission mode of the internal network host to the external server, and when the data packets are continuously sent to a preset external IP, and the transmission rate, traffic size and periodicity characteristics all deviate from the historical steady-state invariant baseline, it is determined that it is an anonymous external transmission behavior, and an abnormal traffic event is generated.

4. The network traffic anomaly perception method based on big data according to claim 3, characterized in that, The method for mapping abnormal traffic events into node and edge relationships of different dimensions comprises: Map key entities in abnormal traffic events to graph nodes, including source IP, destination IP corresponding host nodes, protocol nodes involved in abnormal access, port nodes, session nodes corresponding to abnormal connection sessions, event nodes identifying abnormal types and time windows; Establish graph edges according to the relationships between entities in abnormal traffic events, including host-to-host communication edges between source IPs and destination IPs, host-protocol edges between hosts and protocol nodes, host-port edges between hosts and port nodes, session-host edges between session nodes and involved host nodes, and event-entity edges between event nodes and corresponding host, port and protocol nodes.

5. The method of claim 4, wherein, The method for forming a dynamic network behavior graph comprises: In a cloud computing environment, for abnormal behavior of virtual machines accessing databases, a preset access port set and access port distribution are counted, and a port conditional entropy is calculated, and the port conditional entropy is normalized to distinguish between random port scanning and abnormal access behavior disguised as fixed ports; Define abnormal events of virtual machines accessing databases as VM-DB abnormal events, and define VPN login operations as VPN login events; establish a matching set for the two types of events, and the matching condition is that the events have the same unique identifier, and the time interval between the occurrence of the two events is within a preset time window threshold, when the matching condition is met, it is considered that the VM abnormal access and the VPN login event are associated; Connect the VPN account node with the corresponding virtual machine node, and then connect the virtual machine node with the database node to form a path from the VPN account to the corresponding virtual machine, and then from the virtual machine to the database, and mark the abnormal type, event timestamp and port conditional entropy attribute on the path; realize automatic cascading across events, associate single VM-DB abnormal events with corresponding VPN login events to form a causal path, and finally build a dynamic network behavior graph.

6. The network traffic anomaly perception method based on big data according to claim 5, characterized in that, The acquisition method of the causal chain abnormal evidence chain comprises: Identify the abnormal trigger nodes in the dynamic network behavior graph, including traffic abnormal sources, virtual machine abnormal access events, VPN login abnormal events and database abnormal operation events, and record the abnormal type, event time and related attribute information of each abnormal trigger node; Track the propagation impact path of the abnormal trigger node along the edge relationship between the nodes in the dynamic network behavior graph, which includes access, session establishment or data transmission behavior from the source node to the target node, connect the abnormal trigger node and the nodes tracked along the propagation impact path in turn to form a complete causal chain abnormal evidence chain.

7. The method of claim 6, wherein, The method for solidifying the necessary abnormal evidence sub-chain comprises: Using the minimum cut algorithm, find the minimum edge weight set that can separate the source node and the terminal node in the complete causal chain, identify the key event segment in the abnormal propagation process, and extract the necessary abnormal evidence sub-chain; generate a hash digest according to the key event occurrence time sequence for the extracted necessary abnormal evidence sub-chain, and mark the timestamp to realize the integrity and tamper-proof solidification of the sub-chain. 8.The network traffic anomaly perception method based on big data according to claim 7, characterized in that, The method for realizing the collaborative perception of network traffic anomaly comprises: Collect each subnet abnormal event information involved in the solidified necessary abnormal evidence sub-chain, including source IP, target IP, event time, access port and abnormal event type; summarize each subnet abnormal event according to the subject entity, and count the abnormal access times, access ports and active times of each subject in different subnets; For abnormal access behavior of the same subject between different subnets, calculate the autocorrelation function of the access sequence, and find the maximum periodicity score in the preset time window range; if the periodicity score of the same subject exceeds the preset periodicity score threshold, and the number of subnet abnormal events in at least m subnets exceeds the corresponding preset periodicity score threshold, it is determined that the subject exists a potential horizontal movement attack; For the access behavior of the internal network host to the external network server, count the access frequency, traffic size and port distribution in the preset time window; when the access behavior deviates from the historical steady-state invariant baseline and meets the preset abnormal access mode, it is determined as a potential abnormal control instruction; realize the collaborative perception of network traffic anomaly. 9.A big data based network traffic anomaly perception system for implementing the big data based network traffic anomaly perception method of any one of claims 1 to 8, characterized in that, It comprises: A steady-state mode analysis module acquires and analyzes historical network traffic data, extracts cross-protocol dependencies and traffic periodic patterns, and generates a historical steady-state invariant baseline; An abnormal traffic identification module compares a historical steady-state invariant baseline with a real-time steady-state invariant baseline generated by real-time traffic data, identifies sudden increase in traffic rate and deviation in protocol distribution in a short-time burst traffic window, identifies slow scanning and hidden external transmission behavior in a long-time periodic traffic window, and generates abnormal traffic events; A behavior graph construction module maps abnormal traffic events into node and edge relationships of different dimensions, and in a cloud computing environment, cascades abnormal behavior of virtual machine access to a database with VPN account login records to form a dynamic network behavior graph; A verifiable solidification module forms a causal chain of abnormal evidence chains by connecting abnormal trigger points and corresponding propagation impact paths in the dynamic network behavior graph, extracts necessary abnormal evidence subchains by using a minimum cut evidence extraction method, and solidifies the necessary abnormal evidence subchains by using a time stamp and a hash chain digest; A collaborative threat judgment module performs cross-domain abnormal synthesis between different subnets based on the solidified necessary abnormal evidence subchains, judges that it is a potential horizontal movement attack if the same IP periodically scans in different subnets, and judges that it is an abnormal control instruction if an intranet host abnormally accesses an extranet server within a preset time window, thereby achieving collaborative perception of network traffic abnormalities.

Images