Flow anomaly monitoring method and system based on big data and storage medium

Abstract

The invention discloses a traffic anomaly monitoring method and system based on big data and a storage medium, and belongs to the field of network security. The method comprises the following steps: determining a monitoring target and a monitoring range, and defining a network range in which traffic monitoring needs to be carried out; data collection and arrangement: using a traffic monitoring tool to collect network traffic data, and performing data cleaning and preprocessing; analyzing traffic characteristics, including statistical analysis, protocol and port analysis and traffic mode identification; setting a baseline threshold, setting the baseline threshold based on the flow characteristic analysis result and the service demand, and establishing an adaptive threshold adjustment mechanism; and continuous verification and optimization: monitoring whether the network flow exceeds a baseline threshold in real time, triggering an anomaly detection mechanism, generating early warning information, executing a response strategy, and optimizing the baseline threshold according to feedback. By establishing the network flow base line and monitoring the deviation condition in real time, network abnormity can be found in time, the network security protection capability is improved, and the method is suitable for security monitoring of various network environments.

Description

Technical Field

[0001] This invention relates to network information security technology, and in particular to a method, system, and storage medium for monitoring abnormal traffic based on big data. Background Technology

[0002] With the rapid development and widespread application of network technology, network security issues have become increasingly prominent. Network traffic monitoring, as an important means of network security protection, has received widespread attention. Network traffic monitoring technology, through the collection, analysis, and processing of network traffic data, can promptly detect abnormal behavior and potential threats in the network, providing strong support for network security protection.

[0003] Currently, network traffic monitoring technologies mainly include rule-based detection methods, statistical analysis-based detection methods, and machine learning-based detection methods. Rule-based detection methods match network traffic against a pre-defined rule base to identify known attack patterns; statistical analysis-based detection methods analyze network traffic data to establish a traffic baseline and detect abnormal traffic; and machine learning-based detection methods learn network traffic characteristics using machine learning algorithms to build detection models and detect unknown attacks.

[0004] However, existing network traffic monitoring technologies still have the following problems:

[0005] 1. Traditional network traffic monitoring methods typically employ constant baseline threshold detection, which struggles to adapt to the hidden impacts of changes in network data and structure, and is unable to effectively identify and respond to complex and ever-changing network attacks. This fixed threshold setting fails to reflect the dynamic characteristics of network traffic, easily leading to false alarms or missed alarms.

[0006] 2. Existing network security monitoring methods primarily rely on known rule bases, rendering them ineffective against unknown threats and unable to provide complete attribution evidence and loss assessment. This rule-based detection approach has limited ability to identify new and variant attacks, making it difficult to address increasingly complex network security threats.

[0007] 3. Existing network security management technologies lack effective management of different network slices and cannot provide customized abnormal traffic detection and response strategies for specific network slices. With the development of network virtualization and slicing technologies, the business characteristics and security requirements of different network slices vary, and a unified monitoring strategy is difficult to meet the differentiated security needs.

[0008] 4. Existing technologies lack adaptive confidence interval adjustment strategies, making them unable to adapt to different network traffic characteristics and attack patterns, resulting in room for improvement in the accuracy and timeliness of anomaly detection. The volatility and seasonality of network traffic require monitoring systems to dynamically adjust detection thresholds based on actual conditions, but existing technologies fall short in this regard.

[0009] 5. Traditional network traffic monitoring technologies are insufficient in terms of processing performance and construction costs, making it difficult to meet the real-time monitoring needs of large-scale network environments. As network scale expands and traffic increases, monitoring systems need to process massive amounts of data, posing challenges to existing technologies in terms of data processing efficiency and resource utilization.

[0010] Therefore, this invention proposes a method, system, and storage medium for monitoring traffic anomalies based on big data. Summary of the Invention

[0011] This invention provides a method, system, and storage medium for monitoring traffic anomalies based on big data. It can solve the problems that traditional network traffic monitoring methods are unable to adapt to the hidden impacts brought about by changes in network data and structure, cannot effectively identify and respond to complex and ever-changing network attacks, and lack adaptive confidence interval adjustment strategies in existing technologies.

[0012] The technical solution adopted by this invention to solve its technical problem is: to provide a traffic anomaly monitoring method based on big data, comprising:

[0013] Includes the following steps:

[0014] Step 1: Determine the monitoring target and scope, and define the network area that needs to be monitored for traffic.

[0015] Step 2: Data collection and organization. Use traffic monitoring tools to collect network traffic data and perform data cleaning and preprocessing.

[0016] Step 3: Analyze traffic characteristics, including statistical analysis, protocol and port analysis, and traffic pattern identification;

[0017] Step 4: Set baseline thresholds based on traffic characteristic analysis results and business requirements;

[0018] Step 5: Continuous verification and optimization, including real-time monitoring, feedback analysis, and periodic evaluation.

[0019] Preferably, step 1 includes: determining the target network scope for monitoring based on the network environment and business needs; analyzing the network topology and determining the specific implementation plan for monitoring; and formulating monitoring strategies.

[0020] Furthermore, the step of determining the monitoring range in step 1 includes:

[0021] Step 101: Based on the network topology and business requirements, select the network interface, device, or application to be monitored;

[0022] Step 102: Develop differentiated monitoring strategies for different network slices.

[0023] Preferably, step 2 includes: selecting a suitable network traffic monitoring tool; configuring the monitoring tool; starting the monitoring tool to begin data collection; preprocessing the collected raw traffic data; and standardizing the preprocessed data.

[0024] Furthermore, the data preprocessing in step 2 includes:

[0025] Step 204: Denoise the raw traffic data to remove outliers and duplicate data;

[0026] Step 205: Standardize the preprocessed data and convert it into JSON or CSV format.

[0027] Furthermore, in step 2, the traffic monitoring tool includes the open-source tool Wireshark or Snort, and in step 5, the real-time monitoring frequency is once per minute, and a monitoring report containing the cause of the anomaly, the scope of impact, and handling suggestions is generated.

[0028] Preferably, step 3 includes: statistically analyzing traffic data, calculating statistical indicators such as the mean, median, and standard deviation of traffic; analyzing the usage of different protocols and ports; and using machine learning algorithms to identify traffic patterns.

[0029] Furthermore, the traffic pattern recognition in step 3 is implemented using machine learning algorithms, including:

[0030] Step 3031: Classify traffic data using a clustering algorithm to identify normal traffic patterns and potential abnormal patterns;

[0031] Step 3032: Analyze the correlation between traffic features using association rule mining techniques.

[0032] Preferably, step 4 includes: setting a baseline threshold based on statistical analysis results, combined with business needs and risk tolerance; establishing a threshold adjustment mechanism; and storing the baseline threshold in a database.

[0033] Furthermore, the setting of the baseline threshold in step 4 includes:

[0034] Step 401: Calculate the dynamic threshold range based on the traffic mean, standard deviation, and business risk tolerance;

[0035] Step 403: Store the threshold in the database and manage its version to support historical baseline backtracking.

[0036] Furthermore, the adaptive threshold adjustment mechanism in step 4 includes:

[0037] Step 4021: Dynamically adjust the baseline threshold based on traffic fluctuations and set an adaptive incremental adjustment strategy;

[0038] Step 4022: In response to sudden network events, trigger emergency threshold adjustments and update the baseline model.

[0039] Preferably, step 5 includes: real-time monitoring of whether network traffic exceeds the baseline threshold; triggering an anomaly detection mechanism; generating monitoring reports and early warning information; executing corresponding response strategies; collecting monitoring feedback information; optimizing the baseline threshold and monitoring strategy based on the feedback information; and periodically evaluating the accuracy and effectiveness of the baseline.

[0040] Furthermore, the anomaly detection mechanism in step 5 includes:

[0041] Step 5021: When the traffic exceeds the baseline threshold, extract traffic features for multi-dimensional analysis;

[0042] Step 5022: Determine the anomaly type based on protocol type, port activity, and traffic distribution characteristics.

[0043] Furthermore, the response strategy in step 5 includes:

[0044] Step 5041: Block, discard, or log abnormal traffic;

[0045] Step 5042: Generate graded alarm information based on the severity of the anomaly and notify the administrator.

[0046] Furthermore, step 5 also includes:

[0047] Step 505: Collect false alarm rate and detection timeliness indicators;

[0048] Step 506: Optimize the baseline threshold and machine learning model parameters based on the feedback data.

[0049] The beneficial effects of this invention are as follows: By establishing a traffic baseline and combining steps such as data collection, analysis, and preprocessing, it can accurately identify and respond to complex and ever-changing network attacks, overcoming the limitations of traditional constant baseline threshold detection and significantly improving the accuracy and timeliness of abnormal traffic detection; by adopting multi-dimensional traffic feature analysis and an adaptive threshold adjustment mechanism, it can effectively cope with the hidden impacts brought about by changes in network data and structure, ensuring the accuracy and stability of the baseline and solving the problem of insufficient accuracy of baseline models in existing technologies; by collecting and analyzing diverse traffic data, including traffic information in different time periods, business scenarios, and user behavior patterns, a comprehensive and accurate traffic baseline model is established, improving the comprehensiveness and accuracy of network security monitoring; it supports customized management of different network slices, and can flexibly adjust the baseline threshold according to business needs and risk tolerance, realizing fine-grained control of network traffic and solving the problem of lack of effective management of different network slices in existing technologies; through real-time monitoring and feedback mechanisms, combined with continuous evaluation and optimization processes, it can promptly detect and handle abnormal traffic, effectively ensuring network security and preventing network attacks and abnormal traffic. Attached Figure Description

[0050] Figure 1 The flowchart illustrates the traffic anomaly monitoring method based on big data provided by this invention. Detailed Implementation

[0051] The specific embodiments of the present invention will be described in detail below, but it should be understood that the scope of protection of the present invention is not limited to the specific embodiments.

[0052] like Figure 1 As shown, the traffic anomaly monitoring method based on big data provided in this embodiment of the invention includes:

[0053] Step 1: Determine the monitoring target and scope, and define the network area that needs to be monitored for traffic.

[0054] Specifically, step 1 includes:

[0055] Step 101: Based on the network topology and business requirements, select the network interface, device, or application to be monitored;

[0056] This step begins by obtaining the enterprise network topology map and identifying key nodes in the network architecture, including core switches, border routers, firewalls, and server clusters. Based on a business importance assessment, network areas requiring focused monitoring are determined, such as database servers containing sensitive data, web application servers providing external services, and border devices connected to the internet. Simultaneously, key network interfaces are identified, such as WAN ports connecting to the internet, LAN ports connecting to the internal network, and DMZ area interfaces. For large enterprise networks, the monitoring scope can be divided into multiple monitoring domains, with each domain configured with different monitoring parameters based on its business characteristics.

[0057] Step 102: Develop differentiated monitoring strategies for different network slices;

[0058] In this step, differentiated monitoring strategies are developed for different network slices based on their business characteristics and security requirements. For network slices carrying critical business operations, such as financial transactions and real-time control systems, stricter monitoring strategies are implemented, including higher sampling rates, lower detection thresholds, and faster response mechanisms. For ordinary business network slices, such as office networks, standard monitoring strategies are used. For network slices in test or development environments, monitoring strategies can be appropriately relaxed to reduce false alarms. Simultaneously, an independent traffic baseline model is established for each network slice to ensure the accuracy of monitoring.

[0059] Step 2: Data collection and organization. Use traffic monitoring tools to collect network traffic data and perform data cleaning and preprocessing.

[0060] In this step, traffic monitoring tools are deployed to collect network traffic data. These tools include open-source tools like Wireshark or Snort, which can capture network packets and provide detailed protocol analysis capabilities. Wireshark is suitable for deep packet inspection and protocol analysis, while Snort focuses on intrusion detection and real-time alerts. Depending on the network size and monitoring needs, traffic acquisition probes are deployed at key network nodes to obtain traffic data through port mirroring or network splitters.

[0061] During data collection, setting appropriate sampling rates and filtering rules is crucial to avoid collecting too much irrelevant data and causing excessive system load. For large networks, a distributed data collection architecture can be adopted, deploying collection nodes in various network areas and aggregating the data to a central analysis platform. Collected data includes, but is not limited to: IP addresses, port numbers, protocol types, packet sizes, timestamps, and connection status.

[0062] Data preprocessing includes:

[0063] Step 204: Denoise the raw traffic data to remove outliers and duplicate data;

[0064] During the noise reduction process, obviously abnormal data points are first identified and marked, such as traffic peaks exceeding the normal range and data packets that clearly do not conform to business patterns. Statistical methods such as Z-score or modified Z-score are used to identify outliers, marking data points exceeding three standard deviations from the mean as potential anomalies. For time series data, moving window technology is used to detect abnormal fluctuations within a short period. Simultaneously, hash algorithms or content comparison techniques are used to identify and remove duplicate data packets, reducing data redundancy. For retransmitted data packets caused by network jitter or temporary connection failures, intelligent merging is performed to retain valid information while reducing data volume.

[0065] Step 205: Standardize the preprocessed data and convert it to JSON or CSV format.

[0066] During the standardization process, traffic data from different sources and in different formats are uniformly converted into a standard format to facilitate subsequent analysis and processing. Structured data is converted to JSON format to preserve its hierarchical structure and relationships; tabular data is converted to CSV format for batch processing and import into analysis tools. During the conversion process, it is ensured that field names are standardized, data types are consistent, and timestamps are standardized to UTC or local time. Special characters or non-ASCII characters are appropriately encoded to ensure data integrity and readability. Metadata information, including data source, collection time, and preprocessing methods, is added to the converted data files to facilitate data traceability and quality control.

[0067] Step 3: Analyze traffic characteristics, including statistical analysis, protocol and port analysis, and traffic pattern identification;

[0068] In this step, the preprocessed traffic data is analyzed from multiple dimensions to uncover traffic characteristics and patterns.

[0069] In terms of statistical analysis, basic statistical indicators of flow are calculated, including average flow, peak flow, flow variance, and flow distribution characteristics. Flow time series graphs are plotted to observe the trend of flow changes over time and identify periodic patterns and abnormal fluctuations. Flow statistical characteristics at different time granularities (hourly, daily, weekly) are calculated to establish a multi-level flow model.

[0070] In terms of protocol and port analysis, the usage ratio of various protocols in the network, such as TCP, UDP, ICMP, HTTP, and HTTPS, is statistically analyzed to identify non-standard protocols or encrypted traffic. The distribution of active ports is analyzed to identify the usage of commonly used service ports and non-standard ports. Combining protocol and port information, a network service mapping is constructed to understand the applications and services running on the network.

[0071] Traffic pattern recognition is implemented using machine learning algorithms, including:

[0072] Step 3031: Classify traffic data using a clustering algorithm to identify normal traffic patterns and potential abnormal patterns;

[0073] In this step, algorithms such as K-means, DBSCAN, or hierarchical clustering are applied to perform unsupervised classification of traffic data. First, feature vectors are extracted from the traffic data, including multi-dimensional features such as traffic volume, duration, number of packets, average packet size, and protocol type. The feature data is then normalized to eliminate the influence of different units of measurement. Next, clustering algorithms are applied to group similar traffic patterns into one class, forming multiple traffic clusters. For the K-means algorithm, the optimal number of clusters is determined using the silhouette coefficient or elbow rule; for the DBSCAN algorithm, the optimal neighborhood radius and minimum sample size parameters are determined experimentally. After clustering, the characteristics of each cluster are analyzed. Clusters with high frequency and large size are marked as normal traffic patterns, while outliers or small-scale clusters are marked as potential abnormal patterns. A normal traffic pattern library is established as the basis for subsequent anomaly detection.

[0074] Step 3032: Analyze the correlation between traffic features using association rule mining techniques;

[0075] In this step, association rule mining algorithms such as Apriori or FP-Growth are applied to discover association patterns between traffic features. First, the traffic data is converted into a transactional dataset, with each record containing multiple traffic feature items. Appropriate support and confidence thresholds are set to mine frequent itemsets and strong association rules. Through association rule analysis, associations between protocol type and port number, source and destination addresses and traffic volume, and time period and traffic pattern are discovered. Particular attention is paid to traffic patterns that violate common association rules, such as specific protocols using non-standard ports or specific servers generating large amounts of traffic outside of operating hours. An association network graph is constructed to visually display the strength and patterns of associations between traffic features, assisting security analysts in understanding network behavior.

[0076] Step 4: Set baseline thresholds. Based on traffic characteristic analysis results and business requirements, set baseline thresholds and establish an adaptive threshold adjustment mechanism.

[0077] In this step, based on the aforementioned traffic characteristic analysis results and in conjunction with business requirements, a baseline threshold for network traffic is set.

[0078] The setting of the baseline threshold includes:

[0079] Step 401: Calculate the dynamic threshold range based on the traffic mean, standard deviation, and business risk tolerance;

[0080] In this step, statistical methods are used to calculate the dynamic threshold range for traffic. First, the mean μ and standard deviation σ of traffic are calculated based on historical traffic data. A threshold coefficient k is set according to the business risk tolerance, typically ranging from 2 to 3. The upper threshold is calculated as μ + k × σ, and the lower threshold as μ - k × σ (if applicable). Different threshold calculation methods can be used for different types of traffic metrics: for metrics such as total traffic and number of connections, the above statistical methods can be used; for metrics such as protocol distribution ratio, a percentage threshold can be set; for metrics such as new connection rate, an exponentially weighted moving average method can be used to calculate the dynamic threshold. Considering the business cycle, differentiated threshold ranges are set for different time periods (working hours, non-working hours, weekends, holidays, etc.).

[0081] Step 403: Store the threshold in the database and manage its version, supporting historical baseline backtracking;

[0082] In this step, a baseline threshold management system is established to store the calculated threshold parameters in a database. A relational database or time-series database is used to store threshold data, recording metadata such as threshold type, applicable scope, effective time, expiration time, and version number. A threshold version management mechanism is implemented, creating a new version each time a threshold is updated and retaining historical version records. A historical baseline backtracking function is provided, allowing security analysts to view threshold settings at specific points in time, facilitating post-event analysis and auditing. A threshold change approval process is established; adjustments to important thresholds require authorized approval, and the reason for the change and the operator's information are recorded. Threshold data is backed up regularly to ensure data security and recoverability.

[0083] The adaptive threshold adjustment mechanism includes:

[0084] Step 4021: Dynamically adjust the baseline threshold based on traffic fluctuations and set an adaptive incremental adjustment strategy;

[0085] This step implements an adaptive adjustment mechanism for the baseline threshold, enabling it to automatically adjust as network traffic changes over a long period. A sliding window technique is used to recalculate traffic statistics periodically (e.g., weekly or monthly) and update the baseline threshold. Incremental limits for threshold adjustments are set, ensuring that each adjustment does not exceed a certain percentage (e.g., 10%) of the original threshold to avoid drastic fluctuations. For trends of continuous traffic growth or decline, a trend prediction model is used to adjust the threshold in advance to adapt to business development needs. A feedback mechanism for threshold adjustments is established to record the detection effect after each adjustment and optimize the adjustment strategy. For businesses with significant seasonal variations, a seasonal threshold model is established, and threshold change curves are preset based on historical data from the same period.

[0086] Step 4022: In response to sudden network events, trigger emergency threshold adjustments and update the baseline model;

[0087] This step establishes an emergency response mechanism to quickly adjust threshold settings in special circumstances. When events such as network architecture changes, business system launches, large-scale events, or known network attacks are detected, the emergency threshold adjustment process is triggered. Depending on the event type and scope of impact, relevant threshold parameters are temporarily adjusted, which may include relaxing thresholds to accommodate normal business changes or tightening thresholds to enhance security protection. Emergency threshold adjustments have a validity period, after which they are automatically restored or reassessed. Traffic characteristics of emergencies are recorded, and valuable patterns are incorporated into the baseline model to enhance the model's adaptability to similar scenarios. An emergency knowledge base is established to accumulate handling experience and optimize future threshold adjustment strategies.

[0088] Step 5: Continuous verification and optimization, including real-time monitoring of network traffic to see if it exceeds the baseline threshold, triggering anomaly detection mechanisms, generating early warning information, executing response strategies, and optimizing the baseline threshold based on feedback;

[0089] In this step, a continuous monitoring, verification, and optimization mechanism is established to ensure the effectiveness and accuracy of the baseline thresholds.

[0090] To monitor network traffic in real time and ensure it does not exceed baseline thresholds, a real-time monitoring system is deployed, monitoring every minute to continuously collect network traffic data and compare it with the baseline thresholds. A multi-level comparison mechanism is employed to differentiate between minor, moderate, and severe threshold exceedances, with corresponding processing strategies. A visual monitoring interface is implemented to intuitively display traffic status and threshold comparison results, supporting multi-dimensional data display and drill-down analysis.

[0091] Anomaly detection mechanisms include:

[0092] Step 5021: When the traffic exceeds the baseline threshold, extract traffic features for multi-dimensional analysis;

[0093] In this step, a deep analysis process is immediately initiated when traffic exceeding the baseline threshold is detected. Detailed characteristics of the abnormal traffic are extracted, including source IP address distribution, destination IP address distribution, port usage, protocol type, packet size distribution, connection duration, and other multi-dimensional information. Comparison with historical normal traffic patterns is performed to identify anomalies and trends. Principal component analysis or t-SNE and other dimensionality reduction techniques are applied to map the multi-dimensional features to a two- or three-dimensional space, visually demonstrating the difference between abnormal and normal traffic. An anomalousness score is calculated based on a comprehensive evaluation of multiple features, rather than a single indicator. A feature summary report of the abnormal traffic is generated to provide a basis for subsequent analysis and processing.

[0094] Step 5022: Determine the anomaly type based on protocol type, port activity, and traffic distribution characteristics;

[0095] In this step, based on the results of multi-dimensional feature analysis, the type and possible causes of abnormal traffic are determined. The distribution of protocol types in the abnormal traffic is analyzed to identify uncommon protocols or abnormal protocol usage (e.g., a large amount of ICMP traffic may indicate network scanning). Port activity is checked to identify newly emerging active ports or abnormal usage patterns of common ports (e.g., a sudden increase in traffic to common ports may indicate an attack targeting a specific service). The temporal distribution characteristics of the traffic are analyzed to distinguish between sudden and persistent anomalies. Traffic directionality analysis is combined to distinguish between inbound and outbound anomalies (e.g., a large amount of outbound traffic may indicate data leakage). A predefined anomaly pattern library is applied to match the current anomaly with known attack patterns, such as DDoS attacks, network scanning, data leakage, and malware communication. Based on the matching results, the severity of the anomaly and its processing priority are determined.

[0096] Response strategies include:

[0097] Step 5041: Block, discard, or log abnormal traffic;

[0098] In this step, appropriate response actions are performed based on the type and severity of the abnormal traffic. For traffic confirmed to be malicious, such as DDoS attack traffic or connection requests from known malicious IPs, the relevant traffic is directly blocked by configuring firewall rules or ACL policies. For suspicious but uncertain traffic, traffic rate limiting or bandwidth control measures can be taken to reduce network pressure and avoid mistakenly blocking normal services. For abnormal traffic requiring further analysis, the full traffic logging function is activated to capture complete packet content, supporting subsequent forensic analysis. The response policy is executed automatically, triggering corresponding actions automatically according to preset rules, reducing manual intervention time. An audit log for the response actions is established, recording detailed information for each response behavior, including triggering conditions, execution time, operation type, and scope of impact.

[0099] Step 5042: Generate graded alarm information based on the severity of the anomaly and notify the administrator;

[0100] This step establishes a tiered alerting mechanism, generating different levels of alert information based on the severity and potential impact of anomalies. An alert level system is defined, including four levels: Informational, Warning, Critical, and Urgent, each corresponding to different handling procedures and response time requirements. Alert information includes key information such as the cause of the anomaly, the scope of impact, and handling suggestions, facilitating rapid understanding and resolution by administrators. Appropriate notification channels, such as email, SMS, instant messaging, and telephone, are selected based on the alert level to ensure timely delivery of critical alerts to relevant personnel. An alert aggregation and deduplication mechanism is implemented to prevent a large number of duplicate alerts for the same issue, avoiding alert fatigue. An alert escalation mechanism is established, automatically escalating lower-level alerts and notifying higher-level administrators if they are not handled within a specified time. Regular alert statistical reports are generated to analyze alert trends and distribution, providing a basis for security strategy optimization.

[0101] Step 505: Collect false alarm rate and detection timeliness indicators;

[0102] In this step, a performance evaluation mechanism for the monitoring system is established, and key performance indicator data is collected. False alarm rate metrics are regularly compiled, including the number of false alarms, the false alarm rate (number of false alarms / total number of alarms), and the distribution of false alarm types. The causes of false alarms are analyzed to identify common patterns and triggering conditions. Detection timeliness metrics are evaluated, including anomaly detection latency (the time difference between anomaly occurrence and detection), alarm generation time, and response execution time. A baseline is established to compare performance changes over different periods and identify trends of performance improvement or degradation. User feedback is collected, including subjective evaluation data such as alarm accuracy assessments and response measure effectiveness assessments. A comprehensive system performance evaluation report is generated by integrating objective metrics and subjective evaluations.

[0103] Step 506: Optimize the baseline threshold and machine learning model parameters based on feedback data;

[0104] In this step, the monitoring system is continuously optimized using collected performance metrics and feedback data. False positives are analyzed, and relevant threshold parameters are adjusted to reduce the occurrence of similar false positives. For missed detections, the causes are analyzed, and corresponding detection rules are strengthened to improve detection sensitivity. Based on detection timeliness metrics, data processing workflows and algorithm efficiency are optimized to reduce detection latency. Newly collected anomaly samples are used to update the training dataset of the machine learning model, improving the model's detection accuracy. Hyperparameters of the machine learning algorithm are adjusted, such as distance metrics for clustering algorithms and decision thresholds for classification algorithms, to balance detection sensitivity and specificity. An A / B testing mechanism is established to evaluate the effectiveness of optimization adjustments, ensuring that each update brings performance improvements. A continuous learning and adaptive optimization mechanism is implemented, enabling the system to continuously adapt to changes in the network environment and threat landscape.

[0105] The above-disclosed embodiments are merely a few specific examples of the present invention. However, the embodiments of the present invention are not limited thereto, and any variations that can be conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A traffic anomaly monitoring method based on big data, characterized in that, Comprise: Step S1: Determine the monitoring target and scope, define the network scope that needs to be monitored; Step S2: Data collection and arrangement, use traffic monitoring tools to collect network traffic data, and perform data cleaning and preprocessing; Step S3: Analyze traffic characteristics, including statistical analysis, protocol and port analysis, and traffic pattern recognition; Step S4: Set baseline threshold, set baseline threshold based on traffic characteristic analysis results and business requirements; Step S5: Continuous verification and optimization, including real-time monitoring, feedback analysis and periodic evaluation.

2. The big data based traffic anomaly monitoring method of claim 1, wherein, Step S1 includes: Determine the monitoring target network range according to network environment and business requirements; Sort out the network topology structure and determine the specific implementation scheme of monitoring; Develop monitoring strategy.

3. The big data based traffic anomaly monitoring method of claim 2, wherein, The step of determining the monitoring target network range includes: Step 101: Select the network interface, device or application to be monitored based on network topology structure and business requirements; Step 102: Develop differentiated monitoring strategies for different network slices. 4.The big data based traffic anomaly monitoring method of claim 1, wherein, Step S2 includes: Select appropriate network traffic monitoring tools; Configure monitoring tools; Start data collection; Preprocess the collected raw traffic data; Standardize the preprocessed data; Data preprocessing includes: Step 204: Denoising the raw traffic data to remove outliers and duplicate data; Step 205: Standardize the preprocessed data to JSON or CSV format. 5.The big data based traffic anomaly monitoring method of claim 1, wherein, The step S3 includes: Statistical analysis of traffic data, calculation of mean, median, standard deviation and other statistical indicators; Analysis of the use of different protocols and ports; Use machine learning algorithm to identify traffic patterns; Traffic pattern recognition is achieved by using machine learning algorithm, including: Step 3031: Classify traffic data by clustering algorithm to identify normal traffic patterns and potential abnormal patterns; Step 3032: Use association rule mining technology to analyze the association between traffic characteristics.

6. The big data-based traffic anomaly monitoring method of claim 1, wherein The step S4 includes: Based on the statistical analysis results, set the baseline threshold according to the business requirements and risk tolerance; Establish threshold adjustment mechanism; Store the baseline threshold in the database; The setting of baseline threshold includes: Step 401: Calculate the dynamic threshold interval according to the traffic mean, standard deviation and business risk tolerance; Step 403: Store the threshold in the database and perform version management to support historical baseline backtracking; The adaptive threshold adjustment mechanism includes: Step 4021: Dynamically adjust the baseline threshold based on traffic fluctuations and set adaptive incremental adjustment strategy; Step 4022: Trigger threshold emergency adjustment and update baseline model for sudden network events.

7. The big data based traffic anomaly monitoring method of claim 1, wherein, The step S5 includes: Real-time monitoring of network traffic whether it exceeds the baseline threshold; Trigger anomaly detection mechanism; Generate monitoring report and warning information; Execute corresponding response strategy; Collect monitoring feedback information; Optimize baseline threshold and monitoring strategy according to feedback information; Periodically evaluate the accuracy and effectiveness of the baseline. 8.The big data based traffic anomaly monitoring method of claim 7, wherein, The anomaly detection mechanism includes: Step 5021: When the traffic exceeds the baseline threshold, extract the traffic features for multi-dimensional analysis; Step 5022: Determine the type of anomaly according to the protocol type, port activity and traffic distribution characteristics; Response strategies include: Step 5041: Perform blocking, discarding or recording operations on abnormal traffic; Step 5042: Generate hierarchical alarm information according to the severity of the anomaly and notify the administrator.

9. A traffic anomaly monitoring system based on big data, characterized by, Including: Definition module, determine the monitoring target and range, define the network range that needs to be monitored; Collection and processing module, use traffic monitoring tools to collect network traffic data, and perform data cleaning and preprocessing; Feature analysis module, used for statistical analysis, protocol and port analysis, and traffic pattern recognition of network traffic data; Setting module, used for setting baseline threshold, setting baseline threshold based on traffic feature analysis results and business requirements; Verification and optimization module, for continuous verification and optimization, including real-time monitoring, feedback analysis and regular evaluation.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to realize the traffic anomaly monitoring method based on big data in any one of claims 1 to 8.

Images