A data center dynamic data flow monitoring system and method

Abstract

The present invention relates to the technical field of server load balancing, and specifically to a data center dynamic data traffic monitoring system and method to achieve flexible and efficient dynamic traffic monitoring. The system includes a traffic collection module, a traffic prediction module, a monitoring threshold adjustment and graded alarm module, and a network traffic path allocation module; the traffic collection module collects traffic data of each node in real time; the traffic prediction module predicts the traffic value in the next N minutes through a traffic prediction model to obtain a traffic prediction value; the monitoring threshold adjustment and graded alarm module obtains the anomaly score of each node through an anomaly scoring model, and sets multi-level dynamic thresholds based on the traffic baseline and traffic prediction value to perform real-time monitoring and graded alarm on the traffic data; the network traffic path allocation module generates flow table rules based on service priority and real-time traffic status, dynamically allocates traffic paths for service instances, and automatically triggers cross-node migration of service instances when the secondary alarm persists.

Description

Technical Field

[0001] The present invention relates to the technical field of server load balancing, and in particular to a data center dynamic data flow monitoring system and method. Background Art

[0002] As data centers continue to expand and network applications become increasingly complex, traffic management and load balancing have become critical components for ensuring network service stability and improving resource utilization efficiency. Network traffic in data centers fluctuates rapidly and is highly dynamic. This dynamism stems not only from fluctuations in network service demand but also from a variety of factors, including changes in the external environment and user behavior. To ensure efficient and stable operation of data center networks, load balancing technologies based on real-time monitoring and dynamic scheduling are essential. Through precise traffic monitoring and resource scheduling, data centers can achieve higher performance and resource utilization, reduce network bottlenecks, improve service quality, and reduce system maintenance costs.

[0003] In the field of server load balancing technology, dynamic data traffic monitoring and intelligent scheduling play a crucial role. By collecting and analyzing each server's load status, response time, and traffic in real time, the system can accurately identify the load distribution of each server and dynamically adjust the load distribution strategy based on traffic fluctuations. This dynamic adjustment mechanism ensures that data traffic is efficiently distributed across servers, avoiding the formation of overloaded servers and thereby improving the resource utilization and overall processing capacity of the server cluster. For example, using software-defined networking (SDN) technology, a centralized controller can obtain real-time data flow information and dynamically schedule the load and bandwidth of each node, enabling more flexible and optimized distribution of network traffic and improving overall system performance.

[0004] Furthermore, modern load balancing technology continues to evolve towards intelligence and automation. In complex multi-domain network environments, dynamic load balancing technology optimizes cross-domain traffic management through precise traffic monitoring and adaptive scheduling algorithms. For example, F5 load balancers automatically monitor the load on each server and intelligently distribute traffic based on real-time network performance data and historical traffic patterns. This not only effectively avoids server overload and response delays, but also manages sudden surges in traffic, ensuring high system availability and stability. Through this dynamic adjustment and intelligent decision-making mechanism, data centers can efficiently manage large-scale concurrent traffic flows, improving service quality and user experience.

[0005] However, while existing traffic monitoring technologies and load balancing methods have addressed traffic scheduling and resource allocation issues to a certain extent, numerous challenges remain. First, existing load balancing technologies often struggle to respond quickly to sudden traffic fluctuations when handling large-scale, complex traffic flows. When network traffic fluctuates dramatically over a short period of time, existing systems may not be able to adjust loads quickly enough, leading to server overloads or network bottlenecks, impacting overall system stability. Second, traditional traffic monitoring methods often rely on static threshold settings and pre-set rules to determine whether traffic is abnormal. With the increasing diversification of network environments and traffic patterns, static thresholds are no longer adaptable to rapidly changing traffic conditions, are prone to false positives and false negatives, and fail to effectively capture all types of abnormal traffic. Finally, traditional load balancing solutions often lack flexible adaptive mechanisms and intelligent algorithms in multi-tiered, multi-domain environments, limiting the system's load distribution capabilities when faced with large numbers of concurrent requests. Therefore, a more flexible and intelligent dynamic traffic monitoring and load balancing method is needed that can identify traffic changes in real time and intelligently adjust to improve data center traffic management efficiency and resource allocation accuracy.

[0006] To this end, a data center dynamic data flow monitoring system and method are proposed. Summary of the Invention

[0007] The purpose of the present invention is to provide a data center dynamic data traffic monitoring system and method, which realizes flexible and efficient dynamic traffic monitoring through the collaborative work of multiple modules. The system includes a traffic collection module, a traffic prediction module, a monitoring threshold adjustment and graded alarm module, and a network traffic path allocation module; the traffic collection module collects traffic data of each node in real time; the traffic prediction module predicts the traffic value for the next N minutes through a traffic prediction model to obtain a traffic prediction value; the monitoring threshold adjustment and graded alarm module obtains the anomaly score of each node through an anomaly scoring model, and sets multi-level dynamic thresholds based on the traffic baseline and traffic prediction value to perform real-time monitoring and graded alarm on the traffic data; the network traffic path allocation module generates flow table rules based on service priority and real-time traffic status, dynamically allocates traffic paths for service instances, and automatically triggers cross-node migration of service instances when the secondary alarm persists.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] A data center dynamic data flow monitoring system, comprising:

[0010] Traffic collection module, which is used to deploy data collection probes on SDN switches and physical servers in the data center to collect traffic data of each node in real time;

[0011] The traffic prediction module is used to establish a traffic baseline corresponding to the network service type based on historical traffic data, and predict the traffic value in the next N minutes through the traffic prediction model to obtain the traffic prediction value;

[0012] The monitoring threshold adjustment and graded alarm module is used to obtain the anomaly score of each node through the anomaly scoring model, and set multi-level dynamic thresholds based on the traffic baseline and the traffic prediction value to perform real-time monitoring and graded alarms on the traffic data. The specific process is as follows: if the current traffic value exceeds the multi-level dynamic threshold for M consecutive detection cycles, a level 1 alarm is triggered and pushed to the management platform via the low-latency transmission channel of the HTTP / 3 protocol; if abnormal service access pattern characteristics are detected at the same time, a level 2 alarm is triggered and service traffic mirror analysis is initiated;

[0013] The network traffic path allocation module generates flow table rules based on service priority and real-time traffic status, dynamically allocates traffic paths for service instances, and automatically triggers cross-node migration of service instances when the second-level alarm persists.

[0014] Preferably, a data pre-processing unit and a data aggregation unit are further included after the traffic collection module;

[0015] The data preprocessing unit preprocesses the collected traffic data and stores the preprocessed traffic data in a distributed traffic database; the preprocessing includes: deduplication, format conversion and time synchronization;

[0016] The data aggregation unit aggregates the traffic data according to a preset time window based on the characteristics of the traffic data.

[0017] Preferably, the traffic prediction model includes: a traffic data input unit, a traffic feature calculation unit, a traffic prediction unit and a traffic prediction output unit;

[0018] The flow data input unit inputs the flow data into the flow prediction model;

[0019] The flow characteristic calculation unit calculates the characteristics of the flow data to obtain flow characteristics; the flow characteristics include: historical average flow, peak period flow characteristics and flow patterns in different periods;

[0020] The traffic prediction unit predicts the traffic value for the next N minutes based on the traffic data and the traffic characteristics through an LSTM network to obtain the traffic prediction value;

[0021] The traffic prediction output unit outputs the traffic prediction value.

[0022] Preferably, the anomaly scoring model comprises: a data input unit, an attention weight calculation unit, an anomaly scoring calculation unit and an anomaly scoring output unit;

[0023] The data input unit inputs the historical traffic data of each node, the traffic data and the traffic prediction value into the anomaly scoring model;

[0024] The attention weight calculation unit uses a self-attention mechanism to extract features from the historical traffic data and the traffic prediction value to obtain corresponding historical attention weights and predicted attention weights;

[0025] The anomaly score calculation unit calculates an anomaly score for each node based on the historical traffic data, the traffic data, the traffic prediction value, the historical attention weight, and the predicted attention weight;

[0026] The anomaly score output unit outputs the anomaly score.

[0027] Preferably, the formula for the abnormality score is:

[0028] ;

[0029] in, Score anomalies; is the number of historical data points; is the historical attention weight; is the current traffic data; For the Historical traffic data; To predict the attention weight; is the traffic forecast value.

[0030] Preferably, the formula for the multi-level dynamic threshold is:

[0031] ;

[0032] in, It is a multi-level dynamic threshold; is a series; is the flow baseline; is the flow volatility weight; is the flow volatility; is the prediction error weight; is the traffic prediction value; is the anomaly score weight; Score anomalies; is the external environment weight; is the environmental impact factor; is the current environment variable value; is the historical average environmental value; is the activation function.

[0033] Preferably, a method for monitoring dynamic data flow in a data center includes:

[0034] Deploy data collection probes on the SDN switches and physical servers in the data center to collect traffic data from each node in real time;

[0035] Based on historical traffic data, a traffic baseline corresponding to the network service type is established, and the traffic value for the next N minutes is predicted using a traffic prediction model to obtain a traffic forecast value;

[0036] The anomaly scoring model is used to obtain the anomaly score of each node, and multi-level dynamic thresholds are set in combination with the traffic baseline and the traffic prediction value to conduct real-time monitoring and graded alarms on the traffic data. The specific process is as follows: if the current traffic value exceeds the multi-level dynamic threshold for M consecutive detection cycles, a level 1 alarm is triggered and pushed to the management platform via the low-latency transmission channel of the HTTP / 3 protocol; if abnormal service access pattern characteristics are detected at the same time, a level 2 alarm is triggered and service traffic mirror analysis is initiated;

[0037] Generate flow table rules based on service priority and real-time traffic status, dynamically allocate traffic paths for service instances, and automatically trigger cross-node migration of service instances when the second-level alarm persists.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1. This paper proposes a traffic prediction model that uses an LSTM model to accurately predict traffic values ​​for the next N minutes by deeply analyzing traffic data and combining historical average traffic, peak traffic characteristics, and traffic patterns at different time periods. This model can predict traffic trends in advance and adapt to dynamic changes in data center traffic, improving prediction accuracy and reducing the system burden caused by sudden traffic changes. Accurate traffic predictions can better support subsequent abnormal traffic detection and network traffic routing, ensuring efficient utilization of network resources.

[0040] 2. The present invention proposes an anomaly scoring model that combines the self-attention mechanism and traffic prediction results. By integrating historical traffic data, current traffic data, traffic prediction values, and the influence of environmental factors, the anomaly score of each node is calculated. This model can accurately determine whether there is an anomaly when traffic fluctuates or is abnormal, and quantify the severity of the anomaly. This anomaly scoring model can more accurately identify abnormal traffic, reduce false positives and false negatives, and improve the accuracy and response speed of the monitoring system. At the same time, the use of the self-attention mechanism improves the model's attention to key data points, further enhances the accuracy of anomaly detection, and provides a basis for the subsequent setting of multi-level dynamic thresholds for real-time monitoring of traffic data.

[0041] 3. The present invention proposes a method for setting a multi-level dynamic threshold. By comprehensively considering the traffic baseline, traffic volatility, traffic prediction error, anomaly score and external environmental factors, a multi-level dynamic threshold is set, which can flexibly adjust the monitoring standard according to the actual traffic conditions. When the traffic exceeds the dynamic threshold, the system can trigger the corresponding graded alarm in time to ensure that the abnormal traffic is responded to and processed in a timely manner. The multi-level dynamic threshold can not only adapt to traffic fluctuations, but also cope with changes in different types of network services, reducing the errors and inaccuracies caused by static thresholds. The setting of this multi-level dynamic threshold improves the flexibility and reliability of dynamic traffic monitoring, helps to ensure the stable operation of the data center network, and makes dynamic traffic monitoring more flexible and efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A structural diagram of a data center dynamic data flow monitoring system provided by an embodiment of the present invention;

[0043] Figure 2 A flow chart of a method for monitoring dynamic data traffic in a data center provided by an embodiment of the present invention;

[0044] Figure 3 A flowchart of anomaly scoring provided in an embodiment of the present invention;

[0045] Figure 4 A flowchart of multi-level dynamic threshold traffic monitoring and graded alarms provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0047] With the continuous expansion of data centers and the increasing complexity of network applications, traffic management and optimization have become critical for ensuring network service stability and improving resource utilization efficiency. Network traffic in data centers changes rapidly and is highly dynamic. This dynamism stems not only from fluctuations in network service demand but also from a variety of factors, including changes in the external environment and user behavior. Therefore, to ensure efficient and stable operation of data center networks, dynamic traffic monitoring is essential to identify traffic anomalies in real time and implement timely adjustments and optimizations.

[0048] The present invention proposes a data center dynamic data flow monitoring system, which realizes flexible and efficient real-time monitoring of rapidly changing and highly dynamic network dynamic flow. This system is implemented through a data center dynamic data flow monitoring method. For a specific method flowchart and system structure diagram, please refer to Figure 1 and Figure 2 In order to illustrate that the system and method of the present invention can flexibly and efficiently monitor dynamic network traffic, the effectiveness of the present invention will be described below using two embodiments.

[0049] See also Figures 1 to 4 The present invention provides a data center dynamic data flow monitoring system and method, and the technical solution is as follows.

[0050] Example 1

[0051] In the embodiment of the present application, the system and method proposed by the present invention are used to describe in detail the real-time monitoring process of dynamic network traffic data. In the embodiment of the present application, the real-time monitoring of dynamic network traffic data is aimed at the real-time monitoring of network traffic data in a large cloud data center A. Figure 1 and Figure 2 The content describes in detail the real-time monitoring process of network traffic data in the large cloud data center; Figure 1 The specific structure of the system proposed in the present invention includes a traffic collection module, a traffic prediction module, a monitoring threshold adjustment and graded alarm module, and a network traffic path allocation module. Figure 2The flowchart of the method proposed in the present invention includes: deploying data collection probes on the SDN switches and physical servers in the data center to collect the traffic data of each node in real time; establishing a traffic baseline corresponding to the network service type based on historical traffic data, and predicting the traffic value in the next N minutes through the traffic prediction model to obtain the traffic prediction value; obtaining the abnormal score of each node through the abnormal scoring model; setting multi-level dynamic thresholds based on the traffic baseline and traffic prediction value to monitor the traffic data in real time and issue graded alarms; coordinating with the SDN controller through the OpenFlow protocol to generate flow table rules according to service priority and real-time traffic status, dynamically allocating the traffic path of the service instance, and automatically triggering the cross-node migration of the service instance when the secondary alarm persists. Figure 1 and Figure 2 The following describes the contents:

[0052] A data center dynamic data flow monitoring system, comprising:

[0053] Traffic collection module, which is used to deploy data collection probes on SDN switches and physical servers in the data center to collect traffic data of each node in real time;

[0054] Specifically, in an embodiment of the present application, the data acquisition probe captures network data packets entering and leaving the switch and physical server by sniffing the network interface; and obtains traffic data by parsing network packet header information; the network packet header information includes IP address, port number, protocol type and traffic size.

[0055] The collection frequency of data collection probes is dynamically adjusted according to changes in network traffic.

[0056] Preferably, a data pre-processing unit and a data aggregation unit are further included after the traffic collection module;

[0057] The data preprocessing unit preprocesses the collected traffic data and stores the preprocessed traffic data in a distributed traffic database; the preprocessing includes: deduplication, format conversion and time synchronization;

[0058] The data aggregation unit aggregates the traffic data according to a preset time window based on the characteristics of the traffic data.

[0059] Specifically, the deduplication removes and cleans the collected traffic data to remove duplicate or erroneous data;

[0060] The format conversion converts data in different formats into a unified format for subsequent processing;

[0061] The time synchronization synchronizes the flow data collected by different probes to ensure that the flow data of each node can be analyzed in an accurate time sequence.

[0062] In this embodiment, the traffic collection module deploys data collection probes on the data center's SDN switches and physical servers to collect and process traffic data in real time, providing basic data support for subsequent traffic analysis and anomaly monitoring. After preprocessing, aggregation, and transmission, the collected traffic data provides efficient and accurate input for traffic prediction and dynamic threshold adjustment, thereby enabling real-time monitoring and optimization of network traffic.

[0063] Preferably, the traffic prediction module is used to establish a traffic baseline corresponding to the network service type based on historical traffic data, and predict the traffic value in the next N minutes through a traffic prediction model to obtain a traffic prediction value;

[0064] The flow prediction model includes: a flow data input unit, a flow feature calculation unit, a flow prediction unit and a flow prediction output unit;

[0065] The flow data input unit inputs the flow data into the flow prediction model;

[0066] The flow characteristic calculation unit calculates the characteristics of the flow data to obtain flow characteristics; the flow characteristics include: historical average flow, peak period flow characteristics and flow patterns in different periods;

[0067] The traffic prediction unit predicts the traffic value for the next N minutes based on the traffic data and the traffic characteristics through an LSTM network to obtain the traffic prediction value;

[0068] The traffic prediction output unit outputs the traffic prediction value.

[0069] Specifically, the traffic data input unit inputs the traffic data into the traffic prediction model; the traffic data includes: real-time traffic data, network service type information and timestamp information;

[0070] The traffic characteristics include historical average traffic, peak time traffic characteristics and traffic patterns in different time periods; the historical average traffic includes: the average traffic value over a period of time;

[0071] The peak period traffic characteristics include the traffic peak value of the network traffic during a specific peak period of each day;

[0072] The traffic patterns in different time periods include the traffic change characteristics of the morning peak, noon and evening peak, and analyze the periodic changes and sudden fluctuations of the traffic.

[0073] Table 1 shows the output results and performance comparison of the traffic prediction model.

[0074]

[0075] The traffic prediction module in this embodiment accurately predicts traffic for the next N minutes through the collaborative work of a traffic data input unit, a traffic feature calculation unit, a traffic prediction unit, and a traffic prediction output unit. Based on an LSTM network, this module fully considers the temporal characteristics and complexity of traffic data, helping to improve the accuracy of data center network traffic predictions and providing support for subsequent traffic monitoring, anomaly detection, and network optimization.

[0076] Preferably, the monitoring threshold adjustment and graded alarm module is used to obtain the abnormality score of each node through the abnormality scoring model, and the abnormality scoring model includes: a data input unit, an attention weight calculation unit, an abnormality score calculation unit and an abnormality score output unit; Figure 3 ;

[0077] The data input unit inputs the historical traffic data of each node, the traffic data and the traffic prediction value into the anomaly scoring model;

[0078] The attention weight calculation unit uses a self-attention mechanism to extract features from the historical traffic data and the traffic prediction value to obtain corresponding historical attention weights and predicted attention weights;

[0079] The anomaly score calculation unit calculates an anomaly score for each node based on the historical traffic data, the traffic data, the traffic prediction value, the historical attention weight, and the predicted attention weight;

[0080] The anomaly score output unit outputs the anomaly score.

[0081] Specifically, the attention weight calculation unit uses dot product attention to calculate the degree of influence of historical traffic data on current traffic data to obtain the historical attention weight; calculates the credibility of the traffic prediction value to the current traffic data to obtain the predicted attention weight.

[0082] Table 2 is the result data table of the abnormality scoring model based on the attention mechanism.

[0083]

[0084] The anomaly scoring model in this embodiment is based on a self-attention mechanism. By extracting features from historical traffic data and traffic predictions, and calculating historical and predicted attention weights, it accurately assesses the degree of anomaly in the current traffic data. This model can dynamically adapt to different network environments and service types, improving the accuracy of anomaly detection while effectively reducing false positives and false negatives by combining predicted data. This model can capture complex time series features, enhance the ability to identify sudden abnormal traffic, and provide data centers with a more intelligent and real-time traffic monitoring and alerting mechanism.

[0085] Preferably, the formula for the abnormality score is:

[0086] ;

[0087] in, Score anomalies; is the number of historical data points; is the historical attention weight; is the current traffic data; For the Historical traffic data; To predict the attention weight; is the traffic forecast value.

[0088] The anomaly scoring formula of the embodiment of the present application achieves an accurate assessment of the degree of anomaly by introducing historical attention weights and predicted attention weights, combining current traffic data, historical traffic data and traffic forecast values. Compared with the traditional scoring mechanism based on simple statistical methods, this formula can fully consider historical trends, short-term fluctuations and future forecast information, thereby improving the sensitivity and reliability to abnormal traffic. Through the self-attention mechanism, the system can automatically identify key data points that affect the anomaly score, reduce the false alarm rate and enhance the response capability to sudden anomalies, which contributes to the intelligence and refinement of data center traffic management, and provides a reference for the subsequent setting of multi-level dynamic thresholds for real-time monitoring of traffic data.

[0089] Preferably, a multi-level dynamic threshold is set in combination with the traffic baseline and the traffic prediction value to perform real-time monitoring and graded alarm on the traffic data; the specific process is: if the current traffic value exceeds the multi-level dynamic threshold for M consecutive detection cycles, a level 1 alarm is triggered and pushed to the management platform through the low-latency transmission channel of the HTTP / 3 protocol; if abnormal service access mode characteristics are detected at the same time, a level 2 alarm is triggered and service traffic mirror analysis is started; refer to Figure 4 ;

[0090] The formula for the multi-level dynamic threshold is:

[0091] ;

[0092] in, It is a multi-level dynamic threshold; is a series; is the flow baseline; is the flow volatility weight; is the flow volatility; is the prediction error weight; is the traffic prediction value; is the anomaly score weight; Score anomalies; is the external environment weight; is the environmental impact factor; is the current environment variable value; is the historical average environmental value; is the activation function.

[0093] The embodiments of the present application use multi-level dynamic threshold settings and real-time traffic monitoring, allowing the system to flexibly respond to changes in the network environment and improve the accuracy of detecting abnormal traffic. The multi-level dynamic threshold formula combines factors from multiple dimensions to ensure that the dynamic threshold can adapt to different traffic patterns, environmental changes, and service requirements. Compared with the traditional static threshold method, the system can dynamically adjust the alarm threshold according to different service requirements, environmental conditions, and traffic forecast values, avoiding false positives and missed reports. At the same time, traffic mirroring analysis and cross-node migration functions can also respond to emergencies in a timely manner to ensure the efficient and secure operation of the data center network.

[0094] Preferably, the network traffic path allocation module generates flow table rules according to service priority and real-time traffic status, dynamically allocates traffic paths for service instances, and automatically triggers cross-node migration of service instances when the second-level alarm persists.

[0095] Table 3 shows the performance test data of SDN-based dynamic traffic path allocation obtained through multi-level dynamic threshold setting and hierarchical alarm.

[0096] Specifically, in this embodiment, the flow table generation response time is measured by simulating different traffic change scenarios and measuring the total time from the SDN controller receiving the traffic status update information to the generation and issuance of the flow table rules;

[0097] The path switching success rate is determined by monitoring whether network traffic can be successfully rerouted according to flow table rules during the switching process. In the experiment, some traffic anomalies are intentionally generated, and then the proportion of traffic that successfully completes the path switching is counted.

[0098] Cross-node migration latency: We simulated migration operations, measured the time from migration start to migration completion, and used internal data center monitoring tools to record the latency during the migration process.

[0099] The system stability score is obtained by recording the network stability under different circumstances in the experiment.

[0100]

[0101] The embodiment of the present application proposes a data center dynamic data traffic monitoring system, which has the comprehensive capabilities of traffic prediction, anomaly detection and dynamic adjustment of thresholds, and can effectively cope with the challenges in data center traffic monitoring. The traffic collection module obtains traffic data on SDN switches and physical servers in real time, and uses the traffic prediction module to accurately predict future traffic based on historical data and LSTM networks to obtain traffic prediction values. Combining information such as traffic baselines, traffic volatility, and external environmental factors, the monitoring threshold adjustment and graded alarm module uses an anomaly scoring model to intelligently monitor and grade alarms for traffic, and uses a self-attention mechanism to extract features of historical traffic and predicted traffic, effectively improving the detection accuracy of abnormal traffic. Based on multi-level dynamic thresholds, the system can automatically adjust the monitoring threshold according to real-time traffic conditions, trigger a first-level alarm when traffic exceeds the standard, and push it to the management platform through the HTTP / 3 protocol with low latency; if an abnormal service access pattern is detected, a second-level alarm is further triggered and service traffic mirror analysis is performed. Finally, by integrating the OpenFlow protocol, the traffic routing module works in conjunction with the SDN controller to automatically adjust traffic routing based on traffic priority and traffic predictions, and trigger cross-node migration of service instances in the event of a Level 2 alarm. This optimizes data center traffic management and improves the system's fault tolerance, efficiency, and resource utilization. The entire system ensures efficient and accurate real-time monitoring and response of data center network traffic, enhancing the stability of the data center system.

[0102] Example 2

[0103] In Example 1, the system proposed by the present invention successfully achieved real-time monitoring of dynamic traffic in the data center network. To further verify the effectiveness of the present invention, the network traffic data of another large cloud data center B was also monitored in real time in this embodiment.

[0104] To better monitor subsequent network traffic data in real time, large cloud data center B applies a data center dynamic data traffic monitoring method, including the following steps:

[0105] Deploy data collection probes on the SDN switches and physical servers in the data center to collect traffic data from each node in real time;

[0106] Preferably, a data pre-processing unit and a data aggregation unit are further included after the traffic collection module;

[0107] The data preprocessing unit preprocesses the collected traffic data and stores the preprocessed traffic data in a distributed traffic database; the preprocessing includes: deduplication, format conversion and time synchronization;

[0108] The data aggregation unit aggregates the traffic data according to a preset time window based on the characteristics of the traffic data.

[0109] Preferably, a traffic baseline corresponding to the network service type is established based on historical traffic data, and the traffic value for the next N minutes is predicted using a traffic prediction model to obtain a traffic prediction value;

[0110] The flow prediction model includes: a flow data input unit, a flow feature calculation unit, a flow prediction unit and a flow prediction output unit;

[0111] The flow data input unit inputs the flow data into the flow prediction model;

[0112] The flow characteristic calculation unit calculates the characteristics of the flow data to obtain flow characteristics; the flow characteristics include: historical average flow, peak period flow characteristics and flow patterns in different periods;

[0113] The traffic prediction unit predicts the traffic value for the next N minutes based on the traffic data and the traffic characteristics through an LSTM network to obtain the traffic prediction value;

[0114] The traffic prediction output unit outputs the traffic prediction value.

[0115] Preferably, the anomaly score of each node is obtained through an anomaly scoring model; the anomaly scoring model includes: a data input unit, an attention weight calculation unit, an anomaly score calculation unit and an anomaly score output unit;

[0116] The data input unit inputs the historical traffic data of each node, the traffic data and the traffic prediction value into the anomaly scoring model;

[0117] The attention weight calculation unit uses a self-attention mechanism to extract features from the historical traffic data and the traffic prediction value to obtain corresponding historical attention weights and predicted attention weights;

[0118] The anomaly score calculation unit calculates an anomaly score for each node based on the historical traffic data, the traffic data, the traffic prediction value, the historical attention weight, and the predicted attention weight;

[0119] The anomaly score output unit outputs the anomaly score.

[0120] Preferably, the formula for the abnormality score is:

[0121] ;

[0122] in, Score anomalies; is the number of historical data points; is the historical attention weight; is the current traffic data; For the Historical traffic data; To predict the attention weight; is the traffic forecast value.

[0123] Preferably, a multi-level dynamic threshold is set in combination with the traffic baseline and the traffic prediction value to perform real-time monitoring and graded alarm on the traffic data; the specific process is as follows: if the current traffic value exceeds the multi-level dynamic threshold for M consecutive detection cycles, a level 1 alarm is triggered and pushed to the management platform through the low-latency transmission channel of the HTTP / 3 protocol; if abnormal service access pattern characteristics are detected at the same time, a level 2 alarm is triggered and service traffic mirror analysis is started;

[0124] The formula for the multi-level dynamic threshold is:

[0125] ;

[0126] in, It is a multi-level dynamic threshold; is a series; is the flow baseline; is the flow volatility weight; is the flow volatility; is the prediction error weight; is the traffic prediction value; is the anomaly score weight; Score anomalies; is the external environment weight; is the environmental impact factor; is the current environment variable value; is the historical average environmental value; is the activation function.

[0127] Preferably, flow table rules are generated according to service priority and real-time traffic status, traffic paths of service instances are dynamically allocated, and cross-node migration of service instances is automatically triggered when the second-level alarm persists.

[0128] Table 4 shows the performance test data of SDN-based dynamic traffic path allocation obtained through multi-level dynamic threshold setting and hierarchical alarm.

[0129]

[0130] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A data center dynamic data flow monitoring system, characterized in that: include: Traffic collection module: deploys data collection probes on the SDN switches and physical servers in the data center to collect traffic data from each node in real time; The collection frequency of data collection probes is dynamically adjusted according to changes in network traffic; The traffic prediction module establishes a traffic baseline corresponding to the network service type based on historical traffic data, and predicts the traffic value in the next N minutes through the traffic prediction model to obtain the traffic prediction value; The monitoring threshold adjustment and graded alarm module obtains the anomaly score of each node through the anomaly scoring model, and sets multi-level dynamic thresholds based on the traffic baseline and traffic forecast value to conduct real-time monitoring and graded alarms for traffic data; The anomaly scoring model includes: an attention weight calculation unit, which uses the self-attention mechanism to extract features from historical traffic data and traffic prediction values ​​to obtain the corresponding historical attention weights and predicted attention weights; an anomaly score calculation unit, which calculates the anomaly score of each node based on historical traffic data, traffic data, traffic prediction values, historical attention weights, and predicted attention weights. The formula is: ； in, Score anomalies; is the number of historical data points; is the historical attention weight; is the current traffic data; For the Historical traffic data; To predict attention weights; is the traffic prediction value; The specific real-time monitoring and graded alarm process is as follows: if the current traffic value exceeds the multi-level dynamic threshold for M consecutive detection cycles, a level 1 alarm is triggered and pushed to the management platform via the low-latency transmission channel of the HTTP / 3 protocol; if abnormal service access pattern characteristics are detected at the same time, a level 2 alarm is triggered and service traffic mirror analysis is initiated; The network traffic path allocation module generates flow table rules based on service priority and real-time traffic status, dynamically allocates traffic paths for service instances, and automatically triggers cross-node migration of service instances when the second-level alarm persists.

2. A data center dynamic data flow monitoring system according to claim 1, characterized in that: A data pre-processing unit and a data aggregation unit are also included after the traffic collection module; The data preprocessing unit preprocesses the collected traffic data and stores the preprocessed traffic data in a distributed traffic database; The preprocessing includes: deduplication, format conversion and time synchronization; The data aggregation unit aggregates the traffic data according to a preset time window based on the characteristics of the traffic data.

3. A data center dynamic data flow monitoring system according to claim 1, characterized in that: The flow prediction model includes: a flow data input unit, a flow feature calculation unit, a flow prediction unit and a flow prediction output unit; The flow data input unit inputs the flow data into the flow prediction model; The flow characteristic calculation unit calculates the characteristics of the flow data to obtain flow characteristics; the flow characteristics include: historical average flow, peak period flow characteristics and flow patterns in different periods; The traffic prediction unit predicts the traffic value for the next N minutes based on the traffic data and the traffic characteristics through an LSTM network to obtain the traffic prediction value; The traffic prediction output unit outputs the traffic prediction value.

4. A data center dynamic data flow monitoring system according to claim 1, characterized in that: The anomaly scoring model includes: a data input unit, an attention weight calculation unit, an anomaly score calculation unit and an anomaly score output unit; The data input unit inputs the historical traffic data of each node, the traffic data and the traffic prediction value into the anomaly scoring model; The anomaly score output unit outputs the anomaly score.

5. A data center dynamic data flow monitoring system according to claim 1, characterized in that: The formula for the multi-level dynamic threshold is: ； in, It is a multi-level dynamic threshold; is a series; is the flow baseline; is the flow volatility weight; is the flow volatility; is the prediction error weight; is the traffic prediction value; is the anomaly score weight; Score anomalies; is the external environment weight; is the environmental impact factor; is the current environment variable value; is the historical average environmental value; is the activation function.

6. A data center dynamic data flow monitoring method, characterized in that: include: Deploy data collection probes on the SDN switches and physical servers in the data center to collect traffic data from each node in real time; The collection frequency of data collection probes is dynamically adjusted according to changes in network traffic; Based on historical traffic data, a traffic baseline corresponding to the network service type is established, and the traffic value for the next N minutes is predicted using a traffic prediction model to obtain a traffic forecast value; The anomaly scoring model is used to obtain the anomaly score of each node, and multi-level dynamic thresholds are set based on the traffic baseline and traffic prediction value to conduct real-time monitoring and graded alarms for traffic data. The anomaly scoring model includes: an attention weight calculation unit, which uses the self-attention mechanism to extract features from historical traffic data and traffic prediction values ​​to obtain the corresponding historical attention weights and predicted attention weights; an anomaly score calculation unit, which calculates the anomaly score of each node based on historical traffic data, traffic data, traffic prediction values, historical attention weights, and predicted attention weights. The formula is: ； in, Score anomalies; is the number of historical data points; is the historical attention weight; is the current traffic data; For the Historical traffic data; To predict attention weights; is the traffic prediction value; The specific real-time monitoring and graded alarm process is as follows: if the current traffic value exceeds the multi-level dynamic threshold for M consecutive detection cycles, a level 1 alarm is triggered and pushed to the management platform via the low-latency transmission channel of the HTTP / 3 protocol; if abnormal service access pattern characteristics are detected at the same time, a level 2 alarm is triggered and service traffic mirror analysis is initiated; Generate flow table rules based on service priority and real-time traffic status, dynamically allocate traffic paths for service instances, and automatically trigger cross-node migration of service instances when the second-level alarm persists.

Images