Network flow monitoring method and system based on machine learning

Abstract

The invention discloses a network flow monitoring method and system based on machine learning, and belongs to the technical field of network security, and the method comprises the steps: S1, capturing a network data packet in real time, and carrying out the preprocessing of the network data packet; s2, generating a burst feature set based on the preprocessed network data packet; s3, inputting the burst feature set into a pre-trained anomaly monitoring model, and outputting an anomaly detection result; and S4, generating abnormal data samples according to the abnormal labels and the confidence scores in the abnormal detection result, triggering preset model incremental training when the number of the abnormal data samples reaches a second preset threshold value, and performing parameter optimization on a pre-trained abnormal detection model. The system comprises an acquisition module, a generation module, a training module and an optimization module. Through the steps from S1 to S5, the defect that the missing report rate of a traditional method is high in a high-speed network environment is overcome, and continuous self-adaptive adjustment of model parameters is achieved.

Description

Technical Field

[0001] This invention relates to the field of network security technology, and in particular to a network traffic monitoring method and system based on machine learning. Background Technology

[0002] In the field of cybersecurity technology, especially in technologies related to network traffic monitoring and anomaly detection, the efficient and accurate identification and classification of normal and abnormal behaviors in network traffic has always been a research focus.

[0003] Existing network traffic monitoring methods capture data packets flowing through network nodes by deploying probes, parsing their protocol headers and payloads, extracting quintuple information and traffic statistical features, and then matching the feature data with predefined rule bases or static behavior models. Based on the matching results, they determine whether the traffic is abnormal and generate alarm information. With the increasing complexity of network applications and the widespread use of encrypted traffic, traditional traffic detection methods have gradually revealed significant shortcomings in terms of real-time performance, adaptability, and multimodal feature fusion.

[0004] Existing technologies have the following drawbacks: Due to the covert and dynamic nature of encrypted traffic, traditional methods cannot extract fine-grained temporal features (such as burst density and interval time) in real time, resulting in a decrease in the accuracy of identifying abnormal behavior, especially in high-speed network environments where the false negative rate is high; Existing machine learning models mostly rely on static datasets for training and lack continuous learning mechanisms. Faced with constantly evolving network threats and new attack methods, these models cannot adapt to changes in real time, resulting in insufficient generalization ability and detection lag, making it difficult to maintain long-term effectiveness.

[0005] Therefore, there is an urgent need to provide network traffic monitoring methods and systems based on machine learning to solve the above problems. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a network traffic monitoring method and system based on machine learning.

[0007] To address the aforementioned technical problems, one technical solution adopted by this invention is to provide a network traffic monitoring method based on machine learning, comprising the following steps: S1. Capture network data packets in real time and preprocess the network data packets; S2. Based on the preprocessed network data packets, identify network data packets with an arrival time interval less than a first preset threshold and the same transmission direction as burst units, calculate the duration, data volume and packet number of each burst unit, and generate a burst feature set; S3. Input the set of sudden features into a pre-trained anomaly detection model and output anomaly detection results, the anomaly detection results including anomaly labels and confidence scores; S4. Based on the anomaly label and the confidence score in the anomaly detection result, generate anomaly data samples whose confidence scores are higher than a preset threshold, add the anomaly data samples to a preset training sample library, and trigger incremental training of the preset model when the number of anomaly data samples reaches a second preset threshold to optimize the parameters of the pre-trained anomaly detection model.

[0008] The present invention is further configured such that: the network data packets in step S1 are captured in real time by monitoring tools deployed on network nodes, including but not limited to network sniffers, traffic mirroring ports or deep packet inspection devices; The preprocessing in step S1 includes extracting the five-tuple information and traffic statistics features of the network data packets.

[0009] The present invention is further configured such that the method for generating the arrival time interval and transmission direction of the network data packets in step S2 is as follows: S21. Parse the five-tuple information of the network data packet, extract the source address, destination address and protocol type features of the network data packet, and determine the transmission direction of the network data packet according to the numerical relationship between the source address and the destination address; S22. Start countdown monitoring within a preset sampling period. If no network data packet arrives at the target address during the countdown, generate and insert a preset first virtual data packet at the end of the countdown, and record the current system time as a virtual timestamp. If the network data packet arrives during the countdown, record the actual arrival timestamp of the network data packet and insert a preset second virtual data packet. Calculate the arrival time interval based on the difference in arrival timestamps of consecutive network data packets. The method for inserting the first virtual data packet in step S22 is as follows: S221. When the countdown ends and no network data packet is detected to arrive, the current system time is obtained as the base timestamp, and a first virtual data packet is generated based on the preset virtual data packet template. S222. Insert the first virtual data packet into the end of the current network data packet sequence, add a preset virtual tag and the virtual timestamp to the first virtual data packet, calculate the first difference between the virtual timestamp and the adjacent second arrival timestamp, and use the first difference as the virtual arrival time interval. The method for inserting the second virtual data packet in step S22 is as follows: S2201. When the arrival of the network data packet is detected during the countdown, the countdown monitoring is immediately terminated and the actual arrival timestamp of the network data packet is recorded. A second virtual data packet is created based on the preset data packet generation rules. S2202, Insert the second virtual data packet into the corresponding position of the monitoring time point of the current network data packet sequence, and add an actual arrival identifier and an actual arrival timestamp to the second virtual data packet. Calculate the actual arrival time interval of adjacent network data packets based on the actual arrival timestamp sequence of the continuously inserted second virtual data packets. Both the virtual arrival time interval and the actual arrival time interval are used as arrival time intervals.

[0010] The present invention is further configured such that: the protocol type feature in step S21 includes connection-oriented protocols and connectionless protocols; the determination of the transmission direction of the network data packets specifically includes: S211. Extract the binary numerical representations of the source address and the target address, perform a bit-to-bit comparison operation on the binary numerical representations to generate an address comparison result, determine the address size relationship based on the difference of the most significant bit in the address comparison result, and define the direction indicated by the target address with the smaller value as the initial transmission direction. S212. Perform protocol-specific correction on the initial transmission direction based on the protocol type characteristics. When the protocol type is the connection-oriented protocol, maintain the initial transmission direction. When the protocol type is the connectionless protocol, reverse the initial transmission direction. S213. Mark the corrected initial transmission direction as the transmission direction of the network data packet, and generate a direction identifier for the transmission direction based on the protocol type feature and the address comparison result.

[0011] The present invention is further configured such that the generation step of the burst feature set in step S2 is as follows: S23. Based on the arrival time interval and the transmission direction, identify consecutive network data packet sequences whose arrival time interval is less than the first preset threshold and whose transmission direction is the same as the burst unit; S24. For each burst unit, extract its start timestamp and end timestamp, and calculate the difference between the start timestamp and the end timestamp as the duration of the burst unit. S25. Accumulate the load size of all network data packets in the burst unit to obtain the total data volume of the burst unit; S26. Count the number of network data packets in the burst unit to obtain the number of packets in the burst unit, and calculate the burst density of the burst unit based on the ratio of the number of packets to the duration. S27. Combine the duration, the total data volume, the number of packets, and the burst density into a burst feature set.

[0012] The present invention is further configured such that the pre-trained anomaly detection model in step S3 is generated through the following steps: S31. Select several sets of burst features within the preset historical sampling period, sort all the burst feature sets according to the time series, and divide all the sorted burst feature sets into an initial training dataset, a validation dataset, and a correction dataset according to a preset segmentation period. S32. Construct a network topology structure, wherein the network topology structure includes a nested first ring, a second ring, and a third ring, and map the initial training dataset to the first ring, the validation dataset to the second ring, and the correction dataset to the third ring; S33. Calculate the connection factor between two adjacent burst feature sets in the initial training dataset, the validation dataset, and the calibration dataset. The connection factor is obtained by dividing the duration, the total data volume, the number of packets, and the burst density in the two adjacent burst feature sets respectively, and then summing the results by weight. S34. A first buffer space is set at the connection between the first ring and the second ring, and a second buffer space is set at the connection between the second ring and the third ring. A virtual dataset is generated in the first buffer space and the second buffer space according to the connection factor. The initial training dataset and the virtual dataset are jointly trained using a machine learning training algorithm to generate an initial anomaly detection model. S35. The initial anomaly detection model is evaluated using the validation dataset and the calibration dataset. The parameters of the initial anomaly detection model are adjusted according to the performance evaluation results until a preset accuracy threshold is reached, thereby generating an optimized anomaly detection model. The optimized anomaly detection model is then saved as a pre-trained anomaly detection model.

[0013] The present invention is further configured such that the step of generating the abnormal data sample in step S4 is as follows: S41. Based on the anomaly label and the confidence score in the anomaly detection results, select anomaly detection results with a confidence score higher than a preset threshold as candidate anomaly samples. S42. Extract the duration, total data volume, number of packets, and burst density from the burst feature set corresponding to the candidate anomaly sample, and construct an anomaly data sample by combining the anomaly label and the confidence score. The preset training sample library is set in the first buffer space and the second buffer space of the pre-trained anomaly detection model.

[0014] The present invention is further configured such that: the specific content of triggering incremental training of the preset model in step S4, and optimizing the parameters of the pre-trained anomaly detection model, is as follows: S43. Replace the virtual dataset in the first buffer space and the second buffer space with the abnormal data samples. When the number of the replaced abnormal data samples reaches a second preset threshold, recalculate the connection factor based on the duration, total data volume, number of packets and burst density in the abnormal data samples. S44. Based on the recalculated connection factors and the original connection factors in the first ring, the second ring, and the third ring, a ring structure adjustment factor is generated using the following formula: ; in, It is a ring structure adjustment factor; , , All are preset values; For the recalculated connection factor; This is the average value of all the original connection factors in the first ring; This is the average value of all the original connection factors in the second ring; The average value of all the original connection factors in the third ring; S45. Adjust the hidden layer parameters in the pre-trained anomaly detection model according to the ring structure adjustment factor, and use the validation dataset and the correction dataset to verify the convergence of the adjusted pre-trained anomaly detection model. When the output stability of the pre-trained anomaly detection model reaches the preset standard, the parameter optimization is completed.

[0015] The present invention is further configured as: a network traffic monitoring system based on machine learning, including a collection module, a generation module, a training module and an optimization module; The acquisition module is used to capture network data packets in real time and preprocess the network data packets; The generation module identifies burst units based on the preprocessed network data packets, calculates the duration, data volume, and packet count of each burst unit, and generates a burst feature set. The training module is used to input the set of sudden features into a pre-trained anomaly detection model and output anomaly detection results, which include anomaly labels and confidence scores. The optimization module generates abnormal data samples with confidence scores higher than a preset threshold based on the abnormal labels and confidence scores in the abnormal detection results, and optimizes the parameters of the pre-trained abnormal detection model based on the abnormal data samples.

[0016] The beneficial effects of this invention are as follows: 1. This invention effectively solves the problem of incomplete extraction of temporal features in encrypted traffic environments by using a virtual data packet insertion mechanism and burst feature extraction technology. By utilizing the collaborative calculation of virtual timestamps and actual arrival timestamps, it achieves accurate capture of network traffic burst density and interval time, significantly improving the accuracy of abnormal behavior identification and real-time detection, and overcoming the shortcomings of traditional methods with high false negative rates in high-speed network environments. 2. This invention employs a ring topology training structure and a dynamic incremental learning mechanism, and achieves continuous adaptive adjustment of model parameters through connection factor calculation and buffer space optimization. This enables the anomaly monitoring model to respond in real time to the evolution of new network threats and attack methods, effectively solving the problems of insufficient generalization ability and detection lag caused by traditional static model training mechanisms, and ensuring the effectiveness and reliability of long-term monitoring. Attached Figure Description

[0017] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a flowchart of the method for generating the arrival time interval and transmission direction according to the present invention; Figure 3 This is a flowchart illustrating the steps involved in generating the pre-trained anomaly detection model of the present invention. Figure 4 This is a system flowchart of the present invention. Detailed Implementation

[0018] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0019] Please see Figure 1 - Figure 4 A machine learning-based network traffic monitoring method includes the following steps: S1. Real-time capture of network data packets and preprocessing of network data packets; S2. Based on the preprocessed network data packets, identify network data packets with an arrival time interval of less than a first preset threshold and the same transmission direction as burst units, calculate the duration, data volume and packet number of each burst unit, and generate a burst feature set. S3. Input the set of sudden features into the pre-trained anomaly detection model and output the anomaly detection results, which include anomaly labels and confidence scores. S4. Based on the anomaly labels and confidence scores in the anomaly detection results, generate anomaly data samples with confidence scores higher than a preset threshold, add the anomaly data samples to the preset training sample library, and trigger incremental training of the preset model when the number of anomaly data samples reaches the second preset threshold to optimize the parameters of the pre-trained anomaly detection model.

[0020] The monitoring method consisting of steps S1-S5 ensures comprehensive monitoring of network traffic through real-time packet capture and preprocessing, effectively captures the spatiotemporal characteristics of traffic behavior by utilizing burst unit identification and feature extraction mechanisms, achieves high-precision anomaly identification by leveraging a pre-trained anomaly monitoring model, and finally realizes the self-optimization and self-adaptation of the pre-trained anomaly monitoring model through an incremental learning mechanism based on confidence screening. Overall, it significantly improves the real-time performance, accuracy, and continuous evolution capability of network threat detection, while reducing reliance on manual intervention and enhancing the defense capability against new threats.

[0021] The network packets in step S1 are captured in real time by monitoring tools deployed on network nodes. These monitoring tools include, but are not limited to, network sniffers, traffic mirroring ports, or deep packet inspection devices. The preprocessing in step S1 includes extracting the quintuple information and traffic statistics features of network data packets.

[0022] The steps for extracting the five-tuple information and traffic statistics features of network packets are as follows: Parse the protocol header structure of the network packets, extract the source address, destination address, source port number, destination port number, and protocol type features, and combine them to generate five-tuple information; classify the network packets into flows based on the five-tuple information, and statistically analyze the arrival time interval distribution and load size distribution characteristics of network packets within the same flow; calculate the instantaneous transmission rate and cumulative data volume of the network packets, and record the packet number change trend within the transmission time window of the network packets; generate a set of traffic statistics features based on the arrival time interval distribution and load size distribution characteristics, and use the five-tuple information and the set of traffic statistics features as the preprocessing output.

[0023] One embodiment of the present invention is as follows: The method for generating the arrival time interval and transmission direction of network data packets in step S2 is as follows: S21. Parse the five-tuple information of the network data packet, extract the source address, destination address and protocol type characteristics of the network data packet, and determine the transmission direction of the network data packet based on the numerical relationship between the source address and the destination address. Specifically, the protocol type characteristics in step S21 include connection-oriented protocols and connectionless protocols; the determination of the transmission direction of network data packets specifically includes: S211. Extract the binary numerical representations of the source address and the destination address, perform bit-to-bit comparison operations on the binary numerical representations to generate address comparison results, determine the address size relationship based on the difference of the most significant bit in the address comparison results, and define the direction indicated by the smaller target address as the initial transmission direction. S212. Combine the characteristics of the protocol type to perform protocol-specific correction on the initial transmission direction. When the protocol type is a connection-oriented protocol, the initial transmission direction is maintained. When the protocol type is a connectionless protocol, the initial transmission direction is reversed. S213. Mark the corrected initial transmission direction as the transmission direction of the network data packet, and generate a direction identifier for the transmission direction based on the protocol type characteristics and address comparison results. Extract the same direction identifier as the same transmission direction, and use it to generate the burst feature set. The transmission direction determination method in step S21 parses the binary numerical representations of the source and destination addresses in the five-tuple information of the network data packet, generates address comparison results using bit-to-bit comparison operations, determines the address size relationship based on the difference of the most significant bit, and defines the direction indicated by the smaller target address as the initial transmission direction. Then, it combines protocol type characteristics for specific correction (connection-oriented protocols maintain direction, connectionless protocols perform reverse processing), and finally generates a direction identifier to achieve accurate marking of the transmission direction and rapid filtering of data packets with the same direction. This effectively improves the accuracy of transmission direction determination and protocol adaptability, and provides a reliable direction consistency guarantee for the subsequent generation of burst feature sets. S22. Start countdown monitoring within the preset sampling period. If no network data packet arrives at the target address during the countdown, generate and insert a preset first virtual data packet at the end of the countdown, and record the current system time as a virtual timestamp. If a network data packet arrives during the countdown, record the actual arrival timestamp of the network data packet and insert a preset second virtual data packet. Calculate the arrival time interval based on the difference in arrival timestamps of consecutive network data packets. Preset sampling period: Preferably dynamically configured within the range of 10 to 500 milliseconds. For high-speed network environments (such as data center intranets or bandwidth scenarios exceeding 10 Gigabit), due to dense traffic and frequent bursts, a shorter sampling period (10-100 milliseconds) is recommended to accurately capture microsecond-level traffic fluctuations and burst characteristics. For ordinary enterprise networks or internet access scenarios, a medium sampling period (100-300 milliseconds) can be used to reduce system overhead while ensuring detection accuracy. For wide area networks or high-latency network environments, a longer sampling period (300-500 milliseconds) is recommended to adapt to the inherent latency characteristics of the network. Furthermore, this value should be coordinated with the first preset threshold (1-100 milliseconds). It is generally recommended that the sampling period be at least 3 to 5 times the first preset threshold to ensure that a complete burst behavior sequence can be captured within each sampling window. The method for inserting the first virtual data packet in step S22 is as follows: S221. When the countdown ends and no network data packet is detected, obtain the current system time as the base timestamp, and generate the first virtual data packet based on the preset virtual data packet template. The preset virtual data packet template is a structured data packet model built based on the system time base and protocol framework. It includes a virtual tag field, a virtual timestamp field, a protocol type mapping field, and a direction identifier field. The virtual tag field is used to distinguish actual data packets, the virtual timestamp field is used to provide a time reference base, the protocol type mapping field is used to maintain protocol consistency, and the direction identifier field is used to maintain the continuity of transmission direction. This ensures that the generated virtual data packets can be seamlessly inserted into the network data packet sequence and participate in the time interval calculation. S222. Insert the first virtual data packet into the end of the current network data packet sequence, add a preset virtual tag and virtual timestamp to the first virtual data packet, calculate the first difference between the virtual timestamp and the adjacent second arrival timestamp, and use the first difference as the virtual arrival time interval. The method for inserting the second virtual data packet in step S22 is as follows: S2201. When a network data packet is detected to arrive during the countdown, the countdown monitoring is immediately terminated and the actual arrival timestamp of the network data packet is recorded. A second virtual data packet is created based on the preset data packet generation rules. Preset data packet generation rules: Based on the dynamic creation criteria of data packets triggered by real-time network status, it includes arrival monitoring condition judgment rules, actual arrival timestamp recording rules, actual arrival identifier addition rules, and sequence insertion position determination rules. The arrival monitoring condition judgment rules determine the timing of data packet generation, the actual arrival timestamp recording rules capture the precise arrival time, the actual arrival identifier addition rules distinguish data packet types, and the sequence insertion position determination rules ensure the continuity of the time series, ensuring that the generated data packets can accurately reflect the characteristics of network traffic and support reliable time interval calculation; S2202, Insert the second virtual data packet into the corresponding position of the monitoring time point of the current network data packet sequence, and add an actual arrival identifier and an actual arrival timestamp to the second virtual data packet. Calculate the actual arrival time interval of adjacent network data packets based on the actual arrival timestamp sequence of the continuously inserted second virtual data packets. Both the virtual arrival time interval and the actual arrival time interval are used as arrival time intervals.

[0024] Specifically, the steps for generating the burst feature set in step S2 are as follows: S23. Based on the arrival time interval and transmission direction, identify a sequence of consecutive network data packets with an arrival time interval less than a first preset threshold and the same transmission direction as a burst unit; First preset threshold: Usually set in the range of 1 millisecond to 100 milliseconds. For high-speed networks, it is recommended to use a lower value (1-10 milliseconds) to capture fine-grained burst behavior; for ordinary Internet or WAN traffic, a higher value (10-100 milliseconds) can be used to balance detection accuracy and computational overhead. S24. For each burst unit, extract its start timestamp and end timestamp, and calculate the difference between the start timestamp and end timestamp as the duration of the burst unit. S25. Accumulate the load size of all network data packets in the burst unit to obtain the total data volume of the burst unit; S26. Count the number of network data packets in the burst unit to obtain the number of packets in the burst unit. Calculate the burst density of the burst unit based on the ratio of the number of packets to the duration. S27. Combine the duration, total data volume, number of packets, and burst density into a burst feature set.

[0025] Example: In a data center network environment, a preset sampling period of 50 milliseconds and a first preset threshold of 5 milliseconds are set. By parsing the five-tuple information of network data packets, the binary values ​​of the source and destination addresses are extracted. After determining the initial transmission direction by bit-to-bit comparison, the direction is maintained by combining the protocol type characteristics, and a direction identifier is generated. During the sampling period, a countdown monitoring mechanism is used to insert virtual data packets with virtual tags into the data packets that have not yet arrived and record the system timestamp. For the data packets that have actually arrived, the actual timestamp is recorded and the time interval is calculated. Finally, based on the sequence of continuous data packets with a time interval of less than 5 milliseconds and the same direction, the duration, total data volume, number of packets, and burst density of the burst unit are calculated, and a burst feature set is generated by combining them.

[0026] This embodiment effectively solves the problem of extracting temporal features of encrypted traffic in high-speed network environments through precise transmission direction determination and virtual data packet insertion mechanism, significantly improving the integrity of sudden behavior capture and the accuracy of anomaly detection. At the same time, it achieves the best balance between detection accuracy and system overhead through adaptive sampling period and threshold configuration.

[0027] One embodiment of the present invention is as follows: the pre-trained anomaly detection model in step S3 is generated through the following steps: S31. Select several burst feature sets within a preset historical sampling period, sort all burst feature sets according to time series, and divide all sorted burst feature sets into initial training dataset, validation dataset and correction dataset according to preset segmentation period. Preset segmentation period: The optimal value is recommended to be set within the range of 5 to 15 minutes. For high-speed network environments (such as within data centers), a shorter period (5 to 8 minutes) is recommended to quickly respond to traffic changes; for ordinary enterprise networks, a medium period (8 to 12 minutes) can be used; for high-latency environments such as wide area networks, a longer period (12 to 15 minutes) is recommended. This value should be coordinated with the preset sampling period (50-500 milliseconds). It is generally recommended that the segmentation period contain at least 1000 sampling periods to ensure that each data segment contains a sufficient set of burst features for effective training. S32. Construct a network topology structure, which includes nested first ring, second ring and third ring (the first ring is located in the center, and the second ring and third ring are nested outside the first ring in turn). Map the initial training dataset to the first ring, the validation dataset to the second ring, and the calibration dataset to the third ring. S33. Calculate the connection factor between two adjacent burst feature sets in the initial training dataset, validation dataset, and calibration dataset. The connection factor is obtained by dividing the duration, total data volume, number of packets, and burst density of the two adjacent burst feature sets respectively, and then summing the results by weight. In step S33, the weighting coefficients for the duration, total data volume, number of packets, and burst density ratio of two adjacent burst feature sets in the weighted summation are 0.3, 0.3, 0.2, and 0.2, respectively. S34. Set a first buffer space at the connection between the first ring and the second ring, set a second buffer space at the connection between the second ring and the third ring, and generate a virtual dataset in the first and second buffer spaces according to the connection factor. Use a machine learning training algorithm to jointly train the initial training dataset and the virtual dataset to generate an initial anomaly detection model. The steps for generating the virtual dataset are as follows: First, determine the size distribution of the virtual dataset based on the calculated value of the connection factor. Then, generate the virtual duration based on the statistical characteristics of the initial training dataset. Next, generate the virtual total data volume by referring to the data distribution pattern of the validation dataset. Then, generate the virtual packet number based on the packet number distribution characteristics of the calibration data packet. Finally, calculate the virtual burst density by combining the burst density ratio. Finally, combine these virtual features into a complete virtual dataset for model training. S35. Use the validation dataset and calibration dataset to evaluate the performance of the initial anomaly detection model. Adjust the parameters of the initial anomaly detection model according to the performance evaluation results until the preset accuracy threshold is reached, generate the optimized anomaly detection model, and save the optimized anomaly detection model as a pre-trained anomaly detection model.

[0028] Preset accuracy threshold: The optimal value is recommended to be set within the range of 92% to 98%. Specific recommended values ​​are as follows: Basic safety requirements scenario: 92%~94%; Typical enterprise network scenarios: 94%~96%; High security requirements scenarios (such as finance and government): 96%–98%; Specifically, the steps for outputting the anomaly detection results in step S3 are as follows: a. Input the set of sudden features into the pre-trained anomaly detection model, and perform nonlinear transformation on the set of sudden features through the multi-layer feature extraction network inside the pre-trained anomaly detection model to generate a high-dimensional feature vector. b. Calculate the anomaly probability score based on the high-dimensional feature vector, convert the anomaly probability score into anomaly label using a preset threshold comparison mechanism, and generate the corresponding confidence score through the confidence calculation function (existing technology); The calculation steps for the anomaly probability score are as follows: Matrix multiplication and accumulation are performed on the high-dimensional feature vector and the weight parameters stored in the pre-trained anomaly monitoring model. Potential anomaly pattern features are gradually extracted through multiple nonlinear transformation layers to generate a preliminary anomaly index vector. Feature saliency weights are assigned to the preliminary anomaly index vector, and the matching score between each feature dimension and historical anomaly patterns is calculated. A sliding window weighted aggregation algorithm is used to generate the original anomaly score. The original anomaly score is dynamically adjusted by combining the historical normal range boundaries recorded in the validation dataset and the anomaly cases labeled in the correction dataset. System bias is eliminated through a reliability calibration function to generate a standardized anomaly probability score. The standardized anomaly probability score is input into a probability mapping function, and a nonlinear transformation is performed based on historical anomaly distribution characteristics to finally generate an anomaly probability score in the range of 0 to 1 for threshold comparison. c. Cross-validate the anomaly labels and confidence scores by combining the performance evaluation results of the validation dataset and the calibration dataset to ensure the reliability of the output results; d. Combine the anomaly label and confidence score into the anomaly detection result.

[0029] Example: In a data center network monitoring scenario, a set of burst features within a preset 15-minute timeframe is selected, sorted by time series, and divided into an initial training dataset, a validation dataset, and a calibration dataset. A three-ring nested network topology is constructed, and the datasets are mapped to the corresponding ring regions. The connection factors of adjacent burst feature sets in each dataset are calculated (duration ratio weight 0.3, total data volume ratio weight 0.3, packet number ratio weight 0.2, burst density ratio weight 0.2). Virtual datasets are generated in the first and second buffer spaces based on the connection factors (virtual duration is generated based on the statistical features of the initial training dataset, virtual total data volume is generated by referring to the distribution of the validation dataset, virtual packet number is generated by combining the packet number features of the calibration dataset, and finally, virtual burst density is calculated). After generating an initial anomaly monitoring model using joint training, its performance is evaluated using the validation and calibration datasets. The model parameters are adjusted until a 96% accuracy threshold is reached, and the model is saved as a pre-trained anomaly monitoring model. In actual detection, the real-time burst feature set is input into the model, and a high-dimensional feature vector is generated through multi-layer feature extraction. After calculating the anomaly probability score, it is converted into anomaly labels and confidence scores. Finally, the anomaly detection results are output after cross-validation using the validation and calibration datasets. This example effectively solves the problems of sparsity and imbalanced distribution of training data by using a three-ring topology and a virtual dataset generation mechanism, significantly improving the accuracy and adaptability of the anomaly detection model in high-speed network environments. By using connection factor weighted calculation and buffer space design, it enhances the model's ability to capture temporal feature correlations. At the same time, the combination of a multi-level verification mechanism ensures the reliability of the output results, providing a high-precision, adaptive, and intelligent solution for network traffic anomaly detection.

[0030] One embodiment of the present invention is as follows: the abnormal data sample generation step in step S4 is as follows: S41. Based on the anomaly label and confidence score in the anomaly detection results, select anomaly detection results with confidence scores higher than a preset threshold as candidate anomaly samples. Preset threshold: The optimal value is set within the range of 0.85 to 0.95; S42. Extract the duration, total data volume, number of packets, and burst density from the burst feature set corresponding to the candidate anomaly sample, and construct anomaly data samples by combining anomaly labels and confidence scores. The preset training sample library is set in the first and second buffer spaces of the pre-trained anomaly detection model.

[0031] Specifically, the details of triggering incremental training of the preset model in step S4, which optimizes the parameters of the pre-trained anomaly detection model, are as follows: S43. Replace the virtual datasets in the first and second buffer spaces with abnormal data samples. When the number of replaced abnormal data samples reaches the second preset threshold, recalculate the connection factor based on the duration, total data volume, number of packets and burst density of the abnormal data samples. The second preset threshold is optimally set within the range of 100 to 500 abnormal data samples. Specific recommended values ​​are as follows: High-speed network environment (data center / 10 Gigabit or higher): 100 to 200 samples; Ordinary enterprise network: 200 to 300 samples; High-latency network environment (WAN): 300 to 500 samples. This value needs to be coordinated with the preset sampling period and the first preset threshold. It is recommended that the amount of data corresponding to the second preset threshold can cover at least 10 complete burst unit sequences to ensure the effectiveness of incremental training. S44. Based on the recalculated connection factors and the original connection factors in the first, second, and third rings, the ring structure adjustment factor is generated using the following formula: ; in, It is a ring structure adjustment factor; , , All are preset values; For the recalculated connection factor; This is the average of all original connection factors in the first ring; This is the average of all original connection factors in the second ring; This is the average of all original connection factors in the third ring; Example: Take =0.5, =0.3, =0.2; Assuming the recalculated connection factor =1.25; Average value of the first ring original connection factor =0.85, second ring =1.10, third ring =1.40. Therefore: R=0.5×|1.25-0.85|+0.3×|1.25-1.10|+0.2×|1.25-1.40|=0.5×0.4+0.3×0.15+0.2×0.15=0.20+0.045+0.03=0.275; S45. Adjust the hidden layer parameters in the pre-trained anomaly detection model according to the ring structure adjustment factor, and use the validation dataset and calibration dataset to verify the convergence of the adjusted pre-trained anomaly detection model. When the output stability of the pre-trained anomaly detection model reaches the preset standard, the parameter optimization is completed.

[0032] The specific steps for adjusting hidden layer parameters are as follows: Analyze the numerical magnitude and directional characteristics of the ring structure adjustment factor to generate a parameter adjustment guidance vector; calculate the update magnitude and update direction of the hidden layer weight matrix based on the parameter adjustment guidance vector; optimize the gradient direction of the connection weights of the hidden layer neurons based on the update magnitude and update direction; verify the convergence of the adjusted hidden layer parameters using the validation dataset and the calibration dataset; terminate the adjustment process when the stability of the hidden layer output reaches the preset standard, thus completing the parameter optimization.

[0033] Preset standards: Basic safety scenario: output volatility <8%, loss reduction rate <0.3%, accuracy >92%; General enterprise scenario: output volatility <5%, loss reduction rate <0.2%, accuracy >95%; High safety scenario: output volatility <3%, loss reduction rate <0.1%, accuracy >98%.

[0034] This embodiment achieves incremental training by replacing the virtual dataset in the buffer space with abnormal data samples, calculates the ring structure adjustment factor based on the difference in connection factors, accurately adjusts the hidden layer parameters of the model, and performs convergence verification by combining the validation dataset and the calibration dataset, which significantly improves the adaptive ability and detection accuracy of the anomaly detection model, while ensuring the stability and reliability of the output.

[0035] A machine learning-based network traffic monitoring system includes a data acquisition module, a generation module, a training module, and an optimization module. The acquisition module is used to capture network data packets in real time and preprocess the network data packets; The generation module identifies burst units based on preprocessed network data packets, calculates the duration, data volume, and number of packets for each burst unit, and generates a burst feature set. The training module is used to input the set of sudden features into the pre-trained anomaly detection model and output the anomaly detection results, which include anomaly labels and confidence scores. The optimization module generates abnormal data samples with confidence scores higher than a preset threshold based on the abnormal labels and confidence scores in the abnormality detection results, and optimizes the parameters of the pre-trained abnormality detection model based on the abnormal data samples.

[0036] The above are merely embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A network traffic monitoring method based on machine learning, characterized in that: Includes the following steps: S1. Capture network data packets in real time and preprocess the network data packets; S2. Based on the preprocessed network data packets, identify network data packets with an arrival time interval less than a first preset threshold and the same transmission direction as burst units, calculate the duration, data volume and packet number of each burst unit, and generate a burst feature set; S3. Input the set of sudden features into a pre-trained anomaly detection model and output anomaly detection results, the anomaly detection results including anomaly labels and confidence scores; S4. Based on the anomaly label and the confidence score in the anomaly detection result, generate anomaly data samples whose confidence scores are higher than a preset threshold, add the anomaly data samples to a preset training sample library, and trigger incremental training of the preset model when the number of anomaly data samples reaches a second preset threshold to optimize the parameters of the pre-trained anomaly detection model.

2. The network traffic monitoring method based on machine learning according to claim 1, characterized in that: The network data packets in step S1 are captured in real time by monitoring tools deployed on network nodes. These monitoring tools include, but are not limited to, network sniffers, traffic mirroring ports, or deep packet inspection devices. The preprocessing in step S1 includes extracting the five-tuple information and traffic statistics features of the network data packets.

3. The network traffic monitoring method based on machine learning according to claim 2, characterized in that: The method for generating the arrival time interval and transmission direction of the network data packets in step S2 is as follows: S21. Parse the five-tuple information of the network data packet, extract the source address, destination address and protocol type features of the network data packet, and determine the transmission direction of the network data packet according to the numerical relationship between the source address and the destination address; S22. Start countdown monitoring within the preset sampling period. If the network data packet of the target address is not detected to arrive during the countdown, generate and insert a preset first virtual data packet when the countdown ends, and record the current system time as a virtual timestamp. If the arrival of the network data packet is detected during the countdown, the actual arrival timestamp of the network data packet is recorded and a preset second virtual data packet is inserted. The arrival time interval is calculated based on the difference in arrival timestamps of consecutive network data packets. The method for inserting the first virtual data packet in step S22 is as follows: S221. When the countdown ends and no network data packet is detected to arrive, the current system time is obtained as the base timestamp, and a first virtual data packet is generated based on the preset virtual data packet template. S222. Insert the first virtual data packet into the end of the current network data packet sequence, add a preset virtual tag and the virtual timestamp to the first virtual data packet, calculate the first difference between the virtual timestamp and the adjacent second arrival timestamp, and use the first difference as the virtual arrival time interval. The method for inserting the second virtual data packet in step S22 is as follows: S2201. When the arrival of the network data packet is detected during the countdown, the countdown monitoring is immediately terminated and the actual arrival timestamp of the network data packet is recorded. A second virtual data packet is created based on the preset data packet generation rules. S2202, Insert the second virtual data packet into the corresponding position of the monitoring time point of the current network data packet sequence, and add an actual arrival identifier and an actual arrival timestamp to the second virtual data packet. Calculate the actual arrival time interval of adjacent network data packets based on the actual arrival timestamp sequence of the continuously inserted second virtual data packets. Both the virtual arrival time interval and the actual arrival time interval are used as arrival time intervals.

4. The network traffic monitoring method based on machine learning according to claim 3, characterized in that: The protocol type characteristics in step S21 include connection-oriented protocols and connectionless protocols; the specific content of determining the transmission direction of the network data packets is as follows: S211. Extract the binary numerical representations of the source address and the target address, perform a bit-to-bit comparison operation on the binary numerical representations to generate an address comparison result, determine the address size relationship based on the difference of the most significant bit in the address comparison result, and define the direction indicated by the target address with the smaller value as the initial transmission direction. S212. Perform protocol-specific correction on the initial transmission direction based on the protocol type characteristics. When the protocol type is the connection-oriented protocol, maintain the initial transmission direction. When the protocol type is the connectionless protocol, reverse the initial transmission direction. S213. Mark the corrected initial transmission direction as the transmission direction of the network data packet, and generate a direction identifier for the transmission direction based on the protocol type feature and the address comparison result.

5. The network traffic monitoring method based on machine learning according to claim 4, characterized in that: The steps for generating the burst feature set in step S2 are as follows: S23. Based on the arrival time interval and the transmission direction, identify consecutive network data packet sequences whose arrival time interval is less than the first preset threshold and whose transmission direction is the same as the burst unit; S24. For each burst unit, extract its start timestamp and end timestamp, and calculate the difference between the start timestamp and the end timestamp as the duration of the burst unit. S25. Accumulate the load size of all network data packets in the burst unit to obtain the total data volume of the burst unit; S26. Count the number of network data packets in the burst unit to obtain the number of packets in the burst unit, and calculate the burst density of the burst unit based on the ratio of the number of packets to the duration. S27. Combine the duration, the total data volume, the number of packets, and the burst density into a burst feature set.

6. The network traffic monitoring method based on machine learning according to claim 5, characterized in that: The pre-trained anomaly detection model in step S3 is generated through the following steps: S31. Select several sets of burst features within the preset historical sampling period, sort all the burst feature sets according to the time series, and divide all the sorted burst feature sets into an initial training dataset, a validation dataset, and a correction dataset according to a preset segmentation period. S32. Construct a network topology structure, wherein the network topology structure includes a nested first ring, a second ring, and a third ring, and map the initial training dataset to the first ring, the validation dataset to the second ring, and the correction dataset to the third ring; S33. Calculate the connection factor between two adjacent burst feature sets in the initial training dataset, the validation dataset, and the calibration dataset. The connection factor is obtained by dividing the duration, the total data volume, the number of packets, and the burst density in the two adjacent burst feature sets respectively, and then summing the results by weight. S34. A first buffer space is set at the connection between the first ring and the second ring, and a second buffer space is set at the connection between the second ring and the third ring. A virtual dataset is generated in the first buffer space and the second buffer space according to the connection factor. The initial training dataset and the virtual dataset are jointly trained using a machine learning training algorithm to generate an initial anomaly detection model. S35. The initial anomaly detection model is evaluated using the validation dataset and the calibration dataset. The parameters of the initial anomaly detection model are adjusted according to the performance evaluation results until a preset accuracy threshold is reached, thereby generating an optimized anomaly detection model. The optimized anomaly detection model is then saved as a pre-trained anomaly detection model.

7. The network traffic monitoring method based on machine learning according to claim 6, characterized in that: The steps for generating the abnormal data sample in step S4 are as follows: S41. Based on the anomaly label and the confidence score in the anomaly detection results, select anomaly detection results with a confidence score higher than a preset threshold as candidate anomaly samples. S42. Extract the duration, total data volume, number of packets, and burst density from the burst feature set corresponding to the candidate anomaly sample, and construct an anomaly data sample by combining the anomaly label and the confidence score. The preset training sample library is set in the first buffer space and the second buffer space of the pre-trained anomaly detection model.

8. The network traffic monitoring method based on machine learning according to claim 7, characterized in that: The specific details of triggering incremental training of the preset model in step S4, and optimizing the parameters of the pre-trained anomaly detection model, are as follows: S43. Replace the virtual dataset in the first buffer space and the second buffer space with the abnormal data samples. When the number of the replaced abnormal data samples reaches a second preset threshold, recalculate the connection factor based on the duration, total data volume, number of packets and burst density in the abnormal data samples. S44. Based on the recalculated connection factors and the original connection factors in the first ring, the second ring, and the third ring, a ring structure adjustment factor is generated using the following formula: ； in, It is a ring structure adjustment factor; , , All are preset values; For the recalculated connection factor; This is the average value of all the original connection factors in the first ring; This is the average value of all the original connection factors in the second ring; The average value of all the original connection factors in the third ring; S45. Adjust the hidden layer parameters in the pre-trained anomaly detection model according to the ring structure adjustment factor, and use the validation dataset and the correction dataset to verify the convergence of the adjusted pre-trained anomaly detection model. When the output stability of the pre-trained anomaly detection model reaches the preset standard, the parameter optimization is completed.

9. The network traffic monitoring system based on machine learning according to claim 8, characterized in that: It includes a data acquisition module, a generation module, a training module, and an optimization module; The acquisition module is used to capture network data packets in real time and preprocess the network data packets; The generation module identifies burst units based on the preprocessed network data packets, calculates the duration, data volume, and packet count of each burst unit, and generates a burst feature set. The training module is used to input the set of sudden features into a pre-trained anomaly detection model and output anomaly detection results, which include anomaly labels and confidence scores. The optimization module generates abnormal data samples with confidence scores higher than a preset threshold based on the abnormal labels and confidence scores in the abnormal detection results, and optimizes the parameters of the pre-trained abnormal detection model based on the abnormal data samples.

Images