Intelligent flow monitoring system for network communication

Abstract

The invention discloses an intelligent flow monitoring system for network communication, and relates to the technical field of network communication monitoring. Comprising a traffic data acquisition module, a data preprocessing module, a data analysis and storage module, a data result display module, a real-time traffic analysis module, a communication traffic use monitoring module, a security threat detection module and a performance optimization processing module. The flow data acquisition module is used for capturing network communication flow data from data sources of network equipment and a server, an acquisition method of the flow data acquisition module selects one or more of SNMP, Net Flow and s Flow, flow data acquisition is realized by each node in a network, and acquisition equipment comprises one or more of a router and a switch. The intelligent flow monitoring system is composed of a plurality of modules and functions, the modules and the functions work cooperatively, real-time monitoring, analysis and optimization of network flow are achieved jointly, and through the components, a network administrator can effectively manage network resources and guarantee safety and stability of a network.

Description

Technical Field

[0001] The present invention relates to the technical field of network communication monitoring, in particular to an intelligent flow monitoring system for network communication. Background Art

[0002] Network communication refers to connecting isolated devices through the network and realizing communication between people, people and computers, and computers through information exchange. This communication method relies on network protocols, which are agreements that computers must follow when communicating in the network. They stipulate the information transmission rate, transmission code, code structure, transmission control steps, error control, etc. In network communication, there are usually multiple possible transmission paths for data flow from one location to another. Therefore, nodes can distribute data flow to multiple possible paths to improve the utilization of network resources.

[0003] At present, some network communications lack a monitoring system for traffic usage, which increases network security risks, such as DDOS attacks, malware propagation and other potential security threats. It also leads to a decline in network performance and the inability to reasonably allocate traffic usage, resulting in programs or devices occupying too much traffic. Therefore, implementing effective traffic monitoring is an important measure to ensure the stable operation and security of the network. Summary of the invention

[0004] In view of the deficiencies of the prior art, the present invention provides an intelligent traffic monitoring system for network communication, which solves the problems raised in the above-mentioned background technology.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: an intelligent traffic monitoring system for network communication, including a traffic data acquisition module, a data preprocessing module, a data analysis and storage module, a data result display module, a real-time traffic analysis module, a communication traffic usage monitoring module, a security threat detection module, and a performance optimization processing module;

[0006] The flow data collection module is used to capture network communication flow data from the data sources of network devices and servers. The collection method is one or more of SNMP (Simple Network Management Protocol), Net Flow, and s Flow. The flow data collection is realized by each node in the network. The collection equipment includes one or more of routers and switches.

[0007] The data preprocessing module cleans, filters, and formats the collected raw traffic data to ensure data quality and consistency for subsequent analysis. During the preprocessing process, noise and redundant information in the traffic data are removed, and the data is converted into a consistent format for comparison and analysis of data sets.

[0008] The data analysis and storage module uses the traffic data statistical analysis algorithm to conduct in-depth analysis on the pre-processed data. Through statistical analysis of the traffic data, it obtains the network traffic distribution information, peak time, abnormal data distribution information, extracts traffic data features, and stores the analysis results in the database or data warehouse to analyze the trend and pattern of network traffic and help optimize network traffic and performance.

[0009] Real-time traffic analysis module, used to capture and analyze network data packets in real time, ensuring that the network management system can detect and resolve abnormal problems in a timely manner;

[0010] Security threat detection module, used to identify suspicious behaviors and potential threats in the network, including DDOS attacks, malware propagation, excessive traffic values, and abnormal traffic values;

[0011] The performance optimization module analyzes and detects network traffic, identifies traffic usage bottlenecks and software performance issues, optimizes operations based on the problem, and improves the overall performance of the network.

[0012] Optionally, the flow data collection module has the following process:

[0013] (1) Investigate existing network problems, determine the type of traffic collection required, and analyze the direction of traffic data collection based on network requirements;

[0014] (2) Design a hierarchical and scalable architecture for the traffic collection network based on demand, and complete the hardware deployment and software installation of the traffic collection network;

[0015] (3) Online trial operation to test the traffic quality and transmission efficiency of physical devices and virtualized environments;

[0016] (4) Using network probes to deploy hardware or software devices at network nodes to capture and analyze traffic passing through the node to achieve low-latency data collection;

[0017] (5) Process the data to ensure that it is accurately sent to each traffic analysis system.

[0018] Optionally, the traffic data collection module includes the IP address, destination IP address, port number, and protocol type of the network traffic.

[0019] Optionally, the data preprocessing module includes filtering operations on the data to remove invalid information, and classifies the traffic data, and saves different types of traffic categories, including TXT record files, video traffic data, file transfer traffic data, web browsing traffic data, and audio traffic.

[0020] Optionally, the data preprocessing module uses one or more of Python, Hadoop, Spark, and Scikit-learn;

[0021] The process of the data preprocessing module is as follows:

[0022] (1) Data cleaning, including deleting duplicate data, filling in accurate data, and detecting abnormal data;

[0023] Deduplication, using hash values ​​or unique identifiers to identify and remove duplicate records;

[0024] Fill missing data and use mean, median, mode or interpolation methods to fill missing values;

[0025] Detect abnormal data, identify and deal with outliers through one of the statistical or machine learning methods;

[0026] (2) Data integration, merging multiple data sources into a unified data set to increase the data volume, including flow data alignment, entity recognition, and attribute redundancy;

[0027] Data alignment ensures that data formats and units are consistent across different data sources.

[0028] Entity recognition solves different expressions of the same entity in different data sources.

[0029] Attribute redundancy, handle duplicate attributes and avoid redundancy.

[0030] (3) Data transformation: converting data into a unified format to reduce the complexity of data types.

[0031] (4) Data reduction: screening data features based on selecting the smallest attribute subset.

[0032] Optionally, the data analysis and storage module includes an analysis and reporting function for analyzing the trend of traffic usage, with equal time periods as measurement units, and the equal time period units include s, min, h, d;

[0033] The specific units are as follows:

[0034] 00.01 to 00.10; 00.11 to 00.20; 00.21 to 00.30; 00.31 to 00.40; ...; 23.31 to 23.40; 23.41 to 23.50; 23.51 to 24.00;

[0035] The specific units are as follows:

[0036] 00.01 to 01.00; 01.01 to 02.00; 02.01 to 03.00; 03.01 to 04.00; ...; 21.01 to 22.00; 22.01 to 23.00; 03.01 to 24.00;

[0037] Use the Max and Min values ​​of traffic within a certain period of time to determine whether there are abnormal traffic fluctuations, and use the usage of the analysis protocol to find out whether there are potential security threats.

[0038] Optionally, the real-time traffic analysis module includes an abnormal traffic type identification unit, a data packet, and a traffic analysis unit;

[0039] The abnormal traffic type identification unit includes DDOS attack, DOS attack, Port Scan and IPSpoofing. The abnormal traffic type identification adopts IDS, which realizes the detection operation of abnormal traffic in the network by analyzing data packets, logs and network traffic information.

[0040] The data packet and traffic analysis unit includes mobile users' online behavior data, content analysis, user portraits, terminal analysis, network system monitoring, and online detailed query. The collected data is aggregated through the FTP server, maintained in a directory for 5 minutes, and accumulated in the server.

[0041] The packet and traffic analysis unit includes two threads;

[0042] Thread A detects whether the folder in the FTP server has been generated every 5 minutes, and temporarily puts the generated files on the server where the logs are stored;

[0043] Thread B transfers data every 15 minutes to HDFS. Data is classified in HDFS. During the transfer process, the files need to be marked with the beginning and end. If there is an abnormality during the transfer, the previously transferred data is deleted and retransmitted.

[0044] After the data is stored in HDFS, it is stored centrally and SCA is used to enhance content recognition and behavior trajectory in units of h.

[0045] Optionally, the real-time traffic analysis module further includes a result display unit, which displays the analysis results of traffic usage through a visual interface and provides a network digital chart for the user;

[0046] Network Digital Chart reports include traffic statistics, network performance indicators, and security analysis results.

[0047] Optionally, the traffic statistics information specifically includes the total traffic value, the maximum value and minimum value of the instantaneous traffic, the average value of the total traffic, the proportion of each type of traffic, and the average value of each type of traffic;

[0048] Network performance indicators specifically include bandwidth utilization and network latency;

[0049] The traffic statistics information and network performance indicators use the same time interval as the unit when they are counted, and a trend chart is generated for comparison.

[0050] The present invention provides an intelligent flow monitoring system for network communication, which has the following beneficial effects:

[0051] This intelligent traffic monitoring system for network communications can identify the root cause of network congestion problems and reduce troubleshooting time through traffic analysis; effectively improve network security, and use intelligent traffic monitoring to detect abnormal traffic in real time and identify potential security threats, so as to take timely protective measures to ensure network security; and can provide detailed network usage reports to help network administrators identify and solve network bottlenecks and improve network performance; for example, by analyzing network traffic, it can be found which applications or devices occupy too much bandwidth, so as to optimize the configuration and ensure the reasonable allocation of network resources;

[0052] By collecting traffic data, network bandwidth can be scientifically allocated to ensure the stable operation of key user-specific programs. At the same time, data streams can be identified and classified, and abnormal traffic can be located. By monitoring network traffic, network protocols, IP addresses, port numbers and other information, abnormal traffic in the network can be quickly and accurately located, and user feedback can be generated. Processing instructions can be selected based on the appropriateness and analysis of abnormal traffic nodes. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0054] The present invention provides a technical solution: an intelligent traffic monitoring system for network communication, comprising a traffic data acquisition module, a data preprocessing module, a data analysis and storage module, a data result display module, a real-time traffic analysis module, a communication traffic usage monitoring module, a security threat detection module, and a performance optimization processing module;

[0055] The flow data collection module is used to capture network communication flow data from the data sources of network devices and servers. The collection method uses one or more of SNMP (Simple Network Management Protocol), Net Flow, and s Flow. The flow data collection is realized by each node in the network. The collection equipment includes one or more of routers and switches. The flow data collection module process is as follows:

[0056] (1) Investigate existing network problems, determine the type of traffic collection required, and analyze the direction of traffic data collection based on network requirements;

[0057] (2) Design a hierarchical and scalable architecture for the traffic collection network based on demand, and complete the hardware deployment and software installation of the traffic collection network;

[0058] (3) Online trial operation to test the traffic quality and transmission efficiency of physical devices and virtualized environments;

[0059] (4) Using network probes to deploy hardware or software devices at network nodes to capture and analyze traffic passing through the node to achieve low-latency data collection;

[0060] (5) Process the data to ensure that it is accurately sent to each traffic analysis system;

[0061] The traffic data collection module includes the IP address, destination IP address, port number, and protocol type of the network traffic;

[0062] The data preprocessing module cleans, filters, and formats the collected raw traffic data to ensure data quality and consistency for subsequent analysis. During the preprocessing process, noise and redundant information in the traffic data are removed, and the data is converted into a consistent format for comparison and analysis of data sets.

[0063] The data preprocessing module includes filtering operations on data to remove invalid information, and classifies traffic data to save different types of traffic, including TXT record files, video traffic data, file transfer traffic data, web browsing traffic data, and audio traffic;

[0064] The data preprocessing module uses one or more of Python, Hadoop, Spark, and Scikit-learn;

[0065] The process of the data preprocessing module is as follows:

[0066] (1) Data cleaning, including deleting duplicate data, filling in accurate data, and detecting abnormal data;

[0067] Deduplication, using hash values ​​or unique identifiers to identify and remove duplicate records;

[0068] Fill missing data and use mean, median, mode or interpolation methods to fill missing values;

[0069] Detect abnormal data, identify and deal with outliers through one of the statistical or machine learning methods;

[0070] (2) Data integration, merging multiple data sources into a unified data set to increase the data volume, including flow data alignment, entity recognition, and attribute redundancy;

[0071] Data alignment ensures that data formats and units are consistent across different data sources.

[0072] Entity recognition solves different expressions of the same entity in different data sources.

[0073] Attribute redundancy, handle duplicate attributes and avoid redundancy.

[0074] (3) Data transformation: converting data into a unified format to reduce the complexity of data types.

[0075] (4) Data reduction, which is based on selecting the smallest attribute subset to screen data features;

[0076] The data analysis and storage module uses the traffic data statistical analysis algorithm to conduct in-depth analysis on the pre-processed data. Through statistical analysis of the traffic data, it obtains the network traffic distribution information, peak time, abnormal data distribution information, extracts traffic data features, and stores the analysis results in the database or data warehouse to analyze the trend and pattern of network traffic and help optimize network traffic and performance.

[0077] The data analysis and storage module includes analysis and reporting functions for analyzing traffic usage trends, with equal time periods as the measurement unit. The equal time period units include s, min, h, d;

[0078] The specific units are as follows:

[0079] 00.01 to 00.10; 00.11 to 00.20; 00.21 to 00.30; 00.31 to 00.40; ...; 23.31 to 23.40; 23.41 to 23.50; 23.51 to 24.00;

[0080] The specific units are as follows:

[0081] 00.01 to 01.00; 01.01 to 02.00; 02.01 to 03.00; 03.01 to 04.00; ...; 21.01 to 22.00; 22.01 to 23.00; 03.01 to 24.00;

[0082] Use the maximum and minimum values ​​of traffic in a certain period of time to determine whether there are abnormal traffic fluctuations, and use the usage of the analysis protocol to find out whether there are potential security threats;

[0083] Real-time traffic analysis module, used to capture and analyze network data packets in real time, ensuring that the network management system can detect and resolve abnormal problems in a timely manner;

[0084] The real-time traffic analysis module includes an abnormal traffic type identification unit, a data packet, and a traffic analysis unit;

[0085] The abnormal traffic type identification unit includes DDOS attack, DOS attack, Port Scan and IPSpoofing. The abnormal traffic type identification adopts IDS, which realizes the detection operation of abnormal traffic in the network by analyzing data packets, logs and network traffic information.

[0086] The data packet and traffic analysis unit includes mobile users' online behavior data, content analysis, user portraits, terminal analysis, network system monitoring, and online detailed query. The collected data is aggregated through the FTP server, maintained in a directory for 5 minutes, and accumulated in the server.

[0087] The packet and traffic analysis unit includes two threads;

[0088] Thread A detects whether the folder in the FTP server has been generated every 5 minutes, and temporarily puts the generated files on the server where the logs are stored;

[0089] Thread B transfers data every 15 minutes to HDFS. Data is classified in HDFS. During the transfer process, the files need to be marked with the beginning and end. If there is an abnormality during the transfer, the previously transferred data is deleted and retransmitted.

[0090] After the data is stored in HDFS, the data is stored centrally and SCA is used to enhance the content recognition and behavior trajectory in units of h;

[0091] The real-time traffic analysis module also includes a result display unit, which displays the analysis results of traffic usage through a visual interface and provides users with network digital charts;

[0092] Network Digital Chart reports include traffic statistics, network performance indicators, and security analysis results.

[0093] Traffic statistics include total traffic value, maximum and minimum values ​​of instantaneous traffic, average total traffic value, proportion of each type of traffic, and average value of each type of traffic;

[0094] Network performance indicators specifically include bandwidth utilization and network latency;

[0095] The traffic statistics and network performance indicators use the same time interval as the unit when they are counted, and a trend chart is generated for comparison;

[0096] Security threat detection module, used to identify suspicious behaviors and potential threats in the network, including DDOS attacks, malware propagation, excessive traffic values, and abnormal traffic values;

[0097] The performance optimization module analyzes and detects network traffic, identifies traffic usage bottlenecks and software performance issues, optimizes operations based on the problem, and improves the overall performance of the network.

[0098] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An intelligent traffic monitoring system for network communication, characterized in that: It includes flow data collection module, data preprocessing module, data analysis and storage module, data result display module, real-time flow analysis module, communication flow usage monitoring module, security threat detection module, and performance optimization processing module; The flow data collection module is used to capture network communication flow data from the data sources of network devices and servers. The collection method is one or more of SNMP, Net Flow, and s Flow. The flow data collection is realized by each node in the network. The collection equipment includes one or more of routers and switches. The data preprocessing module cleans, filters, and formats the collected raw traffic data to ensure the quality and consistency of the data for subsequent analysis; During preprocessing, noise and redundant information in traffic data are removed and the data is converted into a consistent format for comparison and analysis of data sets; The data analysis and storage module uses the traffic data statistical analysis algorithm to conduct in-depth analysis on the pre-processed data. Through statistical analysis of the traffic data, it obtains the network traffic distribution information, peak time, abnormal data distribution information, extracts traffic data features, and stores the analysis results in the database or data warehouse to analyze the trend and pattern of network traffic and help optimize network traffic and performance. Real-time traffic analysis module, used to capture and analyze network data packets in real time, ensuring that the network management system can detect and resolve abnormal problems in a timely manner; Security threat detection module, used to identify suspicious behaviors and potential threats in the network, including DDOS attacks, malware propagation, excessive traffic values, and abnormal traffic values; The performance optimization module analyzes and detects network traffic, identifies traffic usage bottlenecks and software performance issues, optimizes operations based on the problem, and improves the overall performance of the network.

2. The intelligent traffic monitoring system for network communication according to claim 1, characterized in that: The flow of the flow data acquisition module is as follows: (1) Investigate existing network problems, determine the type of traffic collection required, and analyze the direction of traffic data collection based on network requirements; (2) Design a hierarchical and scalable architecture for the traffic collection network based on demand, and complete the hardware deployment and software installation of the traffic collection network; (3) Online trial operation to test the traffic quality and transmission efficiency of physical devices and virtualized environments; (4) Using network probes to deploy hardware or software devices at network nodes to capture and analyze traffic passing through the node to achieve low-latency data collection; (5) Process the data to ensure that it is accurately sent to each traffic analysis system.

3. The intelligent traffic monitoring system for network communication according to claim 1, characterized in that: The flow data collection module includes the IP address, destination IP address, port number, and protocol type of the network flow.

4. The intelligent traffic monitoring system for network communication according to claim 1, characterized in that: The data preprocessing module includes filtering operations on data to remove invalid information, and at the same time classifies the flow data and saves different types of flow categories, including TXT record files, video flow data, file transfer flow data, web browsing flow data, and audio flow.

5. The intelligent traffic monitoring system for network communication according to claim 1, characterized in that: The data preprocessing module uses one or more of Python, Hadoop, Spark, and Scikit-learn; The process of the data preprocessing module is as follows: (1) Data cleaning, including deleting duplicate data, filling in accurate data, and detecting abnormal data; Deduplication, using hash values ​​or unique identifiers to identify and remove duplicate records; Fill missing data and use mean, median, mode or interpolation methods to fill missing values; Detect abnormal data, identify and deal with outliers through one of the statistical or machine learning methods; (2) Data integration, merging multiple data sources into a unified data set to increase the data volume, including flow data alignment, entity recognition, and attribute redundancy; Data alignment, ensuring that data formats and units are consistent across different data sources; Entity recognition, solving different expressions of the same entity in different data sources; Attribute redundancy, handle duplicate attributes and avoid redundancy; (3) Data transformation: converting data into a unified format to reduce the complexity of data types. (4) Data reduction: screening data features based on selecting the smallest attribute subset.

6. The intelligent traffic monitoring system for network communication according to claim 1, characterized in that: The data analysis and storage module includes an analysis and reporting function for analyzing the trend of traffic usage, using equal time periods as measurement units, and the equal time period units include s, min, h, d; The specific units are as follows: 00.01 to 00.10; 00.11 to 00.20; 00.21 to 00.30; 00.31 to 00.40; ...; 23.31 to 23.40; 23.41 to 23.50; 23.51 to 24.00; The specific units are as follows: 00.01 to 01.00; 01.01 to 02.00; 02.01 to 03.00; 03.01 to 04.00; ...; 21.01 to 22.00; 22.01 to 23.00; 03.01 to 24.00; Use the Max and Min values ​​of traffic within a certain period of time to determine whether there are abnormal traffic fluctuations, and use the usage of the analysis protocol to find out whether there are potential security threats.

7. The intelligent traffic monitoring system for network communication according to claim 1, characterized in that: The real-time traffic analysis module includes an abnormal traffic type identification unit, a data packet and a traffic analysis unit; The abnormal traffic type identification unit includes DDOS attack, DOS attack, Port Scan and IP Spoofing; abnormal traffic type identification adopts IDS, which realizes the detection operation of abnormal traffic in the network by analyzing data packets, logs and network traffic information; The data packet and traffic analysis unit includes mobile users' online behavior data, content analysis, user portraits, terminal analysis, network system monitoring, and online detailed query. The collected data is aggregated through the FTP server, maintained in a directory for 5 minutes, and accumulated in the server. The packet and traffic analysis unit includes two threads; Thread A detects whether the folder in the FTP server has been generated every 5 minutes, and temporarily puts the generated files on the server where the logs are stored; Thread B transfers data every 15 minutes to HDFS. Data is classified in HDFS. During the transfer process, the files need to be marked with the beginning and end. If there is an abnormality during the transfer, the previously transferred data is deleted and retransmitted. After the data is stored in HDFS, it is stored centrally and SCA is used to enhance content recognition and behavior trajectory in units of h.

8. The intelligent traffic monitoring system for network communication according to claim 1, characterized in that: The real-time traffic analysis module also includes a result display unit, which displays the analysis results of traffic usage through a visual interface and provides network digital charts for users; Network Digital Chart reports include traffic statistics, network performance indicators, and security analysis results.

9. The intelligent traffic monitoring system for network communication according to claim 8, characterized in that: The traffic statistics information specifically includes the total traffic value, the maximum and minimum values ​​of the instantaneous traffic, the average of the total traffic, the proportion of each type of traffic, and the average of each type of traffic; Network performance indicators specifically include bandwidth utilization and network latency; The traffic statistics information and network performance indicators use the same time interval as the unit when they are counted, and a trend chart is generated for comparison.