Network monitoring system for big data

Abstract

The invention discloses a network monitoring system for big data. The system comprises a data input layer, the data input layer is connected with a data processing layer, the data processing layer is connected with a monitoring and prediction layer, the monitoring and prediction layer is connected with a strategy adjustment layer, and the strategy adjustment layer is connected with a user interaction and transparency layer. And the data input layer comprises real-time data collection and historical data storage. Compared with traditional monitoring, the system provided by the invention supports dynamic adaptation to the change of a network environment through the identification and prediction capability of a network abnormal mode, provides a more intelligent and automatic management mode, simplifies the deployment and update of the system through the modular design, enables the expansion to be easier, reduces the technical investment requirement, and improves the monitoring efficiency. The flexibility and the efficiency are improved, the network administrator can respond to potential problems more quickly through the highly customizable instrument board and the real-time alarm function, and the accuracy and the response speed of network monitoring are comprehensively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of network monitoring, and in particular to a big data network monitoring system. Background Art

[0002] Network monitoring is the process of observing and analyzing the traffic, performance, and overall operation status of a computer network in real time. Its main purpose is to ensure the normal operation of the network, identify faults, improve performance, and ensure network security. By monitoring network devices (such as routers, switches, servers, etc.) and connections, network monitoring helps administrators to keep abreast of network conditions, optimize configurations, and prevent potential problems.

[0003] The existing network monitoring system relies on a variety of technologies and tools to achieve effective network management. Its core technologies include Simple Network Management Protocol (SNMP), NetFlow, and sFlow, which are used to collect and analyze network data. These technologies help monitoring tools (such as Nagios, Zabbix, and SolarWinds) collect network traffic, performance indicators, and other key data by deploying agents or using sensors. This data is then visualized through dashboard software (such as Grafana or Kibana), allowing network administrators to intuitively grasp the real-time status of the network. In addition, administrators can predefine thresholds and set alarm rules so that they can be notified immediately when any anomalies or performance degradation are detected. However, the existing network monitoring system has obvious limitations in dealing with complex network environments and adapting to dynamic changes. Traditional monitoring mainly relies on preset rules and static parameters, which lacks sufficient flexibility and responsiveness when dealing with sudden anomalies. In addition, the architecture of traditional monitoring systems is usually complex, and deployment, maintenance, and updates often require high technical costs and human resources. Such limitations have limited the efficiency and effectiveness of traditional monitoring systems in the rapidly changing modern network environment. Summary of the invention

[0004] In view of the defects in the prior art, the present invention provides a big data network monitoring system, comprising a data input layer, the data input layer is connected to a data processing layer, the data processing layer is connected to a monitoring and prediction layer, the monitoring and prediction layer is connected to a strategy adjustment layer, and the strategy adjustment layer is connected to a user interaction and transparency layer;

[0005] The data input layer includes real-time data collection and historical data storage, the data processing layer includes data preprocessing, natural language processing and behavior pattern analysis, the monitoring and prediction layer includes health indicator detection, perception enhancement and prediction modules, the strategy adjustment layer includes self-organization coordination, dynamic feedback mechanism and time-sensitive management, and the user interaction and transparency layer includes open data interface and public participation platform.

[0006] Preferably, the real-time data collection is to collect network traffic data and user behavior log information from various data sources in real time, and use the Kafka message queue tool to realize data collection from smart devices and network nodes. The historical data storage is to use the data lake architecture Hadoop or Amazon S3 to store unstructured and structured data, and support data traceability and long-term behavior analysis.

[0007] Preferably, the data preprocessing is to clean, format and denoise the collected data, including missing value filling, outlier processing and normalization processing; the natural language processing is to identify the user's cultural background and language pattern by analyzing text data, and improve data classification; the behavior pattern analysis is to identify user behavior patterns and abnormal activities through machine learning models, and use clustering, classification K-means or decision trees and association analysis to identify and mark specific behavior patterns.

[0008] Preferably, the health indicator detection uses ELK Stack (Elasticsearch, Logstash, Kibana) to define and monitor key indicators of network health, such as traffic, latency, and packet loss rate. The visualization indicator status uses rule engines and machine learning algorithms, such as outlier detection, to identify anomalies, detect abnormal activities in real time, and generate alarm notifications. The perception enhancement uses visualization tools to view data flows and network status in real time, and uses augmented reality (AR) technology to provide a visualization interface for system operation. The prediction module uses time series LSTM or ARIMA analysis to perform short-term and long-term predictions, and predicts future trends and possible problems by analyzing historical and real-time data.

[0009] Preferably, the self-organizing coordination adopts self-learning optimization algorithms, such as ant colony optimization algorithm (ACO) and particle swarm optimization algorithm (PSO), to implement automatic resource allocation and task scheduling among different nodes; the dynamic feedback mechanism automatically updates the system operation strategy through event triggers and rule engines, and makes real-time strategy adjustments according to external events or internal analysis results; the time-sensitive management uses scheduling algorithms to adjust the monitoring depth and strategy, and adjusts resource allocation and priority according to time periods, such as peak and non-peak periods.

[0010] Preferably, the open data interface displays system data and operating status through API and Dashboard, and provides an interface for external users and systems to access monitoring data. The public participation platform collects user experience and suggestions through forums, questionnaires, etc.

[0011] The beneficial effects of the present invention are embodied in:

[0012] The system of the present invention is inspired by the behavior of ant colonies or bird flocks and adopts a self-organizing mechanism. The network nodes in the system communicate with each other through simple rules and automatically share information. The nodes can dynamically adjust the traffic routing and monitoring strategies according to the real-time traffic data, significantly reducing the dependence on centralized management. At the same time, it has fault tolerance and recovery capabilities and can adjust its own behavior according to the status of neighboring nodes, thereby enhancing the robustness of the system. In addition, the system introduces a cultural computing model. By identifying the user's language and cultural background, the system can adjust the monitoring parameters in a personalized manner and establish a behavior pattern library to enhance the recognition and understanding of different cultural behavior patterns. This helps to dynamically adjust the monitoring focus under different cultural backgrounds, improve the adaptability of monitoring, and improve network health. In terms of monitoring, the system has set up a set of key performance indicator systems to regularly assess the network health status, which is similar to regular inspections in health management. In order to enhance the intuitive perception of network administrators, the system of the present invention uses XR technology and comprehensive prediction models to enhance the perception and prediction of network status. It can perform short-term real-time data analysis and long-term trend prediction based on historical data. In terms of environmental impact feedback, the system is equipped with a dynamic feedback loop, which can automatically trigger an enhanced monitoring mechanism when a major global event occurs, and adaptively adjust the depth and breadth of monitoring according to demand. Through the combination of these core technologies, the system realizes all-round intelligent monitoring and improves the flexibility and efficiency of network management. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0014] Figure 1 The figure is a schematic diagram of the system flow. DETAILED DESCRIPTION

[0015] The following embodiments of the technical solution of the present invention are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and cannot be used to limit the protection scope of the present invention.

[0016] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by those skilled in the art to which the present invention belongs.

[0017] like Figure 1 As shown, a big data network monitoring system includes a data input layer, the data input layer is connected to a data processing layer, the data processing layer is connected to a monitoring and prediction layer, the monitoring and prediction layer is connected to a strategy adjustment layer, and the strategy adjustment layer is connected to a user interaction and transparency layer;

[0018] The data input layer includes real-time data collection and historical data storage; the data processing layer includes data preprocessing, natural language processing and behavior pattern analysis; the monitoring and prediction layer includes health indicator detection, perception enhancement and prediction modules; the strategy adjustment layer includes self-organization coordination, dynamic feedback mechanism and time-sensitive management; the user interaction and transparency layer includes open data interface and public participation platform.

[0019] Real-time data collection is to collect network traffic data and user behavior log information from various data sources in real time, and use Kafka message queue tools to realize data collection from smart devices and network nodes. Historical data storage is to use data lake architecture Hadoop or Amazon S3 to store unstructured and structured data, and support data traceability and long-term behavior analysis.

[0020] Data preprocessing is to clean, format and denoise the collected data, including missing value filling, outlier processing and normalization. Natural language processing is to analyze text data, identify users' cultural background and language patterns, and improve data classification. Behavioral pattern analysis is to identify user behavior patterns and abnormal activities through machine learning models, and use clustering, classification K-means or decision trees and association analysis to identify and mark specific behavior patterns.

[0021] Health indicator detection uses ELK Stack (Elasticsearch, Logstash, Kibana) to define and monitor key indicators of network health, such as traffic, latency, and packet loss rate. Visual indicator status uses rule engines and machine learning algorithms, such as outlier detection, to identify anomalies, detect abnormal activities in real time, and generate alarm notifications. Perception enhancement uses visualization tools to view data flows and network status in real time, and uses augmented reality (AR) technology to provide a visual interface for system operation. The prediction module uses time series LSTM or ARIMA analysis to perform short-term and long-term predictions, and predicts future trends and possible problems by analyzing historical and real-time data.

[0022] In health indicator monitoring, you first need to determine the key health indicators that need to be monitored, including traffic, latency, packet loss rate, and CPU usage. Then build a data collection infrastructure and use tools such as Prometheus or Nagios to regularly extract data for these indicators. Use ELK Stack (Elasticsearch, Logstash, Kibana) or InfluxDB time series database to store and analyze this data. Create a dashboard through Grafana or Kibana to provide real-time graphical display to facilitate identification of problems and trends.

[0023] Anomaly detection and alerting systems use historical data to build a baseline model of normal behavior, which can be achieved through statistical analysis or machine learning methods, such as clustering algorithms. Use stream processing frameworks such as Apache Flink or Spark Streaming to analyze incoming data streams in real time and compare them with the baseline. Use rule engines such as Drools and machine learning algorithms such as Isolation Forest or Support Vector Machine to identify abnormal behavior. Set up alert rules, such as sending alert information to administrators via email, SMS, or instant messaging services such as Slack. Record all abnormal events and cooperate with ticket systems such as Jira Service Desk to track the progress of problem handling.

[0024] In perception enhancement, you first need to determine which system indicators and information need to be displayed in augmented reality, and communicate with the team to optimize the user experience. Select AR tools and platforms, ARKit, ARCore, or Unity, to provide an augmented view of the system. Display AR projections of key indicator data to ensure that users can view the system status from all angles. Then integrate the AR solution into the monitoring system and test it with the goal of user-friendliness and efficiency.

[0025] In the prediction model, the data must be collated and cleaned first, including historical data verification and the correctness of real-time data collection. Select an appropriate time series prediction model according to the specific usage scenario, such as ARIMA (autoregressive integrated moving average model) or LSTM (long short-term memory network). Use historical data sets to train the model and verify its accuracy and stability through cross-validation methods. Deploy the trained prediction model in a real-time environment, such as TensorFlow Serving or using a microservice architecture. Analyze the prediction results and provide feedback on inaccurate results for regular updates and optimization of the model.

[0026] Self-organization and coordination uses self-learning optimization algorithms, such as the ant colony optimization algorithm (ACO) and the particle swarm optimization algorithm (PSO), to implement automatic resource allocation and task scheduling among different nodes. The dynamic feedback mechanism automatically updates the system operation strategy through event triggers and rule engines, and makes real-time strategy adjustments based on external events or internal analysis results. Time-sensitive management uses scheduling algorithms to adjust monitoring depth and strategies, and adjusts resource allocation and priority based on time periods, such as peak and off-peak periods.

[0027] In self-organizing coordination, it is necessary to clearly define the goals that need to be optimized, such as minimizing network latency, maximizing throughput, or improving resource utilization. Use self-organizing algorithms, such as ant colony optimization (ACO) and particle swarm optimization (PSO). These algorithms can automatically discover the best solution by simulating natural phenomena (such as ants looking for the shortest path). Simulate the selected algorithm in a virtual environment to evaluate its performance under different loads and scenarios. Integrate the optimization algorithm into the operating environment and ensure communication with other parts of the system through APIs or microservice interfaces.

[0028] Identify system components and parameters that need to be adjusted dynamically in the dynamic feedback mechanism, such as traffic priority and bandwidth configuration. Collect real-time indicator data from the internal health monitoring module, as well as information about changes in the external environment (such as fluctuations in user demand). Design feedback loops, define inputs and outputs and corresponding rules, and use control theory or machine learning techniques (such as reinforcement learning) to model. Based on the results of model inference, dynamically adjust strategies through automation tools such as Ansible or Chef. Test the adaptability and robustness of the feedback mechanism in different scenarios, and continuously optimize based on test feedback.

[0029] Identify key time periods in time-sensitive management, such as peak / off-peak hours, and analyze the system load characteristics at different times. Develop corresponding strategies for different time periods, such as increasing bandwidth or resource allocation during peak hours, and performing system maintenance or data backup during off-peak hours. Use dynamic scheduling algorithms to drive task scheduling to implement resource management and automatically adjust resource allocation according to actual needs. Configure automated scripts or use schedule software Cron jobs to ensure that adjustment tasks are automatically executed at the set time. By monitoring system performance in real time, dynamically adjust strategies to adapt to instantaneous demand changes to ensure efficient operation of the system.

[0030] The open data interface displays system data and operating status through API and Dashboard, and provides an interface for external users and systems to access monitoring data. The public participation platform collects user experience and suggestions through forums and questionnaires.

[0031] The system of the present invention adopts a self-organizing mechanism, and its design concept is derived from the natural behavior of ant colonies and bird flocks. By implementing simple regular communications between network nodes, nodes can autonomously share and disseminate monitoring information. Such a decentralized structure enables the network to respond to dynamic changes more flexibly. Each node can autonomously adjust its own traffic routing and monitoring strategy based on real-time traffic data, greatly reducing the dependence on traditional centralized management systems. At the same time, these nodes have fault tolerance and recovery functions, and can automatically adjust their own configuration and behavior when detecting changes in the status of neighboring nodes, thereby enhancing the robustness of the entire network system and ensuring that the overall operation can be maintained when some nodes fail.

[0032] In terms of cultural computing models, the system focuses on understanding and adapting to the cultural and language backgrounds of different user groups. The system of the present invention can identify the language habits of users through multilingual support functions, and adjust the monitoring parameters accordingly. In addition, the system has established a behavior pattern library based on big data analysis, covering typical behavior patterns under different cultural backgrounds. This not only improves the accuracy of identifying normal and abnormal behaviors, but also dynamically adjusts the monitoring focus based on cultural differences to better meet the needs of users from different cultural backgrounds.

[0033] In terms of network health monitoring, the system sets key performance indicators (KPIs) for real-time monitoring of network status. These indicators are similar to "network traffic anomaly rate" and "delay time". On this basis, the system draws on the periodic diagnosis mechanism in health management to conduct regular network health status assessments to help administrators prevent and promptly discover potential problems. In order to enhance the intuitive perception of network status, the system uses XR (extended reality) technology, combined with a comprehensive prediction model, so that network administrators can more intuitively perceive the real-time status of the network through augmented reality devices (such as VR / AR helmets) and implement short-term and long-term traffic forecasts. This combination can not only help administrators analyze traffic data in real time and propose immediate response strategies, but also conduct long-term trend forecasts through historical data analysis and optimize the overall planning of network resources.

[0034] In terms of environmental impact feedback, the system introduces a dynamic feedback loop mechanism. When encountering major global events (such as breaking news events or natural disasters), the system will automatically trigger enhanced monitoring mode to ensure the efficient operation of the network. According to specific needs, the system can also adaptively expand the depth and breadth of monitoring to adapt to the impact that changes in the external environment may have on the network. Through this series of technical integration and optimization, the network monitoring system not only improves the accuracy and efficiency of monitoring, but also significantly improves the user experience and management convenience of network administrators.

[0035] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A big data network monitoring system, comprising a data input layer, characterized in that: The data input layer is connected to the data processing layer, the data processing layer is connected to the monitoring and prediction layer, the monitoring and prediction layer is connected to the strategy adjustment layer, and the strategy adjustment layer is connected to the user interaction and transparency layer; The data input layer includes real-time data collection and historical data storage, the data processing layer includes data preprocessing, natural language processing and behavior pattern analysis, the monitoring and prediction layer includes health indicator detection, perception enhancement and prediction modules, the strategy adjustment layer includes self-organization coordination, dynamic feedback mechanism and time-sensitive management, and the user interaction and transparency layer includes open data interface and public participation platform.

2. A big data network monitoring system according to claim 1, characterized in that: The real-time data collection is to collect network traffic data and user behavior log information from various data sources in real time, and use the Kafka message queue tool to realize data collection from smart devices and network nodes. The historical data storage is to use the data lake architecture Hadoop or Amazon S3 to store unstructured and structured data, and support data traceability and long-term behavior analysis.

3. A big data network monitoring system according to claim 1, characterized in that: The data preprocessing is to clean, format and denoise the collected data, including missing value filling, outlier processing and normalization. The natural language processing is to analyze text data, identify the user's cultural background and language pattern, and improve data classification. The behavior pattern analysis is to identify user behavior patterns and abnormal activities through machine learning models, and use clustering, classification K-means or decision trees and association analysis to identify and mark specific behavior patterns.

4. The big data network monitoring system according to claim 1, characterized in that: The health indicator detection uses ELK Stack to define and monitor key indicators of network health, including traffic, latency, and packet loss rate.

5. The big data network monitoring system according to claim 1, characterized in that: The visualized indicator status uses rule engines and machine learning algorithms, such as outlier detection, to identify anomalies, detect abnormal activities in real time, and generate alarm notifications.

6. A big data network monitoring system according to claim 1, characterized in that: The perception enhancement is to view the data flow and network status in real time through visualization tools, and use augmented reality technology to provide a visualization interface for system operation.

7. The big data network monitoring system according to claim 1, characterized in that: The prediction module uses time series LSTM or ARIMA analysis and performs short-term and long-term forecasts to predict future trends and possible problems by analyzing historical and real-time data.

8. The big data network monitoring system according to claim 1, characterized in that: The self-organization coordination adopts self-learning optimization algorithm, ant colony optimization algorithm and particle swarm optimization algorithm to implement automatic resource allocation and task scheduling among different nodes.

9. The big data network monitoring system according to claim 1, characterized in that: The dynamic feedback mechanism automatically updates the system operation strategy through event triggers and rule engines, and makes real-time strategy adjustments based on external events or internal analysis results. The time-sensitive management uses scheduling algorithms to adjust monitoring depth and strategy, and adjusts resource allocation and priority based on time periods, including peak and non-peak periods.

10. The big data network monitoring system according to claim 1, characterized in that: The open data interface displays system data and operating status through API and Dashboard, and provides an interface for external users and systems to access monitoring data. The public participation platform collects user experience and suggestions through forums and questionnaires.

Images