Data monitoring management system based on cloud computing

Abstract

The invention discloses a data monitoring management system based on cloud computing, which is applied to the technical field of data monitoring management, and comprises a real-time monitoring module, a data acquisition module, a data processing module, a data transmission module, a cloud storage module, a data analysis module and an alarm module, and the data acquisition module is bidirectionally and electrically connected with the data processing module. The computing resources can be dynamically adjusted through the elastic expansion capability of the cloud platform, and the capacity is automatically expanded at the traffic peak to guarantee the real-time performance; the cost is reduced in low ebb, and the resource utilization rate is improved. By means of a distributed processing architecture of cloud native service, the system can efficiently process millions of data points per second, and supports unified collection and standardization of multi-source heterogeneous data. Compared with a traditional alarm mode based on a fixed rule, the system can present a multi-dimensional analysis result, and finally closed-loop management from data acquisition, intelligent analysis to decision response is formed.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data monitoring management, and particularly relates to a data monitoring management system based on cloud computing. BACKGROUND

[0002] The data monitoring management system is a software system for real-time or periodic collection, processing, analysis and display of various data, which automatically collects information from databases, APIs, sensors and other data sources, continuously monitors key indicators, and triggers early warning when abnormalities are found, helping to quickly grasp the system operation status and business situation.

[0003] However, the traditional localized monitoring scheme is limited by fixed hardware resources and cannot cope with data processing delays or losses caused by sudden traffic. Decentralized independent monitoring tools form information silos due to the lack of a unified data platform, making cross-system correlation analysis difficult. The alarm mechanism based on static thresholds has a high false positive rate and cannot adapt to dynamic business changes; the high availability guarantee of the self-built data center requires additional investment in load balancing and disaster recovery equipment, and the operation and maintenance cost increases sharply; the storage of historical data is limited by the capacity of local disks and is forced to be cleaned up regularly, making it difficult to support long-term trend analysis. In addition, the traditional scheme generally has problems such as insufficient encryption strength, such as HTTP clear text transmission, and imperfect audit logs. SUMMARY

[0004] The purpose of the application is to provide a data monitoring management system based on cloud computing, which has the advantages of realizing end-to-end monitoring closed loop through cloud native technology stack, having flexibility, intelligence, security and other characteristics, and being able to quickly adapt to financial, e-commerce, industrial and other scenes, thereby reducing the operation and maintenance complexity.

[0005] The above technical purpose of the application is achieved by the following technical scheme: a data monitoring management system based on cloud computing, comprising a real-time monitoring module, a data collection module, a data processing module, a data transmission module, a cloud storage module, a data analysis module and an alarm module, wherein the real-time monitoring module and the data collection module are bidirectionally electrically connected, the data collection module and the data processing module are bidirectionally electrically connected, the data processing module and the data transmission module are bidirectionally electrically connected, the data transmission module and the cloud storage module are bidirectionally electrically connected, the cloud storage module and the data analysis module are bidirectionally electrically connected, and the data analysis module and the alarm module are bidirectionally electrically connected.

[0006] By means of the above technical solutions: the elastic expansion capability of the cloud platform can dynamically adjust the computing resources, automatically expand in the traffic peak to guarantee the real-time performance, shrink in the valley to reduce the cost, realize the improvement of resource utilization, and the global high-availability deployment and the built-in security mechanism of the cloud service ensure the reliability of the system, which can quickly adapt to financial, e-commerce, industrial and other scenes, and reduce the operation and maintenance complexity. By means of the distributed processing architecture of the cloud native service, the system can efficiently process the million-level data points per second, and support unified collection and standardization of multi-source heterogeneous data, and break the data island problem of the traditional monitoring system. Compared with the traditional alarm mode based on fixed rules, the system can present multi-dimensional analysis results, and finally form a closed-loop management from data collection, intelligent analysis to decision response.

[0007] The application further provides that the data collection module collects data through a heterogeneous data source, and the data collection module realizes on-demand triggering collection by using a cloud function such as AWS Lambda / Azure Functions.

[0008] The application further provides that the data transmission module unifies the data format into a JSON / Protobuf format and buffers the data stream through Kafka / RabbitMQ / AWS Kinesis to avoid loss.

[0009] The application further provides that the cloud storage module adopts distributed storage data, wherein cold data is object storage AWS S3 / Azure Blob Storage, structured data of hot data is a cloud database, and time series data is InfluxDB / TimescaleDB.

[0010] The application further provides that the data analysis module processes historical data by using Spark on Kubernetes / AzureDatabricks.

[0011] The application further provides that the engine of the alarm module is based on Prometheus Alertmanager, and the alarm channel is integrated with mail / Slack / WeChat / Webhook.

[0012] The application further provides that the other side of the data analysis module away from the cloud storage module is further bidirectionally electrically connected with a user interaction interface, and the user interaction interface is constructed into a real-time dashboard by using Grafana / Power BI.

[0013] In summary, the application has the following beneficial effects: 1. The elastic scaling capability of the cloud platform can dynamically adjust the computing resources. In the peak of traffic, it automatically expands to ensure real-time performance. In the trough, it shrinks to reduce costs, realizes the improvement of resource utilization, and the global high-availability deployment of cloud services and the built-in security mechanism ensure the reliability of the system. It can quickly adapt to financial, e-commerce, industrial and other scenarios, and reduce the complexity of operation and maintenance. 2. By means of the distributed processing architecture of cloud native services, the system can efficiently process millions of data points per second and support unified collection and standardization of multi-source heterogeneous data, breaking the data island problem of traditional monitoring systems. Compared with the traditional fixed rule-based alarm mode, this system can present multi-dimensional analysis results, and finally form a closed-loop management from data collection, intelligent analysis to decision response. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is the workflow schematic diagram of the present application. DETAILED DESCRIPTION

[0015] The present application will be further described in detail below in conjunction with the drawings.

[0016] Example 1: Referring to Figure 1 A data monitoring and management system based on cloud computing includes a real-time monitoring module, a data collection module, a data processing module, a data transmission module, a cloud storage module, a data analysis module, and an alarm module. The real-time monitoring module is bidirectionally electrically connected to the data collection module. The data collection module is bidirectionally electrically connected to the data processing module. The data processing module is bidirectionally electrically connected to the data transmission module. The data transmission module is bidirectionally electrically connected to the cloud storage module. The cloud storage module is bidirectionally electrically connected to the data analysis module. The data analysis module is bidirectionally electrically connected to the alarm module. The elastic scaling capability of the cloud platform can dynamically adjust the computing resources. In the peak of traffic, it automatically expands to ensure real-time performance. In the trough, it shrinks to reduce costs, realizes the improvement of resource utilization, and the global high-availability deployment of cloud services and the built-in security mechanism ensure the reliability of the system. It can quickly adapt to financial, e-commerce, industrial and other scenarios, and reduce the complexity of operation and maintenance. By means of the distributed processing architecture of cloud native services, the system can efficiently process millions of data points per second and support unified collection and standardization of multi-source heterogeneous data, breaking the data island problem of traditional monitoring systems. Compared with the traditional fixed rule-based alarm mode, this system can present multi-dimensional analysis results, and finally form a closed-loop management from data collection, intelligent analysis to decision response.

[0017] Referring to Figure 1 The data collection module collects data through heterogeneous data sources. The data collection module uses cloud functions such as AWS Lambda / Azure Functions to realize on-demand triggering of collection.

[0018] Referring to Figure 1, the data format of the data transmission module is unified as JSON / Protobuf format, and the data stream is buffered through Kafka / RabbitMQ / AWS Kinesis to avoid loss.

[0019] With reference to Figure 1 The cloud storage module adopts distributed storage data, wherein cold data is object storage AWS S3 / AzureBlob Storage, structured data of hot data is a cloud database, and time series data is InfluxDB / TimescaleDB.

[0020] With reference to Figure 1 The data analysis module adopts Spark on Kubernetes / Azure Databricks to process historical data.

[0021] With reference to Figure 1 The engine of the alarm module is based on Prometheus Alertmanager, and the alarm channel is integrated through mail / Slack / WeChat / Webhook.

[0022] With reference to Figure 1 The data analysis module is also bidirectionally electrically connected to a user interaction interface on the other side away from the cloud storage module, and the user interaction interface adopts Grafana / Power BI to build a real-time dashboard.

[0023] Use process: through the elastic scaling ability of the cloud platform, such as AWS Auto Scaling or Azure elastic pool, the computing resources can be dynamically adjusted, and the system can be automatically expanded during traffic peak to ensure real-time performance and reduced during low valley to reduce costs, thereby improving resource utilization. Secondly, with the help of cloud native services, such as the distributed processing architecture of AWS Kinesis and Azure Stream Analytics, the system can efficiently process millions of data points per second and support unified collection and standardization of multi-source heterogeneous data databases, IoT devices, logs, etc., breaking the data island problem of traditional monitoring systems. And the global high-availability deployment of cloud services across availability zones / regions for disaster recovery and built-in security mechanisms, such as TLS encryption and IAM permission system, ensure the reliability of the system, while the on-demand payment mode, such as Lambda function billing by execution times, reduces the operation and maintenance cost. In addition, the system can also deeply integrate AI capabilities, such as Azure Anomaly Detector or AWS SageMaker, to realize intelligent threshold detection and root cause analysis. Compared with traditional fixed rule-based alarm methods, the false positive rate is effectively reduced, and multi-dimensional analysis results can be presented in real time through visualization tools such as Grafana, ultimately forming a closed-loop management from data collection, intelligent analysis to decision response. It is suitable for fine-grained monitoring needs in complex scenarios such as e-commerce promotion and industrial Internet of Things, and can quickly adapt to financial, e-commerce, and industrial scenarios to reduce operation and maintenance complexity.

[0024] Note that: parts have a life cycle, and can be replaced during regular maintenance when they do not meet the performance requirements in the later stage. The use effect is poor due to long-term use of parts, which is not a design defect of the application.

[0025] The specific embodiments are only an explanation of the application, and are not a limitation of the application. Those skilled in the art can make modifications to the embodiments without creative contribution after reading the specification, as long as the modifications are within the scope of the claims of the application.

Claims

1. A cloud computing-based data monitoring management system, comprising a real-time monitoring module, a data acquisition module, a data processing module, a data transmission module, a cloud storage module, a data analysis module, and an alarm module, characterized in that: The real-time monitoring module is bidirectionally electrically connected with the data acquisition module, the data acquisition module is bidirectionally electrically connected with the data processing module, the data processing module is bidirectionally electrically connected with the data transmission module, the data transmission module is bidirectionally electrically connected with the cloud storage module, the cloud storage module is bidirectionally electrically connected with the data analysis module, and the data analysis module is bidirectionally electrically connected with the alarm module.

2. The data monitoring management system based on cloud computing according to claim 1, characterized in that: The data acquisition module collects data through heterogeneous data sources, and the data acquisition module realizes on-demand triggering collection by using cloud functions such as AWS Lambda / Azure Functions.

3. The data monitoring management system based on cloud computing according to claim 1, characterized in that: The data format of the data transmission module is unified as JSON / Protobuf format and the data stream is buffered through Kafka / RabbitMQ / AWS Kinesis, so as to avoid loss.

4. The data monitoring management system based on cloud computing according to claim 1, characterized in that: The cloud storage module adopts distributed storage data, wherein cold data is object storage AWS S3 / Azure Blob Storage, structured data of hot data is a cloud database, and time series data is InfluxDB / TimescaleDB.

5. The cloud computing based data monitoring management system as claimed in claim 1, wherein: The data analysis module adopts Spark on Kubernetes / Azure Databricks to process historical data.

6. The cloud computing based data monitoring management system as claimed in claim 1, wherein: The engine of the alarm module is based on Prometheus Alertmanager, and the alarm channel is integrated by mail / Slack / WeChat / Webhook.

7. The cloud computing based data monitoring management system as claimed in claim 1, wherein: The other side of the data analysis module away from the cloud storage module is also bidirectionally electrically connected with a user interaction interface, and the user interaction interface adopts Grafana / Power BI to build a real-time dashboard.

Images