Data real-time monitoring and efficient storage system and method under distributed cloud platform

Abstract

The invention relates to the technical field of cloud computing and big data, in particular to a data real-time monitoring and efficient storage system and method under a distributed cloud platform, and the method comprises the following steps: data collection, data transmission, data storage, data monitoring, data query and analysis, and fault tolerance and recovery. The method has the beneficial effects that real-time monitoring and efficient storage of mass time sequence data are realized through a distributed architecture, a stream processing technology, consistency management and a fault-tolerant mechanism, and the method has the remarkable advantages of high performance, high availability, strong consistency, high expansibility, low cost, flexible analysis, wide applicability and the like; the data processing efficiency and the reliability of the system can be remarkably improved, and the method has important application value.

Description

Technical Field

[0001] This invention relates to the field of cloud computing and big data technology, specifically to a real-time data monitoring and efficient storage system and method under a distributed cloud platform. Background Technology

[0002] With the rapid development of cloud computing, the Internet of Things (IoT), big data, and artificial intelligence technologies, the scale of data is growing exponentially, especially the generation speed and volume of time-series data. Time-series data is widely used in equipment monitoring, sensor data acquisition, financial transaction records, log analysis, and other fields, characterized by its chronological generation, dense data points, and high real-time requirements. Traditional centralized data storage and monitoring systems are gradually revealing performance bottlenecks, insufficient scalability, and poor fault tolerance when faced with massive amounts of time-series data, making it difficult to meet the demands of modern distributed cloud platforms for real-time data monitoring and efficient storage.

[0003] Traditional monitoring systems typically use relational databases (such as MySQL and PostgreSQL) or file systems to store data. These technologies have significant limitations when processing time-series data. Relational databases are designed to handle transactional data, not time-series data, resulting in insufficient performance when faced with high-frequency writes and real-time queries. Centralized storage architectures are difficult to scale horizontally and cannot effectively cope with the rapid growth of data volume and access volume. In addition, traditional storage methods cannot effectively compress highly redundant time-series data, leading to high storage costs and difficulty in supporting real-time data acquisition, monitoring, and analysis.

[0004] To address the shortcomings of traditional technologies, distributed storage technology has emerged. Distributed storage achieves high availability, high scalability, and fault tolerance by distributing data across multiple nodes. Typical distributed storage systems include Hadoop HDFS, Ceph, and Amazon S3. These systems solve the problems of large-scale data storage and high availability through data sharding, replication mechanisms, and consistency protocols. However, distributed storage technology is primarily geared towards general-purpose data storage and lacks specific optimizations for time-series data, making it difficult to meet the demands for efficient storage and retrieval in real-time monitoring scenarios.

[0005] To address the unique characteristics of time-series data, specialized time-series databases (TSDBs) have gradually emerged. By optimizing data models, storage engines, and query languages, TSDBs significantly improve the storage and query efficiency of time-series data. Typical TSDBs include InfluxDB, Prometheus, and TimescaleDB. While these databases offer advantages such as efficient storage, real-time querying, and time-series optimization, they still face challenges in distributed cloud platforms, including limited cluster support, complex data consistency management, and insufficient fault tolerance and recovery capabilities.

[0006] Modern distributed cloud platforms require a technology that can simultaneously meet the needs of real-time monitoring and efficient storage. This technology should possess characteristics such as high scalability, high performance, high availability, strong consistency, and real-time performance. However, existing technologies often focus on a single domain (such as storage or monitoring), lacking an integrated solution that combines real-time monitoring with efficient storage. Furthermore, achieving a balance between high performance and data consistency in a distributed environment remains a challenge; building and maintaining a high-performance, highly available distributed monitoring and storage system typically requires significant technical complexity and cost. Summary of the Invention

[0007] The purpose of this invention is to provide a real-time data monitoring and efficient storage system and method under a distributed cloud platform to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a real-time data monitoring and high-efficiency storage system under a distributed cloud platform, comprising:

[0009] The data acquisition module is used to collect data from multiple data sources in real time.

[0010] The data transmission module is used to transmit the collected data to the distributed cloud platform;

[0011] The data storage module is used for efficient storage of massive amounts of time-series data;

[0012] The data monitoring module is used to monitor data status in real time and provide alarm functions;

[0013] The data query and analysis module is designed to support efficient data querying and complex analysis.

[0014] The fault tolerance and recovery module is used to ensure the high availability and data reliability of the system.

[0015] Preferably, the data acquisition module collects data in real time through data acquisition agents deployed on multiple data source nodes, and supports multiple data source protocols.

[0016] Preferably, the data transmission module employs efficient transmission protocols and data compression technologies to ensure efficient data transmission.

[0017] Preferably, the data storage module adopts a distributed storage system, which achieves efficient storage of massive time-series data through data sharding, replication mechanisms and compression technology.

[0018] Preferably, the data monitoring module is based on stream processing technology to realize real-time data monitoring and provides data filtering, aggregation and anomaly detection functions;

[0019] The data query and analysis module provides an efficient data query interface and integrates data analysis tools to support complex data analysis and visualization;

[0020] The fault tolerance and recovery module monitors the node status in real time through heartbeat detection and failover mechanisms, and automatically migrates data and restores services when a node fails.

[0021] The system supports dynamic node expansion to adapt to rapid growth in data volume and access volume.

[0022] A method for a real-time data monitoring and efficient storage system under a distributed cloud platform includes the following steps:

[0023] Data acquisition is used to collect data from multiple data sources in real time.

[0024] Data transmission is used to transmit the collected data to a distributed cloud platform;

[0025] Data storage, used for efficient storage of massive amounts of time-series data;

[0026] Data monitoring is used to monitor data status in real time and provide alarm functions;

[0027] Data query and analysis, used to support efficient data querying and complex analysis;

[0028] Fault tolerance and recovery are used to ensure high availability and data reliability of the system.

[0029] Preferably, data acquisition is performed in real time by data acquisition agents deployed on multiple data source nodes, and supports multiple data source protocols.

[0030] Preferably, data transmission employs efficient transmission protocols and data compression technologies to ensure efficient data transmission.

[0031] Preferably, data storage adopts a distributed storage system, which achieves efficient storage of massive time-series data through data sharding, replication mechanisms and compression technology.

[0032] Preferably, the data monitoring is based on stream processing technology to achieve real-time data monitoring and provides data filtering, aggregation and anomaly detection functions;

[0033] Data query and analysis provides an efficient data query interface and integrates data analysis tools to support complex data analysis and visualization;

[0034] Fault tolerance and recovery are achieved through heartbeat detection and failover mechanisms, which monitor node status in real time and automatically migrate data and restore services when a node fails.

[0035] The method supports dynamically expanding nodes to adapt to rapid growth in data volume and access volume.

[0036] Compared with the prior art, the beneficial effects of the present invention are:

[0037] This invention proposes a real-time data monitoring and efficient storage system and method under a distributed cloud platform, achieving high-performance data processing. Through stream processing technologies (such as Apache Kafka and Apache Flink), the system can collect, process, and analyze massive amounts of data in real time, meeting the needs of scenarios with high real-time data requirements (such as the Internet of Things and financial transaction monitoring). Simultaneously, by employing a distributed storage system and a time-series database (such as InfluxDB), and through data sharding, columnar storage, and compression technologies, data storage and query efficiency are significantly improved.

[0038] It supports dynamically adding nodes, enabling it to adapt to rapid growth in data volume and access volume. Through elastic resource allocation and load balancing, the system can dynamically adjust resource allocation according to load changes, maximizing resource utilization and avoiding resource waste and performance bottlenecks.

[0039] It supports multi-dimensional data queries based on time, tags, and other dimensions, and provides powerful query languages ​​(such as InfluxQL, PromQL, and Flux) to meet complex data processing and analysis needs. Integration with visualization tools (such as Grafana) further enhances the intuitiveness of data display and monitoring.

[0040] Through distributed architecture, stream processing technology, consistency management, and fault tolerance mechanisms, it achieves real-time monitoring and efficient storage of massive time series data. It has significant advantages such as high performance, high availability, strong consistency, high scalability, low cost, flexible analysis, and wide applicability. It can significantly improve the efficiency of data processing and the reliability of the system, and has important application value. Attached Figure Description

[0041] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the present invention clear and complete, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only some, not all, embodiments of the present invention, and are merely illustrative of the embodiments of the present invention. They are not intended to limit the embodiments of the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] Example 1: This invention provides a technical solution: a real-time data monitoring and high-efficiency storage system under a distributed cloud platform, comprising:

[0044] (1) Data Acquisition Module: Collects data in real time from multiple data sources (such as sensors, logs, and databases). A lightweight data acquisition agent is deployed on each data source node, supporting multiple data source protocols (such as MQ, HTTP, and Kafka), and cleaning, filtering, and formatting the collected data.

[0045] (2) Data Transmission Module: This module efficiently stores massive amounts of time-series data. It transmits the collected data to a distributed cloud platform. High-efficiency transmission protocols (such as MQ) ensure rapid data transmission, and the transmitted data is compressed. TLS / SSL encryption technology is used to ensure data transmission security.

[0046] (3) Data storage module: Efficiently stores massive amounts of time-series data. It uses the distributed database InfluxDB for data storage. Targeting the characteristics of time-series data, it employs columnar storage and compression techniques, with data sharded and stored on different nodes, supporting horizontal scaling.

[0047] (4) Data Monitoring Module: Real-time monitoring of data status and provision of alarm functions. Real-time data monitoring is achieved based on stream processing technologies (such as Apache Kafka and Apache Flink). It provides data filtering, aggregation, and anomaly detection functions, and supports custom monitoring rules. It integrates with alarm systems (such as Prometheus Alertmanager), supporting flexible alarm rules and notification mechanisms. It integrates with visualization tools (such as Grafana), providing an intuitive data display and monitoring interface.

[0048] (5) Data Query and Analysis Module: Supports efficient data querying and complex analysis. Provides efficient data query interfaces (such as SQL, InfluxQL, Flux), supports fast retrieval of real-time and historical data, and supports dedicated query languages ​​for time series data (such as PromQL, Flux), simplifying the processing of time series data.

[0049] (6) Fault tolerance and recovery module, used to ensure high availability and data reliability of the system; through heartbeat detection and failover mechanism, it monitors the node status in real time and automatically migrates data and restores services when a node fails.

[0050] Example 2, based on Example 1, proposes a method for a real-time data monitoring and efficient storage system under a distributed cloud platform, including the following steps:

[0051] (1) Define encryption requirements: Communicate with business departments to determine the specific requirements for data collection, monitoring, storage, and analysis. Design the system architecture, determine the technology selection and data flow for each module. Develop system performance metrics (such as throughput, latency, and availability).

[0052] (2) Environment Preparation and Resource Deployment: Select Inspur Cloud as the cloud platform. Plan the cluster architecture and determine the roles of nodes (such as NameNode and DataNode for HDFS, and seed nodes for Cassandra). Install and configure relevant software to ensure normal communication between nodes, and set data sharding and replication strategies to optimize storage and fault tolerance. For the stream processing platform, deploy a Kafka cluster (including Broker and Zookeeper) and a Flink cluster (JobManager and TaskManager), and configure topics and parallel tasks.

[0053] (3) Data acquisition and transmission implementation: When deploying data acquisition agents at each data source node.

[0054] a. Configure input plugins: Configure input plugins according to the data source type (such as log files, sensors, databases) to ensure that data can be collected correctly.

[0055] b. Configure the output plugin: Send data to the target system Kafka and set the data transmission protocol HTTP and compression algorithm GZIP to reduce network bandwidth usage and improve transmission efficiency.

[0056] c. Simulate high-load scenarios: Test the performance of data acquisition and transmission, and monitor throughput, latency, and resource usage. Based on the test results, optimize parameter configurations (such as batch size, buffer size, and number of concurrent threads) to ensure that the system can still run stably under high load.

[0057] d. Set up logging and monitoring mechanisms: track the status of data collection and transmission in real time, and promptly identify and resolve problems.

[0058] (4) Data Encryption, Data Storage and Monitoring Implementation: Achieve efficient data storage and real-time monitoring.

[0059] a. Configure a distributed storage system: Set data sharding strategies (such as sharding by time or hash value) and replication strategies (such as 3 replicas) according to the data scale and access patterns to ensure high availability and fault tolerance of the data.

[0060] b. Deploy a stream processing platform: Kafka is used for high-throughput data buffering and distribution, while Flink is used for real-time data stream processing and analysis. Define data processing logic in Flink, such as data filtering, aggregation, and anomaly detection, and set window functions (such as scrolling windows and sliding windows) to support real-time computation.

[0061] c. Integrate with the Prometheus alerting system, configure alerting rules (such as threshold alerts and abnormal fluctuation alerts), and set up notification channels (such as email and SMS). Simultaneously, integrate visualization tools (such as Grafana), configure data sources (such as InfluxDB and Prometheus), design monitoring dashboards, and display key metrics (such as CPU utilization, request latency, and error rate). Through real-time data streams and a visual interface, users can intuitively monitor system status and promptly identify and resolve problems.

[0062] (5) Data Query and Analysis Implementation: Deploy data query interfaces (such as InfluxDB, Prometheus) and data analysis tools (such as Elasticsearch). Configure time series query languages ​​(such as InfluxQL, PromQL) to optimize query performance. Develop data analysis scripts and visualization reports to support business needs.

[0063] (6) Fault Tolerance and Recovery Implementation: Deploy heartbeat detection and failover mechanisms to monitor node status. Configure distributed consensus algorithms to ensure data consistency. Test fault tolerance and recovery functions to verify the high availability of the system.

[0064] (7) System Testing and Optimization: Conduct stress tests to verify the system's throughput, latency, and availability. Based on the test results, optimize system parameter configurations (such as data sharding and the number of replicas). Fix system defects to ensure stable system operation.

[0065] (8) Security Assessment and Optimization: Regular security assessments are a key step in evaluating the strength of encryption schemes, and they help to develop...

[0066] This involves identifying and fixing security vulnerabilities in the encryption process. Through simulated attacks, vulnerability scanning, and security testing, organizations can identify weaknesses in their encryption strategies and implementation, enabling them to take steps to strengthen security. Furthermore, as encryption technologies and security threats continue to evolve, optimizing encryption schemes to adapt to new security challenges is also crucial. This may include updating encryption algorithms, improving key management practices, or adopting more advanced security measures.

[0067] (9) Emergency Response and Fault Handling: Establishing an emergency response mechanism is crucial for addressing potential problems during the encryption process.

[0068] Measures. This includes developing detailed emergency response plans and procedures to ensure swift action in the event of emergencies such as encryption algorithm leaks, key loss, or security vulnerability exploitation. By training staff and conducting emergency drills, organizations can improve their ability to respond to security incidents, thereby minimizing the impact of encryption failures on business operations and data security.

[0069] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A real-time data monitoring and high-efficiency storage system under a distributed cloud platform, characterized in that: include: The data acquisition module is used to collect data from multiple data sources in real time. The data transmission module is used to transmit the collected data to the distributed cloud platform; The data storage module is used for efficient storage of massive amounts of time-series data; The data monitoring module is used to monitor data status in real time and provide alarm functions; The data query and analysis module is designed to support efficient data querying and complex analysis. The fault tolerance and recovery module is used to ensure the high availability and data reliability of the system.

2. The real-time data monitoring and efficient storage system under a distributed cloud platform according to claim 1, characterized in that: The data acquisition module collects data in real time through data acquisition agents deployed on multiple data source nodes and supports multiple data source protocols.

3. The real-time data monitoring and efficient storage system under a distributed cloud platform according to claim 2, characterized in that: The data transmission module employs efficient transmission protocols and data compression technology to ensure efficient data transmission.

4. The real-time data monitoring and efficient storage system under a distributed cloud platform according to claim 3, characterized in that: The data storage module adopts a distributed storage system, which achieves efficient storage of massive time-series data through data sharding, replication mechanisms and compression technology.

5. A real-time data monitoring and efficient storage system under a distributed cloud platform according to claim 4, characterized in that: The data monitoring module uses stream processing technology to achieve real-time data monitoring and provides data filtering, aggregation, and anomaly detection functions. The data query and analysis module provides an efficient data query interface and integrates data analysis tools to support complex data analysis and visualization; The fault tolerance and recovery module monitors the node status in real time through heartbeat detection and failover mechanisms, and automatically migrates data and restores services when a node fails. The system supports dynamic node expansion to adapt to rapid growth in data volume and access volume.

6. A method for a real-time data monitoring and efficient storage system under a distributed cloud platform according to claim 5, characterized in that: Includes the following steps: Data acquisition is used to collect data from multiple data sources in real time. Data transmission is used to transmit the collected data to a distributed cloud platform; Data storage, used for efficient storage of massive amounts of time-series data; Data monitoring is used to monitor data status in real time and provide alarm functions; Data query and analysis, used to support efficient data querying and complex analysis; Fault tolerance and recovery are used to ensure high availability and data reliability of the system.

7. A method according to claim 6, characterized in that: Data acquisition is achieved by using data acquisition agents deployed on multiple data source nodes to collect data in real time, and supports multiple data source protocols.

8. A method according to claim 7, characterized in that: Data transmission employs efficient transmission protocols and data compression technologies to ensure efficient data transmission.

9. A method according to claim 8, characterized in that: Data storage employs a distributed storage system, utilizing data sharding, replication mechanisms, and compression technology to achieve efficient storage of massive amounts of time-series data.

10. A method according to claim 9, characterized in that: Data monitoring is based on stream processing technology to achieve real-time data monitoring and provides data filtering, aggregation and anomaly detection functions; Data query and analysis provides an efficient data query interface and integrates data analysis tools to support complex data analysis and visualization; Fault tolerance and recovery are achieved through heartbeat detection and failover mechanisms, which monitor node status in real time and automatically migrate data and restore services when a node fails. The method supports dynamically expanding nodes to adapt to rapid growth in data volume and access volume.

Images