Cloud data acquisition control system

Abstract

The invention provides a cloud data acquisition control system, which belongs to the field of computer software, adopts a Spring Boot micro-service architecture, integrates the technologies of MySQL, InfluxDB, Kafka, Nginx and the like, and constructs a stable and extensible architecture. Core components comprise an acquisition control center, an acquisition device, a cloud monitoring center and a set of complete management and execution mechanism. The cloud data acquisition system not only supports multi-mode data acquisition, but also has a unified management platform, security auditing, plug-in expansibility, usability design, real-time monitoring, efficient data processing and alarm mechanism, so that the comprehensive performance of cloud data acquisition is effectively improved, the operation and maintenance cost is reduced, and the service continuity is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer software, and in particular to a cloud data acquisition and control system. Background Art

[0002] With the rapid development of cloud computing technology, enterprises have an increasing demand for cloud environment monitoring. Cloud environments are dynamic, complex, and distributed, and traditional data collection and monitoring methods are difficult to meet enterprises' needs for cloud environment monitoring.

[0003] Traditional data collection and monitoring methods have the following main problems:

[0004] Data dispersion: There are multiple data sources in the cloud environment, such as servers, storage, network devices, applications, etc. Data is dispersed and stored in different systems, making it difficult to manage them uniformly.

[0005] Collection is difficult: There are various types of data in the cloud environment, including structured data, unstructured data, log data, performance data, etc., which makes collection difficult. Different collection programs need to be developed for different data sources.

[0006] Complex management: Traditional data collection and monitoring systems often lack a unified management platform, making it difficult to centrally manage and monitor, resulting in low management efficiency.

[0007] Insufficient security: Traditional data collection and monitoring systems often lack security audit mechanisms and are vulnerable to security threats, leading to data leaks and system attacks.

[0008] Poor scalability: It is difficult to support the collection of more device types and data sources, and it is difficult to adapt to the development and changes of the cloud environment.

[0009] Poor usability: lack of graphical interface and API interface, difficult to manage and use, and high threshold for use.

[0010] In order to solve the above problems, some companies have tried to develop cloud data acquisition and control systems, but the existing systems still have the following shortcomings:

[0011] Single collection method: Existing systems often only support one or several collection methods, which makes it difficult to meet the diverse data collection needs of enterprises.

[0012] Lack of a unified management platform: Existing systems often lack a unified management platform, making it difficult to centrally manage and monitor, resulting in low management efficiency.

[0013] Insufficient security: Existing systems often lack security audit mechanisms and are vulnerable to security threats, leading to data leaks and system attacks.

[0014] Poor scalability: It is difficult to support the collection of more device types and data sources, and it is difficult to adapt to the development and changes of the cloud environment.

[0015] Poor usability: lack of graphical interface and API interface, difficult to manage and use, and high threshold for use.

[0016] Therefore, it is necessary to develop a new cloud data acquisition and control system that can address the shortcomings of the existing system and meet the needs of enterprises for cloud environment monitoring. Summary of the invention

[0017] In order to solve the above technical problems, the present invention provides a cloud data acquisition control system, which aims to solve the shortcomings of the existing cloud data acquisition control system and achieve the following objectives:

[0018] Support multiple collection methods: The present invention supports multiple collection methods, including SNMP, HTTP API, database, etc., to meet the diverse data collection needs of enterprises.

[0019] Providing a unified management platform: The present invention provides a unified management platform that can centrally manage collection tasks, collectors, plug-ins, etc., thereby improving management efficiency.

[0020] Enhanced data security: The present invention conducts security audits on the collection process and sets up a whitelist to ensure the security of the collection process and prevent malicious attacks and data leaks.

[0021] Improve the scalability of data collection: The present invention adopts a plug-in mechanism, which can easily expand the collection function, support the collection of more device types and data sources, and meet the development and changes of the cloud environment.

[0022] Improve the usability of data collection: The present invention provides a graphical interface and REST API interface to facilitate users to manage and use the collection system and lower the threshold for use.

[0023] Improve the real-time and reliability of data collection: The present invention adopts the collector automatic registration and heartbeat mechanism to monitor the running status of the collector in real time, thereby ensuring the real-time and reliability of data collection.

[0024] Improve the efficiency of data collection: The present invention adopts a temporary storage and cleaning mechanism for collected data to optimize data storage and management and improve the efficiency of data collection.

[0025] Improve the timeliness of alarms: The present invention uses alarm rule configuration and notification mechanism to promptly remind users to handle abnormal situations and improve operation and maintenance efficiency.

[0026] The technical solution of the present invention is:

[0027] A cloud data acquisition and control system, comprising:

[0028] a. The collection control center adopts the Spring Boot microservice architecture, equipped with MySQL database as the management library, InfluxDB as the collection database, as well as Kafka message middleware and Nginx proxy equipment, providing REST API interface to realize task management, collector management, collection plug-in management and data transfer services;

[0029] b. Collector, based on Spring Boot microservice architecture, integrates XXL-JOB task management framework and H2 lightweight Java database, with modules for management, task control, executor, data temporary storage, data reporting adapter, data cleaning and security audit;

[0030] c. Collection plug-in, which supports SNMP collection implemented by Java and HTTP API data collection implemented by Python script, and has execution program and metadata definition;

[0031] d. The cloud monitoring center, based on the Spring Boot microservice architecture, provides modules for device management, data display, and alarm notification, and supports business applications with the task types and plug-in metadata of the cloud acquisition and control center.

[0032] Furthermore,

[0033] The task management module of the acquisition control center is responsible for adding, deleting, modifying, and checking acquisition tasks, enabling and disabling operations, monitoring task status, and obtaining abnormal information.

[0034] The collector management module of the collection control center can automatically discover and manage collectors, notify task start and stop, monitor running status, and manage plug-in downloads.

[0035] The acquisition plug-in management module of the acquisition control center is responsible for the addition, deletion, modification and query operations of the acquisition plug-ins and defines metadata to support various data acquisition requirements.

[0036] The data transfer service module of the acquisition control center is responsible for storing the acquired data in a database, CMDB, etc., and supports several data source docking and processing methods.

[0037] The management module of the collector is responsible for registering the collector to the control center, maintaining the heartbeat connection, responding to control instructions, and reporting health status.

[0038] The security audit module of the collector performs a security audit on the execution content of the collection plug-in, sets a whitelist, and ensures the security of the collection process.

[0039] The data reporting adapter module in the collector reports the data to the collection control center through the REST API, and the data cleaning module regularly cleans up the reported data.

[0040] Collection plug-ins implemented through Java and Python scripts support SNMP and HTTP API data collection respectively, and provide detailed metadata configuration.

[0041] The device management module of the cloud monitoring center configures the device to use collectors and collection components, the data display module provides a data query interface, and the alarm notification module generates alarms according to rules.

[0042] The beneficial effects of the present invention are

[0043] Improve data collection efficiency: By providing a unified management platform and automated task scheduling mechanism, the configuration and management process of collection tasks can be simplified, thereby improving data collection efficiency.

[0044] Enhance data security: Ensure the security of the collection process through security audits and whitelist mechanisms to prevent malicious attacks and data leaks.

[0045] Improve the scalability of data collection: Through the plug-in mechanism, the collection function can be easily expanded to support the collection of more device types and data sources, meeting the diverse data collection needs of enterprises.

[0046] Improve the usability of data collection: By providing a graphical interface and REST API interface, it is easier for users to manage and use the collection system and lower the usage threshold.

[0047] Improve the real-time and reliability of data collection: Through the automatic registration and heartbeat mechanism of the collector, the running status of the collector is monitored in real time to ensure the real-time and reliability of data collection.

[0048] Improve the efficiency of data collection: Optimize data storage and management through data temporary storage and cleaning mechanisms, and improve the efficiency of data collection.

[0049] Improve the timeliness of alarms: Through alarm rule configuration and notification mechanism, users can be reminded to handle abnormal situations in a timely manner to improve operation and maintenance efficiency.

[0050] Reduce operation and maintenance costs: The present invention provides a unified management platform and automation tools, which can reduce operation and maintenance costs and improve operation and maintenance efficiency.

[0051] Improve operation and maintenance efficiency: The present invention can monitor the status of the cloud environment in real time, discover and solve problems in a timely manner, and improve operation and maintenance efficiency.

[0052] Optimize resource allocation: The present invention can analyze the performance data of the cloud environment to help enterprises optimize resource allocation and improve resource utilization.

[0053] Improve business continuity: The present invention can timely discover and resolve cloud environment failures, thereby improving business continuity. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a schematic diagram of the overall structure of the present invention;

[0055] Figure 2 It is a business process diagram of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0057] The present invention comprises four main parts: a collection control center, a collector, a collection plug-in, and a cloud monitoring center.

[0058] 1. Collection control center:

[0059] ① Adopt Spring Boot microservice architecture, use MySQL database as the management library, InfluxDB database as the collection database, Kafka message middleware and Nginx proxy device, etc. to ensure the system is stable, scalable and decoupled.

[0060] ② Design REST API interface to interact with other system components (such as collector, cloud monitoring, etc.).

[0061] ③Includes functional modules such as task management module, collector management module, collection plug-in management module and data transfer service module.

[0062] ④The task management module is responsible for operations such as adding, deleting, modifying, checking, enabling and disabling collection tasks, monitoring whether the collection tasks are working normally, and obtaining task exception information.

[0063] ⑤ The collector management module is responsible for automatically discovering the collector, notifying the collector to enable tasks, stop tasks, etc., monitoring the running status of the collector, and notifying the collector to download plug-ins, etc.

[0064] ⑥The collection plug-in management module is responsible for adding, deleting, modifying and checking collection plug-ins and defining the metadata of the plug-ins.

[0065] ⑦The data transfer service module transfers the collected data to the database, CMDB, etc., and supports multiple data source docking and data processing methods.

[0066] ⑧The security audit module performs security audits on the content, scripts, commands, etc. executed by the collection plug-in and sets a whitelist.

[0067] 2. Collector:

[0068] ① Adopt Spring Boot microservice architecture, integrate XXL-JOB task management framework to implement control center and executor, integrate H2 lightweight Java database, temporarily store some data, and realize task control and scheduling.

[0069] ② Mainly includes functional modules such as management module, task control module, task executor module, data register module, data reporting adapter module, data cleaning module and security audit module.

[0070] ③The management module is responsible for registering with the collection control center, maintaining heartbeat, reporting the health status of its own services, listening to the instructions of the collection control center, and operating according to the instructions.

[0071] ④The task control module calls the corresponding executor to perform the collection operation according to the task parameters and strategies, generates a scheduling log, and records the execution log.

[0072] ⑤The task executor module is responsible for loading the plug-in into the memory, referencing and calling it, providing an interface for updating the plug-in to the management module, and providing a scheduling interface to the task control module.

[0073] ⑥The data temporary storage module temporarily stores the data produced by the actuator in the local H2 database, marks whether the data has been successfully reported, and performs regular cleaning of the data that has been successfully reported.

[0074] ⑦ The data reporting adapter module passes the data generated by the actuator collection task to the data reporting adapter for processing, and reports the data to the interface of the collection control center through the REST API.

[0075] ⑧The data cleaning module regularly cleans up the data reported by the data buffer.

[0076] ⑨ The security audit module intercepts the scripts, programs, and commands again when the final executor executes, records the execution log and reports it, and simultaneously uses the whitelist configured by the collection control center.

[0077] 3. Collection plug-in:

[0078] ① The plug-in implements the SNMP collection plug-in through Java and the HTTP API data collection plug-in through Python script.

[0079] ②The plug-in consists of two parts: the execution program and metadata. The metadata provides the necessary parameters for the execution program and the acquisition control center, including input parameter definition, output parameter definition, the mapping relationship between data objects and database tables, and other configuration items for program execution.

[0080] ③The SNMP data collection plug-in supports the standard SNMP specification interface. The plug-in data needs to include the OID set of the SNMP indicators to be queried and the corresponding indicator attribute code, data type, parsing method, etc.

[0081] ④The HTTP API data collection plug-in involves various and complex API calls. It is recommended to develop it by writing Python scripts. It is also necessary to define the indicator attribute code, data type, parsing method, etc. of the returned data.

[0082] ⑤The database data collection plug-in can use Python scripts to collect data by connecting to the remote database, executing query statements, and then organizing the query results into a JSON array for printing.

[0083] ⑥The Jar package plug-in can use Spring Boot to dynamically load the jar package and configure it to the container to implement more complex business operations, extract business commonalities to write abstract classes as templates, implement abstract business interfaces, and unify the calling methods and return data formats.

[0084] 4. Cloud Monitoring Center:

[0085] ① Based on the Spring Boot microservice architecture, supported by the cloud procurement and control center task type and its plug-in metadata, to express related business applications.

[0086] ②Includes functional modules such as device management, data display, alarm notification and settings.

[0087] ③The device management module manages the devices, configures the collectors and collection components to be used by the devices, configures the execution strategy of the collection tasks, etc.

[0088] ④ The data display module writes a custom general query interface, parses and calculates the data in the collection library, and provides a query interface for data query and rendering.

[0089] ⑤The alarm notification module generates alarm notifications by setting alarm rules and monitoring data changes.

[0090] ⑥Set the module to manage SNMP credential information, etc.

[0091] The above description is only a preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, and is not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A cloud data acquisition and control system, comprising: a. The collection control center adopts the Spring Boot microservice architecture, equipped with MySQL database as the management library, InfluxDB as the collection database, as well as Kafka message middleware and Nginx proxy equipment, providing REST API interface to realize task management, collector management, collection plug-in management and data transfer services; b. Collector, based on Spring Boot microservice architecture, integrates XXL-JOB task management framework and H2 lightweight Java database, with modules for management, task control, executor, data temporary storage, data reporting adapter, data cleaning and security audit; c. Collection plug-in, which supports SNMP collection implemented by Java and HTTP API data collection implemented by Python script, and has execution program and metadata definition; d. The cloud monitoring center, based on the Spring Boot microservice architecture, provides modules for device management, data display, and alarm notification, and supports business applications with the task types and plug-in metadata of the cloud acquisition and control center.

2. The system according to claim 1, characterized in that The task management module of the acquisition control center is responsible for adding, deleting, modifying, and checking acquisition tasks, enabling and disabling operations, monitoring task status, and obtaining abnormal information.

3. The system according to claim 1, characterized in that The collector management module of the collection control center can automatically discover and manage collectors, notify task start and stop, monitor running status, and manage plug-in downloads.

4. The system according to claim 1, characterized in that The acquisition plug-in management module of the acquisition control center is responsible for the addition, deletion, modification and query operations of the acquisition plug-ins and defines metadata to support several data acquisition requirements.

5. The system according to claim 1, characterized in that The data transfer service module of the acquisition control center is responsible for storing the acquired data in the database and CMDB, and supports several data source docking and processing methods.

6. The system according to claim 1, characterized in that The management module of the collector is responsible for registering the collector to the control center, maintaining the heartbeat connection, responding to control instructions, and reporting health status.

7. The system according to claim 1, characterized in that The security audit module of the collector performs a security audit on the execution content of the collection plug-in, sets a whitelist, and ensures the security of the collection process.

8. The system according to claim 1, characterized in that The data reporting adapter module in the collector reports the data to the collection control center through the REST API, and the data cleaning module regularly cleans up the reported data.

9. The system according to claim 1, characterized in that The collection plug-ins implemented through Java and Python scripts support SNMP and HTTP API data collection respectively and provide metadata configuration.

10. The system according to claim 1, characterized in that The device management module of the cloud monitoring center configures the device to use collectors and collection components, the data display module provides a data query interface, and the alarm notification module generates alarms according to rules.

Images