Data acquisition system based on user-defined plug-in

Abstract

The invention provides a user-defined plug-in-based data acquisition system, which belongs to the technical field of data acquisition and comprises an acquisition control center, an acquisition device, a monitoring system and a user-defined plug-in. The method comprises the following steps: firstly, capturing target data through a custom script, and then preprocessing the captured data, including data cleaning, duplicate removal and other operations; and then, storing and managing the processed data by utilizing a database so as to facilitate subsequent data analysis and application. The system also provides a user interface, which is convenient for a user to customize scripts and configure data acquisition parameters. The method has high flexibility and expandability, and can be applied to various complex data acquisition scenes. Through practical application, the system achieves a good effect in the aspects of data acquisition and analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of data acquisition, and in particular to a data acquisition system based on a custom plug-in. Background Art

[0002] In today's information age, data is an important basis for corporate and personal decision-making. However, with the explosive growth of information, how to effectively collect, process and store this data has become a huge challenge. Existing data collection systems often require professional developers to develop specific collection programs based on different data sources and collection tasks, which is not only time-consuming and labor-intensive, but also difficult to adapt to rapidly changing data collection needs. In addition, these systems often only work for specific data sources and collection tasks, lacking flexibility and scalability. Summary of the invention

[0003] In order to solve the above technical problems, the present invention provides a data acquisition system based on a custom plug-in.

[0004] The technical solution of the present invention is:

[0005] A data collection system based on a custom plug-in, comprising:

[0006] The collection control center is used to manage collection tasks, collectors, plug-ins, and data dumps;

[0007] Collectors, deployed in different network environments, are used to perform collection tasks and report the collected data to the collection control center;

[0008] Monitoring system, used to configure and display collected data, and provide alarm and notification functions;

[0009] Custom plug-ins, including jar packages, script plug-ins, and hybrid plug-ins, are used to implement the logic and parsing rules of data collection.

[0010] Furthermore,

[0011] The acquisition control center includes:

[0012] Task management module, used to manage the configuration, execution and monitoring of acquisition tasks;

[0013] Collector management module, used to manage the registration, configuration and monitoring of collectors;

[0014] Plugin management module, used to manage the creation, update, deletion and version control of plugins;

[0015] The data dump module is used to parse, clean and store the collected data.

[0016] The collection control center periodically updates the security certificate, and the collector needs to verify the compliance and validity of the certificate when registering with the collection control center.

[0017] Furthermore,

[0018] The collection control center and collector both have built-in script auditing programs to limit the execution of risky scripts, avoid security risks, and add a plug-in script execution whitelist.

[0019] Furthermore,

[0020] The collector includes:

[0021] The management module is responsible for registering with the acquisition control center, maintaining heartbeats, and reporting the health status of its own services; it is responsible for listening to the control instructions of the acquisition control center, operating according to the instructions, monitoring enable / disable instructions, triggering task updates, obtaining the version of the execution task plug-in and comparing it with the plug-in version in its own library and updating it, and adding, updating, enabling, disabling, and deleting the task control module; generating task execution logs and reporting them as monitoring data; providing query and operation and maintenance operations;

[0022] Task control is responsible for calling the corresponding executor to perform the collection operation according to the task parameters and strategies; generating scheduling logs and recording execution logs; the output result data of the task execution is stored in the data buffer; built-in task module, data cleaning of the buffer, data scheduled batch reporting, log data and service node monitoring data reporting operations;

[0023] The executor module automatically loads the plug-in into the memory, and references and calls it; it provides an interface for updating the plug-in to the management module, and a scheduling interface to the task control module, and writes the returned data into the temporary storage library;

[0024] Data cache module, the cache data contains fields such as partition creation time, reporting time, status, and content fields; implement the data writing interface of the executor, and the implementation method is temporarily set to write to the H2 lightweight database, and the data is retained for a set time; implement the batch acquisition and batch update interface of the data reporting adapter, return the data of the H2 database to the data reporting adapter for reporting, and update the data status in batches according to conditions;

[0025] Data reporting adapter module, data partition batch reporting, batch acquisition of data from the data temporary storage module for reporting, update data status to reported after successful reporting, add intermediate verification status in case of reporting failure to avoid repeated data reporting, issue an alarm after an abnormality occurs, and after troubleshooting, decide whether the data needs to be reported again and record the reporting log; built-in task execution log reporting component, independent of business data;

[0026] Data cleaning module: The task control module executes this component to regularly perform cleaning operations on reported data and logs; built-in UPMS plug-in, which is a Jar type plug-in, built into the system, and is used to enable the data of UPMS service devices; defines the received input parameters: IP, version, credentials, attribute OID comparison table metadata and data parsing rules.

[0027] Among them, the executor performs script security audits, performs security checks on scripts to be executed, avoids executing high-risk commands, and records logs for executed scripts.

[0028] Furthermore,

[0029] The monitoring system comprises:

[0030] Device management module, used to manage the configuration and monitoring of acquisition devices;

[0031] Customized display page, used to display the visual interface of collected data;

[0032] General query interface, used to provide flexible data query functions;

[0033] The alarm and notification module is used to configure alarm rules and notification strategies.

[0034] Furthermore,

[0035] The custom plug-in includes:

[0036] The jar package plug-in implements data collection logic by uploading the jar package;

[0037] Script plug-in, supporting four scripting languages: Python, PowerShell, Shell and Java;

[0038] Hybrid plug-in combines jar package and script plug-in, and defines the execution process through bpmn2.0 specification.

[0039] Furthermore,

[0040] The data transmission link uses real-time key generation to ensure the security of data transmission.

[0041] The data cache module built into the collector supports offline collection and ensures that data is not lost.

[0042] The beneficial effects of the present invention are

[0043] Enhanced data collection flexibility: By customizing plug-ins, users can flexibly customize the logic and scope of data collection according to their own needs without having to rely on fixed data interfaces or modules, thereby improving the adaptability and scalability of the system.

[0044] Improve data collection efficiency: Custom plug-ins can be optimized for specific data sources or formats, reducing data conversion and processing time and improving data collection efficiency.

[0045] Reduce maintenance costs: Due to the plug-in design, system maintenance and upgrades become easier. When the data collection logic needs to be updated, only the corresponding plug-in needs to be replaced or updated, without the need to reconstruct or upgrade the entire system.

[0046] Enhanced data security: Custom plug-ins can collect data locally, reducing direct interactions with external systems and thus reducing the risk of data leakage or attacks.

[0047] Support diverse data sources: The custom plug-in-based design can support various types of data sources, such as databases, files, network services, etc., so that the system can adapt to data collection needs in different scenarios.

[0048] User-friendly configuration interface: By providing an intuitive and easy-to-use configuration interface, users can set and adjust data collection parameters without writing code, lowering the user's usage threshold.

[0049] Portability: The system is usually designed to run on multiple operating systems and hardware platforms, which improves the portability of the system and enables the same data acquisition system to be used in different environments.

[0050] Compatibility: The design of custom plug-ins usually takes into account compatibility with other systems, such as the ability to seamlessly integrate with existing enterprise information systems, data analysis tools, etc.

[0051] Strong customizability: Users can customize the functions of the plug-in according to their own business logic and needs, making the data collection system more in line with actual business needs.

[0052] Support distributed collection: The system supports distributed data collection, which can realize efficient processing of large-scale data and is especially suitable for big data scenarios.

[0053] Easy to monitor and manage: Equipped with comprehensive monitoring functions, it is convenient for administrators to discover and solve problems in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a schematic diagram of the overall structure of the present invention;

[0055] Figure 2 It is a schematic diagram of the working structure of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0057] The present invention first captures the target data through a custom script, and then pre-processes the captured data, including data cleaning, deduplication and other operations. Next, the processed data is stored and managed using a database for subsequent data analysis and application. The present invention also provides a user interface to facilitate users to customize scripts and configure data collection parameters. The present invention is divided into three subsystems and a built-in plug-in, namely, the collection control center system, the collector system, the monitoring system, and the UPMS collection plug-in. The collection control center is responsible for the comprehensive management and monitoring of collection tasks; the collector is used to be deployed in different VPC network environments, execute specific collection tasks through remote or local collection, and return data; the purpose of the monitoring system is to make specific business presentations of the collected data through visual interface configuration, general query interface, alarm strategies and other methods.

[0058] 1. Collection control center

[0059] The collection control center mainly covers four areas: task management, collector management, plug-in management, and data dump. The collection control center is mainly responsible for managing the business data of collection tasks, and controlling, coordinating, monitoring, and operating the collectors to perform collection tasks.

[0060] Task management: used to manage tasks. By selecting the device to be collected, add one or more collection task data. A collection task contains the configuration information of a device including IP, description, access credentials, etc., and is associated with a collection plug-in and a collector to ensure that the network where the collector is located can access this device. A collection task contains multiple execution strategies. Collection tasks can be added, edited, enabled, disabled, and deleted. Click the details of the collection task to see the specific execution record and operation history of the task.

[0061] Collector management: You can edit, enable, disable, query, and perform other operations on the collector. First, install the collector on the device manually. After the collector service is started, you can register to the collection control center by manually adding or automatically registering. The collection control center verifies the data reported by the collector. If the verification is passed, the registration is completed. The collector calls the interface of the collection control center to upload monitoring data and maintain the heartbeat. The collection control center will monitor the health status of the collector based on the reported monitoring and log data. Click to enter the collector details page to check the connection status, communication delay, runtime monitoring, etc. of the collector.

[0062] Plugin management: add, delete, modify, upload, query, and version management of plugins. Add, delete, modify, upload, query, and version management of plugin metadata.

[0063] There are three types of plug-ins: jar type, script type and combination type. The jar type is managed by uploading the jar package. The script has 4 sub-categories: Python, Java, Shell and PowerShell. The script provides an editor and file upload for management.

[0064] Hybrid plug-in: refers to a plug-in that contains both jar type and different script types. It is combined through two xml files. One xml defines the plug-in program function, and the other xml uses bpmn2.0 process specifications to deal with complex collection execution processes, making complex process specifications reusable and improving execution efficiency.

[0065] Plugin definition: Both jar packages and script files need to follow certain development specifications to facilitate docking with subsequent processing logic.

[0066] Metadata: managed through interface input, metadata types: input, output, global variables, parsing rules, custom types, etc. The plug-in defines parameter specifications, data specifications, and database specifications, generates library tables by parsing metadata, performs type conversion on data, and stores data.

[0067] Data dump: First, the reported data will be stored in the database. A plug-in generates an InfluxDB table. By obtaining the plug-in metadata, the data returned by the interface is calculated, verified, cleaned, transformed, and stored.

[0068] Data dump includes a series of data analysis tools, which parse the data through the analysis rules defined by the plug-in and generate the standard return data structure defined by the plug-in. After storage, the data is dumped according to the dump rules defined by the plug-in.

[0069] Data dump provides an interface for customizing dump targets. By implementing different interfaces, it can be easily connected to other systems or applications including CMDB, message queues, databases, etc.

[0070] Security: The collection control center generates an installation instruction and an update instruction for each collector. The instruction contains the public key and updates the public key regularly to ensure that all registered collectors are safe and compliant.

[0071] Script plug-in security audit: perform security audit on the script content of uploads or data, avoid using high-risk commands, and add plug-in whitelist.

[0072] 2. Collector

[0073] The collector consists of six modules, namely management module, task control module, executor (plug-in) module, data cache module, data reporting adapter module, and cache data cleaning module (which should be a special executor with built-in tasks for control).

[0074] Management module: responsible for registering with the acquisition control center, maintaining heartbeat, and reporting the health status of its own services. Responsible for listening to the control instructions of the acquisition control center, operating according to the instructions, monitoring enable / disable instructions, triggering task updates, obtaining the version of the execution task plug-in and comparing and updating it with the plug-in version in its own library, and adding, updating, enabling, disabling, and deleting the task control module.

[0075] Generate task execution logs and report them as monitoring data. Provide a simple single-machine task management interface, operation reports, scheduling logs and other query and operation operations.

[0076] Task control: mainly responsible for calling the corresponding executor to perform the collection operation according to the task parameters and strategies. Generate scheduling logs and record execution logs. The task control module is the core module. The output result data of the execution task is stored in the data register.

[0077] Built-in task modules: data cleaning of temporary storage, scheduled batch reporting of data, reporting of log data and service node monitoring data, etc.

[0078] Executor: Automatically loads the plug-in into the memory, references and calls it. Provides an interface for updating the plug-in to the management module, a scheduling interface to the task control module, and writes the returned data to the temporary storage library.

[0079] Data temporary storage, temporary data contains fields such as partition (differentiated by plug-in), creation time, reporting time, status (not reported, reported, reporting failed), content (data text), etc. Implement the data writing interface of the executor, and the implementation method is temporarily set to write to the H2 lightweight database. The data is retained for a certain period of time to avoid reporting failures. Implement the batch acquisition and batch update interface of the data reporting adapter, return the data of the H2 database to the data reporting adapter for reporting, and update the data status in batches according to conditions.

[0080] Data reporting adapter: data partition batch reporting, batch acquisition of a certain amount of data from the data temporary storage module for reporting, update the data status to reported after successful reporting, pay attention to the situation of reporting failure to add intermediate verification status to avoid repeated data reporting, issue an alarm after an abnormality occurs, manually troubleshoot the problem, and then decide whether the data needs to be reported again, record the reporting log, and ensure the final consistency of the data. Built-in task execution log reporting component, independent of business data.

[0081] Data cleaning: The task control module executes this component to regularly clean up the reported data and logs.

[0082] Built-in plug-in: UPMS plug-in, which is a Jar type plug-in, built into the system, and used to enable the data of UPMS service devices. It defines the received input parameters: IP, version, credentials, attribute OID comparison table metadata and data parsing rules.

[0083] Data transmission encryption: The data reported by the collector is encrypted using the public key. The collection control center will decrypt the data according to the private key of the corresponding collector to ensure the security of real-time data transmission and avoid data tampering.

[0084] The executor performs script security audits, performs security checks on scripts to be executed, avoids executing high-risk commands, and logs the executed scripts.

[0085] 3. Cloud Monitoring

[0086] As the specific business embodiment of the data collection system, cloud monitoring has built-in device management module, custom page, general query interface, alarm and notification module.

[0087] Device management: Manage the device information to be collected, including device classification, name, description, IP address, etc. By adding collection tasks to the device, selecting the collector, collection plug-in, and customizing the collection strategy, define which device should be used for scheduled collection and which data should be collected. Operations such as device query list, add, batch move, batch enable, batch disable, and batch delete. Error log function, obtain error log data and display it.

[0088] Management of equipment categories: add, delete, modify, and query.

[0089] Customized display page: The customized page consists of information panel, list page, monitoring chart, and relies on the metadata of the collection plug-in and the public query interface to fill and display data. The customized page can be displayed as a new function item or device details page. Through configuration, a rich combination display page can be generated.

[0090] General query interface: general paging interface, aggregate grouping query, data condition query, etc., to meet most analysis and display functions.

[0091] Alarm information: manage the basic information of the alarm, including name, description, etc.

[0092] Alarm rules analyze the plugin metadata, perform rule matching and logical operations based on the output attribute field type, including greater than, less than, including, equal to, and, or, etc., and analyze the data matched by the conditions and the time when the data was generated. Define the level of alarm generated by the condition.

[0093] Alarm triggering strategy: multiple consecutive or cumulative triggering.

[0094] Alarm recovery strategy: If there are multiple consecutive and cumulative failures to match, recovery is triggered.

[0095] Notification method: in-site message, mobile phone text message, email. Notification strategy, interval length, maximum number of notifications, etc.

[0096] Notification target: Notification by user group or specified user.

[0097] The system allows users to customize scripts according to specific needs, so that information from various data sources can be easily collected, and the data collection, parsing and storage processes can be automatically performed to reduce the complexity and error rate of manual operations and improve the efficiency and accuracy of data collection. In addition, the system aims to reduce the development and maintenance costs of the data acquisition system so that it can adapt to the ever-changing data collection needs, thereby having a wide range of application value in various data-driven application scenarios. The present invention realizes flexible collection of multiple data sources through user-defined scripts. Users can write custom scripts according to their own needs, and the scripts include the logic of data collection, the method of data parsing and the method of data storage. The data acquisition system automatically generates the corresponding collection program based on the script provided by the user, and deploys the program to the data acquisition server. The data acquisition system automatically executes the collection program and collects the required data according to the collection frequency and collection range set by the user. After the collected data is parsed and processed, it is stored in the specified data storage system

[0098] The above description is only a preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, and is not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A data collection system based on a custom plug-in, characterized in that: include: The collection control center is used to manage collection tasks, collectors, plug-ins, and data dumps; Collectors, deployed in different network environments, are used to perform collection tasks and report the collected data to the collection control center; Monitoring system, used to configure and display collected data, and provide alarm and notification functions; Custom plug-ins, including jar packages, script plug-ins, and hybrid plug-ins, are used to implement the logic and parsing rules of data collection.

2. The system according to claim 1, characterized in that The acquisition control center includes: Task management module, used to manage the configuration, execution and monitoring of acquisition tasks; Collector management module, used to manage the registration, configuration and monitoring of collectors; Plugin management module, used to manage the creation, update, deletion and version control of plugins; The data dump module is used to parse, clean and store the collected data.

3. The system according to claim 1 or 2, characterized in that: The collection control center periodically updates the security certificate, and the collector needs to verify the compliance and validity of the certificate when registering with the collection control center.

4. The system according to claim 1 or 2, characterized in that: The collection control center and collector both have built-in script auditing programs to limit the execution of risky scripts, avoid security risks, and add a plug-in script execution whitelist.

5. The system according to claim 1, characterized in that The collector includes: The management module is responsible for registering with the acquisition control center, maintaining heartbeats, and reporting the health status of its own services; it is responsible for listening to the control instructions of the acquisition control center, operating according to the instructions, monitoring enable / disable instructions, triggering task updates, obtaining the version of the execution task plug-in and comparing it with the plug-in version in its own library and updating it, and adding, updating, enabling, disabling, and deleting the task control module; generating task execution logs and reporting them as monitoring data; and providing query and operation and maintenance operations; Task control is responsible for calling the corresponding executor to perform the collection operation according to the task parameters and strategies; generating scheduling logs and recording execution logs; the output result data of the task execution is stored in the data buffer; built-in task module, data cleaning of the buffer, data scheduled batch reporting, log data and service node monitoring data reporting operations; The executor module automatically loads the plug-in into the memory, and references and calls it; it provides an interface for updating the plug-in to the management module, and a scheduling interface to the task control module, and writes the returned data into the temporary storage library; Data cache module, the cache data contains fields such as partition creation time, reporting time, status, and content fields; implement the data writing interface of the executor, and the implementation method is temporarily set to write to the H2 lightweight database, and the data is retained for a set time; implement the batch acquisition and batch update interface of the data reporting adapter, return the data of the H2 database to the data reporting adapter for reporting, and update the data status in batches according to conditions; Data reporting adapter module, data partition batch reporting, batch acquisition of data from the data temporary storage module for reporting, update data status to reported after successful reporting, add intermediate verification status in case of reporting failure to avoid repeated data reporting, issue an alarm after an abnormality occurs, and after troubleshooting, decide whether the data needs to be reported again and record the reporting log; built-in task execution log reporting component, independent of business data; Data cleaning module: The task control module executes this component to regularly perform cleaning operations on reported data and logs; built-in UPMS plug-in, which is a Jar type plug-in, built into the system, and is used to enable the data of UPMS service devices; defines the received input parameters: IP, version, credentials, attribute OID comparison table metadata and data parsing rules.

6. The system according to claim 5, characterized in that The executor performs script security audits, performs security checks on scripts to be executed, avoids executing high-risk commands, and logs the executed scripts.

7. The system according to claim 1, characterized in that The monitoring system comprises: Device management module, used to manage the configuration and monitoring of acquisition devices; Customized display page, used to display the visual interface of collected data; General query interface, used to provide flexible data query functions; The alarm and notification module is used to configure alarm rules and notification strategies.

8. The system according to claim 1, characterized in that The custom plug-in includes: The jar package plug-in implements data collection logic by uploading the jar package; Script plug-in, supporting four scripting languages: Python, PowerShell, Shell and Java; Hybrid plug-in combines jar package and script plug-in, and defines the execution process through bpmn2.0 specification.

9. The system according to claim 1, characterized in that The data transmission link uses real-time key generation to ensure the security of data transmission.

10. The system according to claim 1, characterized in that The data cache module built into the collector supports offline collection and ensures that data is not lost.

Images