Prometheus data collection method, device and equipment based on plug-in fusion collector and medium

Abstract

The invention discloses a Prometheus data collection method, device and equipment based on a plug-in fusion collector and a medium, and the method comprises the steps: receiving an index pulling request of a Prometheus Server based on a fusion collector Agent, obtaining index data from a cache, and returning the index data through a set Prometheus interface; wherein the fusion collector Agent is configured as follows: reading and analyzing a configuration file based on a configuration monitor; dynamically discovering and loading a collector plug-in based on an SPI engine; wherein the collector plug-in is configured to run in an independent Jar packet; the collector plug-ins are matched according to the content of the configuration file, and the matched collector plug-ins are registered to a collector management center for unified management; based on an acquisition task management center, creating and scheduling a corresponding data acquisition task for the acquisition instance under the registered collector plug-in; and the collector plug-in executes the data collection task and caches the collected index data to the index data management center. According to the method and the device, the multi-process resource consumption in the Prometheus data acquisition process is reduced, and unified management and control are realized.

Description

Technical Field

[0001] This invention relates to the field of computer system monitoring technology, and in particular to a Prometheus data acquisition method, apparatus, device and medium based on a plug-in fusion data acquisition device. Background Technology

[0002] With the rapid development of cloud-native technologies and microservice architectures, system monitoring and data collection have become crucial means to ensure stable service operation. Among these, Prometheus, a widely used open-source monitoring system, has gradually become one of the mainstream monitoring solutions due to its efficient data model, flexible query language, and deep integration with ecosystems such as Kubernetes.

[0003] In the Prometheus system, data collection relies on components called exporters. An exporter is a standalone process that extracts monitoring metrics from a target system or application and exposes them via an interface in a Prometheus-supported format, which is then periodically pulled by the Prometheus Server. A widely used approach is to use a single exporter process to pull metrics from a single data source and expose them as an HTTP interface. This exporter is typically a small application with an embedded HTTP service, and its configuration file defines parameters such as data source connection information, collection cycle, and exposure path.

[0004] As application complexity increases, various implementation methods have emerged around the data acquisition system of exporters, mainly including: 1. Official and community standard exporters: such as Node Exporter, MySQL Exporter, BlackboxExporter, etc. These exporters are usually independent binary programs, maintained by the community or the official team, and have good versatility and stability. 2. Custom exporter development: Developers can develop custom exporters according to business needs to achieve support for data collection from specific systems or private protocols; 3. Sidecar container mode: In a Kubernetes environment, the exporter is often used as a sidecar container and deployed together with the main service to achieve local data collection and reporting. 4. Monitoring probes in the service mesh: such as the Prometheus metrics interface integrated with Envoy, which also provides indirect exporter functionality; Metrics are forwarded through intermediate collection platforms: for example, metrics are collected through middleware such as Telegraf and OpenTelemetry and then converted into Prometheus format for retrieval.

[0005] These solutions collectively build Prometheus's powerful ecosystem and data collection capabilities. However, with a large number of exporters and increasingly heterogeneous data collection tasks, these data collection methods, centered around independent exporters, require each exporter to run independently, consuming separate system resources such as CPU and memory. Especially in large microservice systems, dozens or even hundreds of exporters often need to be deployed, leading to wasted server resources and increased operating costs. Moreover, because exporters are distributed and independent entities, operations and maintenance personnel need to configure, monitor, and maintain each exporter individually. Different exporters may have different configuration methods and monitoring metrics, greatly increasing the complexity and manpower costs of operations and maintenance, making unified management difficult. Summary of the Invention

[0006] This invention provides a Prometheus data acquisition method, apparatus, device, and medium based on a plug-in fusion collector, which solves the problems of multi-process resource consumption and unified management during Prometheus data acquisition.

[0007] To achieve the above objectives, this application adopts the following technical solution: Firstly, a Prometheus data acquisition method based on a plug-in fusion collector is provided, including: The fusion collector agent receives metrics retrieval requests from the Prometheus server. In response to the metric retrieval request, metric data is retrieved from the cache of the fusion collector Agent and returned via the Prometheus interface. The fusion collector Agent is configured as follows: Based on the configuration listener, read and parse the configuration file; The SPI engine dynamically discovers and loads one or more data acquisition plugins into the container to execute preset monitoring data acquisition logic; wherein, the data acquisition plugins are configured to run as independent JAR packages; The collector plugins are matched according to the parsed configuration file content, and the matched collector plugins are registered to the collector management center for unified management; the configuration file content includes the collector plugins to be enabled and their corresponding collection instance configurations; Based on the data acquisition task management center, corresponding data acquisition tasks are created and scheduled for the data acquisition instances under the registered data acquisition plugins; In response to task scheduling, the collector plugin executes a data collection task on the target data source corresponding to the collection instance, and caches the collected indicator data in the indicator data management center.

[0008] Secondly, a Prometheus data acquisition device based on a plug-in fusion collector is provided, comprising: The request receiving module is used to receive metric retrieval requests from the Prometheus Server based on the fusion collector agent; The indicator data acquisition module is used to respond to the indicator retrieval request, retrieve indicator data from the cache of the fusion collector Agent, and return it through the Prometheus interface. The fusion collector Agent is configured as follows: Based on the configuration listener, read and parse the configuration file; The SPI engine dynamically discovers and loads one or more data acquisition plugins into the container to execute preset monitoring data acquisition logic; wherein, the data acquisition plugins are configured to run as independent JAR packages; The collector plugins are matched according to the parsed configuration file content, and the matched collector plugins are registered to the collector management center for unified management; the configuration file content includes the collector plugins to be enabled and their corresponding collection instance configurations; Based on the data acquisition task management center, corresponding data acquisition tasks are created and scheduled for the data acquisition instances under the registered data acquisition plugins; In response to task scheduling, the collector plugin executes a data collection task on the target data source corresponding to the collection instance, and caches the collected indicator data in the indicator data management center.

[0009] Thirdly, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the Prometheus data acquisition method based on a plug-in fusion acquisition device as described in the first aspect.

[0010] Fourthly, a readable storage medium is provided, on which a program or instructions are stored, which, when executed by a processor, implement the steps of the Prometheus data acquisition method based on a plug-in fusion collector as described in the first aspect. Attached Figure Description

[0011] Figure 1A schematic diagram of a Prometheus data acquisition system architecture based on a plug-in fusion collector is provided for an embodiment of this application; Figure 2 This application provides a schematic diagram of the runtime logic of a plug-in runtime layer. Figure 3 A schematic flowchart illustrating a Prometheus data acquisition method based on a plug-in fusion collector, provided for an embodiment of this application; Figure 4 A schematic diagram illustrating the process of acquiring and outputting indicator data as provided in an embodiment of this application; Figure 5 A schematic flowchart illustrating the process of loading a Collector using an SPI engine, provided for an embodiment of this application; Figure 6 A schematic flowchart illustrating the startup process of a data collector manager provided in an embodiment of this application; Figure 7 A schematic flowchart illustrating the hot-loading process of a Collector manager provided in an embodiment of this application; Figure 8 A schematic diagram of the structure of a Prometheus data acquisition device based on a plug-in fusion acquisition device provided in this application embodiment; Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the technical solutions in the embodiments of this application are clearly described. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art are within the scope of protection of this application.

[0013] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0014] The steps described in this application and the flowcharts in the accompanying drawings are not necessarily strictly executed according to the step numbers; the execution order of the steps can be changed. Furthermore, certain steps can be omitted, multiple steps can be combined into one step, and / or one step can be broken down into multiple steps.

[0015] In the Prometheus ecosystem, a common way to collect data from applications or devices is by implementing an exporter. Prometheus periodically pulls data through the corresponding exporter. However, when faced with large-scale, heterogeneous, and dynamically changing data collection tasks, there are insurmountable bottlenecks in terms of resource efficiency, unified scheduling, and ease of operation and maintenance.

[0016] Based on this, this specification provides a Prometheus data acquisition system architecture based on a plug-in fusion acquisition device, and also relates to a Prometheus data acquisition method based on a plug-in fusion acquisition device, a Prometheus data acquisition device based on a plug-in fusion acquisition device, a computer device, and a computer-readable storage medium. The following describes each of these in detail with reference to the accompanying drawings and preferred embodiments.

[0017] First, the architecture of the Prometheus data acquisition system based on a plug-in fusion collector, as described in this application embodiment, will be explained below.

[0018] Please see Figure 1-2 The system includes an Agent core layer and a plugin runtime layer. The Agent runs as a unified process, integrating and managing multiple data collection tasks.

[0019] The Agent core layer includes the following functional modules: The HTTP Server module provides an HTTP service interface to the outside world, receives pull requests from the Prometheus Server, and returns the collected metric data.

[0020] The configuration listener is responsible for reading and parsing the configuration file, and loading the corresponding collector plugins based on the configuration content. Simultaneously, it monitors changes to the configuration file and synchronizes the changes to the collector management center, driving hot updates and start / stop operations for the collectors. Main functions include: 1. Load and parse the configuration files under the configuration path; 2. Maintain the currently valid configuration for use by other modules; 3. Detect configuration file changes via scheduled tasks; 4. Analyze the configuration changes and synchronize them with the data collector management center.

[0021] The collector management center is responsible for loading, initializing, controlling, and managing the lifecycle of collector plugins. Its functions include: 1. Obtain available collector plugins from the SPI engine; 2. Complete the registration, initialization, and start / stop of the plugin according to the configuration; 3. Respond to changes in the configuration monitoring center and dynamically adjust the collector status; 4. Unify the management of all registered data collectors.

[0022] The data acquisition task management center centrally manages data acquisition tasks, coordinates the task scheduling and execution of each data acquisition device, and provides interfaces for starting and stopping data acquisition tasks.

[0023] The SPI engine enables service discovery and dynamic loading of plugins, and its main functions include: 1. Scan the preset plugin directory and identify JAR files that meet the specifications; 2. Extract the plugin classes that implement the collector interface; 3. Provide plugin examples for the collector management center to use; 4. Implement dynamic loading, unloading, and reloading of plugins.

[0024] The metrics data management center receives and manages all metrics data collected by the collectors, and decides which data to report to the Prometheus Server.

[0025] like Figure 2 As shown, the plugin runtime layer is responsible for executing the actual data collection logic. All plugins are loaded as independent JAR files and run in an isolated environment managed by the SPI engine, possessing independent dependencies and runtime contexts. Each data collector plugin must perform the following core functions: 1. Complete its own initialization based on system configuration; 2. Multiple data collection instances specified in the management configuration; 3. Create a data collection task and register it with the data collection task management center for unified scheduling; 4. The collected results are then handed over to the indicator data management center for management.

[0026] Therefore, the specific objective of this application embodiment is to implement a unified collector as an Agent, integrating all exporters into a single process to reduce resource consumption and improve the convenience of operation and management. Simultaneously, each collector is encapsulated as a Collector through Java's SPI mechanism and integrated into the Agent as a plugin. This way, when a Collector undergoes a hot update, the plugin is also hot-updated via the SPI mechanism, eliminating the need to restart the entire Agent and improving system stability and availability.

[0027] Please see Figure 3-4 This application provides a Prometheus data acquisition method based on a plug-in fusion collector, including: Step S1: Receive the metrics pull request from the Prometheus Server based on the fusion collector agent; Step S2: In response to the metric retrieval request, retrieve metric data from the cache of the fusion collector Agent and return it through the Prometheus interface. The fusion collector Agent is configured as follows: Based on the configuration listener, read and parse the configuration file; The SPI engine dynamically discovers and loads one or more data acquisition plugins into the container to execute preset monitoring data acquisition logic; wherein, the data acquisition plugins are configured to run as independent JAR packages; The collector plugins are matched according to the parsed configuration file content, and the matched collector plugins are registered to the collector management center for unified management; the configuration file content includes the collector plugins to be enabled and their corresponding collection instance configurations; Based on the data acquisition task management center, corresponding data acquisition tasks are created and scheduled for the data acquisition instances under the registered data acquisition plugins; In response to task scheduling, the collector plugin executes a data collection task on the target data source corresponding to the collection instance, and caches the collected indicator data in the indicator data management center.

[0028] Based on the above embodiments, this application proposes a Prometheus fusion data acquisition device design based on the SPI mechanism. Addressing the problems of high resource consumption, complex operation and maintenance, and lack of flexibility in existing Prometheus data acquisition methods, it constructs a modular, pluggable, and dynamically expandable data acquisition system. By introducing the Java SPI (Service Provider Interface) mechanism, automatic discovery, dynamic loading, and runtime isolation of data collector plugins are achieved. Each data collector plugin is independently encapsulated and decoupled from others, supporting hot-swapping and hot updates, greatly improving the system's scalability and flexibility.

[0029] The original multiple independently running exporters are integrated into a single Agent process for management. Through internal task scheduling and plugin control mechanisms, different collectors are scheduled in a unified manner, reducing resource consumption and simplifying deployment and maintenance.

[0030] The system can achieve centralized management, dynamic configuration loading, and automatic updates of data collectors through a configured monitoring center. It supports periodic detection of configuration file changes, accurately locates changed items, and drives plugin-level overloads to ensure that the runtime system state always remains consistent with the configuration.

[0031] It supports fine-grained management of data collection tasks at the collector instance level. Each instance can be independently configured with collection intervals and start / stop status, improving the controllability of task scheduling and the overall operating efficiency of the system.

[0032] By employing a caching mechanism in the indicator data management center, the data collection frequency is decoupled from the Prometheus fetch frequency. Timestamps are used to identify data validity, effectively avoiding data loss or overwriting issues caused by frequency mismatches and improving system robustness.

[0033] Therefore, the embodiments of this application achieve the following beneficial effects: 1. Significantly reduce resource consumption Compared to the traditional approach where each exporter runs independently, this invention integrates multiple collectors into a unified process, significantly reducing the number of processes and system resource (CPU, memory) overhead, making it suitable for centralized monitoring scenarios in large-scale microservice systems.

[0034] 2. Operation and maintenance costs are significantly reduced. It supports unified plugin management, centralized configuration control, and hot updates, which simplifies the deployment, version control, and troubleshooting process of exporter and significantly improves system maintainability.

[0035] 3. Easy to integrate and migrate Plugin development only requires implementing a unified interface specification; there's no need to worry about the main framework logic. Existing exporter functionality can be quickly migrated and integrated by rewriting it as a plugin, and the system boasts excellent compatibility and adaptability.

[0036] 4. Seamlessly compatible with the Prometheus ecosystem All data collectors output data in a format fully compatible with the Prometheus pull protocol, requiring no additional modifications to the Prometheus Server, enabling seamless replacement and upgrades without any server-side concerns.

[0037] 5. The system has strong stability and scalability.

[0038] Furthermore, based on the SPI engine, one or more data acquisition plugins for executing preset monitoring data acquisition logic are dynamically discovered and loaded into the container, specifically including: The SPI engine is based on Java's SPI mechanism and scans the plugin directory under a preset path at runtime; The discovered data collector plugins are instantiated and provided to the data collector management center for invocation. One data collector plugin can correspond to multiple data collection instances, and each instance corresponds to an independent data collection task.

[0039] like Figure 5 As shown, in the specific implementation of this step, the path of the JAR files to be scanned by the SPI engine is first set, which is the preset directory where the collector plugins are stored. The SPI engine reads and loads all the JAR files under this path and checks whether there is an implementation class of the collector interface. If it cannot be scanned, the process ends directly. If it is scanned, it is determined whether the implementation class of the collector interface has corresponding configuration information. If the corresponding configuration is not found, the process ends. If the corresponding configuration is found, the plugin loading process is entered: the implementation class of the collector interface that matches the configuration is instantiated to obtain an executable collector plugin object. The instantiated collector plugin object is stored in a container for subsequent calls by the collector management center.

[0040] The call process of the data collector management center is as follows: Figure 6 As shown: First, the collector-related configuration information is read and parsed from the external configuration file. The Collector implementation class stored in the SPI engine is retrieved. Based on the parsed configuration information, the configuration corresponding to a collector plugin to be enabled is selected and matched with the loaded Collector implementation class. If the match fails, the next configuration is matched. If the match is successful, the corresponding collector plugin is initialized according to the matched configuration, and then the initialized collector plugin is started to execute the monitoring data collection logic. The started collector plugin is then registered to the collector management center. It can also be further checked whether there are any unmatched collector configurations. If there are unmatched configurations, the matching steps above are returned, and the process of matching → initialization → startup → registration is repeated. If there are no configurations to be matched, the process ends.

[0041] Furthermore, the aforementioned data acquisition task management center creates and schedules corresponding data acquisition tasks for the acquisition instances under the registered acquisition plugin. Specifically, the data acquisition task management center maintains a unified task scheduler; each acquisition instance registers a scheduled task with the data acquisition task management center during initialization according to its own acquisition cycle configuration; the data acquisition task management center concurrently schedules and executes all registered acquisition tasks through the unified task scheduler.

[0042] Furthermore, the step of caching the collected indicator data to the indicator data management center specifically includes: The indicator data management center pre-sets a global indicator data cache area; After each data collection task is completed, the latest collected indicator data, along with the timestamp, will be updated to the indicator data cache, overwriting the old data from the same data source. When the PrometheusServer fetches data, it retrieves the latest values ​​of all metrics from the metric data cache and returns them. This decouples the data collection frequency from the PrometheusServer fetch frequency, avoiding data overwriting or loss caused by a mismatch between collection and fetch frequencies.

[0043] In some possible implementations, a dynamic update process is also included. The fusion collector agent is further configured as follows: A scheduled task based on a configuration listener monitors changes to configuration files. In response to the change event, the collector management center performs a dynamic update operation, including: Register the new collector plugin based on the added configuration items; Update the configuration of the collection instances of the registered collector plugins according to the modified configuration items; Stop and unregister the corresponding collector plugin and / or collector instance based on the deleted configuration item.

[0044] The above dynamic update operation is completed without restarting the fusion collector agent process, achieving hot updating and hot-swapping of collector plugins. For example... Figure 7 As shown, in the specific implementation of this step, external modifications to the collector's configuration, such as adding, modifying, or deleting configuration items, are made. The configuration listener's scheduled task detects the change event in the configuration file, compares the old and new versions of the configuration, identifies the specific changes, and extracts the configuration change items containing configuration change information that need to be synchronized to the collector management center. The collector management center receives the configuration change items and finds the registered collector plugin corresponding to the change item in the collector management center's container. First, it determines whether it is a type-level change. If so, it involves replacing the collector plugin itself, requiring the corresponding plugin to be stopped and deregistered, and a new collector plugin to be registered. If it is not a type-level change, such as a modification to the collection cycle or target address, the configuration modification is passed to the corresponding collector plugin. The collector plugin receives the change information and updates its own configuration internally.

[0045] It is worth noting that, compared with the prior art, this application: (i) Related technologies utilize centralized Exporter middleware, which can forward metric data from multiple child nodes or services by deploying an intermediate Exporter aggregation proxy. This aggregation proxy uniformly exposes the Prometheus pull interface, thereby reducing the number of Prometheus access targets. Its limitations relative to this application are: 1. Aggregator proxies themselves usually do not have plug-in capabilities, and their collection logic is fixed, making it difficult to adapt to diverse collection needs; 2. It does not support plugin-level hot updates and dynamic loading, resulting in insufficient system scalability and ease of operation and maintenance; 3. The exporter component still needs to be deployed on each node or service side, and the overall architecture has not deviated from the multi-exporter model.

[0046] (ii) Unified deployment of the collector via a sidecar container approach: In a containerized environment, the exporter can be deployed alongside business services as a sidecar container, achieving partial centralized control through unified container images and configuration management. Its limitations compared to this application are: 1. Each Sidecar still needs to run a separate process, resulting in high system resource consumption and making it difficult to adapt to resource-sensitive scenarios; 2. Unable to resolve the issue of unified scheduling and management across exporter tasks; 3. Plugin updates require container rebuilding, making it difficult to support runtime dynamic updates.

[0047] (iii) Plug-in acquisition is implemented through general acquisition frameworks (such as Telegraf or OpenTelemetryCollector). These general acquisition frameworks support multiple data source inputs and extend acquisition capabilities through plug-in mechanisms, theoretically integrating the data acquisition functions required by Prometheus. Their limitations relative to this application are: 1. Tools such as Telegraf are not natively designed for the Prometheus pull mechanism and need to be converted into an exposed metrics interface; 2. The plugin system and language ecosystem differ significantly from the mainstream Prometheus environment, resulting in high migration costs; 3. It is difficult to integrate seamlessly with the existing Prometheusexporter community ecosystem, and compatibility issues exist.

[0048] Although several technical approaches exist that can achieve some of the objectives to a certain extent, existing alternatives are difficult to match the comprehensive advantages of this invention in key aspects such as flexibility, scalability, compatibility, and system resource efficiency.

[0049] See Figure 8 Corresponding to the above-described Prometheus data acquisition method embodiment based on a plug-in fusion collector, this application embodiment provides a Prometheus data acquisition device based on a plug-in fusion collector, comprising: The request receiving module 1001 is used to receive the metric retrieval request from the Prometheus Server based on the fusion collector Agent; The indicator data acquisition module 1002 is used to respond to the indicator retrieval request, retrieve indicator data from the cache of the fusion collector Agent, and return it through the Prometheus interface. The fusion collector Agent is configured as follows: Based on the configuration listener, read and parse the configuration file; The SPI engine dynamically discovers and loads one or more data acquisition plugins into the container to execute preset monitoring data acquisition logic; wherein, the data acquisition plugins are configured to run as independent JAR packages; The collector plugins are matched according to the parsed configuration file content, and the matched collector plugins are registered to the collector management center for unified management; the configuration file content includes the collector plugins to be enabled and their corresponding collection instance configurations; Based on the data acquisition task management center, corresponding data acquisition tasks are created and scheduled for the data acquisition instances under the registered data acquisition plugins; In response to task scheduling, the collector plugin executes a data collection task on the target data source corresponding to the collection instance, and caches the collected indicator data in the indicator data management center.

[0050] Furthermore, the method of dynamically discovering and loading one or more data acquisition plugins for executing preset monitoring data acquisition logic into the container based on the SPI engine includes: The SPI engine is based on Java's SPI mechanism and scans the plugin directory under a preset path at runtime; The discovered collector plugins are instantiated and made available for use by the collector management center. One collector plugin can correspond to multiple collection instances, and each collection instance corresponds to an independent collection task.

[0051] Furthermore, the step of caching the collected indicator data to the indicator data management center includes: The indicator data management center pre-sets a global indicator data cache area; After each data collection task is completed, the latest collected indicator data, along with the timestamp, will be updated to the indicator data cache, overwriting the old data from the same data source. When the PrometheusServer pulls data, it retrieves the latest values ​​of all metric data from the metric data cache and returns them.

[0052] Furthermore, the fusion collector agent is also configured as follows: A scheduled task based on a configuration listener monitors changes to configuration files. In response to the change event, the collector management center performs a dynamic update operation, including: Register the new collector plugin based on the added configuration items; Update the configuration of the collection instances of the registered collector plugins according to the modified configuration items; Stop and unregister the corresponding collector plugin and / or collector instance based on the deleted configuration item.

[0053] The Prometheus data acquisition device based on the plug-in fusion collector described above implements the steps and processes of the Prometheus data acquisition method embodiment based on the plug-in fusion collector described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0054] See Figure 9 Corresponding to the above-described embodiment of the Prometheus data acquisition method based on a plug-in fusion collector, this application provides an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps and processes of the above-described embodiment of the Prometheus data acquisition method based on a plug-in fusion collector, and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0055] The memory 1009 can be used to store software programs and various data. The memory 1009 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback function, image playback function, etc.). Furthermore, the memory 1009 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1009 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0056] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into the processor 1010.

[0057] Corresponding to the above-described Prometheus data acquisition method embodiment based on a plug-in fusion collector, this application embodiment also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the steps and processes of the above-described Prometheus data acquisition method embodiment based on a plug-in fusion collector, and achieve the same technical effect. To avoid repetition, these will not be described again here.

[0058] The processor is the processor in the electronic device described in the above embodiments of this application. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0059] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0060] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0061] It is understood that the embodiments of this application have been described above in conjunction with the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. As those skilled in the art will know, various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, those skilled in the art, under the guidance or instruction of this application, can modify these features and embodiments to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, this invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of this invention.

Claims

1. A Prometheus data acquisition method based on a plug-in fusion collector, characterized in that, include: The fusion collector agent receives metric retrieval requests from the Prometheus server. In response to the metric retrieval request, metric data is retrieved from the cache of the fusion collector Agent and returned via the Prometheus interface. The fusion collector Agent is configured as follows: Based on the configuration listener, read and parse the configuration file; The SPI engine dynamically discovers and loads one or more data acquisition plugins into the container to execute preset monitoring data acquisition logic; wherein, the data acquisition plugins are configured to run as independent JAR packages; The collector plugins are matched according to the parsed configuration file content, and the matched collector plugins are registered to the collector management center for unified management; the configuration file content includes the collector plugins to be enabled and their corresponding collection instance configurations; Based on the data acquisition task management center, corresponding data acquisition tasks are created and scheduled for the data acquisition instances under the registered data acquisition plugins; In response to task scheduling, the collector plugin executes a data collection task on the target data source corresponding to the collection instance, and caches the collected indicator data in the indicator data management center.

2. The Prometheus data acquisition method based on a plug-in fusion collector according to claim 1, characterized in that, The method of dynamically discovering and loading one or more data acquisition plugins for executing preset monitoring data acquisition logic into the container based on the SPI engine includes: The SPI engine is based on Java's SPI mechanism and scans the plugin directory under a preset path at runtime; The discovered collector plugins are instantiated and made available for use by the collector management center. One collector plugin can correspond to multiple collection instances, and each collection instance corresponds to an independent collection task.

3. The Prometheus data acquisition method based on a plug-in fusion collector according to claim 1, characterized in that, The aforementioned caching of the collected indicator data to the indicator data management center includes: The indicator data management center pre-sets a global indicator data cache area; After each data collection task is completed, the latest collected indicator data, along with the timestamp, will be updated to the indicator data cache, overwriting the old data from the same data source. When the Prometheus Server pulls data, it retrieves the latest values ​​of all metric data from the metric data cache and returns them.

4. The Prometheus data acquisition method based on a plug-in fusion collector according to claim 1, characterized in that, The fusion collector agent is also configured as follows: A scheduled task based on a configuration listener monitors changes to configuration files. In response to the change event, the collector management center performs a dynamic update operation, including: Register the new collector plugin based on the added configuration items; Update the configuration of the collection instances of the registered collector plugins according to the modified configuration items; Stop and unregister the corresponding collector plugin and / or collector instance based on the deleted configuration item.

5. A Prometheus data acquisition device based on a plug-in fusion acquisition unit, characterized in that, include: The request receiving module is used to receive metric retrieval requests from the Prometheus Server based on the fusion collector agent; The indicator data acquisition module is used to respond to the indicator retrieval request, retrieve indicator data from the cache of the fusion collector Agent, and return it through the Prometheus interface. The fusion collector Agent is configured as follows: Based on the configuration listener, read and parse the configuration file; The SPI engine dynamically discovers and loads one or more data acquisition plugins into the container to execute preset monitoring data acquisition logic; wherein, the data acquisition plugins are configured to run as independent JAR packages; The collector plugins are matched according to the parsed configuration file content, and the matched collector plugins are registered to the collector management center for unified management; the configuration file content includes the collector plugins to be enabled and their corresponding collection instance configurations; Based on the data acquisition task management center, corresponding data acquisition tasks are created and scheduled for the data acquisition instances under the registered data acquisition plugins; In response to task scheduling, the collector plugin executes a data collection task on the target data source corresponding to the collection instance, and caches the collected indicator data in the indicator data management center.

6. The Prometheus data acquisition device based on a plug-in fusion acquisition unit according to claim 5, characterized in that, The method of dynamically discovering and loading one or more data acquisition plugins for executing preset monitoring data acquisition logic into the container based on the SPI engine includes: The SPI engine is based on Java's SPI mechanism and scans the plugin directory under a preset path at runtime; The discovered collector plugins are instantiated and made available for use by the collector management center. One collector plugin can correspond to multiple collection instances, and each collection instance corresponds to an independent collection task.

7. The Prometheus data acquisition device based on a plug-in fusion acquisition device according to claim 5, characterized in that, The aforementioned caching of the collected indicator data to the indicator data management center includes: The indicator data management center pre-sets a global indicator data cache area; After each data collection task is completed, the latest collected indicator data, along with the timestamp, will be updated to the indicator data cache, overwriting the old data from the same data source. When the Prometheus Server pulls data, it retrieves the latest values ​​of all metric data from the metric data cache and returns them.

8. The Prometheus data acquisition device based on a plug-in fusion acquisition unit according to claim 5, characterized in that, The fusion collector agent is also configured as follows: A scheduled task based on a configuration listener monitors changes to configuration files. In response to the change event, the collector management center performs a dynamic update operation, including: Register the new collector plugin based on the added configuration items; Update the configuration of the collection instances of the registered collector plugins according to the modified configuration items; Stop and unregister the corresponding collector plugin and / or collector instance based on the deleted configuration item.

9. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the steps of the Prometheus data acquisition method based on a plug-in fusion collector as described in any one of claims 1 to 4.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the Prometheus data acquisition method based on a plug-in fusion collector as described in any one of claims 1 to 4.

Images