A Zabbix monitoring system, a data access method and an electronic device

Abstract

The application provides a Zabbix monitoring system, a data access method and an electronic device. A collection processing module, especially a Zabbix index collection receiving module, is added between a Zabbix service module and an OTLP protocol receiving system. The Zabbix index collection receiving module acquires and parses an index file from the Zabbix service module in real time, acquires various Zabbix monitoring index data, performs data conversion on the Zabbix monitoring index data according to an OTLP index protocol specification, generates target monitoring index data conforming to the OTLP index protocol specification, and outputs the target monitoring index data to a corresponding OTLP protocol receiving system. In this way, the application can output the indexes generated by the Zabbix monitoring system as unified standard OTLP index data, seamlessly connect to devices supporting OTLP, and realize seamless access of Zabbix to the OTLP ecology.

Description

Technical Field

[0001] This application relates to the field of software operation and maintenance technology, and in particular to a Zabbix monitoring system, data access method, and electronic device. Background Technology

[0002] With technological advancements and business expansion, enterprises typically utilize various types of IT equipment to support daily operations. These IT devices may be geographically dispersed and exist in multiple data centers. Because the OTLP (Open Telemetry Protocol) defines a unified data protocol and transmission standard, its ecosystem is widely adopted by enterprises in their daily business operations to enable seamless data exchange between IT devices located in different locations and departments.

[0003] With IT equipment distributed across different regions and departments, its operation and maintenance management is crucial for cost control for enterprises. Zabbix, as an open-source enterprise-grade distributed monitoring system, replaces traditional centralized monitoring solutions, enabling rapid fault detection and reducing business downtime.

[0004] However, in practice, Zabbix, a distributed monitoring system, may not be able to seamlessly adapt to the OTLP ecosystem because the data rules for Zabbix monitoring metrics differ from the requirements of the OTLP metric protocol. Summary of the Invention

[0005] In view of this, embodiments of this application provide a Zabbix monitoring system, a data access method, and an electronic device to enable seamless Zabbix access to the OTLP ecosystem.

[0006] In a first aspect, embodiments of this application provide a Zabbix monitoring system, wherein the system includes:

[0007] The Zabbix service module and the data acquisition and processing module include a Zabbix metric acquisition and receiving module, wherein the data acquisition and processing module comprises a Zabbix metric acquisition and receiving module, which is connected to both the Zabbix service module and the OTLP protocol receiving system; the Zabbix metric acquisition and receiving module is used for:

[0008] The Zabbix service module obtains and parses the indicator file in real time to obtain the various Zabbix monitoring indicator data recorded in the indicator file.

[0009] According to the OTLP indicator protocol specification, the Zabbix monitoring indicator data is converted to generate target monitoring indicator data that conforms to the OTLP indicator protocol specification, and the target monitoring indicator data is output to the OTLP protocol receiving system.

[0010] Secondly, embodiments of this application provide a data access method, wherein the method is applied to the Zabbix monitoring system as described in the first aspect, the system comprising: a Zabbix service module and a data acquisition and processing module, wherein the data acquisition and processing module includes: a Zabbix metric acquisition and receiving module, and the method comprises:

[0011] The Zabbix service module obtains and parses the indicator file in real time to obtain the various Zabbix monitoring indicator data recorded in the indicator file.

[0012] According to the OTLP indicator protocol specification, the Zabbix monitoring indicator data is converted to generate target monitoring indicator data that conforms to the OTLP indicator protocol specification, and the target monitoring indicator data is output to the OTLP protocol receiving system.

[0013] Thirdly, embodiments of this application provide an electronic device, wherein the electronic device includes: a processor; and a memory storing a program; wherein the program includes instructions, which, when executed by the processor, cause the processor to perform the data access method described in the second aspect.

[0014] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the data access method described in the second aspect.

[0015] The beneficial effects of this application are:

[0016] This application provides a Zabbix monitoring system, a data access method, and an electronic device. By adding a data acquisition and processing module between the Zabbix service module and the OTLP protocol receiving system, and specifically by adding a Zabbix indicator acquisition and receiving module within the acquisition and processing module, this application obtains and parses indicator files from the Zabbix service module in real time. It acquires various Zabbix monitoring indicator data recorded in the indicator files, transforms the Zabbix monitoring indicator data according to the OTLP indicator protocol specification, generates target monitoring indicator data conforming to the OTLP indicator protocol specification, and outputs this target monitoring indicator data to the corresponding OTLP protocol receiving system. Thus, the Zabbix indicator acquisition and receiving module provided in this application can output the indicators generated by the Zabbix monitoring system as unified standard OTLP protocol data, seamlessly connecting to OTLP-supporting devices to achieve seamless Zabbix access to the OTLP ecosystem. Attached Figure Description

[0017] Further details, features, and advantages of this application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0018] Figure 1 This diagram illustrates a workflow architecture for a traditional timed reading of the Zabbix metrics database.

[0019] Figure 2 This diagram illustrates a workflow architecture for streaming metric data using the traditional MySQL binlog method.

[0020] Figure 3 This diagram illustrates a workflow architecture for pushing metrics data via the HTTP protocol in traditional Zabbix 7.0.

[0021] Figure 4 This application provides a schematic diagram of a system architecture for the Zabbix monitoring system.

[0022] Figure 5 This application illustrates another system architecture diagram of the Zabbix monitoring system provided in this application;

[0023] Figure 6 This paper illustrates a processing flow diagram of the Zabbix acquisition and processing module provided in this application.

[0024] Figure 7 This paper presents a schematic diagram illustrating the working principle of the Zabbix index acquisition and receiving module provided in this application.

[0025] Figure 8This paper illustrates another working principle flowchart of the Zabbix indicator acquisition and receiving module provided in this application;

[0026] Figure 9 This application provides a schematic flowchart of a data access method.

[0027] Figure 10 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of this application is shown. Detailed Implementation

[0028] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0029] It should be understood that the steps described in the method embodiments of this application may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.

[0030] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in this application are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0031] It should be noted that the terms "a" and "a plurality of" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0032] The methods for obtaining metric data in the Zabbix distributed monitoring system can be specifically divided into, for example: Figure 1 The scheme shown is to periodically read the Zabbix metrics database. Figure 2 The scheme shown uses MySQL binlog for streaming parsing of Zabbix metric data and Figure 3The example shown is a scheme for pushing metric data based on the HTTP protocol of Zabbix 7.0.

[0033] In actual deployment, the Zabbix service works by deploying the Zabbix Agent monitoring module on the devices to be monitored for real-time monitoring. Then, the Zabbix Proxy data acquisition unit interacts with the Zabbix Agent to obtain metric data for each device. Finally, the Zabbix data collector uploads the collected metric data to the Zabbix server, which then stores the data in a MySQL relational database. The Zabbix server provides a visual data interface, allowing maintenance personnel to view the data in the database.

[0034] Figure 1 The illustrated solution for periodically reading Zabbix metrics database involves developing a client application to read Zabbix metrics data, configuring the database tables, database server IP address, database account, and database password, and then starting the periodic reading of data from the MySQL table. However, this solution cannot guarantee data timeliness due to the periodic data queries. Furthermore, because the MySQL table lacks a unique identifier for the query location, there is a risk of querying too much or too little data, and it is incompatible with OTLP, especially the processor and export modules of the OT Collector in the OTLP data protocol.

[0035] Figure 2 The solution shown uses MySQL binlog for streaming metric data parsing. Because it uses a binary log for streaming, any changes to any table in MySQL are synchronized, requiring the application to shield against reading irrelevant data. Furthermore, it necessitates enabling specific binlog modes and managing username and password data in MySQL, making the process relatively cumbersome. Additionally, this solution is incompatible with OTLP, particularly the OT Collector's processor and export modules within the OTLP data protocol.

[0036] Figure 3The solution for pushing metric data via HTTP in Zabbix 7.0, as shown, relies on application stability. If the application is unstable, the accuracy of the acquired monitoring metric data will be affected. Furthermore, HTTP-based metric push scenarios only push metric data and timestamps, easily missing crucial information such as host information, host kernel version, host model, and description. Additionally, this solution is incompatible with OTLP, particularly the OT Collector's processor and exporter modules within the OTLP data protocol.

[0037] In summary, it can be seen that traditional Zabbix distributed monitoring systems cannot be seamlessly adapted to the OTLP ecosystem, especially the data processing module (processor) and data export module (exporter) provided by the OT collector in the OTLP ecosystem.

[0038] In view of this, this application provides a Zabbix monitoring system, a data access method, and an electronic device. Specifically, in a first aspect, this application provides a Zabbix monitoring system, which is a software system that can be deployed in any data system employing a Zabbix distributed monitoring system.

[0039] In some possible embodiments, it may be as follows: Figure 4 As shown, the Zabbix monitoring system may include a Zabbix service module and a data acquisition and processing module. The data acquisition and processing module includes a Zabbix metric acquisition and receiving module, which is connected to both the Zabbix service module and the OTLP protocol receiving system.

[0040] In this embodiment of the application, the Zabbix metric acquisition and receiving module is used for:

[0041] The Zabbix service module obtains and parses the indicator file in real time to obtain the various Zabbix monitoring indicator data recorded in the indicator file.

[0042] According to the OTLP indicator protocol specification, the Zabbix monitoring indicator data is converted to generate target monitoring indicator data that conforms to the OTLP indicator protocol specification, and the target monitoring indicator data is output to the OTLP protocol receiving system.

[0043] This application adds a data acquisition and processing module between Zabbix and the OTLP protocol receiving system, specifically a Zabbix indicator acquisition and receiving module within the data acquisition and processing module. This Zabbix indicator acquisition and receiving module acquires and parses indicator files from the Zabbix service module in real time, obtains various Zabbix monitoring indicator data recorded in the indicator files, performs data transformation on each Zabbix monitoring indicator data, generates target monitoring indicator data that conforms to the OTLP indicator protocol specification, and outputs the target monitoring indicator data to the corresponding OTLP protocol receiving system.

[0044] Thus, the Zabbix indicator acquisition and receiving module provided in this application can output the indicators generated by the Zabbix monitoring system as target monitoring indicator data of the unified standard OTLP protocol, seamlessly connecting with and supporting OTLP protocol receiving systems.

[0045] The following section will provide a detailed explanation of the Zabbix monitoring system using specific examples:

[0046] In this application, the Zabbix monitoring system is broadly divided into a service side and an operations and maintenance side. The service side comprises a three-layer architecture: Server ←→ Proxy ←→ Agent / Monitored Device. The Server can be understood as the Zabbix server, the Proxy as the Zabbix data collector (acting as a regional data collection center), and the Agent as the software component deployed on the monitored device to perform real-time monitoring. Because the Zabbix Server communicates directly with all monitoring devices within the enterprise in real time, frequent data requests can overload the Server process and cause a surge in data write pressure. Therefore, Zabbix uses the Zabbix Proxy data collection unit to collect, temporarily store, and pre-process data from devices within its assigned area. This data is then aggregated to obtain metric data and event information, which is synchronized to the Zabbix Server, reducing the data processing load on the Zabbix Server.

[0047] After receiving data uploaded by the Zabbix Proxy, the Zabbix Server stores the received data in the corresponding tables of the database according to different data types (such as numeric and character). For example, numeric monitoring metric data (such as CPU utilization and memory usage) is written to the history table, aggregated trend data is written to the trends table, and event and alarm related data is written to the events table.

[0048] Alternatively, after receiving data uploaded by the Zabbix Proxy, the Zabbix Server aggregates the received data according to the configured metric aggregation rules and writes the metrics to the corresponding metric file. For example, it can perform calculations and aggregation operations on some metric data that require further processing. For instance, it can aggregate the original monitoring metric data at certain time intervals (such as hourly or daily), calculate the average, maximum, and minimum values, generate trend data, and then summarize the calculated data along with the original monitoring metric data into the metric file to help users analyze metric changes.

[0049] Alternatively, after receiving data uploaded by the Zabbix Proxy, the Zabbix Server can associate and integrate the received data with other relevant data. For example, it can associate various indicator data of the same device from different Zabbix Proxy sources, or integrate monitoring indicator data with the device's metadata, configuration information, topology information, etc., to generate indicator files.

[0050] Based on this, the Zabbix metric acquisition and receiving module can obtain metric files generated from the metric data collected by the Zabbix data collector in real time by the Zabbix service module, thereby achieving the effect of real-time metric data acquisition. In this embodiment, the acquisition and processing module can be understood as an improved OT Collector. Specifically, compared to the traditional OT Collector, the OT Collector corresponding to the acquisition and processing module of this application has a pre-written execution plugin for the Zabbix metric acquisition and receiving module provided in this application. By running the execution plugin of the Zabbix metric acquisition and receiving module, the running logic corresponding to the Zabbix metric acquisition and receiving module is executed.

[0051] In some possible embodiments, it may be as follows Figure 5 As shown, the acquisition and processing module provided in this application includes, in addition to the Zabbix indicator acquisition and receiving module, a data processing module and a data export module. The data processing module and data export module are software units developed to adapt to the OTLP data processing and data export modules.

[0052] As an example, it can be seen that... Figure 6 As shown, the data processing module includes: Batch (batch data processing module), Transform (transformation unit), Attributes (attribute data processing module), Generation (metric generation unit), GroupBy (grouping unit), and Sampling (sampling unit).

[0053] The system comprises several modules: a batch data processing module for batch processing of received data to improve efficiency; a conversion unit for data format conversion, data type conversion, and data mapping; an attribute data processing module for managing and manipulating data attribute information, allowing for the addition, modification, and deletion of attributes; an indicator generation unit for generating new derived indicators based on existing data, such as creating new indicators through mathematical operations and logical judgments on original monitoring data; and an indicator for the comprehensive utilization of system resources based on CPU and memory usage through weighted calculations. A grouping unit for grouping data according to specified attributes or conditions allows data with similar attributes to be grouped together, facilitating the same processing and analysis of similar data. Finally, a sampling unit for extracting and analyzing a portion of the original data using random sampling, timed sampling, or other sampling algorithms when dealing with large datasets or situations where not all data needs to be processed.

[0054] In some possible embodiments, each data processing module can be understood as a processor; that is, in the embodiments of this application, multiple processors may be pre-configured in the data processing module. Figure 7 As shown, the data processing module includes: a FilterProcessor, an AttributeProcessor, a MetricsTransformProcessor, a BatchProcessor, and a CumulativetodeltaProcessor. Each processor has its own pre-defined data processing logic. Each processor receives monitoring indicator data conforming to the OTLP indicator protocol specification from the Zabbix indicator acquisition and receiving module, processes the data according to its pre-defined logic, and then outputs the processing results to the data export module. The data export module outputs the processed data according to an indicated path, which includes output to the OTLP protocol receiving system.

[0055] The specific data processing principles of each processor can be found in relevant existing technical documents and are not the focus of this application; therefore, they will not be described in detail here. The data export module can output corresponding data according to an indicated path, either by pulling or pushing data. The indicated path is a pre-configured output path, which may include output to an OTLP protocol receiving system, or... Figure 6 As shown, the output is sent to the operation and management system, or sent to other devices via Kafka data stream. Alternatively, it can be... Figure 7As shown, the data export module may include: OtlpGrpcExporter, OtlpHttpExporter, KafkaExporter, PrometheusExporter, FileExporter, etc. Different data export modules can be selected for data export according to the final export path.

[0056] In this application, the Zabbix indicator acquisition and reception module is the key to implementing the solution. The following section focuses on explaining the implementation logic and principles of the Zabbix indicator acquisition and reception module:

[0057] In some possible embodiments, it may be as follows Figure 7 As shown, the Zabbix metric acquisition and receiving module includes: a real-time acquisition unit, a filtering unit, a metadata loading unit, and a metric data enrichment unit, wherein:

[0058] The real-time acquisition unit is used to obtain the indicator file from the Zabbix service module in real time, and to parse and process the indicator file to obtain the Zabbix monitoring indicator data.

[0059] The filtering unit is used to filter out target Zabbix monitoring indicator data that meet the preset data filtering conditions.

[0060] The metadata loading unit is used to periodically call the metadata information of the Zabbix monitoring indicator data in the Zabbix service module according to the API interface between the Zabbix indicator acquisition and receiving module and the Zabbix service module.

[0061] The indicator data enrichment unit is used to assemble, trim, and merge the target Zabbix monitoring indicator data based on the Zabbix monitoring indicator data metadata information and the monitoring indicator data of the OTLP indicator protocol specification, so as to generate the target monitoring indicator data.

[0062] Before the Zabbix metrics acquisition and receiving module retrieves metric files, the Zabbix Server needs to be started to write the Zabbix monitoring metric data collected by the Zabbix data collector to the specified metric file storage path in real time to generate metric files. Specifically, this can be done by modifying the zabbix_server.conf configuration file to configure the Zabbix Server. After configuring the Zabbix Server, restarting the Zabbix monitoring system will allow you to control the Zabbix Server to start generating metric files in real time. Modifying the zabbix_server.conf configuration file involves changing the "ExportDir (export path)," "ExportFileSize (export data volume)," and "ExportType (export metric category, which can include historical data (=history) and real-time data (=RealTime))" settings.

[0063] Meanwhile, the configuration file of the Zabbix metrics acquisition and receiving module can be used to configure data filtering rules, the Zabbix Server report address, login account, login password (which can be a plaintext password or an encrypted password), and the path to read the metrics files. In this embodiment, the Zabbix metrics acquisition and receiving module can be understood as a software plugin designed specifically for the OTLP ecosystem. The plugin for this Zabbix metrics acquisition and receiving module can be configured with: the path to read Zabbix metrics data, the login address, the login account and password, the Zabbix version, the checkpoint path, and the metadata query frequency.

[0064] After all configuration information is configured, restarting the Zabbix monitoring system will immediately take effect, thus reducing the time required for maintenance personnel to manually configure various parameters.

[0065] One possible implementation method is as follows: Figure 8 As shown, this real-time acquisition unit retrieves indicator files from the Zabbix service module in real time based on the path of the Zabbix indicator data. Alternatively, it retrieves indicator files identified by checkpoints from the Zabbix service module. The indicator files are in n.ndjson format, specifically a file named history-history-syncer-n.ndjson, where -n is used to distinguish different indicator files.

[0066] After the real-time acquisition unit obtains the indicator file, it parses and processes the file to extract the required target indicator data and performs dimensional transformation on the target indicator data. For example, the indicator name of the original indicator `vfs.dev.read.rate[sda]` is converted to the indicator name specified by the OTLP indicator protocol: `system.disk.operations.rate`. Simultaneously, two new indicator dimensions are added: `disk.io.direction` and `[sda]` from the original indicator, as the corresponding dimension `system.device`. This dimensional transformation process can be implemented using the following code snippet:

[0067] vfs.dev.read.rate: / / Virtual file system device read operation rate

[0068] set:

[0069] static: / / static rules

[0070] alias: system.disk.operations.rate / / Rename the original vfs.dev.read.rate metric to system.disk.operations.rate

[0071] labels: / / Custom Dimension Labels

[0072] disk.io.direction: read / / read operation

[0073] system.device: $1

[0074] The filtering unit selects target Zabbix monitoring metrics data that meet the preset data filtering conditions. Specifically, the preset data filtering conditions are set by the operation and maintenance personnel according to the actual operation and maintenance needs, and can be set visually through the Zabbix Web interface. This application does not strictly limit the specific data filtering conditions.

[0075] The metadata loading unit retrieves the metadata information of Zabbix monitoring metrics from the Zabbix service module by calling the API interface of the Zabbix service module.

[0076] Then, the indicator data enrichment unit performs data processing operations such as assembling, renaming, trimming, and dimension supplementation on the target Zabbix monitoring indicator data to generate target monitoring indicator data that conforms to the OTLP indicator protocol specification.

[0077] Metadata consists of descriptive information used to describe monitoring metric data. The metadata loading unit can query the metric key and description (such as CPU utilization) via the metric ID using an API call, or query the host IP, host architecture, host production environment, host server, and other information via the host ID using an API call. Assembly can combine the original metric value with metadata, for example, adding information such as host IP: 192.168.1.100 and metric description: CPU utilization to "CPU utilization 80%". Trimming can specifically remove invalid or redundant data, such as deleting duplicate metadata. Merging refers to aggregating related metrics, such as merging the utilization of CPU core 1 with the utilization of CPU core 2 to generate the overall CPU utilization, or associating metadata from different dimensions, etc.

[0078] In this embodiment, the OTLP Metrics Protocol Specification is a unified observability data transmission protocol defined by the OpenTelemetry project, supporting standardized formats for metrics, logs, and link data. Data encapsulated according to the OTLP specification can be uniformly received, stored, and analyzed by tools within the OTLP ecosystem. These tools include Prometheus, Grafana, Jaeger, and others.

[0079] Specifically, suppose the real-time acquisition unit collects a raw metric: Raw identifier: metric ITEMID=42249, value VALUE=80, metric description NAME, group GROUPS, and tag TAGS. Further, based on this metric ID and host ID, the API interface is called to obtain metadata. Further, the API query retrieves the metric KEY=system.cpu.util[1,nice] based on the metric ITEMID, where 1 in square brackets represents the number of CPU cores (system.cpu.logical_number), and nice represents the CPU mode (cpu.mode). The host IP is obtained by querying the host HOST_ID associated with the ITEMID (host.ip), and the hostname (host.name) is obtained by querying the INTERFACE_ID. Then, the host architecture (host.arch), operating system type (os.type), and operating system version (os.version) are obtained by querying the host metric system.uname.

[0080] Furthermore, the indicator data and metadata are assembled, trimmed, and merged. For specific examples, please refer to the following code to assemble, trim, and merge the data in key-value pair format:

[0081] {

[0082] "resourceMetrics": [

[0084] {

[0085] "resource":

[0086] {

[0087] "attributes": [

[0089] {

[0090] "key": "host.name",

[0091] "value":

[0092] {

[0093] "stringValue": "VM-10-100-centos"

[0094] }

[0095] },

[0096] {

[0097] "key": "host.ip",

[0098] "value":

[0099] {

[0100] "stringValue": "192.168.1.100"

[0101] }

[0102] },

[0103] {

[0104] "key": "host.arch",

[0105] "value":

[0106] {

[0107] "stringValue": "x86_64"

[0108] }

[0109] },

[0110] {

[0111] "key": "os.type",

[0112] "value":

[0113] {

[0114] "stringValue": "Linux"

[0115] }

[0116] },

[0117] {

[0118] "key": "os.version",

[0119] "value":

[0120] {

[0121] "stringValue": "3.10.0-1062.9.1.el7.x86_64"

[0122] }

[0123] } ]

[0125] },

[0126] "scopeMetrics": [

[0128] {

[0129] "metrics": [

[0131] {

[0132] "gauge":

[0133] {

[0134] "dataPoints": [

[0136] {

[0137] "asDouble": 80,

[0138] "attributes": [

[0140] {

[0141] "key": "system.cpu.logical_number",

[0142] "value":

[0143] {

[0144] "intValue": "1"

[0145] }

[0146] },

[0147] {

[0148] "key": "cpu.mode",

[0149] "value":

[0150] {

[0151] "stringValue": "nice"

[0152] }

[0153] }

[0154] ],

[0155] "timeUnixNano": "1725636505000000000"

[0156] } ]

[0158] },

[0159] "name": "system.cpu.util

[0160] } ]

[0162] } ]

[0164] } ]

[0166] }

[0167] In this way, target monitoring indicator data can be generated and output to the data processing module.

[0168] In the embodiments of this application, it can be as follows Figure 7 As shown, the Zabbix metrics acquisition and receiving module includes a checkpoint placement unit and an output unit. The checkpoint compass unit can be understood as a checkpoint module, which contains checkpoint mechanism logic. The checkpoint placement unit is used for:

[0169] During the process of the Zabbix indicator acquisition and receiving module reading the indicator file, corresponding inspection identification information is generated. The inspection identification information includes at least: the indicator file identification information of the indicator file being read at the current moment, and the byte position information in the file being read at the current moment.

[0170] The output unit is used to output and record the inspection identification information to a specified storage path, which includes a specified memory path and a specified disk path. It also outputs the target monitoring data to the data processing module (processor).

[0171] Specifically, after each line of data in the indicator file is parsed by the Zabbix indicator acquisition and receiving module, the current byte position of the parsed file (i.e., which byte in the indicator file is parsed, represented by the file offset) is recorded and saved in the specified memory path. By default, the checkpoint identification information is written from the specified memory path to the specified disk path once per minute. As another implementation, the checkpoint identification information in the specified memory path is also written to the specified disk path when the application stops.

[0172] The checkpoint information can be represented by the checkpoint itself. This checkpoint information includes the parsing time, the target index file being parsed, and the current byte position being parsed. Based on this, in some possible embodiments, the Zabbix index acquisition and receiving module is also used for:

[0173] In response to the abnormal shutdown indication, target check identification information is obtained from the specified disk path. The target check identification information is the check identification information generated by the checkpoint disk placement unit at the time when the abnormal shutdown indication information occurs.

[0174] Based on the target indicator file corresponding to the target inspection identification information and the byte position information in the file, the target indicator file is read continuously.

[0175] Specifically, in data processing workflows (such as collecting metrics from Zabbix and outputting them to downstream storage / analysis systems), if the program abnormally shuts down during the "writing data downstream" process (e.g., server power failure, process crash), two problems are likely to occur:

[0176] Repeated read / write: After the program restarts, it may forget "which data has been successfully output", causing the same data to be written to the downstream multiple times, resulting in data duplication.

[0177] Insufficient read / write operations: The program may have only completed the output of part of the data, leaving the remaining data unprocessed, resulting in missing downstream data.

[0178] In this implementation, after each complete output of data to the downstream system, the program performs a checkpoint operation, persistently saving information such as the current "range of successfully output data and processing progress" to a checkpoint file or checkpoint database. If anomaly recovery is required, the progress recorded in the checkpoint file or checkpoint database can be read first to determine which data has been successfully parsed and which has not, thereby avoiding duplicate processing or omissions.

[0179] Thus, a checkpoint mechanism is completed after each data write operation and output to the downstream process. This ensures that the written data is not read multiple times or incompletely due to improper application closure, primarily to guarantee data integrity. In this application, outputting to the downstream process specifically refers to outputting to the data processing module, particularly to each processor, where the downstream processor continues to process the data.

[0180] Furthermore, in some possible embodiments, the Zabbix metric acquisition and receiving module is also used for:

[0181] Obtain target configuration information, which includes: Zabbix login address, Zabbix login account, Zabbix login password, and Zabbix checkpoint identifier path;

[0182] The Zabbix login account and Zabbix login password are used to perform security verification on the account accessing the Zabbix monitoring system.

[0183] The specific security verification logic can refer to traditional account password verification logic, and is not strictly limited here. In this application, the configuration file of the Zabbix indicator collection and receiving module can be flexibly configured according to actual needs to achieve on-demand acquisition of monitoring indicators. At the same time, by configuring the Zabbix account password, data security can be effectively protected. Furthermore, the introduction of a checkpoint mechanism can ensure data integrity and transmission reliability. In actual engineering practice, it has been found that by using the embodiment of this application and developing data processing and data export modules using an OTLP-based collector, all data processing and data export modules can be directly used, reducing the required resources and achieving higher real-time performance.

[0184] Based on the Zabbix monitoring system provided in the first aspect, this application provides a data access method, which is applied to the Zabbix monitoring system provided in the first aspect, wherein, as... Figure 9As shown, the method includes the following steps:

[0185] S91. Obtain and parse the indicator file from the Zabbix service module in real time, and obtain the various Zabbix monitoring indicator data recorded in the indicator file;

[0186] S92. According to the OTLP indicator protocol specification, perform data conversion on each of the Zabbix monitoring indicator data to generate target monitoring indicator data that conforms to the OTLP indicator protocol specification, and output the target monitoring indicator data to the OTLP protocol receiving system.

[0187] In some possible embodiments, the method further includes:

[0188] The indicator file is obtained in real time from the Zabbix service module, and the indicator file is parsed to obtain the Zabbix monitoring indicator data.

[0189] According to the preset data filtering conditions, filter out the target Zabbix monitoring indicator data that meet the data filtering conditions;

[0190] Based on the API interface between the Zabbix metric acquisition and reception module and the Zabbix service module, the metadata information of the Zabbix monitoring metric data in the Zabbix service module is called periodically.

[0191] Based on the Zabbix monitoring metric data metadata information and combined with the monitoring metric data of the OTLP metric protocol specification, the target Zabbix monitoring metric data are assembled, trimmed, and merged to generate the target monitoring metric data.

[0192] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in this application comply with relevant laws and regulations and do not violate public order and good morals.

[0193] The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0194] Thirdly, exemplary embodiments of this application also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the electronic device to perform a method according to an embodiment of this application.

[0195] An exemplary embodiment of this application also provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of this application.

[0196] An exemplary embodiment of this application also provides a computer program product, including a computer program, wherein, when executed by a computer's processor, the computer program is used to cause the computer to perform a method according to an embodiment of this application.

[0197] refer to Figure 10 The present invention describes a structural block diagram of an electronic device 100 that can serve as a server or client of this application, which is an example of a hardware device that can be applied to various aspects of this application. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0198] like Figure 10 As shown, the electronic device 100 includes a computing unit 101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM 102) or a computer program loaded from a storage unit 108 into a random access memory (RAM 103). The RAM 103 may also store various programs and data required for the operation of the electronic device 100. The computing unit 101, ROM 102, and RAM 103 are interconnected via a bus 104. An input / output interface (I / O interface 105) is also connected to the bus 104.

[0199] Multiple components in electronic device 100 are connected to I / O interface 105, including: input unit 106, output unit 107, storage unit 108, and communication unit 109. Input unit 106 can be any type of device capable of inputting information to electronic device 100. Input unit 106 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 107 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 108 may include, but is not limited to, disks and optical discs. Communication unit 109 allows electronic device 100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0200] The computing unit 101 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 101 performs the various methods and processes described above. For example, in some embodiments, the aforementioned data access methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 108. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 100 via ROM 102 and / or communication unit 109. In some embodiments, the computing unit 101 can be configured to perform the aforementioned data access methods by any other suitable means (e.g., by means of firmware).

[0201] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0202] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0203] As used in this application, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0204] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0205] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0206] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

Claims

1. A Zabbix monitoring system, characterized in that, The Zabbix monitoring system includes a Zabbix service module and a data acquisition and processing module. The data acquisition and processing module further includes a Zabbix metric acquisition and receiving module, which is connected to both the Zabbix service module and the OTLP protocol receiving system. The Zabbix metric acquisition and receiving module is used for: The Zabbix service module obtains and parses the indicator file in real time to obtain the various Zabbix monitoring indicator data recorded in the indicator file. According to the OTLP indicator protocol specification, the Zabbix monitoring indicator data is converted to generate target monitoring indicator data that conforms to the OTLP indicator protocol specification, and the target monitoring indicator data is output to the OTLP protocol receiving system. The Zabbix metric acquisition and reception module includes: a real-time acquisition unit, a filtering unit, a metadata loading unit, and a metric data enrichment unit, wherein: The real-time acquisition unit is used to obtain the indicator file from the Zabbix service module in real time, and to parse and process the indicator file to obtain the Zabbix monitoring indicator data. The filtering unit is used to filter out target Zabbix monitoring indicator data that meet the preset data filtering conditions. The metadata loading unit is used to periodically call the metadata information of the Zabbix monitoring indicator data in the Zabbix service module according to the API interface between the Zabbix indicator acquisition and receiving module and the Zabbix service module. The indicator data enrichment unit is used to assemble, trim, and merge the target Zabbix monitoring indicator data based on the metadata information of the Zabbix monitoring indicator data and the monitoring indicator data of the OTLP indicator protocol specification, so as to generate the target monitoring indicator data.

2. The Zabbix monitoring system according to claim 1, characterized in that, The acquisition and processing module further includes a data processing module and a data export module, wherein the data processing module is pre-configured with multiple processors; Each of the processors is respectively configured to: receive the target monitoring indicator data output by the Zabbix indicator acquisition and receiving module, process the target monitoring indicator data according to the preset data processing logic, and output the data processing result to the data export module; The data export module is used to output the data processing results according to the indicated path, the indicated path including: output to the OTLP protocol receiving system.

3. The Zabbix monitoring system according to claim 1, characterized in that, The Zabbix index acquisition and receiving module includes a checkpoint disk insertion unit and an output unit. The checkpoint disk insertion unit is used for: During the process of the Zabbix indicator acquisition and receiving module reading the indicator file, corresponding inspection identification information is generated. The inspection identification information includes at least: the indicator file identification information of the indicator file being read at the current moment, and the byte position information in the file being read at the current moment. The output unit is used to output and record the inspection identification information to a specified storage path, and to output the target monitoring indicator data to the data processing module; the specified storage path includes a specified memory path and a specified disk path.

4. The Zabbix monitoring system according to claim 3, characterized in that, The Zabbix metric acquisition and reception module is also used for: In response to the abnormal shutdown indication, target check identification information is obtained from the specified disk path. The target check identification information is the check identification information generated by the checkpoint disk placement unit at the time when the abnormal shutdown indication information occurs. Based on the target indicator file corresponding to the target inspection identification information and the byte position information in the file, the target indicator file is read continuously.

5. The Zabbix monitoring system according to claim 1, characterized in that, The Zabbix metric acquisition and reception module is also used for: Obtain target configuration information, which includes: Zabbix login address, Zabbix login account, Zabbix login password, and Zabbix checkpoint identifier path; The Zabbix login account and Zabbix login password are used to perform security verification on the account accessing the Zabbix monitoring system.

6. A data access method, characterized in that, The method is applied to the Zabbix monitoring system as described in any one of claims 1-5, the system comprising: a Zabbix service module and a data acquisition and processing module, wherein the data acquisition and processing module includes: a Zabbix metric data acquisition and receiving module, and the method comprises: The Zabbix service module obtains and parses the indicator file in real time to obtain the various Zabbix monitoring indicator data recorded in the indicator file. According to the OTLP indicator protocol specification, the Zabbix monitoring indicator data is converted to generate target monitoring indicator data that conforms to the OTLP indicator protocol specification, and the target monitoring indicator data is output to the OTLP protocol receiving system. The indicator file is obtained in real time from the Zabbix service module, and the indicator file is parsed to obtain the Zabbix monitoring indicator data. According to the preset data filtering conditions, filter out the target Zabbix monitoring indicator data that meet the data filtering conditions; Based on the API interface between the Zabbix metric acquisition and reception module and the Zabbix service module, the metadata information of the Zabbix monitoring metric data in the Zabbix service module is called periodically. Based on the Zabbix monitoring metric data metadata information and combined with the monitoring metric data of the OTLP metric protocol specification, the target Zabbix monitoring metric data are assembled, trimmed, and merged to generate the target monitoring metric data.

7. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing a program; wherein the program includes instructions that, when executed by the processor, cause the processor to perform the method according to claim 6.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to claim 6.

Images