Observable platform and observable data processing method

Abstract

The invention discloses an observable platform and an observable data processing method, and belongs to the field of network application, and the platform comprises a monitoring module which is used for monitoring the change operation of resources in a cluster and triggering a preset hook function when a change event is detected; the capturing module is used for periodically polling the resources monitored by the monitoring module and determining whether observable information of the target resources is collected or not, and if yes, the corresponding collector is activated; the collector is used for collecting observable information of a target resource and transmitting the observable information to a corresponding data pipeline of the OpenTelemetry according to the data type of the observable information; the data processing module is used for calling a corresponding processor in the data pipeline to process the observable information and transmitting the processed observable information to the output device; and the output device is used for exporting the processed observable information to a target memory or a display system. The method can adapt to constantly changing technology and business requirements, simplifies the data collection and analysis process, and improves the consistency and comparability of data.

Description

Technical Field

[0001] The present invention relates to the field of network application technology, and in particular to an observable platform and a method for processing observable data. Background Art

[0002] With the development of information technology, system observability has become a key factor in ensuring service stability and maintainability. At present, mainstream observability systems mainly rely on three core data types: metrics, traces, and logs, but lack unified data standards. Therefore, although many projects have emerged in the field of observability, observable data generated by different systems are usually not directly interoperable. This means that data collected in different technical environments must undergo complex conversion and analysis before they can be understood and used by other systems, which increases the complexity and cost of data processing, and also makes observable data between different enterprises often form islands, making it difficult to conduct correlation analysis and unified display, reducing the value of data utilization and the overall efficiency of the system.

[0003] At the same time, since different companies use different technology stacks, including programming languages, frameworks, and middleware, observability tools and solutions are highly customized. This diversity and heterogeneity makes it difficult to reuse existing tools across platforms and increases the complexity of development and operation and maintenance work. Summary of the invention

[0004] The purpose of the present invention is to provide an observable platform and a method for processing observable data to solve the problems of complex data processing and low utilization value caused by inconsistent data standards in the prior art, and the problem that observability tools are difficult to reuse across platforms due to the high degree of customization.

[0005] To achieve the above objectives, this application adopts the following technical solutions:

[0006] The present application provides an observability platform, which is built on the OpenTelemetry Collector framework and dynamically configures and extends its data processing plug-ins through an application programming interface defined by a custom resource, including:

[0007] The monitoring module is used to monitor the change operations of resources in the cluster, and when a change event is detected, it triggers the preset hook function to synchronously update the data structure object of the cluster resource in the memory;

[0008] A capture module is used to periodically poll the resources monitored by the monitoring module and determine whether to collect observable information of the target resources according to preset rules, and if so, activate the corresponding collector;

[0009] The collector is used to collect observable information of the target resource and transmit it to the corresponding data pipeline of OpenTelemetry according to the data type of the observable information;

[0010] A data processing module, configured to call a corresponding processor in the data pipeline according to a preset data processing logic to process the observable information, and transmit the processed observable information to an output device;

[0011] The output device is used to export the processed observable information to a target storage or display system to view and analyze the observable information.

[0012] Preferably, the monitoring module includes an acquisition unit and a calling unit;

[0013] The monitoring unit is used to subscribe to the events of the cluster application programming interface service to capture the change operations of the resources in the cluster in real time, and the change operations include adding, updating and deleting;

[0014] The calling unit is used to trigger a preset hook function to intercept the change event when the monitoring unit detects a change event, and call a callback function corresponding to the resource change to update the data structure object of the resource in the memory.

[0015] Preferably, the hook function includes an OnChange function, an OnAdd function and an OnDelete function.

[0016] Preferably, the collector comprises a collection unit, a conversion unit and a transmission unit;

[0017] The collection unit is used to call the scraper interface to interact with the Pod to which the target resource belongs, and obtain observable information of the target resource;

[0018] The conversion unit is used to convert the observable information of the target resource into data that complies with the OpenTelementry standard;

[0019] The transmission unit is used to transmit the converted observable information to a data pipeline with corresponding data type in OpenTelementry according to the data type of the observable information.

[0020] Preferably, there are multiple collectors for collecting observable information of different target resources.

[0021] Preferably, the data types include indicators, links and logs.

[0022] Preferably, there are multiple data pipelines, each of which processes a different type of observable information.

[0023] Preferably, the preset rules and the preset data processing logic are configured through an application programming interface defined by the custom resource.

[0024] Preferably, there are multiple processors connected in series to perform data processing.

[0025] A method for processing observable data, applied to an observable platform as described above, comprises the following steps:

[0026] Monitor the change operations of resources in the cluster, and when a change event is detected, trigger the preset hook function to synchronously update the data structure object of the cluster resource in the memory;

[0027] Regularly poll the monitored resources and determine whether to collect observable information of the target resources based on preset rules. If so, activate the corresponding collector;

[0028] Collect observable information of the target resource through the collector, and transmit it to the corresponding data pipeline of OpenTelemetry according to the data type of the observable information;

[0029] Calling the corresponding processor in the data pipeline according to the preset data processing logic to process the observable information, and transmitting the processed observable information to the output device;

[0030] The processed observable information is exported to a target storage or presentation system through the exporter to view and analyze the observable information.

[0031] The present invention has the following beneficial effects:

[0032] The observability platform based on the OpenTelemetry Collector framework provided in this application can easily expand new functions by selecting plug-ins on demand in a plug-in manner to support more data types and sources, and can also adapt to changing technical and business needs. At the same time, the use of unified standards to process metrics, logs, and traces helps to simplify the process of data collection and analysis and improve data consistency and comparability. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is the platform architecture diagram in this application;

[0034] Figure 2 It is a flowchart of a method for processing observable data provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] To make the technical solution of the present application clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The terms "first", "second", etc. in the claims and specification of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances. This is only to describe the distinction method used when describing objects with the same attributes in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that a process, method, system, product or device containing a series of units is not necessarily limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] In some embodiments, Figure 1 As shown in the figure, an observability platform is provided. The observability platform is built on the OpenTelemetry Collector framework and dynamically configures and extends its data processing plug-ins through the custom resource definition application programming interface (CRDAPI), including:

[0037] The monitoring module is used to monitor the change operations of resources in the cluster, and when a change event is detected, it triggers the preset hook function to synchronously update the data structure object of the cluster resource in the memory;

[0038] The capture module is used to periodically poll the resources monitored by the monitoring module and determine whether to collect observable information of the target resources according to preset rules. If so, the corresponding collector is activated;

[0039] Collector, which is used to collect observable information of target resources and transmit it to the corresponding data pipeline of OpenTelemetry according to the data type of the observable information;

[0040] The data processing module is used to call the corresponding processor in the data pipeline according to the preset data processing logic to process the observable information, and transmit the processed observable information to the output device;

[0041] The output device is used to export the processed observable information to a target storage or display system to view and analyze the observable information.

[0042] The observability platform provided in this embodiment is mainly aimed at the complex environment in the Kubernetes cluster, and can handle a variety of user scenarios, including stateless applications represented by microservice applications and stateful applications represented by databases. The platform is built based on the OpenTelementry Collector framework and can process observable data such as Metrics, Logs, and Traces with a unified standard. The specific implementation adopts a data pipeline (Pipeline) and plug-in architecture, where Pipeline represents the flow of data, usually including the order in which data passes through the Receiver, Processor, and Exporter. The specific plug-in used is determined according to the CRD API.

[0043] Among them, the plug-in architecture of OpenTelemetry Collector means that its design allows the dynamic addition, modification or deletion of functions through plug-ins without large-scale modification or recompilation of the entire Collector. This architectural model makes the Collector highly flexible and scalable. Specifically, the plug-in architecture of OpenTelemetry Collector contains four types of plug-ins: Receiver, Processor, Exporter, and Connector that may be included in some implementations. Each type of plug-in has different functions.

[0044] The Receiver plug-in is responsible for receiving observable data from different services and protocols. For example, the OTLP Receiver can receive observable data that conforms to the OpenTelemetry protocol, and the Prometheus Receiver can actively pull observable data in the Prometheus format.

[0045] The Processor plug-in is responsible for processing the received observable data, such as filtering, modification, and batch processing, especially performing necessary processing on the data before passing it to the Exporter to meet specific monitoring requirements.

[0046] The Exporter plugin is responsible for sending the processed observable data to downstream storage systems or analysis tools, and supports multiple output formats and protocols, enabling Collector to be integrated with different backend systems.

[0047] The Connector plug-in is used to connect different Pipelines to achieve data interoperability, such as converting the tracking data in one Pipeline into indicator data and sending it to another Pipeline for processing.

[0048] Therefore, in this embodiment, the observability platform based on the OpenTelemetry Collector framework can select plug-ins on demand in a plug-in manner, so as to easily expand new functions to support more data types and sources, and can also adapt to changing technical and business needs. At the same time, the use of unified standards to process metrics, logs and traces helps to simplify the process of data collection and analysis and improve the consistency and comparability of data.

[0049] Further, the monitoring module includes an acquisition unit and a calling unit;

[0050] A monitoring unit, which is used to subscribe to the events of the cluster application programming interface service to capture the change operations of resources in the cluster in real time. The change operations include adding, updating, and deleting;

[0051] The calling unit is used to trigger a preset hook function to intercept the change event when the monitoring unit detects a change event, and call the callback function corresponding to the resource change to update the data structure object of the resource in the memory.

[0052] The monitoring module is called K8sObserver component, which is specially designed to monitor resource changes in Kubernetes clusters. The resources here include Pod, Service, and Deployment, etc. It mainly consists of two parts: monitoring unit and calling unit. The monitoring unit captures resource change operations in real time by subscribing to events of the cluster application programming interface service (Kubernetes API Server). Change operations include adding, updating, and deleting. When these change events are detected by the monitoring unit, the calling unit will trigger the preset hook (Hook ) function, Hook functions include OnChange function, OnAdd function and OnDelete function, among which OnChange function is usually triggered when a certain state or value of an object changes, OnAdd function is usually triggered when a new element or record is added, and OnDelete function is usually triggered when an element or record is deleted. After intercepting the resource change event, the triggered Hook function will call the callback function of the corresponding resource change. The K8sObserver component maintains a data structure abstracted from Kubernetes resources in its memory. These callback functions will synchronously update the data structure object of Kubernetes resources in memory. For example, when the OnAdd function is triggered, the callback function of CreateStruct will be called to create a data type corresponding to the Kubernetes resource in its memory, which is equivalent to maintaining a copy of Kubernetes key resources in memory. When obtaining Kubernetes information, it will be selected from its memory first. This can not only ensure that the monitoring system is consistent with the Kubernetes cluster, but also greatly relieve the pressure of the APIServer component in Kubernetes, and ensure that Kubernetes's ability to process data is not affected.

[0053] The capture module is named Receiver_Creator, which is used to periodically poll the resources monitored by the K8sObserver component and decide whether to collect observable information of the target resource according to the preset rules. The preset rules here refer to the configuration files or user-set rules, which are all configured in the CRD API. For example, in the CRD API, collectors define how to collect data, including the type of data source and collection rules. Logs define detailed rules for log collection, including the path of the log file, log type, and filtering conditions, which makes log collection more accurate and efficient. Metrics defines the rules and details of indicator collection, including the collection interval and specific indicator name. If collected, a specific Receiver is activated, because there are multiple Receivers in the platform, each of which collects observable information of different target resources, such as Loki for log data and Prometheus for indicator data. These Receivers can be built-in Receivers such as mysqlReceiver or redisReceiver, or they can be customized by users according to specific needs. This design provides great flexibility, allowing the platform to adapt to different monitoring scenarios and user needs.

[0054] Further, the collector includes a collection unit, a conversion unit and a transmission unit;

[0055] The collection unit is used to call the scraper interface to interact with the Pod to which the target resource belongs to obtain observable information of the target resource;

[0056] The conversion unit is used to convert the observable information of the target resource into data that conforms to the OpenTelementry standard;

[0057] The transmission unit is used to transmit the transformed observable information to the corresponding data pipeline of OpenTelementry according to its data type.

[0058] After a specific Receiver is activated, it will execute specific data collection logic, which includes a collection unit, a conversion unit, and a transmission unit. The collection unit is responsible for calling the scraper interface to establish a connection with the collected Pod, and the Pod is the Pod to which the target resource belongs, so as to obtain the observable information of the target resource. For example, mysqlReceiver will call the scraper interface according to its own configuration information and establish a connection with the collected Mysql Pod, and then query the required information through SQL statements, and then use the conversion unit to convert the collected observable information into data that conforms to the OpenTelementry standard. Finally, the transmission unit determines the data type of the converted observable information, which includes three types: Metrics, Logs, and Traces, and transmits the converted observable information to the matching Pipeline in OpenTelementry according to its data type. Pipeline is the core of the data flow. It defines the entire process from data reception to processing and then export. There are many Pipelines in OpenTelementry, and each Pipeline only processes observable information of one data type. The matching Pipeline refers to the Pipeline whose data type of the observable information that can be processed is the same as the data type of the observable information collected. It should be noted here that there are multiple Receivers and each Receiver can collect different observable information. Therefore, multiple Receivers can be combined to collect all aspects of observable data such as Metrics, Logs and Traces, such as obtaining indicator data from the Kubernetes API, extracting information from container logs, or tracing the call relationship between services.

[0059] The name of the data processing module is OpenTelemetry. In Pipeline, OpenTelemetry determines the data processing logic according to MonitorDefinition in CRD API. Specifically, data processing is completed by triggering the corresponding Processor. In other words, the observable information converted in the collector is actually transmitted to the corresponding Processor in Pipeline. The Processor uses streaming processing to process data in real time, including filtering, aggregation, conversion and other operations. There are multiple Processors in Pipeline, and a Processor can generally only perform one operation. For example, Processor 1 focuses on filtering log data, while Processor 2 is only responsible for processing log tags. These multiple Processors are connected in series according to MonitorDefinition to complete the data processing process. This processing method ensures the real-time and accuracy of data, while improving the efficiency of data processing.

[0060] The observable information processed by OpenTelemetry will flow to the corresponding exporter according to the export direction defined in the CRD API. The Exporter is responsible for exporting this data to external systems such as Loki, Prometheus, Grafana or Elasticsearch. These systems can be log storage, indicator storage or other target storage, or display systems, and provide monitoring dashboards, log analysis and performance monitoring functions. In this way, users can view and analyze data in these systems, thereby achieving in-depth insights and real-time monitoring of the operating status of the Kubernetes cluster.

[0061] This embodiment abstracts a complete set of APIs for operating the observability platform through the CRD mechanism of Kubernetes, which can collect, process and export data such as metrics, logs and traces out of the box, and can easily extend customized collection logic and implement custom plug-ins. At the same time, it natively supports the OpenTelementry standard, which helps to simplify the process of data collection and analysis and improve the consistency and comparability of data.

[0062] This embodiment also builds an efficient, flexible and scalable observability platform that can meet users' needs for resource monitoring and data analysis in a Kubernetes environment.

[0063] In some embodiments, Figure 2 As shown, a method for processing observable data is provided, which is applied to an observable platform as described above, and includes the following steps:

[0064] S110, monitoring the change operation of the resources in the cluster, and when a change event is detected, triggering a preset hook function to synchronously update the data structure object of the cluster resources in the memory;

[0065] S120, regularly polling the monitored resources, and determining whether to collect observable information of the target resources according to preset rules, and if so, activating the corresponding collector;

[0066] S130, collect observable information of the target resource through the collector, and transmit it to the corresponding data pipeline of OpenTelemetry according to the data type of the observable information;

[0067] S140, calling the corresponding processor in the data pipeline according to the preset data processing logic to process the observable information, and transmitting the processed observable information to the output device;

[0068] S150, exporting the processed observable information to a target storage or display system through an output device to view and analyze the observable information.

[0069] The K8sObserver module of the observability platform monitors the change operations of resources in the Kubernetes cluster in real time. When it detects that a resource has changed, it triggers the pre-set Hook function corresponding to the change event. The Hook function intercepts the change event and calls the callback function of the corresponding resource change. The callback function will synchronously update the data structure object of the Kubernetes resource in the memory. The Receiver_Creator module periodically polls the resources monitored by K8sObserver and decides whether to collect the observable information of the target resource based on the configuration file or the rules set by the user. If collected, the corresponding Receiver is activated. The activated Receiver calls the scraper interface to interact with the collected Pod to obtain the observable information of the target resource, and converts the observable information into data that conforms to the OpenTelementry standard. Then, according to its data type, the converted observable information is transmitted to the matching Pipeline in OpenTelemetry. The OpenTelemetry framework determines the triggered Processor based on the MonitorDefinition in the CRD API. The Processor processes the converted observable information through streaming processing, and multiple Processors can be connected in series. Finally, the processed observable information is processed according to the CRD. The export direction defined in the API is output to the corresponding Exporter, which then exports it to the target storage or presentation system so that users can view and analyze the observable data.

[0070] The entire process of the method provided in this embodiment is automated, ensuring the accuracy and real-time nature of the data, while also providing a high degree of flexibility and scalability to meet the needs of different users and scenarios.

[0071] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. An observable platform, characterized in that: The observability platform is built on the OpenTelemetry Collector framework and dynamically configures and extends its data processing plug-ins through custom resource definition application programming interfaces, including: The monitoring module is used to monitor the change operations of resources in the cluster, and when a change event is detected, it triggers the preset hook function to synchronously update the data structure object of the cluster resource in the memory; A capture module is used to periodically poll the resources monitored by the monitoring module and determine whether to collect observable information of the target resources according to preset rules, and if so, activate the corresponding collector; The collector is used to collect observable information of the target resource and transmit it to the corresponding data pipeline of OpenTelemetry according to the data type of the observable information; A data processing module, configured to call a corresponding processor in the data pipeline according to a preset data processing logic to process the observable information, and transmit the processed observable information to an output device; The output device is used to export the processed observable information to a target storage or display system to view and analyze the observable information.

2. An observable platform according to claim 1, characterized in that: The monitoring module includes an acquisition unit and a calling unit; The monitoring unit is used to subscribe to the events of the cluster application programming interface service to capture the change operations of the resources in the cluster in real time, and the change operations include adding, updating and deleting; The calling unit is used to trigger a preset hook function to intercept the change event when the monitoring unit detects a change event, and call a callback function corresponding to the resource change to update the data structure object of the resource in the memory.

3. An observable platform according to claim 2, characterized in that: The hook functions include OnChange function, OnAdd function and OnDelete function.

4. The observable platform according to claim 1, characterized in that: The collector includes a collection unit, a conversion unit and a transmission unit; The collection unit is used to call the scraper interface to interact with the Pod to which the target resource belongs, and obtain observable information of the target resource; The conversion unit is used to convert the observable information of the target resource into data that complies with the OpenTelementry standard; The transmission unit is used to transmit the converted observable information to the corresponding data pipeline of OpenTelementry according to the data type of the observable information.

5. An observable platform according to claim 4, characterized in that: There are multiple collectors for collecting observable information of different target resources.

6. An observable platform according to claim 4, characterized in that: The data types include metrics, links, and logs.

7. An observable platform according to claim 4, characterized in that: There are multiple data pipelines, and each data pipeline processes different types of observable information.

8. The observable platform according to claim 1, characterized in that: The preset rules and the preset data processing logic are configured through the application programming interface defined by the custom resource.

9. The observable platform according to claim 1, characterized in that: There are multiple processors, which are connected in series to perform data processing.

10. A method for processing observable data, characterized in that: An observable platform as claimed in any one of claims 1 to 9, comprising the following steps: Monitor the change operations of resources in the cluster, and when a change event is detected, trigger the preset hook function to synchronously update the data structure object of the cluster resource in the memory; Regularly poll the monitored resources and determine whether to collect observable information of the target resources according to preset rules. If so, activate the corresponding collector; Collect observable information of the target resource through the collector, and transmit it to the corresponding data pipeline of OpenTelemetry according to the data type of the observable information; Calling the corresponding processor in the data pipeline according to the preset data processing logic to process the observable information, and transmitting the processed observable information to the output device; The processed observable information is exported to a target storage or presentation system through the exporter to view and analyze the observable information.

Images