Dynamic index generation method based on skywalking

Abstract

The invention discloses a skywalking-based index dynamic generation method, which comprises the following steps of: reading an oal script to obtain a first oal index generation rule, and initializing the first oal index generation rule to a program memory; according to the service configuration information in the intra-monitor service configuration system, circularly updating the service configuration rule in the program cache; analyzing the first oal index generation rule and the service configuration rule, and dynamically generating a java code; compiling a java code, and constructing an index processor; analyzing first Trace data reported by the application into a first Trace object; sending the first Trace data in the first Trace object to a source receiver for data distribution; the source converter receives the first Trace data distributed by the source receiver; and converting the first Trace data into a unified index, and processing and storing the index. The technical problem that filtering rule configuration is not supported is solved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and particularly to a method for dynamically generating metrics based on SkyWalking. Background Art

[0002] In the process of implementing SkyWalking observability in complex enterprise architectures, link (Trace) metrics are usually generated by predefined OAL scripts and pre-loaded filtering rules. This method has the following limitations: It cannot adapt to the differences between different applications: The definitions of whether a link call is successful are not exactly the same among different software services and different endpoints in an enterprise. For example, an HTTP call with a return status code of 200 is considered successful, and a Dubbo call without an error is considered successful. In the actual production process of an enterprise, it is usually considered that a successful call is when the interface returns the expected data, and it is impossible to adapt to the differences of different endpoints through scripts. It does not support configuration of filtering rules: Based on the metric generation rules of SkyWalking, modifying the link filtering rules usually requires first learning the relevant OAL syntax, rewriting the code, and then restarting and going online the service, which violates the stability requirements of the system in an enterprise. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a method for dynamically generating metrics based on SkyWalking to solve at least some of the above technical problems.

[0004] To achieve the above object, the technical solution adopted by the present invention is as follows: A method for dynamically generating metrics based on SkyWalking, comprising the following steps: S1. Predefine an OAL script; start SkyWalking and read the OAL script to obtain a first OAL metric generation rule and initialize the first OAL metric generation rule into the program memory; S2. According to the service configuration information in the infra-monitor service configuration system, cyclically update the service configuration rules in the program cache, and the service configuration rules are integrated with the first OAL metric generation rule to form a second OAL metric generation rule; S3. Parse the second OAL metric generation rule based on the Antlr4 framework, dynamically generate Java code; compile the Java code to build a metric processor; S4. The OAP-Receiver cluster receives the first Trace data reported by the application. The source listener monitors the first Trace data and sends the first Trace data to the source builder for parsing into a first Trace object; the first Trace data in the first Trace object is sent to the source receiver for data distribution; S5. The source converter receives the first Trace data; according to the service configuration rules, the source converter converts the first Trace data into a unified Trace object; according to the first oal metric generation rule, the unified Trace object is converted into a unified metric; S6. The unified metrics are initially aggregated by the metric processor to obtain initially aggregated metric data; S7. The OAP-Receiver cluster aggregates the initially aggregated metric data again to obtain Trace metrics and sends the Trace metrics to the monitoring data persistence processor for storage.

[0005] Further, S2 includes: S21. Obtain service configuration information by operating the metric configuration page of the infra-monitor service configuration system, bind a configuration ID to the service configuration information, and call the change data interface of the infra-monitor service configuration system; S22. The infra-monitor service configuration system updates the service configuration rules in the database according to the service configuration information and sends a data change message to the message queue; S23. Skywalking cyclically receives the data change message sent by the message queue, reads the service configuration rules containing the configuration ID from the database and updates them to the program cache; S24. The service configuration rules in the program cache are merged with the first oal metric generation rule to form a second oal metric generation rule.

[0006] Further, the database is mysql or redis.

[0007] Further, S5 includes: S51. According to the source of the first Trace data, perform resource type tagging on the first Trace data to obtain a second Trace object; S52. Obtain the service configuration information that needs to be matched in the service configuration rules and perform rule matching on the second Trace object; S53. If the matching is successful, a third Trace object is generated; S54. Obtain the configuration ID of the successfully matched service configuration information, set the object ID of the third Trace object calculated by the generation method, and generate a unified Trace object by adding a fixed string to the object ID. S55. The first oal metric generation rule receives the unified Trace object and converts the unified Trace object into a unified metric. S56. Repeat steps S52 - S56 until all the first Trace data is converted into unified metrics.

[0008] Further, in S51, the sources of the first Trace data include: http, rpc, mq, xxlJob.

[0009] Further, the following process is also included in S53: re - define the call status of the third Trace object: in an http call, determine the first call success status according to the http status code of the response interface, and set the business status code returned after the first call success status as the second call success status.

[0010] Further, in S54, the fixed string includes the metric name and metric type.

[0011] Further, the infra - monitor service configuration system is also used for data query, and the data query process is as follows: Input the metric type to be queried on the metric configuration page of the infra - monitor service configuration system; find the corresponding third Trace object according to the metric type, obtain the object ID of the third Trace object, obtain the corresponding configuration ID and generation method according to the object ID, and determine the service configuration information required for the metric type query based on the configuration ID and generation method.

[0012] Compared with the prior art, the present invention has the following beneficial effects: The present invention is scientifically and reasonably designed and convenient to use. By adding the infra - monitor service configuration system, the user can set the service configuration information in the configuration page of the infra - monitor service configuration system according to needs, and then combine the predefined first oal metric generation rule to complete the hot - loading of the metric generation rule, solving the technical problem that the prior art does not support the configuration of filtering rules.

[0013] The present invention combines the service configuration rule and the first oal metric generation rule to achieve the pre - filtering of Trace data and the re - definition of the link call status, solving the problem that different systems in the prior art have different definitions of the link call status.

[0014] The present invention completes the precise positioning and query of target data in a large amount of data by customizing the generation method of the index ID, reducing the query time. Description of the Drawings

[0015] Figure 1 It is a flowchart of the steps of the present invention. Detailed Embodiments

[0016] Glossary: Skywalking is an open-source distributed tracing and application performance monitoring platform; OAL is a domain-specific language (DSL) in Skywalking for querying and analyzing observed data; Antlr4 is a powerful open-source parser generator tool; The OAP-Receiver cluster is a distributed deployment mode of the OAP (Observability Analysis Platform) component in the Skywalking observability platform; Trace data is tracing data in a distributed system; A Trace object is a tool in a programming framework or library for generating and managing Trace data; Infra-monitor is a tool for infrastructure monitoring; MySQL is an open-source relational database management system; Redis is an open-source, high-performance key-value storage database; HTTP is a network protocol; RPC is a network communication protocol; MQ is a message queue; XXL-Job is a lightweight distributed task scheduling platform.

[0017] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0018] In the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation to the present invention. In addition, the terms "first", "second", "third" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance.

[0019] In the description of the present invention, it should be noted that unless otherwise clearly specified and defined, the terms "installed", "connected", "connected to" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; of course, it can also be a mechanical connection or an electrical connection; additionally, it can be directly connected, or indirectly connected through an intermediate medium, or it can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0020] As Figure 1 shown, a method for dynamically generating metrics based on SkyWalking includes the following steps: S1. Pre-define the OAL script; start SkyWalking and read the OAL script to obtain the first OAL metric generation rule and initialize the first OAL metric generation rule to the program memory; S2. Update the service configuration rules in the program cache cyclically according to the service configuration information in the infra-monitor service configuration system. The service configuration rules are integrated with the first OAL metric generation rule to form the second OAL metric generation rule; S3. Parse the second OAL metric generation rule based on the Antlr4 framework to dynamically generate Java code; compile the Java code to build a metric processor; S4. The OAP-Receiver cluster receives the first Trace data reported by the application. The source listener monitors the first Trace data and sends the first Trace data to the source builder for parsing into a first Trace object; the first Trace data in the first Trace object is sent to the source receiver for data distribution; S5. The source converter receives the first Trace data distributed by the source receiver; according to the service configuration rules, the source converter converts the first Trace data into a unified Trace object; according to the first OAL metric generation rule, the unified Trace object is converted into unified metrics; S6. The unified metrics are initially aggregated by the metrics processor to obtain the initially aggregated metric data; S7. The OAP-Receiver cluster re-aggregates the initially aggregated metric data to obtain the Trace metrics, and sends the Trace metrics to the monitoring data persistence processor for storage.

[0021] In this embodiment, a unified Trace object is obtained by performing a series of processes on the first Trace data. The Trace object contains most of the common attributes in the link. After the aggregation of the unified metrics, the metrics that meet the user's defined requirements are finally generated.

[0022] In one embodiment of the method for dynamically generating metrics based on skywalking according to the present invention, S2 includes: S21. Obtain the service configuration information by operating the metrics configuration page of the infra-monitor service configuration system, bind the configuration ID to the service configuration information, and call the change data interface of the infra-monitor service configuration system; wherein, the content of the service configuration information includes but is not limited to the following. Request application: application name; Request method: http or rpc or mq or xxlJob; Request method name: http path or rpc interface or xxlJob executor name or mq-topic name; Request parameters: http-param parameters or body parameters or rpc method parameters or mq message content; Request middleware type: redis or mysql or mq or es or pgsql or mongo; Middleware address: connection address; Request duration; Success code; During the matching process, determine the required service configuration information according to the service needs; S22. The infra-monitor service configuration system updates the service configuration rules in the database according to the service configuration information and sends a data change message to the message queue; S23. Skywalking repeatedly receives the data change message sent by the message queue, reads the service configuration rules containing the configuration ID from the database and updates them to the program cache; wherein, the database is mysql or redis; S24. The service configuration rules in the program cache are fused with the first oal metric generation rule to form the second oal metric generation rule.

[0023] In this embodiment, SkyWalking itself needs to load the expressions in the OAL file in the configuration file during startup, then generate relevant code, and finally implement the generation of different metric data. The infra-monitor system provides an HTTP interface for the web page to perform business metric configuration operations. Users only need to operate through business configuration. The infra-monitor business configuration system combines with S5 and can generate different metrics according to user needs.

[0024] The present invention relates to a method for dynamically generating metrics based on SkyWalking. In one embodiment, S5 includes: S51. Label the resource type of the first Trace data according to the source of the first Trace data to obtain a second Trace object; S52. Obtain the business configuration information to be matched in the business configuration rule and perform rule matching on the second Trace object; S53. If the matching is successful, generate a third Trace object; S54. Obtain the configuration ID of the successfully matched business configuration information, set the generation method to calculate the object ID of the third Trace object, and generate a unified Trace object by adding a fixed string to the object ID; S55. The first OAL metric generation rule receives the unified Trace object and converts the unified Trace object into a unified metric; S56. Repeat steps S52 - S56 until all the first Trace data is converted into unified metrics.

[0025] Preferably, in S51, the sources of the first Trace data include: http, rpc, mq, xxlJob.

[0026] Preferably, the following process is further included in S53: Redefine the call status of the third Trace object: In an HTTP call, determine the first call success status according to the HTTP status code of the response interface, and set the business status code to the second call success status after the first call success status.

[0027] Preferably, in S54, the fixed string includes the metric name and the metric type.

[0028] The business configuration rule and the first OAL metric generation rule are in an upstream and downstream relationship in the link of the dynamic metric generation rule. The unified Trace object is generated by the business configuration rule, and then the first OAL metric generation rule converts the unified Trace object into a unified metric. The specific steps include: (1)Filter the link data by one or more of the request method, request parameters, middleware type connected, connection address, and request duration in the service configuration rules, and then re-define the call status of the link through the http status code, rpc status code, business status code in the response service data, or response duration, so as to generate a unified Trace object from the link data.

[0029] (2)The first oal metric generation rule receives the unified Trace object and converts it into unified metrics through predefined unified metric generation logic (native skywalking capabilities), such as: cpm() --- calls per minute; percent() --- percentage of the anchor value in the total; longAvg() --- average value of the data; count() --- total number of data items; sum() --- sum of data values, etc.

[0030] An embodiment of the method for dynamically generating metrics based on skywalking in the present invention, the infra-monitor service configuration system has a data query function, and the data query process is as follows: Enter the metric type to be queried on the metric configuration page of the infra-monitor service configuration system; find the corresponding third Trace object according to the metric type, obtain the object ID of the third Trace object, obtain the corresponding configuration ID and generation method according to the object ID, and determine the service configuration information required for metric type query based on the configuration ID and generation method. In this embodiment, when the user performs data query, only the metric type required by the user needs to be known. The metric type is set as a fixed string in the unified Trace object. Based on the unified Trace object, the configuration ID and generation method are obtained. This embodiment can accurately locate and query the target data in a large amount of data, reducing the query time.

[0031] Finally, it should be noted that the above embodiments are only preferred embodiments of the present invention to illustrate the technical solutions of the present invention, rather than limiting it, and certainly not limiting the patent scope of the present invention; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the various embodiments of the present invention; that is to say, any meaningless changes or polishing made in the main design concept and spirit of the present invention, as long as the technical problems solved are still the same as those of the present invention, should be included in the protection scope of the present invention; in addition, directly or indirectly applying the technical solutions of the present invention to other related technical fields shall similarly be included in the patent protection scope of the present invention.

Claims

1. A method for dynamically generating indicators based on skywalking, characterized in that: The following steps are involved: S1, predefine oal script; start skywalking and read the oal script, obtain the first oal indicator generation rule and initialize the first oal indicator generation rule to the program memory; S2, according to the business configuration information in the infra-monitor business configuration system, the business configuration rules in the program cache are cyclically updated, and the business configuration rules are merged with the first oal indicator generation rule to form a second oal indicator generation rule; S3, based on the Antlr4 framework, parse the second oal indicator generation rules and dynamically generate java code; Compile java code and build indicator processor; S4, the OAP-Receiver cluster receives the first Trace data reported by the application, the source listener listens to the first Trace data and sends the first Trace data to the source builder to parse it into a first Trace object; the first Trace data in the first Trace object is sent to the source receiver for data distribution; S5. The source converter receives the first Trace data; according to the business configuration rule, the source converter converts the first Trace data into a unified Trace object; according to the first oal indicator generation rule, the unified Trace object is converted into a unified indicator; S6. The unified indicators are preliminarily aggregated through the indicator processor to obtain preliminary aggregated indicator data; S7. The OAP-Receiver cluster aggregates the preliminary aggregated indicator data again to obtain Trace indicators, and sends the Trace indicators to the monitoring data persistence processor for storage.

2. A method for dynamically generating indicators based on skywalking according to claim 1, characterized in that S2 include: S21. Obtain business configuration information by operating the indicator configuration page of the infra-monitor business configuration system, bind the configuration ID to the business configuration information, and call the change data interface of the infra-monitor business configuration system; S22, the infra-monitor service configuration system updates the service configuration rules in the database according to the service configuration information and sends a data change message to the message queue; S23, skywalking loop receives the data change message sent by the message queue, reads the business configuration rules containing the configuration ID from the database and updates them to the program cache; S24. The business configuration rule in the program cache is merged with the first oal indicator generation rule to form a second oal indicator generation rule.

3. A method for dynamically generating indicators based on skywalking according to claim 2, characterized in that: The database is mysql or redis.

4. The method for dynamically generating indicators based on skywalking according to claim 2, characterized in that S5 include: S51. According to the source of the first Trace data, the first Trace data is marked with a resource type to obtain a second Trace object; S52, obtaining the business configuration information that needs to be matched in the business configuration rule, and performing rule matching on the second Trace object; S53, if the match is successful, generate a third Trace object; S54, obtaining the configuration ID of the successfully matched business configuration information, setting the generation method to calculate and obtain the object ID of the third Trace object, and adding a fixed string to the object ID to generate a unified Trace object; S55, the first oal indicator generation rule receives a unified Trace object and converts the unified Trace object into a unified indicator; S56. Repeat steps S52-S56 until all first Trace data are converted into a unified indicator.

5. The method for dynamically generating indicators based on skywalking according to claim 4, characterized in that: In S51, sources of the first Trace data include: http, rpc, mq, and xxlJob.

6. The method for dynamically generating indicators based on skywalking according to claim 4, characterized in that: S53 also includes the following process: re-defining the call status of the third Trace object: in the http call, determining the first call success status according to the response interface http status code, and setting the business status code returned after the first call success status to be the second call success status.

7. The method for dynamically generating indicators based on skywalking according to claim 4, characterized in that: In S54, the fixed character string includes the indicator name and the indicator type.

8. The method for dynamically generating indicators based on skywalking according to claim 4, characterized in that: The infra-monitor service configuration system is also used for data query. The data query process is as follows: Enter the indicator type to be queried on the indicator configuration page of the infra-monitor business configuration system; find the corresponding third Trace object according to the indicator type, obtain the object ID of the third Trace object, obtain the corresponding configuration ID and generation method according to the object ID, and determine the business configuration information required for the indicator type query based on the configuration ID and generation method.

Images