A method for dynamic generation of indicators based on skywalking

Abstract

The present invention discloses a method for dynamically generating indicators based on SkyWalking, comprising: reading an OAL script to obtain a first OAL indicator generation rule and initializing it to program memory; cyclically updating the business configuration rule in the program cache based on business configuration information in an infra-monitor business configuration system; parsing the first OAL indicator generation rule and the business configuration rule to dynamically generate Java code; compiling the Java code to construct an indicator processor; parsing first Trace data reported by an application into a first Trace object; sending the first Trace data in the first Trace object to a source receiver for data distribution; a source converter receiving the first Trace data distributed by the source receiver; converting the first Trace data into a unified indicator, and processing and storing the indicator. The present invention solves the technical problem of not supporting configurable filtering rules.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method for dynamically generating indicators based on skywalking. Background Art

[0002] When implementing SkyWalking observability in complex enterprise architectures, trace metrics are typically generated using predefined OAL scripts and preloaded filtering rules. This approach has the following limitations:

[0003] Unable to adapt to differences between different applications: Different software services and endpoints in an enterprise have different definitions of whether a link call is successful. For example, an HTTP call is considered successful if the return status code is 200, and a Dubbo call is considered successful if there is no error. However, in actual production processes, enterprises generally consider a call to be successful only if the interface returns the expected data. Scripts cannot adapt to the differences between different endpoints.

[0004] Filter rule configuration is not supported: Based on SkyWalking's indicator generation rules, modifying link filtering rules usually requires learning the relevant OAL syntax, rewriting the code, and then restarting the online service, which is contrary to the stability required by the enterprise for the system. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method for dynamically generating indicators based on skywalking to at least solve some of the above technical problems.

[0006] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A method for dynamically generating indicators based on skywalking, comprising the following steps:

[0008] S1, predefine oal script; start skywalking and read the oal script, obtain the first oal indicator generation rule and initialize the first oal indicator generation rule to the program memory;

[0009] S2. The business configuration rules in the program cache are cyclically updated according to the business configuration information in the infra-monitor business configuration system, and the business configuration rules are merged with the first oal indicator generation rule to form a second oal indicator generation rule;

[0010] S3. Parse the second oal indicator generation rules based on the Antlr4 framework and dynamically generate Java code; compile the Java code and build the indicator processor;

[0011] S4. The OAP-Receiver cluster receives the first Trace data reported by the application. The source listener listens to the first Trace data and sends the first Trace data to the source builder for parsing into a first Trace object. The first Trace data in the first Trace object is sent to the source receiver for data distribution.

[0012] S5. The source converter receives the first Trace data; according to the business configuration rule, the source converter converts the first Trace data into a unified Trace object; according to the first oal indicator generation rule, the unified Trace object is converted into a unified indicator;

[0013] S6. The unified indicators are preliminarily aggregated through the indicator processor to obtain preliminary aggregated indicator data;

[0014] S7. The OAP-Receiver cluster aggregates the preliminary aggregated indicator data again to obtain Trace indicators, and sends the Trace indicators to the monitoring data persistence processor for storage.

[0015] Furthermore, S2 includes:

[0016] S21. Obtain business configuration information by operating the indicator configuration page of the infra-monitor business configuration system, bind the configuration ID to the business configuration information, and call the change data interface of the infra-monitor business configuration system;

[0017] S22, the infra-monitor service configuration system updates the service configuration rules in the database according to the service configuration information and sends a data change message to the message queue;

[0018] S23, skywalking loop receives the data change message sent by the message queue, reads the business configuration rules containing the configuration ID from the database and updates them to the program cache;

[0019] S24: The business configuration rule in the program cache is merged with the first oal indicator generation rule to form a second oal indicator generation rule.

[0020] Furthermore, the database is MySQL or Redis.

[0021] Furthermore, S5 includes:

[0022] S51. Based on the source of the first Trace data, label the first Trace data with a resource type to obtain a second Trace object.

[0023] S52: Obtain the service configuration information that needs to be matched in the service configuration rule, and perform rule matching on the second Trace object;

[0024] S53. If the match is successful, generate a third Trace object;

[0025] S54. Obtain the configuration ID of the successfully matched business configuration information, set the generation method to calculate and obtain the object ID of the third Trace object, and add a fixed string to the object ID to generate a unified Trace object;

[0026] S55. The first oal indicator generation rule receives the unified Trace object and converts the unified Trace object into a unified indicator;

[0027] S56. Repeat steps S52-S56 until all first Trace data are converted into a unified indicator.

[0028] Furthermore, in S51, sources of the first Trace data include: http, rpc, mq, and xxlJob.

[0029] Furthermore, S53 also includes the following process: secondary definition of the call status of the third Trace object: in the http call, the first call success status is determined according to the response interface http status code, and the business status code returned after the first call success status is set to the second call success status.

[0030] Furthermore, in S54, the fixed character string includes the indicator name and the indicator type.

[0031] Furthermore, the infra-monitor service configuration system is also used for data query. The data query process is as follows:

[0032] Enter the indicator type to be queried on the indicator configuration page of the infra-monitor business configuration system; find the corresponding third Trace object based on the indicator type, obtain the object ID of the third Trace object, obtain the corresponding configuration ID and generation method based on the object ID, and determine the business configuration information required for the indicator type query based on the configuration ID and generation method.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The present invention is scientifically and rationally designed and easy to use. By adding an infra-monitor business configuration system, the present invention allows users to set the business configuration information in the configuration page of the infra-monitor business configuration system as needed, and then combines it with the predefined first oal indicator generation rule to complete the hot loading of the indicator generation rule, solving the technical problem that the existing technology does not support the configuration of filtering rules.

[0035] The present invention combines business configuration rules and first oal indicator generation rules to achieve early filtering of Trace data and secondary definition of link call status, solving the problem of different definitions of link call status between different systems in the prior art.

[0036] The present invention achieves accurate positioning and query of target data in massive data by generating a custom indicator ID, thus reducing query time. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flow chart of the steps of the present invention. DETAILED DESCRIPTION

[0038] Explanation of terms:

[0039] Skywalking is an open source distributed tracing and application performance monitoring platform;

[0040] oal is a domain-specific language (DSL) used in skywalking to query and analyze observation data;

[0041] Antlr4 is a powerful open source parser generation tool;

[0042] The OAP-Receiver cluster is a distributed deployment mode of the OAP (Observability Analysis Platform) component in the SkyWalking observability platform;

[0043] Trace data is tracking data in distributed systems;

[0044] Trace objects are tools used in programming frameworks or libraries to generate and manage trace data;

[0045] infra-monitor is a tool for infrastructure monitoring;

[0046] MySQL is an open source relational database management system;

[0047] Redis is an open source, high-performance key-value storage database;

[0048] HTTP is a network protocol;

[0049] RPC is a network communication protocol;

[0050] mq is a message queue;

[0051] xxlJob is a lightweight distributed task scheduling platform.

[0052] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0053] In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have a specific orientation or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0054] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; of course, they may also refer to mechanical connections or electrical connections; in addition, they may refer to direct connections, indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0055] like Figure 1 As shown, a method for dynamically generating indicators based on skywalking includes the following steps:

[0056] S1, predefine oal script; start skywalking and read the oal script, obtain the first oal indicator generation rule and initialize the first oal indicator generation rule to the program memory;

[0057] S2. The business configuration rules in the program cache are cyclically updated according to the business configuration information in the infra-monitor business configuration system, and the business configuration rules are merged with the first oal indicator generation rule to form a second oal indicator generation rule;

[0058] S3. Parse the second oal indicator generation rules based on the Antlr4 framework and dynamically generate Java code; compile the Java code and build the indicator processor;

[0059] S4. The OAP-Receiver cluster receives the first Trace data reported by the application. The source listener listens to the first Trace data and sends the first Trace data to the source builder for parsing into a first Trace object. The first Trace data in the first Trace object is sent to the source receiver for data distribution.

[0060] S5. The source converter receives the first Trace data distributed by the source receiver; according to the business configuration rule, the source converter converts the first Trace data into a unified Trace object; according to the first oal indicator generation rule, the unified Trace object is converted into a unified indicator;

[0061] S6. The unified indicators are preliminarily aggregated through the indicator processor to obtain preliminary aggregated indicator data;

[0062] S7. The OAP-Receiver cluster aggregates the preliminary aggregated indicator data again to obtain Trace indicators, and sends the Trace indicators to the monitoring data persistence processor for storage.

[0063] In this embodiment, a unified Trace object is obtained by performing a series of processing on the first Trace data. The Trace object contains most common attributes in the link, and then undergoes unified indicator aggregation to finally generate indicators that meet user requirements.

[0064] The present invention is based on the method for dynamically generating indicators of skywalking. In one embodiment, S2 includes:

[0065] S21. Obtain the business configuration information by operating the indicator configuration page of the infra-monitor business configuration system, bind the configuration ID to the business configuration information, and call the change data interface of the infra-monitor business configuration system; wherein, the content of the business configuration information includes but is not limited to the following. Request application: application name; request method: http or rpc or mq or xxlJob; request method name: http path or rpc interface or xxlJob executor name or mq-topic name; request parameter: http-param parameter or body parameter or rpc method parameter or mq message content; request middleware type: redis or mysql or mq or es or pgsql or mongo; middleware address: connection address; request duration; success code; during the matching process, determine the required business configuration information according to business needs;

[0066] S22, the infra-monitor service configuration system updates the service configuration rules in the database according to the service configuration information and sends a data change message to the message queue;

[0067] S23, skywalking loop receives the data change message sent by the message queue, reads the business configuration rules containing the configuration ID from the database and updates them to the program cache; the database is MySQL or Redis;

[0068] S24: The business configuration rule in the program cache is merged with the first oal indicator generation rule to form a second oal indicator generation rule.

[0069] In this embodiment, SkyWalking itself needs to load the expressions in the oal file in the configuration file at startup, then generate the relevant code to finally generate different indicator data. The infra-monitor system provides an HTTP interface to the web page for configuring business indicators. Users only need to operate through business configuration. The infra-monitor business configuration system, combined with S5, can generate different indicators according to user needs.

[0070] The present invention is based on the method for dynamically generating indicators of skywalking. In one embodiment, S5 includes:

[0071] S51. Based on the source of the first Trace data, the resource type is marked on the first Trace data to obtain a second Trace object; S52. The business configuration information that needs to be matched in the business configuration rule is obtained, and the rule matching is performed on the second Trace object; S53. If the match is successful, a third Trace object is generated; S54. The configuration ID of the successfully matched business configuration information is obtained, and the generation method is set to calculate the object ID of the third Trace object. A fixed string is added to the object ID to generate a unified Trace object; S55. The first oal indicator generation rule receives the unified Trace object and converts the unified Trace object into a unified indicator; S56. Steps S52-S56 are repeated until all the first Trace data are converted into a unified indicator.

[0072] Preferably, in S51, the source of the first Trace data includes: http, rpc, mq, and xxlJob.

[0073] Preferably, S53 also includes the following process: secondary definition of the call status of the third Trace object: in the http call, the first call success status is determined according to the response interface http status code, and the business status code returned after the first call success status is set to the second call success status.

[0074] Preferably, in S54, the fixed character string includes an indicator name and an indicator type.

[0075] The business configuration rule and the first oal indicator generation rule are in an upstream and downstream relationship in the dynamic indicator generation rule chain. The business configuration rule generates a unified Trace object, and then the first oal indicator generation rule converts the unified Trace object into a unified indicator. The specific steps include:

[0076] (1) Filter link data by limiting one or more of the request method, request parameters, connected middleware type, connection address, and request duration in the business configuration rules, and then redefine the link call status by the http status code or rpc status code or the business status code or response duration in the response business data, so as to achieve the purpose of generating a unified Trace object from the link data.

[0077] (2) The first oal indicator generation rule receives a unified Trace object and converts it into a unified indicator through the pre-defined unified indicator generation logic (SkyWalking native capability), such as: cpm()---call volume per minute; percent()---anchor value as a percentage of the total; longAvg()---average value of the data; count()---total number of data items; sum()----sum of data values ​​and other indicator calculation methods.

[0078] The present invention is based on the dynamic generation method of indicators of skywalking. In one embodiment, the infra-monitor business configuration system has a data query function. The data query process is as follows:

[0079] Enter the indicator type you want to query on the indicator configuration page of the infra-monitor business configuration system; find the corresponding third Trace object based on the indicator type, obtain the object ID of the third Trace object, obtain the corresponding configuration ID and generation method based on the object ID, and determine the business configuration information required for the indicator type query based on the configuration ID and generation method. In this embodiment, when a user performs a data query, they only need to know the indicator type they need. The indicator type is set in a unified Trace object using a fixed string. The configuration ID and generation method are obtained based on the unified Trace object. This embodiment can accurately locate and query target data in massive amounts of data, reducing query time.

[0080] Finally, it should be noted that the above embodiments are merely preferred embodiments of the present invention and are intended to illustrate the technical solutions of the present invention, rather than limiting them, and certainly not limiting the patent scope of the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they may still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention. In other words, any changes or embellishments made to the main design concept and spirit of the present invention that have no substantive significance, provided that the technical problems they solve are still consistent with those of the present invention, should be included in the protection scope of the present invention. In addition, the direct or indirect application of the technical solutions of the present invention to other related technical fields should also be included in the patent protection scope of the present invention.

Claims

1. A method for dynamically generating indicators based on skywalking, characterized in that: The following steps are involved: S1, predefine oal script; start skywalking and read the oal script, obtain the first oal indicator generation rule and initialize the first oal indicator generation rule to the program memory; S2. The business configuration rules in the program cache are cyclically updated according to the business configuration information in the infra-monitor business configuration system, and the business configuration rules are merged with the first oal indicator generation rule to form a second oal indicator generation rule; S3. Parse the second oal indicator generation rules based on the Antlr4 framework and dynamically generate Java code; Compile Java code and build indicator processor; S4. The OAP-Receiver cluster receives the first Trace data reported by the application. The source listener listens to the first Trace data and sends the first Trace data to the source builder for parsing into a first Trace object. The first Trace data in the first Trace object is sent to the source receiver for data distribution. S5. The source converter receives the first Trace data; according to the business configuration rule, the source converter converts the first Trace data into a unified Trace object; according to the first oal indicator generation rule, the unified Trace object is converted into a unified indicator; S6. The unified indicators are preliminarily aggregated through the indicator processor to obtain preliminary aggregated indicator data; S7. The OAP-Receiver cluster aggregates the preliminary aggregated indicator data again to obtain trace indicators, and sends the trace indicators to the monitoring data persistence processor for storage; S2 includes: S21. Obtain business configuration information by operating the indicator configuration page of the infra-monitor business configuration system, bind the configuration ID to the business configuration information, and call the change data interface of the infra-monitor business configuration system; S22, the infra-monitor service configuration system updates the service configuration rules in the database according to the service configuration information and sends a data change message to the message queue; S23, skywalking loop receives the data change message sent by the message queue, reads the business configuration rules containing the configuration ID from the database and updates them to the program cache; S24: The business configuration rule in the program cache is merged with the first oal indicator generation rule to form a second oal indicator generation rule.

2. A method for dynamically generating indicators based on skywalking according to claim 1, characterized in that, The database is MySQL or Redis.

3. The method for dynamically generating indicators based on skywalking according to claim 1, wherein S5 include: S51. Based on the source of the first Trace data, label the first Trace data with a resource type to obtain a second Trace object. S52: Obtain the service configuration information that needs to be matched in the service configuration rule, and perform rule matching on the second Trace object; S53. If the match is successful, generate a third Trace object; S54. Obtain the configuration ID of the successfully matched business configuration information, set the generation method to calculate and obtain the object ID of the third Trace object, and add a fixed string to the object ID to generate a unified Trace object; S55. The first oal indicator generation rule receives the unified Trace object and converts the unified Trace object into a unified indicator; S56. Repeat steps S52-S56 until all first Trace data are converted into a unified indicator.

4. The method for dynamically generating indicators based on skywalking according to claim 3, characterized in that: In S51 , sources of the first Trace data include: http, rpc, mq, and xxlJob.

5. The method for dynamically generating indicators based on skywalking according to claim 3, characterized in that: S53 also includes the following process: secondary definition of the call status of the third Trace object: in the http call, the first call success status is determined according to the response interface http status code, and the business status code returned after the first call success status is set to the second call success status.

6. The method for dynamically generating indicators based on skywalking according to claim 3, characterized in that: In S54, the fixed character string includes the indicator name and the indicator type.

7. The method for dynamically generating indicators based on skywalking according to claim 3, characterized in that: The infra-monitor service configuration system is also used for data query. The data query process is as follows: Enter the indicator type to be queried on the indicator configuration page of the infra-monitor business configuration system; find the corresponding third Trace object based on the indicator type, obtain the object ID of the third Trace object, obtain the corresponding configuration ID and generation method based on the object ID, and determine the business configuration information required for the indicator type query based on the configuration ID and generation method.

Images