A Custom Metric Generation Method Based on a Real-Time Streaming Framework

Abstract

The present invention discloses a method for generating custom indicators based on a real-time stream framework, comprising receiving indicator data to form an indicator data stream; receiving indicator rules to form an indicator rule stream; converting the indicator rule stream into an indicator rule broadcast stream; merging the indicator data stream with the indicator rule broadcast stream, matching indicator data that meets different indicator rules, and determining whether the matched indicator data is single indicator data or associated indicator data; calculating the single indicator data / associated indicator data using a calculation expression in the indicator rule to obtain a single indicator value / associated indicator value; calculating the single indicator value / associated indicator value based on the indicator calculation type in the indicator rule to obtain an indicator value; merging each indicator value and the corresponding indicator name, event association identifier, and indicator time of the indicator data into an alarm indicator and reporting it to a database. The present invention provides a new processing logic based on the Flink framework to realize the generation of custom indicators and alarm behavior.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method for generating custom indicators based on a real-time stream framework. Background Art

[0002] In the internet world, when backend services, such as sending text messages, take too long or cross-service consumption data is lost, developers need to promptly detect system anomalies. If the issue is with the code, they need to debug or optimize the code to improve system stability and smoothness.

[0003] Currently, we primarily calculate and report the execution time for different scenarios to the monitoring platform for unified alert processing. For example, to determine the execution time of a function, we need to print the start and end times of the program function, calculate the execution time, and then report it to the monitoring platform. If the execution time exceeds the threshold configured by the monitoring platform, an alert will be issued to notify R&D that the execution time is too long and requires attention.

[0004] Existing technologies typically calculate specific metrics within business code, such as time consumption, and then generate alerts directly or report them to a third-party monitoring platform for alert processing. Alternatively, third-party alerting programs can be used to generate metrics using specific methods or annotations, and then report them to the monitoring platform for alert processing. There's no unified platform to handle the entire logic for alerts related to data loss across systems, requiring R&D personnel to implement specific code logic at both the data generation and data processing points. Summary of the Invention

[0005] The purpose of the present invention is to provide a custom indicator generation method based on a real-time stream framework to solve the technical problem that the existing method cannot realize the calculation of cross-system correlation data indicators and data loss indicators.

[0006] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A method for generating custom indicators based on a real-time streaming framework includes the following steps:

[0008] S1. Receive indicator data reported by the interface and form an indicator data stream through Flink; the indicator data includes indicator time;

[0009] S2. Receive the indicator rules pushed by the server to Kafka to form an indicator rule stream; the indicator rules are customized and include indicator name, matching event, event association identifier, calculation expression and indicator calculation type;

[0010] S3. Convert the indicator rule stream into the indicator rule broadcast stream and define the broadcast state;

[0011] S4. Merge the indicator data stream with the indicator rule broadcast stream, match the indicator data in the indicator data stream with the indicator rules in the indicator rule stream through the broadcast state, match the indicator data that meets different indicator rules, and determine whether the matched indicator data is single indicator data or associated indicator data;

[0012] S5. Calculate the single indicator data / associated indicator data using the calculation expression in the indicator rule to obtain the single indicator value / associated indicator value, and merge the single indicator value / associated indicator value and the corresponding indicator rule into a single indicator event and rule merged flow / associated indicator event and rule merged flow;

[0013] S6. Determine the indicator calculation type of the single indicator event and rule merge flow / the associated indicator event and rule merge flow based on the indicator rule, calculate the single indicator value / the associated indicator value based on the indicator calculation type, and obtain the reported indicator value;

[0014] S7. Combine each reported indicator value and the corresponding indicator name, event association identifier, and indicator time into an alarm indicator and report it to the database.

[0015] Furthermore, the definition of the broadcast state includes: calling the broadcast method of the indicator rule stream to set the broadcast state, and the broadcast method will also return the indicator rule broadcast stream.

[0016] Furthermore, in S4, the indicator rules in the indicator rule stream are stored in the broadcast state through the indicator rule broadcast stream, the indicator data stream obtains the indicator rules in the broadcast state, and each indicator data in the indicator data stream is matched with the indicator rule; when the matching event of the indicator rule includes one, the indicator data that meets the corresponding indicator rule is single indicator data; when the matching event of the indicator rule includes multiple, the indicator data that meets the corresponding indicator rule is associated indicator data.

[0017] Furthermore, in said S5, the calculation method of the single indicator data includes: for each successfully matched single indicator data, performing calculation according to the calculation expression of the corresponding indicator rule to obtain the single indicator value.

[0018] Furthermore, in said S5, the calculation method of the correlation indicator data includes: S51, for each successfully matched correlation indicator data, grouping according to the event correlation identifier of the corresponding indicator rule, each group is used to cache the correlation indicator data of the same event correlation identifier; S52, calculating the correlation indicator value of the correlation indicator data in each group, the correlation indicator value including the first correlation indicator value and / or the second correlation indicator value;

[0019] The calculation method of the first association index value and the second association index value is: during the grouping process, the first association index value is obtained by caching the first association index data in the group, and the first association index value is equal to -1; after all the association index data of each group are cached in the group, all the association index data in the current group are calculated according to the calculation expression of the corresponding matching index rule to obtain the second association index value.

[0020] Furthermore, in the calculation of the associated indicator data, grouping is performed through Flink's key grouping.

[0021] Furthermore, in S6, the indicator calculation type includes a fixed value and an aggregate value; when the indicator calculation type is a fixed value, the single indicator value / associated indicator value is used as the reported indicator value; when the indicator calculation type is an aggregate value, each single indicator value / associated indicator value is accumulated to obtain the reported indicator value.

[0022] Furthermore, in S7, a rolling time window is set, and the alarm indicator is reported within each rolling time window.

[0023] Furthermore, the database is a Prometheus database.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] This paper, based on the Flink framework, provides a new processing logic to enable custom metric generation and alerting. This paper is applicable to different systems. Metrics data across different systems is automatically generated and alerts configured using metric rules, eliminating the need for each system to perform scenario-specific logical processing. This paper prioritizes generating a -1 indicator value for associated metric data and configures alerts by querying this value, thus addressing the issue of missing alerts for consumption metrics. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0027] Explanation of terms:

[0028] Flink is an open source distributed stream processing framework;

[0029] Kafka is a distributed stream processing platform;

[0030] The Prometheus database is an open source monitoring system and time series database;

[0031] The broadcast method is usually used to broadcast a data stream to all downstream tasks so that these tasks can share the data in the broadcast stream;

[0032] RichSourceFunction is an enhanced data source class in Flink, used to implement custom data sources;

[0033] The run method is the core method in RichSourceFunction, which is used to generate data and send it downstream;

[0034] The addSource method is a core method of StreamExecutionEnvironment, which is used to add a data source to a Flink stream processing job.

[0035] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0036] like Figure 1 As shown, the present invention provides a method for generating a custom indicator based on a real-time stream framework, comprising the following steps:

[0037] S1. Receive indicator data reported by the interface and form an indicator data stream through Flink; the indicator data includes indicator time;

[0038] S2. Receive the indicator rules pushed by the server to Kafka to form an indicator rule stream; the indicator rules are customized and include indicator name, matching event, event association identifier, calculation expression and indicator calculation type;

[0039] S3. Convert the indicator rule stream into the indicator rule broadcast stream and define the broadcast state;

[0040] S4. Merge the indicator data stream with the indicator rule broadcast stream, match the indicator data in the indicator data stream with the indicator rules in the indicator rule stream through the broadcast state, match the indicator data that meets different indicator rules, and determine whether the matched indicator data is single indicator data or associated indicator data;

[0041] S5. Calculate the single indicator data / associated indicator data using the calculation expression in the indicator rule to obtain the single indicator value / associated indicator value, and merge the single indicator value / associated indicator value and the corresponding indicator rule into a single indicator event and rule merged flow / associated indicator event and rule merged flow;

[0042] S6. Determine the indicator calculation type of the single indicator event and rule merge flow / the associated indicator event and rule merge flow based on the indicator rule, calculate the single indicator value / the associated indicator value based on the indicator calculation type, and obtain the reported indicator value;

[0043] S7. Combine each reported indicator value and the corresponding indicator name, event association identifier, and indicator time into an alarm indicator and report it to the database.

[0044] In S1 described in the present invention, the indicator data reported by the receiving interface is reported in a form that supports indicator data received by the Hypertext Transfer Protocol and the message queue. By inheriting the RichSourceFunction class, its built-in run method is mainly implemented to define the logic of receiving indicators, and then the indicator data stream is formed through the built-in addSource method of flink.

[0045] In S2, based on the RichSourceFunction class, the metric rules pushed from the server to Kafka are received. These metric rules are pre-configured and include the metric name, matching event, event-related metric (i.e., the unique identifier field (ID) in the corresponding metric data), calculation expression, and metric calculation type. This is achieved by implementing the RichSourceFunction built-in run method to define the logic for receiving metric rules. Then, the metric rule flow is formed using Flink's built-in addSource method.

[0046] In S3 of the present invention, the indicator rule stream is converted into the indicator rule broadcast stream, specifically by calling the broadcast method. In addition, the broadcast state is defined: the broadcast method of the indicator rule stream is called to set the broadcast state, and the broadcast method also returns the indicator rule broadcast stream.

[0047] In S4 described in the present invention, the indicator data stream is merged with the indicator rule broadcast stream, and the indicator rule broadcast stream is processed in processBroadcastElement by inheriting BroadcastProcessFunction. The indicator rules in the indicator rule stream are stored in the broadcast state, and then the indicator data stream is processed in processElement. The indicator data stream obtains the indicator rules in the broadcast state through ctx.getBroadcastState, and each indicator data in the indicator data stream is matched with the indicator rule. The indicator data that meets different indicator rules is matched, and it is determined whether the matched indicator data is single indicator data or associated indicator data. When the matching event of the indicator rule contains one, the indicator data that meets the corresponding indicator rule is single indicator data; when the matching event of the indicator rule contains multiple, the indicator data that meets the corresponding indicator rule is associated indicator data.

[0048] In S5 of the present invention, the calculation expressions in the matched indicator rules are used to calculate the single indicator data and the associated indicator data to obtain the single indicator value and the associated indicator value. All single indicator data and corresponding single indicator values and indicator rules are merged into a single indicator event and rule merge flow, and all associated indicator data and corresponding associated indicator values and indicator rules are merged into a associated indicator event and rule merge flow.

[0049] In S5, the calculation method of the single indicator data includes: for each successfully matched single indicator data, performing calculation according to the calculation expression of the corresponding indicator rule to obtain the single indicator value.

[0050] In S5, the calculation method of the associated indicator data includes: for each successfully matched associated indicator data, grouping according to the event association identifier (ID) of the corresponding indicator rule, each group is used to cache the associated indicator data of the same event association identifier (ID), preferably grouping by keyby of Flink; in the grouping process, obtaining a first associated indicator value by the first associated indicator data cached in the group, and the first associated indicator value is equal to -1; after the remaining associated indicator data is cached in the group, all the associated indicator data in the current group are calculated according to the calculation expression of the corresponding matching indicator rule to obtain a second associated indicator value.

[0051] For the associated indicator data, a first associated indicator value equal to -1 is first generated, so that the indicator configuration alarm can be queried by the first associated indicator value to fill the gap of the consumption indicator loss alarm.

[0052] In S6 described in the present invention, the indicator calculation type is judged for the single indicator event and rule merged flow, and the associated indicator event and rule merged flow, respectively, and the corresponding indicator values in the single indicator event and rule merged flow, and the associated indicator event and rule merged flow are calculated according to the indicator calculation type. For example, for an indicator calculation type of cumulative indicator, flink's flatMap is used in conjunction with a rolling time window (one time window per second) to accumulate the indicator values in each window, and the indicator value of the maximum time in the window is taken to ensure that at most one indicator is generated per second for the same indicator, and then passed to the downstream; for an indicator type of fixed indicator, since the fixed indicator is a fixed value, no additional calculation is required, and the corresponding single indicator value / associated indicator value can be directly passed to the downstream as the indicator value.

[0053] The S7 described in the present invention is to merge each indicator value and the indicator name, event association identifier and indicator time of the indicator rule in the corresponding single indicator event and rule merge flow / associated indicator event and rule merge flow into an alarm indicator and report it to the database. Preferably, a rolling time window is set, and the alarm indicator is reported in each rolling time window. More preferably, the rolling time window is set to 5 seconds, and the database is a Prometheus database. If the alarm indicator containing the first associated indicator value equal to -1 is obtained first and reported to the database, and then the latest alarm indicator containing the second associated indicator value is generated and reported to the database: first query whether the alarm indicator has been reported based on the indicator name. If it has been reported and the indicator value is -1, delete the previously reported alarm indicator and save the latest reported alarm indicator; otherwise, keep the alarm indicator containing the first associated indicator value equal to -1.

[0054] In one implementation, two indicator rules, Rule X and Rule Y, are pre-set. Rule X addresses the specific business scenario where a message needs to be processed by both systems A and B. If a message is not processed by either system within 10 minutes, an alarm is triggered. If the processing interval between systems A and B exceeds 2 minutes, an alarm is also triggered. Rule Y addresses the specific business scenario where an indicator is recorded after processing by system C. Ultimately, the monitoring system needs to display the cumulative number of times system C has processed the message.

[0055] Rule x includes:

[0056] Metric name: system_cast_time;

[0057] Matching events: system:A--->asystem:B--->b;

[0058] Event association identifier: id;

[0059] Calculation expression: @(b.executeTime)-@(a.executeTime);

[0060] Indicator calculation type: fixed value.

[0061] There are two matching events in rule x, including matching event a: system:A--->a and matching event b: system:B--->b. Matching event a and matching event b respectively indicate that a message has been processed by system A and system B. The calculation expression @(b.executeTime)-@(a.executeTime) represents the processing time of system B minus the processing time of system A.

[0062] Rule y includes:

[0063] Metric name: system_c_execute_count;

[0064] Matching event: system:C--->c;

[0065] Event association identifier: id;

[0066] Calculation expression: 1;

[0067] Metric calculation type: aggregate value.

[0068] There is one matching event in rule y: matching event c: system:C--->c, which means that a message has been processed by system C.

[0069] The method of the present invention first receives the indicator data reported by the interface to form an indicator data stream, including the indicator data reported by the four systems A, B, C, and D: {id:1,executeTime:00:02,system:A}, {id:2,executeTime:00:32,system:B}, {id:1,executeTime:02:03,system:B}, {id:3,executeTime:01:09,system:A}, {id:2,executeTime:00:18,s Each metric contains a time. For example, the metric time in {id:1,executeTime:00:02,system:A} is 00:02, indicating that system A processed the message at 00:02.

[0070] Then, the indicator rules are received to form an indicator rule stream, and then the indicator rule stream is converted into an indicator rule broadcast stream.

[0071] Next, match the indicator data with the preset indicator rules and filter out the indicator data that meets different indicator rules:

[0072] The indicator data that meets rule y is as follows: {id:3,executeTime:03:01,system:C} (this indicator data indicates that system C processed the message at 03:01), {id:3,executeTime:03:01,system:C}, and {id:2,executeTime:03:20,system:C}. There is a millisecond difference between {id:3,executeTime:03:01,system:C} and {id:3,executeTime:03:01,system:C}, so there are two recorded indicator data.

[0073] The indicator data that meets rule x are as follows: {id:1,executeTime:00:02,system:A} (this indicator data indicates that system A processed the message at 00:02), {id:1,executeTime:02:03,system:B}, {id:2,executeTime:00:32,system:B}, {id:2,executeTime:00:18,system:A}, {id:3,executeTime:02:07,system:B}, {id:3,executeTime:01:09,system:A}, {id:4,executeTime:02:18,system:A};

[0074] The three indicator data that meet rule y are single indicator data, and the seven indicator data that meet rule x are associated indicator data.

[0075] For each single indicator data above, calculate according to the calculation expression of rule y, and obtain a single indicator value of 1. Merge the single indicator value of each single indicator data and rule y into a single indicator event and rule merge flow. Preferably, the single indicator event and rule merge flow also includes the indicator name in rule y, the event association identifier, and the corresponding indicator time in the indicator data.

[0076] Because the metric calculation type for rule y is an aggregate value, the single metric values are aggregated according to the event correlation ID. Specifically, the single metric event and rule merge stream is divided into different groups (windows) based on the event correlation ID, and the single metric values within the same group are accumulated. If one group contains two single metric values equal to 1, the accumulated value is 2; if another group contains only one single metric value equal to 1, the aggregated value remains 1. The resulting alarm metrics (system_c_execute_count{id=3} 2 @ 03:01; system_c_execute_count{id=2} 1 @ 03:20) are reported to the database.

[0077] For each of the above correlation indicator data, the event correlation indicator is used as the unique identification field. Grouping is performed according to the event correlation indicator, resulting in four groups with IDs 1, 2, 3, and 4. The correlation indicator data for each group is stored and displayed in a window. Specifically, the first group stores (caches) {id:1,executeTime:00:02,system:A} and {id:1,executeTime:02:03,system:B} in sequence. The second group stores (caches) {id:2,executeTime:00:32,system:B} and {id:2,executeTime:00:18,system:A} in sequence. The third group stores (caches) {id:3,executeTime:02:07,system:B} and {id:3,executeTime:01:09,system:A} in sequence. The fourth group stores (caches) {id:4,executeTime:02:18,system:A}.

[0078] The indicator value of the first cached (i.e., arrived) associated indicator data in each group is recorded as -1 (i.e., the first associated indicator value). The first associated indicator value and the corresponding indicator name, event association identifier, and indicator time are combined into an alarm indicator (system_cast_time{id=1} -1 @00:02; system_cast_time{id=2} -1 @00:32; system_cast_time{id=3} -1 @02:07; system_cast_time{id=4} -1 @02:18) and reported to the database. Next, the associated indicator data in each group is calculated according to the calculation expression @(b.executeTime) - @(a.executeTime) in rule x. The second associated indicator value of each group of associated indicator data is obtained: 121 seconds, 14 seconds, and 58 seconds, respectively. Because the indicator calculation type of rule x is a fixed indicator, the data obtained in the previous step does not need to be calculated again. The second associated indicator value, the corresponding indicator name, event association identifier, and the indicator time of the first cached associated indicator data in the group are combined into an alarm indicator (system_cast_time{id=1} 121 @ 00:02; system_cast_time{id=2} 14 @ 00:32; system_cast_time{id=3} 58 @ 02:07), which is reported to the database.

[0079] In this embodiment, for the associated indicator data, in order to control the window memory usage, a data removal time is set, for example one day. When the grouping window is greater than 1, if a window time is greater than 1 day, all the data in the corresponding window is discarded to recycle resources.

[0080] Finally, it should be noted that the above embodiments are merely preferred embodiments of the present invention and are intended to illustrate the technical solutions of the present invention, rather than limiting them, and certainly not limiting the patent scope of the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features therein may be replaced by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention. In other words, any changes or refinements made to the main design concept and spirit of the present invention that have no substantive significance, provided that the technical problems they solve are still consistent with those of the present invention, should be included in the protection scope of the present invention. In addition, the direct or indirect application of the technical solutions of the present invention to other related technical fields should also be included in the patent protection scope of the present invention.

Claims

1. A method for generating custom indicators based on a real-time streaming framework, characterized in that: The following steps are involved: S1. Receive indicator data reported by the interface and form an indicator data stream through Flink; the indicator data includes indicator time; S2. Receive the indicator rules pushed by the server to Kafka to form an indicator rule stream; the indicator rules are customized and include indicator name, matching event, event association identifier, calculation expression and indicator calculation type; S3. Convert the indicator rule stream into the indicator rule broadcast stream and define the broadcast state; S4. Merge the indicator data stream with the indicator rule broadcast stream, match the indicator data in the indicator data stream with the indicator rules in the indicator rule stream through the broadcast state, match the indicator data that meets different indicator rules, and determine whether the matched indicator data is single indicator data or associated indicator data; when the indicator rule contains one matching event, the indicator data that meets the corresponding indicator rule is single indicator data; when the indicator rule contains multiple matching events, the indicator data that meets the corresponding indicator rule is associated indicator data; S5. Calculate the single indicator data / associated indicator data using the calculation expression in the indicator rule to obtain the single indicator value / associated indicator value, and merge the single indicator value / associated indicator value and the corresponding indicator rule into a single indicator event and rule merged flow / associated indicator event and rule merged flow; In said S5, the method for calculating the associated indicator data includes: S51, for each successfully matched associated indicator data, grouping according to the event association identifier of the corresponding indicator rule, each group is used to cache the associated indicator data of the same event association identifier; S52, calculating the associated indicator value of the associated indicator data in each group, the associated indicator value including the first associated indicator value and / or the second associated indicator value; The first correlation index value and the second correlation index value are calculated as follows: during the grouping process, the first correlation index value is obtained by using the first correlation index data cached in the group, where the first correlation index value is equal to -1; after all the correlation index data of each group is cached in the group, all the correlation index data in the current group are calculated according to the calculation expression corresponding to the matching index rule to obtain the second correlation index value; S6. Determine the indicator calculation type of the single indicator event and rule merge flow / the associated indicator event and rule merge flow based on the indicator rule, calculate the single indicator value / the associated indicator value based on the indicator calculation type, and obtain the reported indicator value; S7, combining each reported indicator value and the corresponding indicator name, event association identifier and indicator time into an alarm indicator and reporting it to the database; In said S7, a rolling time window is set, and alarm indicators are reported within each rolling time window; Prioritize obtaining the alarm indicator containing the first associated indicator value and reporting it to the database, and then generate the latest alarm indicator containing the second associated indicator value and report it to the database: first query whether the alarm indicator has been reported based on the indicator name. If it has been reported and the indicator value is -1, delete the previously reported alarm indicator and save the latest reported alarm indicator; otherwise, keep the alarm indicator containing the first associated indicator value equal to -1.

2. A method for generating custom indicators based on a real-time streaming framework according to claim 1, characterized in that: Defining the broadcast state includes: calling the broadcast method of the indicator rule stream to set the broadcast state, and the broadcast method will also return the indicator rule broadcast stream.

3. A method for generating custom indicators based on a real-time streaming framework according to claim 2, characterized in that: In S4, the indicator rules in the indicator rule stream are stored in the broadcast state through the indicator rule broadcast stream, the indicator data stream obtains the indicator rules in the broadcast state, and each indicator data in the indicator data stream is matched with the indicator rule; when the matching event of the indicator rule contains one, the indicator data that meets the corresponding indicator rule is single indicator data; when the matching event of the indicator rule contains multiple, the indicator data that meets the corresponding indicator rule is associated indicator data.

4. A method for generating custom indicators based on a real-time streaming framework according to claim 1, characterized in that: In S5, the calculation method of the single indicator data includes: for each successfully matched single indicator data, performing calculation according to the calculation expression of the corresponding indicator rule to obtain the single indicator value.

5. A method for generating custom indicators based on a real-time streaming framework according to claim 1, characterized in that: In the calculation of the associated indicator data, grouping is performed using Flink's key grouping.

6. A method for generating custom indicators based on a real-time streaming framework according to claim 1, characterized in that: In S6, the indicator calculation type includes fixed value and aggregate value; when the indicator calculation type is fixed value, the single indicator value / associated indicator value is used as the reported indicator value; when the indicator calculation type is aggregate value, each single indicator value / associated indicator value is accumulated to obtain the reported indicator value.

7. A method for generating custom indicators based on a real-time streaming framework according to claim 1, characterized in that: The database is a Prometheus database.

Images