Custom index generation method based on real-time streaming framework

Abstract

The invention discloses a self-defined index generation method based on a real-time stream framework. The method comprises the following steps: receiving index data to form an index data stream; receiving index rules to form an index rule flow; converting the index rule stream into an index rule broadcast stream; combining the index data stream with the index rule broadcast stream, matching index data meeting different index rules, and judging whether the matched index data is single-index data or associated index data; calculating the single index data / associated index data by adopting a calculation expression in the index rule to obtain a single index value / associated index value; calculating the single index value / associated index value based on an index calculation type in the index rule to obtain an index value; and combining each index value, the corresponding index name, the event association identifier and the index time of the index data into an alarm index, and reporting the alarm index to a database. According to the method, a new processing logic is provided on the basis of the flink framework, and generation and alarm behaviors of the user-defined indexes are achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method for generating a custom indicator based on a real-time stream framework. Background Art

[0002] In the Internet field, when backend services such as sending text messages take too long or cross-service consumption data is lost, R&D personnel need to promptly perceive system abnormalities. If it is a code problem, the code needs to be debugged or optimized to improve the stability and fluency of the system.

[0003] Currently, we mainly calculate different scenarios and report them to the monitoring platform for unified alarm processing. For example, if the function execution time is too long, we need to print the start and end time of the program function, calculate the time, and then report it to the monitoring platform. When the time exceeds the threshold configured by the monitoring platform, an alarm is required to notify R&D that the execution time is too long and needs attention.

[0004] The existing technology usually calculates specific indicators in the business code, such as time consumption indicators, and then directly issues an alarm or reports it to a third-party monitoring platform for alarm processing, or directly uses a third-party alarm program to generate indicators using specific methods or annotations, and reports it to the monitoring platform for alarm processing. There is no unified platform to help complete the entire logical processing of alarms for data loss across systems, and R&D personnel need to perform specific code logic processing in both data generation and data processing. Summary of the invention

[0005] The purpose of the present invention is to provide a method for generating custom indicators based on a real-time stream framework to solve the technical problem that the existing method cannot realize the calculation of cross-system correlation data indicators and data loss indicators.

[0006] To achieve the above purpose, the technical solution adopted by the present invention is as follows: A method for generating a custom indicator based on a real-time streaming framework comprises the following steps: S1. Receive the indicator data reported by the interface and form an indicator data stream through flink; the indicator data includes indicator time; S2. Receive the indicator rules pushed by the server to Kafka to form an indicator rule stream; the indicator rules are customized, including indicator name, matching event, event association identifier, calculation expression and indicator calculation type; S3, convert the indicator rule stream into the indicator rule broadcast stream and define the broadcast state; S4, merging the indicator data stream with the indicator rule broadcast stream, matching the indicator data in the indicator data stream with the indicator rules in the indicator rule stream through the broadcast status, matching the indicator data that meets different indicator rules, and determining whether the matched indicator data is single indicator data or associated indicator data; S5. Calculate the single indicator data / associated indicator data using the calculation expression in the indicator rule to obtain the single indicator value / associated indicator value, and merge the single indicator value / associated indicator value and the corresponding indicator rule into a single indicator event and rule merged flow / associated indicator event and rule merged flow; S6. Determine the indicator calculation type of the single indicator event and rule merge flow / the associated indicator event and rule merge flow according to the indicator rule, calculate the single indicator value / the associated indicator value according to the indicator calculation type, and obtain the reported indicator value; S7. Combine each reported indicator value and the corresponding indicator name, event association identifier and indicator time into one alarm indicator and report it to the database.

[0007] Furthermore, the definition of the broadcast state includes: calling the broadcast method of the indicator rule stream to set the broadcast state, and the broadcast method will also return the indicator rule broadcast stream.

[0008] Furthermore, in S4, the indicator rules in the indicator rule stream are stored in the broadcast state through the indicator rule broadcast stream, the indicator data stream obtains the indicator rules in the broadcast state, and each indicator data in the indicator data stream is matched with the indicator rules; when the matching event of the indicator rule contains one, the indicator data that satisfies the corresponding indicator rule is single indicator data; when the matching event of the indicator rule contains multiple, the indicator data that satisfies the corresponding indicator rule is associated indicator data.

[0009] Furthermore, in S5, the calculation method of the single indicator data includes: for each successfully matched single indicator data, calculating according to the calculation expression of the corresponding indicator rule to obtain the single indicator value.

[0010] Further, in said S5, the calculation method of the associated indicator data includes: S51, for each successfully matched associated indicator data, grouping according to the event association identifier of the corresponding indicator rule, each group is used to cache the associated indicator data of the same event association identifier; S52, calculating the associated indicator value of the associated indicator data in each group, the associated indicator value includes the first associated indicator value and / or the second associated indicator value; The calculation method of the first association index value and the second association index value is: during the grouping process, the first association index value is obtained by caching the first association index data in the group, and the first association index value is equal to -1; after all the association index data of each group is cached in the group, all the association index data in the current group are calculated according to the calculation expression of the corresponding matching index rule to obtain the second association index value.

[0011] Furthermore, in the calculation of the associated indicator data, grouping is performed through Flink's key grouping.

[0012] Furthermore, in S6, the indicator calculation type includes a fixed value and an aggregate value; when the indicator calculation type is a fixed value, the single indicator value / associated indicator value is used as the reported indicator value; when the indicator calculation type is an aggregate value, each single indicator value / associated indicator value is accumulated to obtain the reported indicator value.

[0013] Furthermore, in S7, a rolling time window is set, and the alarm indicator is reported in each rolling time window.

[0014] Furthermore, the database is a Prometheus database.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a new processing logic based on the flink framework to realize the generation and alarm behavior of custom indicators. The present invention is applicable to different systems. The indicator data between different systems automatically generates indicators and configures alarms through indicator rules, without the need for each system to perform logical processing for specific scenarios. The present invention preferentially generates an indicator value of -1 for the associated indicator data, configures an alarm by querying the indicator value, and fills the gap of the alarm of the loss of consumption indicators. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0017] Terminology explanation: Flink is an open source distributed stream processing framework; Kafka is a distributed stream processing platform; The Prometheus database is an open source monitoring system and time series database; The broadcast method is usually used to broadcast a data stream to all downstream tasks so that these tasks can share the data in the broadcast stream; RichSourceFunction is an enhanced data source class in Flink, which is used to implement custom data sources; The run method is the core method in RichSourceFunction, which is used to generate data and send it downstream; The addSource method is a core method of StreamExecutionEnvironment, which is used to add a data source to a Flink stream processing job.

[0018] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0019] like Figure 1 As shown, the present invention provides a method for generating a custom indicator based on a real-time stream framework, comprising the following steps: S1. Receive the indicator data reported by the interface and form an indicator data stream through flink; the indicator data includes indicator time; S2. Receive the indicator rules pushed by the server to Kafka to form an indicator rule stream; the indicator rules are customized, including indicator name, matching event, event association identifier, calculation expression and indicator calculation type; S3, convert the indicator rule stream into the indicator rule broadcast stream and define the broadcast state; S4, merging the indicator data stream with the indicator rule broadcast stream, matching the indicator data in the indicator data stream with the indicator rules in the indicator rule stream through the broadcast status, matching the indicator data that meets different indicator rules, and determining whether the matched indicator data is single indicator data or associated indicator data; S5. Calculate the single indicator data / associated indicator data using the calculation expression in the indicator rule to obtain the single indicator value / associated indicator value, and merge the single indicator value / associated indicator value and the corresponding indicator rule into a single indicator event and rule merged flow / associated indicator event and rule merged flow; S6. Determine the indicator calculation type of the single indicator event and rule merge flow / the associated indicator event and rule merge flow according to the indicator rule, calculate the single indicator value / the associated indicator value according to the indicator calculation type, and obtain the reported indicator value; S7. Combine each reported indicator value and the corresponding indicator name, event association identifier and indicator time into one alarm indicator and report it to the database.

[0020] In S1 described in the present invention, the indicator data reported by the receiving interface, the reporting form supports the indicator data received by the hypertext transfer protocol and the message queue, and by inheriting the RichSourceFunction class, mainly implements its built-in run method to define the logic of receiving indicators, and then forms the indicator data stream through the built-in addSource method of flink.

[0021] In S2 of the present invention, based on the function of the RichSourceFunction class, the indicator rules pushed by the server to Kafka are received. The indicator rules are pre-set, including indicator name, matching event, event-related indicator (i.e., the unique identification field (id) in the corresponding indicator data), calculation expression and indicator calculation type. It mainly defines the logic of indicator rule reception by implementing the built-in run method of RichSourceFunction, and then forms the indicator rule flow through the built-in addSource method of flink.

[0022] In S3 of the present invention, the indicator rule stream is converted into the indicator rule broadcast stream, specifically by calling the broadcast method to convert the indicator rule stream into the broadcast stream. In addition, the broadcast state is defined: the broadcast method of the indicator rule stream is called to set the broadcast state, and the broadcast method will also return the indicator rule broadcast stream.

[0023] In S4 of the present invention, the indicator data stream is merged with the indicator rule broadcast stream, and the indicator rule broadcast stream is processed in processBroadcastElement by inheriting BroadcastProcessFunction, and the indicator rules in the indicator rule stream are stored in the broadcast state, and then the indicator data stream is processed in processElement, and the indicator data stream obtains the indicator rules in the broadcast state through ctx.getBroadcastState, and each indicator data in the indicator data stream is matched with the indicator rule, and the indicator data that meets different indicator rules is matched, and it is determined whether the matched indicator data is single indicator data or associated indicator data. When the matching event of the indicator rule contains one, the indicator data that meets the corresponding indicator rule is single indicator data; when the matching event of the indicator rule contains multiple, the indicator data that meets the corresponding indicator rule is associated indicator data.

[0024] In S5 of the present invention, the calculation expression in the matched indicator rule is used to calculate the single indicator data and the associated indicator data respectively to obtain the single indicator value and the associated indicator value. All single indicator data and the corresponding single indicator value and indicator rule are merged into a single indicator event and rule merge flow, and all associated indicator data and the corresponding associated indicator value and indicator rule are merged into an associated indicator event and rule merge flow.

[0025] In S5, the calculation method of the single indicator data includes: for each successfully matched single indicator data, calculating according to the calculation expression of the corresponding indicator rule to obtain the single indicator value.

[0026] In the S5, the calculation method of the associated indicator data includes: for each successfully matched associated indicator data, grouping according to the event association identifier (id) of the corresponding indicator rule, each group is used to cache the associated indicator data of the same event association identifier (id), preferably grouping by key grouping (keyby) of flink; in the grouping process, a first associated indicator value is obtained by caching the first associated indicator data in the group, and the first associated indicator value is equal to -1; when the remaining associated indicator data is cached in the group, all the associated indicator data in the current group are calculated according to the calculation expression of the corresponding matching indicator rule to obtain a second associated indicator value.

[0027] For the associated indicator data, a first associated indicator value equal to -1 is first generated, so that the indicator configuration alarm can be queried by the first associated indicator value to fill the gap of the consumption indicator loss alarm.

[0028] In S6 described in the present invention, the indicator calculation type is judged for the single indicator event and rule merged flow, and the associated indicator event and rule merged flow, respectively, and the corresponding indicator value in the single indicator event and rule merged flow, and the associated indicator event and rule merged flow is calculated according to the indicator calculation type. For example, for the indicator calculation type of cumulative indicators, use flink's flatMap with the rolling time window (one time window per second) to accumulate the indicator value in each window, take the indicator value of the largest time in the window, ensure that the same indicator generates at most one indicator per second, and then pass it to the downstream; for the indicator type of fixed indicators, since the fixed indicator is a fixed value, no additional calculation is required, and the corresponding single indicator value / associated indicator value can be directly passed to the downstream as the indicator value.

[0029] S7 described in the present invention is to merge the indicator name, event association identifier and indicator time of the indicator rule in each indicator value and the corresponding single indicator event and rule merge flow / associated indicator event and rule merge flow into an alarm indicator and report it to the database. Preferably, a rolling time window is set, and the alarm indicator is reported in each rolling time window. More preferably, the rolling time window is set to 5 seconds, and the database is a Prometheus database. If the alarm indicator containing the first associated indicator value equal to -1 is obtained first and reported to the database, and then the latest alarm indicator containing the second associated indicator value is generated and reported to the database: first query whether the alarm indicator has been reported according to the indicator name. If it has been reported and the indicator value is -1, delete the previously reported alarm indicator and save the latest reported alarm indicator; otherwise, keep the alarm indicator containing the first associated indicator value equal to -1.

[0030] In one of the embodiments, two indicator rules are pre-set, including rule x and rule y. The specific business scenario corresponding to rule x is that a message needs to be processed by both systems A and B at the same time. When a message is not processed by any system within 10 minutes, an alarm needs to be triggered; when the message processing interval between systems A and B exceeds 2 minutes, an alarm needs to be triggered. The specific business scenario corresponding to rule y is that the indicator will be recorded after being processed by system C, and the monitoring system will eventually need to show how many times system C has processed it cumulatively.

[0031] Rule x includes: Metric name: system_cast_time; Matching events: system:A--->asystem:B--->b; Event association identifier: id; Calculation expression: @(b.executeTime)-@(a.executeTime); Indicator calculation type: fixed value.

[0032] There are two matching events in rule x, including matching event a: system:A--->a and matching event b: system:B--->b, where matching event a and matching event b respectively indicate that a message has been processed by system A and system B; the calculation expression @(b.executeTime)-@(a.executeTime) represents the processing time of system B minus the processing time of system A.

[0033] Rule y includes: Metric name: system_c_execute_count; Matching events: system:C--->c; Event association identifier: id; Calculation expression: 1; Metric calculation type: aggregate value.

[0034] There is one matching event in rule y: matching event c: system:C--->c, which means that a message has been processed by system C.

[0035] The method of the present invention first receives the indicator data reported by the interface to form an indicator data stream, including the indicator data reported by the four systems A, B, C, and D: {id:1, executeTime:00:02, system:A}, {id:2, executeTime:00:32, system:B}, {id:1, executeTime:02:03, system:B}, {id:3, executeTime:01:09, system:A}, {id:2, executeTime:00:18, s Each indicator data contains an indicator time. For example, the indicator time in {id:1,executeTime:00:02,system:A} is 00:02, which means that system A processed the message at 00:02.

[0036] Then, the indicator rules are received to form an indicator rule stream, and then the indicator rule stream is converted into an indicator rule broadcast stream.

[0037] Next, match the indicator data with the preset indicator rules to filter out the indicator data that meets different indicator rules: The indicator data that meets rule y are as follows: {id:3,executeTime:03:01,system:C} (this indicator data indicates that system C processed the message at 03:01), {id:3,executeTime:03:01,system:C}, and {id:2,executeTime:03:20,system:C}. There is a millisecond difference between {id:3,executeTime:03:01,system:C} and {id:3,executeTime:03:01,system:C}, so there are two recorded indicator data. The indicator data that meets rule x are as follows: {id:1,executeTime:00:02,system:A} (this indicator data indicates that system A processed the message at 00:02), {id:1,executeTime:02:03,system:B}, {id:2,executeTime:00:32,system:B}, {id:2,executeTime:00:18,system:A}, {id:3,executeTime:02:07,system:B}, {id:3,executeTime:01:09,system:A}, {id:4,executeTime:02:18,system:A}; The three indicator data that meet rule y are single indicator data, and the seven indicator data that meet rule x are associated indicator data.

[0038] For each single indicator data above, calculate according to the calculation expression of rule y, and obtain a single indicator value of 1. Merge the single indicator value of each single indicator data and rule y into a single indicator event and rule merge flow. Preferably, the single indicator event and rule merge flow also includes the indicator name in rule y, the event association identifier and the corresponding indicator time in the indicator data.

[0039] Since the indicator calculation type of rule y is an aggregate value. Aggregate the single indicator values ​​according to the event association identifier id. Specifically, divide the single indicator event and rule merge flow into different groups (windows) according to the event association identifier id, and accumulate the single indicator values ​​in the same group. One group contains two single indicator values ​​of 1, and the accumulation is 2; the other group contains only one single indicator value of 1, which is still 1 after aggregation. Report the final alarm indicators (system_c_execute_count{id=3} 2 @03:01; system_c_execute_count{id=2} 1 @03:20) to the database.

[0040] For each of the above associated indicator data, the event associated indicator is used as a unique identification field, and grouped according to the event associated indicator to obtain four groups with ids 1, 2, 3, and 4, and the associated indicator data of each group is stored in a window for display. Specifically, the first group stores (caches) {id: 1, executeTime: 00: 02, system: A} and {id: 1, executeTime: 02: 03, system: B} in sequence, the second group stores (caches) {id: 2, executeTime: 00: 32, system: B} and {id: 2, executeTime: 00: 18, system: A} in sequence, the third group stores (caches) {id: 3, executeTime: 02: 07, system: B} and {id: 3, executeTime: 01: 09, system: A} in sequence, and the fourth group stores (caches) {id: 4, executeTime: 02: 18, system: A}.

[0041] The indicator value of the associated indicator data of the first cache (i.e., arrival) in each group is recorded as -1 (i.e., the first associated indicator value), and the first associated indicator value and the corresponding indicator name, event association identifier, and indicator time are combined into an alarm indicator (system_cast_time{id=1} -1 @00:02; system_cast_time{id=2} -1 @00:32; system_cast_time{id=3} -1 @02:07; system_cast_time{id=4} -1 @02:18) and reported to the database. Next, the associated indicator data in each group is calculated according to the calculation expression @(b.executeTime) - @(a.executeTime) in rule x, and the second associated indicator value of each group of associated indicator data is obtained, which is 121 seconds, 14 seconds and 58 seconds respectively. Since the indicator calculation type of rule x is a fixed indicator, the data obtained in the previous step does not need to be calculated again. The second associated indicator value and the corresponding indicator name, event association identifier and indicator time of the first cached associated indicator data in the group are merged into an alarm indicator (system_cast_time{id=1} 121 @00:02; system_cast_time{id=2} 14 @00:32; system_cast_time{id=3} 58 @02:07) and reported to the database.

[0042] In this embodiment, for the associated indicator data, in order to control the window memory usage, a data removal time is set, for example one day. When the grouping window is greater than 1, if a window time is greater than 1 day, all the data in the corresponding window is discarded to recycle resources.

[0043] Finally, it should be noted that the above embodiments are only preferred embodiments of the present invention to illustrate the technical solutions of the present invention, rather than limiting them, and certainly not limiting the patent scope of the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention. In other words, any changes or modifications made to the main design concept and spirit of the present invention that have no substantive significance, and the technical problems they solve are still consistent with the present invention, should be included in the protection scope of the present invention. In addition, the direct or indirect application of the technical solutions of the present invention in other related technical fields is also included in the patent protection scope of the present invention.

Claims

1. A method for generating custom indicators based on a real-time streaming framework, characterized in that: The following steps are involved: S1. Receive the indicator data reported by the interface and form an indicator data stream through flink; the indicator data includes indicator time; S2. Receive the indicator rules pushed by the server to Kafka to form an indicator rule stream; the indicator rules are customized, including indicator name, matching event, event association identifier, calculation expression and indicator calculation type; S3, convert the indicator rule stream into the indicator rule broadcast stream and define the broadcast state; S4, merging the indicator data stream with the indicator rule broadcast stream, matching the indicator data in the indicator data stream with the indicator rules in the indicator rule stream through the broadcast status, matching the indicator data that meets different indicator rules, and determining whether the matched indicator data is single indicator data or associated indicator data; S5. Calculate the single indicator data / associated indicator data using the calculation expression in the indicator rule to obtain the single indicator value / associated indicator value, and merge the single indicator value / associated indicator value and the corresponding indicator rule into a single indicator event and rule merged flow / associated indicator event and rule merged flow; S6. Determine the indicator calculation type of the single indicator event and rule merge flow / the associated indicator event and rule merge flow according to the indicator rule, calculate the single indicator value / the associated indicator value according to the indicator calculation type, and obtain the reported indicator value; S7. Combine each reported indicator value and the corresponding indicator name, event association identifier and indicator time into one alarm indicator and report it to the database.

2. According to the method for generating a custom indicator based on a real-time stream framework according to claim 1, it is characterized in that: Defining the broadcast state includes: calling the broadcast method of the indicator rule stream to set the broadcast state, and the broadcast method will also return the indicator rule broadcast stream.

3. According to claim 2, a method for generating a custom indicator based on a real-time stream framework is characterized in that: In S4, the indicator rules in the indicator rule stream are stored in the broadcast state through the indicator rule broadcast stream, the indicator data stream obtains the indicator rules in the broadcast state, and each indicator data in the indicator data stream is matched with the indicator rules; when the matching event of the indicator rule contains one, the indicator data that satisfies the corresponding indicator rule is single indicator data; when the matching event of the indicator rule contains multiple, the indicator data that satisfies the corresponding indicator rule is associated indicator data.

4. According to claim 1, a method for generating a custom indicator based on a real-time stream framework is characterized in that: In S5, the calculation method of the single indicator data includes: for each successfully matched single indicator data, calculating according to the calculation expression of the corresponding indicator rule to obtain the single indicator value.

5. According to claim 1, a method for generating custom indicators based on a real-time streaming framework is characterized in that: In the S5, the calculation method of the associated indicator data includes: S51, for each successfully matched associated indicator data, grouping according to the event association identifier of the corresponding indicator rule, each group is used to cache the associated indicator data of the same event association identifier; S52, calculating the associated indicator value of the associated indicator data in each group, the associated indicator value includes the first associated indicator value and / or the second associated indicator value; The calculation method of the first association index value and the second association index value is: during the grouping process, the first association index value is obtained by caching the first association index data in the group, and the first association index value is equal to -1; after all the association index data of each group is cached in the group, all the association index data in the current group are calculated according to the calculation expression of the corresponding matching index rule to obtain the second association index value.

6. A method for generating custom indicators based on a real-time streaming framework according to claim 5, characterized in that: In the calculation of the associated indicator data, grouping is performed through Flink's key grouping.

7. A method for generating custom indicators based on a real-time streaming framework according to claim 1, characterized in that: In S6, the indicator calculation type includes a fixed value and an aggregate value; when the indicator calculation type is a fixed value, the single indicator value / associated indicator value is used as the reported indicator value; when the indicator calculation type is an aggregate value, each single indicator value / associated indicator value is accumulated to obtain the reported indicator value.

8. The method for generating a custom indicator based on a real-time stream framework according to claim 5, characterized in that: In S7, a rolling time window is set, and alarm indicators are reported in each rolling time window; Prioritize obtaining the alarm indicator containing the first associated indicator value and reporting it to the database, and then generate the latest alarm indicator containing the second associated indicator value and report it to the database: first query whether the alarm indicator has been reported according to the indicator name. If it has been reported and the indicator value is -1, delete the previously reported alarm indicator and save the latest reported alarm indicator; otherwise, keep the alarm indicator containing the first associated indicator value equal to -1.

9. A method for generating custom indicators based on a real-time streaming framework according to claim 1, characterized in that: The database is a Prometheus database.

Images