Data monitoring method based on part-time job business

Abstract

The invention discloses a data monitoring method based on a part-time job service. The method comprises the following steps: S1, configuring a service scene rule, a scene data extraction rule, a monitoring scheduling rule, a triggering rule and an alarm assembly broadcasting rule; s2, hanging the scene data extraction rule, the monitoring scheduling rule and the triggering rule in the step S1 under corresponding data source tasks, and performing periodic data monitoring on the data source tasks; s3, when abnormal data meeting the triggering rule is monitored, assembling and broadcasting the abnormal data according to an alarm assembling and broadcasting rule; s4, generating an analysis report from the data in the historical monitoring period; according to the invention, the data can be accurately and periodically monitored to find abnormal data, and the abnormal data can be timely broadcasted.

Description

Technical Field

[0001] The present invention relates to the field of information processing, and in particular to a data monitoring method based on part-time business. Background Art

[0002] Due to the investment and other means on the operational side, or the influence of the industry's ecology, business indicators fluctuate greatly, but the big data side is insensitive to it. At the same time, it extends to considerations on data quality: data accuracy and real-time. Data accuracy is reflected in whether the monitored data has undergone multiple processing and whether the data monitoring period is consistent. Data real-time is reflected in whether abnormal data can be promptly and clearly reminded. Summary of the invention

[0003] The purpose of the present invention is to provide a data monitoring method based on part-time business. The present invention can accurately monitor data periodically to find abnormal data and report the abnormal data in a timely manner.

[0004] The technical solution of the present invention is a data monitoring method based on part-time business, which is carried out according to the following steps:

[0005] Step S1: configure business scenario rules, scenario data extraction rules, monitoring scheduling rules, triggering rules and alarm assembly and broadcasting rules;

[0006] Step S2: attach the scene data extraction rules, monitoring scheduling rules and triggering rules of step S1 to the corresponding data source task, and perform periodic data monitoring on the data source task;

[0007] Step S3: When abnormal data that meets the triggering rules is monitored, the abnormal data is assembled and broadcast according to the alarm assembly and broadcasting rules;

[0008] Step S4: Generate an analysis report based on the data within the historical monitoring period.

[0009] In the above-mentioned data monitoring method based on part-time business, the business scenario rules include the monitoring name, description, remarks and effective status.

[0010] In the aforementioned data monitoring method based on part-time business, the scenario data extraction rules include data source selection, SQL statements, parameter sets and monitoring indicators. The SQL statements and parameter sets are assembled into execution statements and executed in the corresponding data source tasks to obtain data corresponding to the monitoring indicators.

[0011] In the aforementioned data monitoring method based on part-time business, the monitoring scheduling rules include upstream task dependency, scheduling cycle and cron expression, the upstream task dependency is used to specify the prerequisite task for starting data monitoring, the scheduling cycle is the execution time of data monitoring, and the cron expression is the start time of data monitoring.

[0012] In the aforementioned data monitoring method based on part-time business, the trigger rules include comparison period, comparison operator, comparison threshold and comparison relationship; according to the comparison period and comparison operator, it is judged whether a new monitoring indicator is derived from the data in the historical period, and then according to the comparison threshold and comparison relationship, it is judged whether it is abnormal data.

[0013] In the aforementioned data monitoring method based on part-time business, the alarm assembly broadcast rules include broadcast template, broadcast form, broadcast assembly information and broadcast location, and the abnormal data and broadcast assembly information are assembled accordingly according to the broadcast template and broadcast in the designated broadcast location in the designated broadcast form.

[0014] In the aforementioned data monitoring method based on part-time business, the broadcast assembly information includes configuration items of business scenario rules, configuration items of scenario data extraction rules and configuration items of trigger rules.

[0015] In the aforementioned data monitoring method based on part-time business, the data within the historical monitoring period is stored in the result table of the database, and quick BI is applied to generate reports for all application scenarios.

[0016] Compared with the prior art, the present invention pre-configures business scenario rules, scenario data extraction rules, monitoring scheduling rules, trigger rules and alarm assembly and broadcasting rules. The scenario data extraction rules, monitoring scheduling rules and trigger rules are hung under the corresponding data source tasks, and periodic data monitoring is performed on the data source tasks. The data is directly tested in the same period to ensure the accuracy of data comparison; when data that meets the trigger rules is monitored, the data is assembled and broadcast according to the alarm assembly and broadcasting rules to ensure the timeliness and intuitiveness of data reminders; and analysis reports are generated for the data within the historical monitoring period to assist in data change analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a structural schematic diagram of the present invention. DETAILED DESCRIPTION

[0018] The present invention is further described below in conjunction with the accompanying drawings and embodiments, but they are not intended to limit the present invention.

[0019] Embodiment 1: A data monitoring method based on part-time business, as shown in the attached Figure 1 As shown, follow the steps below:

[0020] Step S1: Configure business scenario rules, scenario data extraction rules, monitoring scheduling rules, triggering rules and alarm assembly and broadcasting rules; the business scenario rules include the name, description, remarks and effective status of the monitoring; the scenario data extraction rules include data source selection (single choice: ODPS, Hologres), SQL statements, parameter sets and monitoring indicators. The SQL statements and parameter sets are assembled into execution statements and executed in the corresponding data source tasks to obtain the data corresponding to the monitoring indicators. The execution statement must contain the "monitoring indicator" field to facilitate logical judgment when triggered; the monitoring scheduling rules include upstream task dependencies, scheduling cycles (single choice: days, hours) and cron expressions. Depends on the pre-task used to specify the start of data monitoring, the scheduling period is the execution time of data monitoring, and the cron expression is the start time of data monitoring; the triggering rules include comparison period (single choice: T-1, T-7, T-30, average value of the past 7 days, average value of the past 30 days, none), comparison operator (single choice: difference, ratio, none), comparison threshold and comparison relationship (single choice: greater than, greater than or equal to, less than, less than or equal to); the alarm assembly broadcasting rules include broadcasting template, broadcasting form (single choice: text, report), broadcasting assembly information and broadcasting position, the abnormal data and broadcasting assembly information are assembled accordingly according to the broadcasting template and broadcast in the specified broadcasting form at the specified broadcasting position.

[0021] Step S2: Attach the scenario data extraction rules, monitoring scheduling rules, and triggering rules of step S1 to the corresponding data source tasks, and perform periodic data monitoring on the data source tasks; data monitoring has two operating modes: 1. SQL statements that are executed at half past every hour without relying on any data warehouse tasks; 2. Generate monitoring tasks only for ODPS data warehouse tasks according to the scheduling cycle. The monitoring tasks are attached to the specified ODPS task node according to the upstream task dependency (supports multiple tasks), and run within the configured execution time to obtain monitoring data.

[0022] Step S3: execute the statement to obtain data, determine whether a new monitoring indicator is derived from the data in the historical period according to the comparison period and the comparison operator, and then determine whether it is abnormal data according to the comparison threshold and the comparison relationship. The sql statement contains monitoring indicators, and the derived indicators are generated according to the configuration item "comparison period". The monitoring data of each period will be stored in the result table. If "comparison period" selects "average value of the past 7 days", the average value of the monitoring indicator of the past 7 days will be derived and generated according to the data in the result table. If "none" is selected, the derived indicator is the monitoring indicator itself. If "T-1" is selected, the derived indicator is the monitoring indicator itself at the T-1 time point. The data comparison rule is based on the configuration items "comparison threshold" and "comparison relationship" for logical judgment. The derived indicator obtained above is compared with the comparison threshold. As long as the conditions are met, it is determined to be abnormal data and stored in the result table. The abnormal data is assembled and broadcast according to the alarm assembly and broadcast rules. There are two forms: 1. When the report form is selected, according to the quick The BI subscription function uses the monitoring data of the result table as the data source to create a dashboard, and broadcasts it to the selected location through the subscription function (suitable for monitoring data fluctuations in multiple time periods); 2. When the text format is selected, assemble the configuration items of the business scenario rules, the configuration items of the scenario data extraction rules, the configuration items of the trigger rules and the abnormal data values, and assemble them with the template "In monitoring scenario a, at the current time b cycle, under the rule of c, the monitored abnormal data is: d". The python script calls the API interface regularly to broadcast the assembled message for alarm.

[0023] Step S4: Generate an analysis report based on the data in the historical monitoring period; store the data in the historical monitoring period in the result table of the database, and apply quick BI to generate reports for all application scenarios to provide data analysis capabilities.

[0024] Example 2: The practical monitoring scenarios of the data monitoring method of this example based on Example 1 are as follows:

[0025] This embodiment monitors whether the traffic supported by the merchant every day meets its own value, and reports the alarm information to the DingTalk group - "Merchant Value Monitoring Group" at 10 o'clock on T+1 day in the form of a report;

[0026] 1. Business scenario rules:

[0027] Monitoring name: Merchant value monitoring;

[0028] Description: The daily consumption of merchants (the income brought to the platform) and the daily traffic support provided by the platform (exposure and registration);

[0029] Note: The report will be broadcast at 10:00 on T+1 day;

[0030] Is it effective: Yes;

[0031] 2. Scene data extraction rules:

[0032] Data source selection: ODPS;

[0033] SQL statement:

[0034] Select merchant, income, merchant exposure on the same day, merchant registration on the same day, income / merchant exposure on the same day as merchant value;

[0035] fromMerchant Value Table;

[0036] where date partition = ${date input parameter};

[0037] and the merchant’s exposure on that day > the average exposure of all merchants on that day;

[0038] order by income / merchant exposure asc on the day;

[0039] limit 20;

[0040] Parameter set: Date input parameter = $[yyyymmdd-1];

[0041] Monitoring indicators: merchant value;

[0042] 3. Monitoring and scheduling rules:

[0043] Upstream task dependency: merchant value table warehouse task;

[0044] Scheduling period: day;

[0045] Cron expression: 0010**?;

[0046] 4. Triggering rules:

[0047] Comparison period: none;

[0048] Comparison operator: none;

[0049] Contrast threshold: 50 per 10,000;

[0050] Contrast relationship: less than;

[0051] 5. Alarm assembly broadcast rules

[0052] Report form: report;

[0053] Report assembly information: merchant, income, merchant exposure on the day, merchant registration on the day, merchant value;

[0054] Broadcast location: the webhook corresponding to the robot in the DingTalk group;

[0055] Under this rule, an execution statement will be created under the original task "Merchant Value Table Data Warehouse Task" of ODPS, and it will be run at 10 o'clock every day after the "Merchant Value Table Data Warehouse Task" is completed. After the execution is completed, the monitoring data will be obtained and compared with the threshold of 50 per 10,000. When it is less than the threshold, it is abnormal data. The abnormal data will be dropped into the result table, and its corresponding quick BI report will be updated in seconds. According to the alarm assembly broadcast rule, the subscription function of quick BI will be called to broadcast the result table where the abnormal information is located to the corresponding DingTalk group, as shown in Table 1 below:

[0056]

[0057]

[0058] Table 1

[0059] To summarize, the present invention pre-configures business scenario rules, scenario data extraction rules, monitoring and scheduling rules, trigger rules and alarm assembly and broadcasting rules. The scenario data extraction rules and monitoring and scheduling rules are hung under the corresponding data source tasks, and periodic data monitoring is performed on the data source tasks. The data is directly tested in the same period to ensure the accuracy of data comparison; when data that meets the trigger rules is monitored, the abnormal data is assembled and broadcast according to the alarm assembly and broadcasting rules to ensure the timeliness and intuitiveness of data reminders; and analysis reports are generated for the data within the historical monitoring period to assist in data change analysis.

Claims

1. A data monitoring method based on part-time business, characterized in that: Follow these steps: Step S1: configure business scenario rules, scenario data extraction rules, monitoring scheduling rules, triggering rules and alarm assembly and broadcasting rules; Step S2: attach the scene data extraction rules, monitoring scheduling rules and triggering rules of step S1 to the corresponding data source task, and perform periodic data monitoring on the data source task; Step S3: When abnormal data that meets the triggering rules is monitored, the abnormal data is assembled and broadcast according to the alarm assembly and broadcasting rules; Step S4: Generate an analysis report based on the data within the historical monitoring period.

2. The data monitoring method based on part-time business according to claim 1 is characterized in that: The business scenario rules include the monitoring name, description, remarks and effective status.

3. The data monitoring method based on part-time business according to claim 1 is characterized in that: The scenario data extraction rules include data source selection, SQL statements, parameter sets and monitoring indicators. The SQL statements and parameter sets are assembled into execution statements and executed in corresponding data source tasks to obtain data corresponding to the monitoring indicators.

4. The data monitoring method based on part-time business according to claim 1 is characterized in that: The monitoring scheduling rules include upstream task dependency, scheduling cycle and cron expression. The upstream task dependency is used to specify the pre-task for starting data monitoring. The scheduling cycle is the execution time of data monitoring. The cron expression is the start time of data monitoring.

5. The data monitoring method based on part-time business according to claim 1 is characterized in that: The triggering rules include a comparison period, a comparison operator, a comparison threshold and a comparison relationship; according to the comparison period and the comparison operator, it is determined whether a new monitoring indicator is derived from the data in the historical period, and then according to the comparison threshold and the comparison relationship, it is determined whether it is abnormal data.

6. The data monitoring method based on part-time business according to claim 1 is characterized in that: The alarm assembly broadcasting rule includes a broadcasting form, a broadcasting template, broadcasting assembly information and a broadcasting position. The abnormal data and the broadcasting assembly information are assembled accordingly according to the broadcasting template and broadcasted in a designated broadcasting form at a designated broadcasting position.

7. The data monitoring method based on part-time business according to claim 6 is characterized in that: The broadcast assembly information includes configuration items of business scenario rules, configuration items of scenario data extraction rules and configuration items of trigger rules.

8. The data monitoring method based on part-time business according to claim 1 is characterized in that: The data within the historical monitoring period is stored in the result table of the database, and quick BI is applied to generate reports for all application scenarios.

Images