Work operation and maintenance method based on index strategy monitoring, storage medium and electronic device

Abstract

The invention provides a job operation and maintenance method based on index strategy monitoring, a storage medium and an electronic device, and relates to the technical field of smart homes.The method comprises the steps that monitoring strategy analysis is conducted on service application logic, a corresponding monitoring strategy is extracted, and a strategy library is updated based on the extracted monitoring strategy; monitoring the execution state of the platform operation corresponding to each monitoring strategy in the strategy library, and triggering the execution of the monitoring strategies under the condition that the operation is executed successfully to obtain the execution result of each monitoring strategy; and when it is determined that the execution result of any monitoring strategy is abnormal, generating corresponding alarm information. According to the operation and maintenance method based on index strategy monitoring provided by the invention, the key dimension and the core data table in the service data are identified by analyzing the index logic of the platform service, and the supplement of the missing service metadata in the operation and maintenance process is completed, so that the exception can be found and processed in time, and the user experience is improved. The operation efficiency of the data platform is improved.

Description

Technical Field

[0001] This application relates to the field of smart home technology, and in particular to an operation and maintenance method, storage medium and electronic device based on indicator strategy monitoring. Background Technology

[0002] As business grows, the complexity of data platform operations, data models, and data flow will increase. If anomalies are identified solely based on data lineage and operation status during maintenance, omissions may occur.

[0003] In related technologies, when the operation is normal, but data loss or changes in data source occur, or when an operation fails but no exception is thrown, the exception can only be discovered by users at the outermost application layer of the data platform and they need to report it. This greatly affects the normal operation of the data platform. Summary of the Invention

[0004] The purpose of this application is to provide an operation and maintenance method, storage medium, and electronic device based on indicator strategy monitoring. By parsing the indicator logic of platform business, the key dimensions and core data tables in business data are identified, and missing business metadata during operation and maintenance is supplemented, so that anomalies can be detected and handled in a timely manner, thereby improving the operating efficiency of the data platform.

[0005] This application provides a job operation and maintenance method based on indicator strategy monitoring, including: The system parses the business application logic in the data platform, extracts the corresponding monitoring strategies, and updates the strategy library based on the extracted monitoring strategies. The business application logic includes at least: indicator definitions, dashboard datasets, and custom SQL statements. The monitoring strategies include: data tables to be monitored, filtering conditions, and statistical strategies. The system monitors the execution status of the platform jobs corresponding to each monitoring strategy in the strategy library, and triggers the execution of the monitoring strategy when the job executes successfully, obtaining the execution result of each monitoring strategy. The system performs anomaly analysis and judgment on the execution result of each monitoring strategy according to a preset anomaly judgment logic, and generates corresponding alarm information when it is determined that the execution result of any monitoring strategy is abnormal. The alarm information includes: strategy-level alarm information to characterize the abnormal execution of the monitoring strategy, and topic-level alarm information to characterize the abnormal business theme to which the data table belongs.

[0006] Optionally, the step of parsing the business application logic in the data platform and extracting the corresponding monitoring strategies includes: parsing the corresponding database tables and SQL statements from the metadata of the indicator definition, and extracting the corresponding filtering conditions and statistical strategies from the parsed SQL statements; and performing structured processing on the parsed data tables and the extracted filtering conditions and statistical strategies according to preset processing rules to generate a first type of monitoring strategy.

[0007] Optionally, the step of parsing the business application logic in the data platform and extracting the corresponding monitoring strategies includes: extracting filtering conditions and statistical strategies from the SQL statements contained in the dashboard dataset, and determining the data tables associated with the SQL statements; performing structured processing on the data tables associated with the SQL statements, as well as the filtering conditions and statistical strategies extracted from the SQL statements, according to preset processing rules, to generate a second type of monitoring strategy.

[0008] Optionally, the step of parsing the business application logic in the data platform and extracting the corresponding monitoring strategy includes: parsing the custom SQL statement, extracting the filtering conditions and statistical strategies, and determining the data table associated with the custom SQL statement; performing structured processing on the data table associated with the custom SQL statement, as well as the filtering conditions and statistical strategies extracted from the custom SQL statement, according to preset processing rules, to generate a third type of monitoring strategy.

[0009] Optionally, updating the policy library based on the extracted monitoring policy includes: comparing the extracted monitoring policy with the policies stored in the policy library and generating a comparison result; if the comparison result indicates that there is no identical monitoring policy in the policy library, then the extracted monitoring policy is appended to the policy library; otherwise, the monitoring policy stored in the policy library is updated.

[0010] Optionally, the strategy library includes a strategy table and a source table; the strategy table is used to store at least one of the following: strategy identifier, table name of the data table, filtering conditions, statistical strategy, strategy type, and update time; the source table is used to store at least one of the following: strategy identifier, table name of the data table, strategy source, relevant personnel, and update time.

[0011] Optionally, the step of monitoring the execution status of the platform job corresponding to each monitoring strategy in the strategy library, and triggering the execution of the monitoring strategy when the job is successfully executed to obtain the execution result of each monitoring strategy, includes: determining the corresponding target platform job based on the target data table in the target monitoring strategy, and monitoring the execution status of the target platform job after the expected completion time of the target platform job; when the target platform job is successfully executed, triggering the execution of all monitoring strategies that depend on the target data table, and obtaining the execution result of each monitoring strategy.

[0012] Optionally, the alarm information includes: policy-level alarm information and topic-level alarm information; the step of performing anomaly analysis and judgment on the execution result of each monitoring policy according to the preset anomaly judgment logic, and generating corresponding alarm information when it is determined that the execution result of any monitoring policy is abnormal, includes: generating corresponding policy-level alarm information when the execution result of any target monitoring policy is abnormal, and notifying the relevant personnel corresponding to the target monitoring policy; when the number of policy-level alarm information triggered by any target data table exceeds a preset threshold, generating corresponding topic-level alarm information based on the target business topic corresponding to the target data table, synchronizing the abnormal status of the target business topic to the data lineage link, and identifying the affected upstream and downstream nodes.

[0013] This application also provides an operation and maintenance device based on indicator strategy monitoring, including: The strategy update module is used to parse the business application logic in the data platform, extract the corresponding monitoring strategies, and update the strategy library based on the extracted monitoring strategies. The business application logic includes at least: indicator definitions, dashboard datasets, and custom SQL statements. The monitoring strategies include: data tables to be monitored, filtering conditions, and statistical strategies. The strategy execution module is used to monitor the execution status of the platform job corresponding to each monitoring strategy in the strategy library, and trigger the execution of the monitoring strategy when the job is successfully executed, obtaining the execution result of each monitoring strategy. The anomaly detection module is used to perform anomaly analysis and judgment on the execution result of each monitoring strategy according to preset anomaly judgment logic, and generate corresponding alarm information when it is determined that the execution result of any monitoring strategy is abnormal. The alarm information includes: strategy-level alarm information to characterize the abnormal execution of the monitoring strategy, and topic-level alarm information to characterize the abnormal business theme to which the data table belongs.

[0014] Optionally, the strategy update module is specifically used to parse the corresponding database table and SQL statement from the metadata of the indicator definition, and extract the corresponding filtering conditions and statistical strategies from the parsed SQL statement; the strategy update module is also specifically used to perform structured processing on the parsed data table and the extracted filtering conditions and statistical strategies according to preset processing rules to generate a first type of monitoring strategy.

[0015] Optionally, the strategy update module is specifically used to extract filtering conditions and statistical strategies from the SQL statements contained in the Kanban dataset, and determine the data table associated with the SQL statement; the strategy update module is also specifically used to perform structured processing on the data table associated with the SQL statement, as well as the filtering conditions and statistical strategies extracted from the SQL statement, according to preset processing rules, to generate a second type of monitoring strategy.

[0016] Optionally, the strategy update module is specifically used to parse the custom SQL statement, extract the filtering conditions and statistical strategies, and determine the data table associated with the custom SQL statement; the strategy update module is also specifically used to perform structured processing on the data table associated with the custom SQL statement, as well as the filtering conditions and statistical strategies extracted from the custom SQL statement, according to preset processing rules, to generate a third type of monitoring strategy.

[0017] Optionally, the policy update module is specifically used to compare the extracted monitoring policy with the policies stored in the policy library and generate a comparison result; the policy update module is further used to append the extracted monitoring policy to the policy library if the comparison result indicates that there is no identical monitoring policy in the policy library, otherwise, update the monitoring policy stored in the policy library.

[0018] Optionally, the strategy execution module is specifically used to determine the corresponding target platform job based on the target data table in the target monitoring strategy, and to monitor the execution status of the target platform job after the expected completion time of the target platform job; the strategy execution module is also specifically used to trigger the execution of all monitoring strategies that depend on the target data table when the target platform job is successfully executed, and to obtain the execution result of each monitoring strategy.

[0019] Optionally, the alarm information includes: policy-level alarm information and topic-level alarm information; the anomaly detection module is specifically used to generate corresponding policy-level alarm information and notify the relevant personnel corresponding to the target monitoring policy when the execution result of any target monitoring policy is abnormal; the anomaly detection module is also specifically used to generate corresponding topic-level alarm information based on the target business topic corresponding to the target data table when the number of policy-level alarm information triggered by any target data table exceeds a preset threshold, and synchronize the abnormal status of the target business topic to the data lineage link, while identifying the affected upstream and downstream nodes.

[0020] This application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the job operation and maintenance method based on indicator strategy monitoring as described above.

[0021] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the operation and maintenance method based on indicator strategy monitoring as described above.

[0022] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the job operation and maintenance method based on indicator strategy monitoring as described above.

[0023] The operation and maintenance method, storage medium, and electronic device based on indicator-based strategy monitoring provided in this application firstly parse the business application logic in the data platform, extract the corresponding monitoring strategies, and update the strategy library based on the extracted monitoring strategies. The business application logic includes at least: indicator definitions, dashboard datasets, and custom SQL statements. The monitoring strategies include: data tables to be monitored, filtering conditions, and statistical strategies. Next, the execution status of the platform jobs corresponding to each monitoring strategy in the strategy library is monitored, and if the job execution is successful, the execution of the monitoring strategy is triggered to obtain the execution result of each monitoring strategy. Finally, anomaly analysis and judgment are performed on the execution results of each monitoring strategy according to preset anomaly judgment logic, and corresponding alarm information is generated when an anomaly is determined in the execution result of any monitoring strategy. In this way, by parsing the indicator logic of the platform business, key dimensions and core data tables in the business data are identified, and missing business metadata during operation and maintenance is supplemented, enabling anomalies to be detected and handled in a timely manner, thereby improving the operating efficiency of the data platform. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of the hardware environment for an interaction method of a smart device according to an embodiment of this application; Figure 2 This is one of the flowcharts of the operation and maintenance method based on indicator strategy monitoring provided in this application; Figure 3 This is the second flowchart of the operation and maintenance method based on indicator strategy monitoring provided in this application; Figure 4 This is a schematic diagram of the operation and maintenance device based on indicator strategy monitoring provided in this application; Figure 5 This is a schematic diagram of the electronic device provided in this application. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be used interchangeably where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices. All actions involving the acquisition of signal information or data in this application are performed in accordance with the relevant data protection laws and policies of the country where the application is located and with authorization from the owner of the relevant device.

[0028] According to one aspect of the embodiments of this application, a job operation and maintenance method based on indicator policy monitoring is provided. This method is widely applicable to whole-house intelligent digital control application scenarios such as smart homes, smart home ecosystems, and intelligence house ecosystems. Optionally, in this embodiment, the above-mentioned job operation and maintenance method based on indicator policy monitoring can be applied to, for example... Figure 2 The hardware environment shown consists of terminal device 102 and server 104. For example... Figure 2 As shown, server 104 is connected to terminal device 102 via a network and can be used to provide services (such as application services) to the terminal or clients installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data processing services for server 104.

[0029] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. The terminal device 102 may not be limited to PC, mobile phone, tablet computer, smart air conditioner, smart range hood, smart refrigerator, smart oven, smart stove, smart washing machine, smart water heater, smart washing equipment, smart dishwasher, smart projector, smart TV, smart clothes rack, smart curtains, smart audio-visual equipment, smart socket, smart speaker, smart speaker box, smart fresh air equipment, smart kitchen and bathroom equipment, smart bathroom equipment, smart robot vacuum cleaner, smart window cleaning robot, smart mopping robot, smart air purifier, smart steam oven, smart microwave oven, smart water heater, smart air purifier, smart water dispenser, smart door lock, etc.

[0030] In related technologies, data platforms mainly face the following problems: 1. Limitations of operation and maintenance relying on technical metadata: Monitoring only through data lineage and job status cannot cover anomalies in the business data itself (such as source data loss, changes in definition, silent failures, etc.).

[0031] 2. Delayed discovery of business anomalies: Problems are only exposed through end-user feedback, affecting data reliability and the efficiency of business decision-making.

[0032] 3. Lack of business metadata linkage: The existing operation and maintenance process does not incorporate application layer logic such as business metrics and dashboards into the monitoring system.

[0033] To address the aforementioned technical problems in related technologies, embodiments of this application provide a job operation and maintenance method based on indicator strategy monitoring, the method comprising: 1. Analyze business application logic Automatically parse dashboards and metric definitions in the data platform, identifying the key business dimensions (such as time, region, and product line) and core data tables they depend on. Business theme abstraction: Based on business meaning, core tables are categorized into different themes (such as sales theme and user theme) to form a business metadata graph.

[0034] 2. Two-layer monitoring strategy Strategy 1: Key Dimension Sampling Monitoring; Applicable Scenarios: Massive data scenarios (such as log tables, transaction logs). Method: Sampling statistics are performed on key dimensions (such as date, channel) to detect data integrity, distribution anomalies, etc. Anomaly Handling: Trigger alarms and notify all downstream nodes in the data lineage (such as ETL jobs, reporting tasks).

[0035] Strategy 2: Metric Trend Threshold Monitoring; Applicable Scenarios: Core application-layer metrics (such as daily active users, sales revenue). Method: Set dynamic thresholds (year-on-year / month-on-month fluctuation ±10%). When the threshold is exceeded, mark the business topic status as "Watchlist". Response Linkage: Synchronize the abnormal status to the lineage chain to assist in locating the root cause table or operation.

[0036] The operation and maintenance method based on indicator strategy monitoring provided in this application analyzes the indicator logic of platform business, identifies key dimensions and core data tables in business data, and completes the supplementation of missing business metadata during operation and maintenance, so that anomalies can be detected and handled in a timely manner, thereby improving the operating efficiency of the data platform.

[0037] The operation and maintenance method based on indicator strategy monitoring provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0038] like Figure 2 As shown in the embodiment of this application, a job operation and maintenance method based on indicator strategy monitoring is provided. This method may include the following steps 201 to 203: Step 201: Analyze the business application logic in the data platform, extract the corresponding monitoring strategies, and update the strategy library based on the extracted monitoring strategies.

[0039] The business application logic includes at least: metric definition, dashboard dataset, and custom SQL statement; the monitoring strategy includes: the data table to be monitored, filtering conditions, and statistical strategies.

[0040] For example, in this embodiment of the application, the monitoring strategy is extracted and updated periodically to facilitate the use of the monitoring strategy for anomaly detection in subsequent steps. Figure 3 As shown, the above monitoring strategy mainly includes three aspects: extracting monitoring strategies by parsing the metric definitions, extracting monitoring strategies by parsing the dashboard dataset, and extracting monitoring strategies by parsing custom SQL statements.

[0041] Specifically, step 201 above, which involves extracting monitoring strategies by parsing the indicator definitions, may further include the following steps 201a1 and 201a2: Step 201a1: Parse the corresponding database table and SQL statement from the metadata of the indicator definition, and extract the corresponding filtering conditions and statistical strategies from the parsed SQL statement.

[0042] Step 201a2: Perform structured processing on the parsed data table, as well as the extracted filtering conditions and statistical strategies, according to the preset processing rules to generate the first type of monitoring strategy.

[0043] For example, typically, each metric corresponds to a system that defines it. The metric's metadata contains its definition. Parsing the metric definition allows the extraction of the corresponding data table and SQL statement. The system can scan the metric definitions daily to automatically add monitoring strategies.

[0044] For example, the following can be automatically parsed from the metadata of the indicator definition system (such as an internally built indicator management platform): ① Which table and which field this indicator depends on (such as the amount field of the order table); ② What is its statistical strategy (such as sum, count(distinct)); ③ What are its filtering conditions (i.e., the content after the where statement, such as order status = 'completed'); ④ The purpose of parsing is to automatically generate a data quality monitoring strategy to monitor whether the core data behind this indicator is normal.

[0045] Specifically, step 201 above, which involves extracting monitoring strategies by parsing the dashboard dataset, may further include the following steps 201b1 and 201b2: Step 201b1: Extract the filtering conditions and statistical strategies from the SQL statements contained in the Kanban dataset, and determine the data tables associated with the SQL statements.

[0046] Step 201b2: Perform structured processing on the data tables associated with the SQL statement, as well as the filtering conditions and statistical strategies extracted from the SQL statement, according to the preset processing rules to generate the second type of monitoring strategy.

[0047] For example, a dashboard is a data visualization product for business users. It consists of multiple charts, tables, and numerical summaries, used to centrally display a set of related metrics to support business monitoring and decision-making. The datasets (most often SQL queries) corresponding to each visualization component behind the dashboard are parsed. By parsing these SQL queries, the following can be extracted: key data tables and data logic not covered by the metric system but actually used by the business; table names, predicate conditions, and statistical strategies that need to be monitored; and monitoring strategies can be automatically generated.

[0048] For example, consider the following dashboard: ① Dashboard Name: After-Sales Service Efficiency Dashboard; ② One of the charts: "Top 5 User Complaints Today by Each Channel"; ③ The dataset (SQL) behind this chart. Parsing this dashboard yields the following: Table Name: Customer Service Work Order Table; Predicate Conditions (i.e., filtering conditions, usually the content specified after "where" in an SQL statement): Work Order Creation Date = Today AND Work Order Type = 'Complaint'; Statistical Strategy: count(*).

[0049] Specifically, step 201 above, which involves extracting monitoring strategies by parsing custom SQL statements, may further include the following steps 201c1 and 201c2: Step 201c1: Parse the custom SQL statement, extract the filtering conditions and statistical strategies, and determine the data table associated with the custom SQL statement.

[0050] Step 201c2: Perform structured processing on the data table associated with the custom SQL statement, as well as the filtering conditions and statistical strategies extracted from the custom SQL statement, according to preset processing rules, to generate a third type of monitoring strategy.

[0051] For example, the extraction of the above monitoring strategies essentially involves parsing SQL statements to determine the associated data tables, filtering conditions (i.e., the content specified after the WHERE clause), and statistical strategies. After the monitoring strategies are extracted, they can be persisted.

[0052] Specifically, step 201 above, the step of updating the policy library based on the extracted monitoring policies, may further include the following steps 201d1 and 201d2: Step 201d1: Compare the extracted monitoring strategy with the strategy stored in the strategy library and generate a comparison result.

[0053] Step 201d2: If the comparison result indicates that there is no identical monitoring policy in the policy library, then the extracted monitoring policy is stored in the policy library; otherwise, the monitoring policy stored in the policy library is updated.

[0054] The strategy library includes a strategy table and a source table. The strategy table stores at least one of the following: strategy identifier, table name, filtering conditions, statistical strategy, strategy type, and update time. The source table stores at least one of the following: strategy identifier, table name, strategy source, relevant personnel, and update time.

[0055] For example, for the tables that need to be scanned and the calculations that need to be performed, this information is compared with the existing strategy table. Existing strategies are not entered again, but their dates are updated and logged to indicate that the strategy is still in use by the business. If no identical strategy exists, a new strategy is entered.

[0056] For example, such as Figure 3 As shown, after extracting a strategy, it is checked whether the strategy exists in the strategy library. If it does, the strategy information in the strategy library is updated; otherwise, the new strategy is added to the strategy library. It should be noted that the execution of the monitoring strategy must be performed after the platform job (e.g., data cleaning (Extraction Transformation Loading, ETL) job) is completed. Therefore, when updating the monitoring strategy, the execution time of the monitoring strategy must also be updated based on the completion time of the corresponding platform job.

[0057] Step 202: Monitor the execution status of the platform job corresponding to each monitoring strategy in the strategy library, and trigger the execution of the monitoring strategy if the job is successfully executed, so as to obtain the execution result of each monitoring strategy.

[0058] For example, after the extracted monitoring policy is persisted, abnormal operations can be judged and handled based on the policy.

[0059] Specifically, step 202 above may also include the following steps 202a1 and 202a2: Step 202a1: Based on the target data table in the target monitoring strategy, determine the corresponding target platform job, and monitor the execution status of the target platform job after its expected completion time.

[0060] Step 202a2: When the target platform job is executed successfully, all monitoring strategies that depend on the target data table are triggered and the execution result of each monitoring strategy is obtained.

[0061] For example, in this embodiment of the application, the execution time of the strategy is not configured separately. Instead, the corresponding job is found based on the table name, and the strategy is executed after the job is completed. For instance, if strategy 10002 scans table B at 01:20 every day, strategy 10002 will check the execution status of the job every minute starting at 01:20. The strategy will start executing when the status is "execution successful".

[0062] Step 203: Perform anomaly analysis and judgment on the execution result of each monitoring strategy according to the preset anomaly judgment logic, and generate corresponding alarm information when it is determined that the execution result of any monitoring strategy is abnormal.

[0063] For example, the alarm information in this application embodiment includes: policy-level alarm information for the execution result of a single policy, and topic-level alarm information for related business entities. The alarm information includes: policy-level alarm information used to characterize abnormal execution of monitoring policies, and topic-level alarm information used to characterize abnormalities in the business topic to which the data table belongs.

[0064] Specifically, step 203 above may also include the following steps: 203a1 or 203a2: Step 203a1: If the execution result of any target monitoring strategy is abnormal, generate the corresponding strategy-level alarm information and notify the relevant personnel corresponding to the target monitoring strategy.

[0065] Step 203a2: If the number of policy-level alarm messages triggered by any target data table exceeds a preset threshold, generate corresponding topic-level alarm messages based on the target business topic corresponding to the target data table, synchronize the abnormal status of the target business topic to the data lineage link, and identify the affected upstream and downstream nodes.

[0066] For example, once the job executes successfully, the corresponding monitoring policy can be executed to determine if any anomalies have occurred, and an alarm notification can be issued if an anomaly is detected. For instance, such as... Figure 3As shown, after the table to be accessed is written, a verification strategy is executed (if the job corresponding to the table to be accessed fails, it continues to wait). The execution results are persisted for subsequent judgment of whether the data is abnormal. Anomalies are judged based on the indicator results (year-on-year comparison, trend chart anomalies, equal to 0, null values, etc.). When an anomaly is detected, the relevant personnel are notified. When the anomaly strategy corresponding to a data table exceeds the threshold, the status of the corresponding topic can also be modified. For example, for strategy 10001, a strategy-level anomaly alarm is generated, and the indicator user confirms whether further investigation is needed; for strategies 10002-10004, strategy-level anomaly alarms are generated, and the status of the topic domain where data table B is located is modified, the person in charge of the topic domain is notified, and a notification is generated for the "normal" status of strategy 10005 (because the data in data table B may be abnormal, and all indicators on this table may be potentially affected). The downstream applications of data table B can be directly displayed through the platform's data lineage function to facilitate confirmation of the scope of impact.

[0067] The job operation and maintenance method based on indicator strategy monitoring provided in this application can be deployed independently outside the scheduling system in a plug-in manner without interfering with normal scheduling. Based on the existing functions of a traditional scheduling platform, business data validation is automatically added for business usage scenarios (indicators, dashboards, etc.) to identify potential data quality issues that may exist when jobs are running normally.

[0068] The operation and maintenance method based on indicator strategy monitoring provided in this application embodiment first parses the business application logic in the data platform, extracts the corresponding monitoring strategies, and updates the strategy library based on the extracted monitoring strategies. The business application logic includes at least: indicator definitions, dashboard datasets, and custom SQL statements. The monitoring strategies include: data tables to be monitored, filtering conditions, and statistical strategies. Then, the execution status of the platform jobs corresponding to each monitoring strategy in the strategy library is monitored, and if the job execution is successful, the execution of the monitoring strategy is triggered to obtain the execution result of each monitoring strategy. Finally, according to the preset anomaly judgment logic, anomaly analysis and judgment are performed on the execution results of each monitoring strategy, and if an anomaly is determined in the execution result of any monitoring strategy, corresponding alarm information is generated. In this way, by parsing the indicator logic of the platform business, key dimensions and core data tables in the business data are identified, and missing business metadata during operation and maintenance is supplemented, so that anomalies can be detected and handled in a timely manner, improving the operating efficiency of the data platform.

[0069] It should be noted that the operation and maintenance method based on indicator policy monitoring provided in this application can be executed by an operation and maintenance device based on indicator policy monitoring, or by a control module within that device for executing the operation and maintenance method based on indicator policy monitoring. This application uses the execution of the operation and maintenance method based on indicator policy monitoring by an operation and maintenance device as an example to illustrate the operation and maintenance device based on indicator policy monitoring provided in this application.

[0070] It should be noted that, in the embodiments of this application, the operation and maintenance methods based on indicator strategy monitoring shown in the accompanying drawings are all illustrated by way of example with reference to one of the accompanying drawings in the embodiments of this application. In specific implementation, the operation and maintenance methods based on indicator strategy monitoring shown in the accompanying drawings of the above methods can also be implemented in conjunction with any other accompanying drawings shown in the above embodiments, which will not be elaborated here.

[0071] The operation and maintenance device based on indicator strategy monitoring provided in this application is described below. The operation and maintenance method based on indicator strategy monitoring described below can be referred to in correspondence with the operation and maintenance method based on indicator strategy monitoring described above.

[0072] Figure 4 This is a schematic diagram of the operation and maintenance device based on indicator strategy monitoring provided in the embodiments of this application, as shown below. Figure 4 As shown, it specifically includes: The strategy update module 401 is used to parse the business application logic in the data platform, extract the corresponding monitoring strategies, and update the strategy library based on the extracted monitoring strategies. The business application logic includes at least: indicator definitions, dashboard datasets, and custom SQL statements. The monitoring strategies include: data tables to be monitored, filtering conditions, and statistical strategies. The strategy execution module 402 is used to monitor the execution status of the platform job corresponding to each monitoring strategy in the strategy library, and trigger the execution of the monitoring strategy when the job is successfully executed, to obtain the execution result of each monitoring strategy. The anomaly detection module 403 is used to perform anomaly analysis and judgment on the execution result of each monitoring strategy according to the preset anomaly judgment logic, and generate corresponding alarm information when it is determined that an anomaly occurs in the execution result of any monitoring strategy.

[0073] Optionally, the strategy update module 401 is specifically used to parse the corresponding database table and SQL statement from the metadata of the indicator definition, and extract the corresponding filtering conditions and statistical strategies from the parsed SQL statement; the strategy update module 401 is also specifically used to perform structured processing on the parsed data table and the extracted filtering conditions and statistical strategies according to preset processing rules to generate a first type of monitoring strategy.

[0074] Optionally, the strategy update module 401 is specifically used to extract filtering conditions and statistical strategies from the SQL statements contained in the Kanban dataset, and determine the data table associated with the SQL statement; the strategy update module 401 is also specifically used to perform structured processing on the data table associated with the SQL statement, as well as the filtering conditions and statistical strategies extracted from the SQL statement, according to preset processing rules, to generate a second type of monitoring strategy.

[0075] Optionally, the strategy update module 401 is specifically used to parse the custom SQL statement, extract the filtering conditions and statistical strategies, and determine the data table associated with the custom SQL statement; the strategy update module 401 is also specifically used to perform structured processing on the data table associated with the custom SQL statement, as well as the filtering conditions and statistical strategies extracted from the custom SQL statement, according to preset processing rules, to generate a third type of monitoring strategy.

[0076] Optionally, the policy update module 401 is specifically used to compare the extracted monitoring policy with the policies stored in the policy library and generate a comparison result; the policy update module 401 is also specifically used to append the extracted monitoring policy to the policy library if the comparison result indicates that there is no identical monitoring policy in the policy library, otherwise, update the monitoring policy stored in the policy library.

[0077] Optionally, the strategy execution module 402 is specifically used to determine the corresponding target platform job based on the target data table in the target monitoring strategy, and to monitor the execution status of the target platform job after the expected completion time of the target platform job; the strategy execution module 402 is also specifically used to trigger the execution of all monitoring strategies that depend on the target data table when the target platform job is successfully executed, and to obtain the execution result of each monitoring strategy.

[0078] Optionally, the alarm information includes: policy-level alarm information and topic-level alarm information; the anomaly detection module 403 is specifically used to generate corresponding policy-level alarm information and notify the relevant personnel corresponding to the target monitoring policy when the execution result of any target monitoring policy is abnormal; the anomaly detection module 403 is also specifically used to generate corresponding topic-level alarm information based on the target business topic corresponding to the target data table when the number of policy-level alarm information triggered by any target data table exceeds a preset threshold, and synchronize the abnormal status of the target business topic to the data lineage link, while identifying the affected upstream and downstream nodes.

[0079] The operation and maintenance device based on indicator-based strategy monitoring provided in this application first parses the business application logic in the data platform, extracts the corresponding monitoring rules and strategies, and updates the strategy library based on the extracted monitoring rules and strategies. The business application logic includes at least: indicator definitions, dashboard datasets, and custom SQL statements. The monitoring strategies include: data tables to be monitored, filtering conditions, and statistical strategies. Then, the execution status of the platform jobs corresponding to each monitoring strategy in the strategy library is monitored, and if the job is executed successfully, the execution of the monitoring strategy is triggered to obtain the execution result of each monitoring strategy. Finally, according to the preset anomaly judgment logic, anomaly analysis and judgment are performed on the execution results of each monitoring strategy, and if an anomaly is determined in the execution result of any monitoring strategy, corresponding alarm information is generated. In this way, by parsing the indicator logic of the platform business, key dimensions and core data tables in the business data are identified, and missing business metadata during operation and maintenance is supplemented, so that anomalies can be detected and handled in a timely manner, improving the operating efficiency of the data platform.

[0080] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a job operation and maintenance method based on indicator strategy monitoring. This method includes: first, parsing the business application logic in the data platform, extracting the corresponding monitoring strategies, and updating the strategy library based on the extracted monitoring strategies; the business application logic includes at least: indicator definitions, dashboard datasets, and custom SQL statements; the monitoring strategies include: data tables to be monitored, filtering conditions, and statistical strategies; then, monitoring the execution status of the platform job corresponding to each monitoring strategy in the strategy library, and triggering the execution of the monitoring strategy when the job is successfully executed, obtaining the execution result of each monitoring strategy; finally, performing anomaly analysis and judgment on the execution result of each monitoring strategy according to preset anomaly judgment logic, and generating corresponding alarm information when it is determined that an anomaly occurs in the execution result of any monitoring strategy. In this way, by analyzing the indicator logic of the platform business, the key dimensions and core data tables in the business data can be identified, and the missing business metadata during the operation and maintenance process can be supplemented, so that anomalies can be detected and handled in a timely manner, thereby improving the operating efficiency of the data platform.

[0081] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0082] On the other hand, this application also provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the operation and maintenance method based on indicator strategy monitoring provided by the above methods. The method includes: first, parsing the business application logic in the data platform, extracting the corresponding monitoring strategies, and updating the strategy library based on the extracted monitoring strategies; the business application logic includes at least: indicator definitions, dashboard datasets, and custom SQL statements; the monitoring strategies include: data tables to be monitored, filtering conditions, and statistical strategies; then, monitoring the execution status of the platform job corresponding to each monitoring strategy in the strategy library, and triggering the execution of the monitoring strategy when the job is successfully executed, obtaining the execution result of each monitoring strategy; finally, performing anomaly analysis and judgment on the execution result of each monitoring strategy according to preset anomaly judgment logic, and generating corresponding alarm information when it is determined that the execution result of any monitoring strategy is abnormal. In this way, by analyzing the indicator logic of the platform business, the key dimensions and core data tables in the business data can be identified, and the missing business metadata during the operation and maintenance process can be supplemented, so that anomalies can be detected and handled in a timely manner, thereby improving the operating efficiency of the data platform.

[0083] On another front, this application also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, this computer program implements the aforementioned job operation and maintenance methods based on indicator-based strategy monitoring. The method includes: first, parsing the business application logic in the data platform, extracting the corresponding monitoring strategies, and updating the strategy library based on the extracted monitoring strategies; the business application logic includes at least: indicator definitions, dashboard datasets, and custom SQL statements; the monitoring strategies include: data tables to be monitored, filtering conditions, and statistical strategies; then, monitoring the execution status of the platform job corresponding to each monitoring strategy in the strategy library, and triggering the execution of the monitoring strategy when the job is successfully executed, obtaining the execution result of each monitoring strategy; finally, performing anomaly analysis and judgment on the execution result of each monitoring strategy according to preset anomaly judgment logic, and generating corresponding alarm information when it is determined that anomalies occur in the execution result of any monitoring strategy. Thus, by parsing the indicator logic of the platform business, identifying key dimensions and core data tables in the business data, and supplementing the missing business metadata during operation and maintenance, anomalies can be detected and handled in a timely manner, improving the operating efficiency of the data platform.

[0084] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0085] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A job operation and maintenance method based on indicator-based strategy monitoring, characterized in that, include: The monitoring strategy is parsed for the business application logic in the data platform, the corresponding monitoring strategy is extracted, and the strategy library is updated based on the extracted monitoring strategy. The business application logic includes at least: indicator definition, dashboard dataset, and custom SQL statement; the monitoring strategy includes: the data table to be monitored, filtering conditions, and statistical strategies. The execution status of the platform job corresponding to each monitoring strategy in the strategy library is monitored, and if the job is executed successfully, the execution of the monitoring strategy is triggered to obtain the execution result of each monitoring strategy. Based on the preset anomaly judgment logic, the execution result of each monitoring strategy is analyzed and judged for anomalies, and when it is determined that the execution result of any monitoring strategy is abnormal, the corresponding alarm information is generated. The alarm information includes: policy-level alarm information used to characterize abnormal execution of monitoring policies, and topic-level alarm information used to characterize abnormal business topics to which data tables belong.

2. The operation and maintenance method based on indicator strategy monitoring according to claim 1, characterized in that, The step of parsing the monitoring strategy for the business application logic in the data platform and extracting the corresponding monitoring strategy includes: Parse the corresponding database tables and SQL statements from the metadata of the indicator definition, and extract the corresponding filtering conditions and statistical strategies from the parsed SQL statements; The parsed data table, along with the extracted filtering conditions and statistical strategies, are structured according to preset processing rules to generate the first type of monitoring strategy.

3. The operation and maintenance method based on indicator strategy monitoring according to claim 1, characterized in that, The step of parsing the monitoring strategy for the business application logic in the data platform and extracting the corresponding monitoring strategy includes: Extract filtering criteria and statistical strategies from the SQL statements contained in the Kanban dataset, and identify the data tables associated with the SQL statements; The data tables associated with the SQL statements, as well as the filtering conditions and statistical strategies extracted from the SQL statements, are structured according to preset processing rules to generate the second type of monitoring strategy.

4. The operation and maintenance method based on indicator strategy monitoring according to claim 1, characterized in that, The step of parsing the monitoring strategy for the business application logic in the data platform and extracting the corresponding monitoring strategy includes: The custom SQL statement is parsed to extract the filtering conditions and statistical strategies, and the data table associated with the custom SQL statement is determined. According to preset processing rules, the data tables associated with the custom SQL statement, as well as the filtering conditions and statistical strategies extracted from the custom SQL statement, are structured to generate a third type of monitoring strategy.

5. The operation and maintenance method based on indicator strategy monitoring according to claim 1, characterized in that, The update of the policy library based on the extracted monitoring policies includes: The extracted monitoring strategies are compared with the strategies stored in the strategy library, and a comparison result is generated. If the comparison result indicates that there is no identical monitoring policy in the policy library, the extracted monitoring policy is appended and stored in the policy library; otherwise, the monitoring policy stored in the policy library is updated.

6. The operation and maintenance method based on indicator strategy monitoring according to claim 1 or 5, characterized in that, The strategy library includes a strategy table and a source table. The strategy table stores at least one of the following: strategy identifier, table name, filtering conditions, statistical strategy, strategy type, and update time. The source table stores at least one of the following: strategy identifier, table name, strategy source, relevant personnel, and update time.

7. The operation and maintenance method based on indicator strategy monitoring according to claim 1, characterized in that, The process of monitoring the execution status of the platform job corresponding to each monitoring strategy in the strategy library, and triggering the execution of the monitoring strategy when the job is successfully executed, to obtain the execution result of each monitoring strategy, includes: Based on the target data table in the target monitoring strategy, the corresponding target platform job is determined, and the execution status of the target platform job is monitored after the expected completion time of the target platform job. When the target platform job is executed successfully, all monitoring strategies that depend on the target data table are triggered, and the execution result of each monitoring strategy is obtained.

8. The operation and maintenance method based on indicator strategy monitoring according to claim 1, characterized in that, The step involves performing anomaly analysis and judgment on the execution results of each monitoring strategy according to a preset anomaly judgment logic, and generating corresponding alarm information when an anomaly is determined in the execution result of any monitoring strategy, including: If the execution result of any target monitoring strategy is abnormal, a corresponding policy-level alarm message will be generated and the relevant personnel corresponding to the target monitoring strategy will be notified. If the number of policy-level alarm messages triggered by any target data table exceeds a preset threshold, a corresponding topic-level alarm message is generated based on the target business topic corresponding to the target data table, and the abnormal status of the target business topic is synchronized to the data lineage link, while the affected upstream and downstream nodes are identified.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the job operation and maintenance method based on indicator strategy monitoring as described in any one of claims 1 to 8.

10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the job operation and maintenance method based on indicator strategy monitoring as described in any one of claims 1 to 8.

Images