Data information monitoring method and device, electronic equipment and storage medium

Abstract

The application relates to the technical field of big data, and provides a data information monitoring method and device, electronic equipment and a storage medium, the method comprises the following steps: acquiring first information of to-be-monitored data; identifying the first information, acquiring a plurality of second information of a plurality of dependent data of the to-be-monitored data; determining a full-link task of the to-be-monitored data according to the plurality of second information of the plurality of dependent data; monitoring the running condition of the full-link task to perform time-effectiveness pre-judgment; and triggering an early warning when the task running time-effectiveness of the full-link task is substandard in the pre-judgment. The application performs time-effectiveness pre-judgment by monitoring the running condition of the full-link task, reduces the time-effectiveness influence on the to-be-monitored data, and improves the monitoring efficiency of data time-effectiveness monitoring.

Description

Technical Field

[0001] This invention relates to the field of big data technology, specifically to a data information monitoring method, device, electronic device, and storage medium. Background Technology

[0002] In traditional technologies, big data timeliness monitoring and early warning only considers the task operation status and does not take into account the data that is ultimately sent to the user terminal. This results in a lack of user value orientation in data timeliness monitoring and low efficiency.

[0003] Furthermore, during the data timeliness monitoring process, the priority of data tasks cannot be effectively determined, resulting in a relatively lagging data timeliness monitoring and early warning, and a lack of effective intelligent task scheduling and timely intervention. Summary of the Invention

[0004] In view of the above, it is necessary to propose a data information monitoring method, device, electronic device and storage medium, which can make timeliness prediction by monitoring the operation of the entire link task, reduce the impact on the timeliness of the monitored data and improve the monitoring efficiency of data timeliness monitoring.

[0005] A first aspect of the present invention provides a data information monitoring method, the method comprising:

[0006] Obtain the first information about the data to be monitored;

[0007] The first information is identified to obtain multiple second information of multiple dependent data of the data to be monitored;

[0008] The full-link task of the data to be monitored is determined based on multiple second pieces of information from the multiple dependent data.

[0009] Monitor the operation of the entire chain of tasks to make a timeliness prediction and determine whether the task execution time of the entire chain of tasks meets the standard;

[0010] When it is pre-determined that the task execution efficiency of the entire chain task will not meet the standard, an early warning will be triggered; or

[0011] When the pre-determined task runtime efficiency of the entire chain task meets the standard, target data is generated.

[0012] Optionally, the full-link task of determining the data to be monitored based on multiple second pieces of information from the multiple dependent data includes:

[0013] A global dependency graph is constructed based on the dependency relationships of the multiple dependency data, wherein each node of the global dependency graph contains the second information of the corresponding dependency data;

[0014] The full-link tasks for the data to be monitored are determined based on the global dependency graph.

[0015] Optionally, constructing a global dependency graph based on the dependency relationships of the multiple dependency data includes:

[0016] Identify multiple first dependent data that the data to be monitored depends on, and determine the multiple first dependent data as the first-level dependency relationship of the data to be monitored;

[0017] Continue to identify multiple second dependent data that each first dependent data depends on in the first-level dependency relationship of the data to be monitored, and determine the multiple second dependent data as the next level dependency relationship of the first-level dependency relationship of the data to be monitored;

[0018] Repeatedly execute the process of constructing the next level dependency relationship of the upper level dependency relationship of the data to be monitored based on the remaining dependency data that each dependency data in the upper level dependency relationship depends on, until all level dependency relationships are constructed, and obtain the global dependency relationship graph of the data to be monitored.

[0019] Optionally, the monitoring of the operation of the entire link task includes a timeliness pre-judgment, which includes determining whether the task execution timeliness of the entire link task meets the standard.

[0020] Monitor the first moment of task execution of the entire link task;

[0021] When the first time is earlier than the second time in the first information of the data to be monitored, it is pre-judged that the task runtime efficiency of the entire link task meets the standard.

[0022] When the first time is later than the second time in the first information of the data to be monitored, it is pre-judged that the task runtime efficiency of the end-to-end task is not up to standard.

[0023] Optionally, the monitoring of the operation of the entire link task includes a timeliness pre-judgment, which includes determining whether the task execution timeliness of the entire link task meets the standard.

[0024] The operation of the entire task chain is monitored based on timeliness anomaly rules;

[0025] When the operation of any task in the entire chain of tasks is monitored to meet the timeliness anomaly rule, the tag information of any task is identified;

[0026] When the label information of any one of the tasks is identified as the preset first level, an early warning is triggered; or, when the label information of any one of the tasks is identified as the preset second level, the operation of any one of the tasks is paused, the resource scheduling module is started, and resources are preferentially allocated from the full-link tasks to the task corresponding to the preset first level of the label information, the resources including: processor resources and memory resources;

[0027] Calculate the third time of task execution for the entire chain task;

[0028] When the third time is earlier than the second time in the first information of the data to be monitored, it is pre-judged that the task runtime efficiency of the entire link task meets the standard.

[0029] When the third time is later than the second time in the first information of the data to be monitored, it is pre-judged that the task execution efficiency of the entire link task is not up to standard.

[0030] Optionally, the calculation of the third time of task execution for the entire link task includes:

[0031] When the tag information of any task that meets the timeliness exception rule is a preset first level, the first duration of the task is obtained, and the historical runtime of the remaining tasks of the full-link task is obtained. The sum of the first duration and the historical runtime of the remaining tasks is determined as the second duration of the task running of the full-link task. Based on the second duration, the third duration of the task running of the full-link task is determined.

[0032] When the tag information of any task that meets the timeliness exception rule is a preset second level, the historical running time of the remaining tasks of the full-link task is obtained; the historical running time of the remaining tasks is determined as the third running time of the full-link task, and the third running time of the full-link task is determined based on the third running time.

[0033] Optionally, before performing a timeliness pre-judgment on the monitoring of the entire link task's operation status, and before pre-judging whether the task's runtime efficiency meets the standard, the method further includes:

[0034] Obtain historical runtime data of the entire chain of tasks for the data to be monitored within a preset time period;

[0035] Extract the target historical data of the entire chain where the task's runtime efficiency is not up to standard from the historical operation data;

[0036] Analyze the abnormal data in the target's historical data to obtain the analysis results;

[0037] Time-sensitive anomaly rules are created based on the analysis results.

[0038] A second aspect of the present invention provides a data information monitoring device, the device comprising:

[0039] The acquisition module is used to acquire the initial information of the data to be monitored.

[0040] The identification module is used to identify the first information and obtain multiple second information of multiple dependent data of the data to be monitored;

[0041] The determination module is used to determine the full-link task of the data to be monitored based on multiple second pieces of information of the multiple dependent data;

[0042] The monitoring module is used to monitor the operation of the entire chain task and make a timeliness prediction to determine whether the task execution time of the entire chain task meets the standard.

[0043] The triggering module is used to trigger an early warning when it is pre-determined that the runtime efficiency of the entire task is substandard; or

[0044] The generation module is used to generate target data when the task runtime efficiency of the pre-determined end-to-end task meets the standard.

[0045] A third aspect of the present invention provides an electronic device comprising a processor and a memory, wherein the processor is configured to implement the data information monitoring method by executing a computer program stored in the memory.

[0046] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the data information monitoring method described above.

[0047] In summary, the data information monitoring method, device, electronic device, and storage medium described in this invention can promote the construction of smart cities and be applied in fields such as smart buildings, smart security, smart communities, smart living, and the Internet of Things. By identifying the first information, multiple second information of multiple dependent data of the data to be monitored is obtained. Based on the multiple second information of the multiple dependent data, the entire chain task of the data to be monitored is determined. During the monitoring process, abnormal task nodes can be located in a timely manner, improving the efficiency and accuracy of data monitoring. The timeliness of the entire chain task is pre-judged. When any task in the entire chain task reports an error or meets the timeliness exception rule, a corresponding intervention strategy is determined based on the impact level of any task reporting an error or meeting the timeliness exception rule on the data to be monitored. Intervention is carried out based on the corresponding intervention strategy to reduce the impact on the timeliness of the data to be monitored, improving the monitoring efficiency of data timeliness monitoring. When the timeliness of the pre-judged entire chain task is not up to standard, an early warning is triggered, improving the accuracy of early warning for data information monitoring. Attached Figure Description

[0048] Figure 1 This is a flowchart of the data information monitoring method provided in Embodiment 1 of the present invention.

[0049] Figure 2 This is a structural diagram of the data information monitoring device provided in Embodiment 2 of the present invention.

[0050] Figure 3 This is a schematic diagram of the structure of the electronic device provided in Embodiment 3 of the present invention. Detailed Implementation

[0051] To better understand the above-mentioned objects, features, and advantages of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0053] Example 1

[0054] Figure 1 This is a flowchart of the data information monitoring method provided in Embodiment 1 of the present invention.

[0055] In this embodiment, the data information monitoring method can be applied to electronic devices. For electronic devices that need to monitor data information, the data information monitoring function provided by the method of this invention can be directly integrated into the electronic device, or it can be run in the electronic device in the form of a software development kit (SDK).

[0056] The embodiments of this invention can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0057] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, as well as machine learning and deep learning.

[0058] like Figure 1 As shown, the data information monitoring method specifically includes the following steps. Depending on different needs, the order of the steps in this flowchart can be changed, and some steps can be omitted.

[0059] S11, obtain the first information of the data to be monitored.

[0060] In this embodiment, the data to be monitored can be a report to be monitored, such as a financial report or a sales report. The first information is the basic information of the report, which is based on the data that will ultimately reach the user terminal. The basic information includes the report ID, report name, report directory, dependent data table name, expected timeframe, importance level, technical manager, and business manager of the report to be monitored.

[0061] In this embodiment, during data monitoring, in order to ensure that the monitoring results obtained from subsequent monitoring are the data required by the user terminal, the first information of the data to be monitored is obtained from the user terminal.

[0062] S12, identify the first information and obtain multiple second information of multiple dependent data of the data to be monitored.

[0063] In this embodiment, the second information can be the task running information for each dependent data, wherein the task running information includes the dependent data table name, task name, task running type, task running status, task running start time, task running end time, task running duration, etc.

[0064] For example, if the data to be monitored is a report, the report contains multiple sub-reports, and there may be dependencies between the multiple sub-reports, with each sub-report containing second information.

[0065] In an optional embodiment, identifying the first information and obtaining multiple pieces of second information from multiple dependent data of the data to be monitored includes:

[0066] Identify multiple report IDs in the first information;

[0067] Determine the corresponding dependent data name based on each report ID;

[0068] Traverse the database based on each dependency data name to obtain the task name for each dependency data;

[0069] The corresponding data interface is called based on the task name of each dependent data to obtain the second information of the corresponding dependent data.

[0070] In this embodiment, the largest and smallest report IDs are found in the database. The database is then divided into intervals based on the largest and smallest report IDs. Multiple threads are started to traverse the database simultaneously based on the interval division results to obtain the task name of each dependent data. Since multiple threads are started to traverse, the efficiency of obtaining the task name of each dependent data is improved.

[0071] For example, if the data to be monitored is a report, all report IDs in the first information corresponding to the report are identified, the corresponding dependent data table name is determined according to each report ID, the task name corresponding to the dependent data table name is found, and the corresponding task running information is obtained according to the task name.

[0072] S13, determine the full-link task of the data to be monitored based on the multiple second pieces of information of the multiple dependent data.

[0073] In this embodiment, the end-to-end task refers to all tasks related to the data to be monitored.

[0074] In an optional embodiment, the end-to-end task of determining the data to be monitored based on multiple pieces of second information from the multiple dependent data includes:

[0075] A global dependency graph is constructed based on the dependency relationships of the multiple dependency data, wherein each node of the global dependency graph contains the second information of the corresponding dependency data;

[0076] The full-link tasks for the data to be monitored are determined based on the global dependency graph.

[0077] In an optional embodiment, constructing a global dependency graph based on the dependency relationships of the plurality of dependency data includes:

[0078] Identify multiple first dependent data that the data to be monitored depends on, and determine the multiple first dependent data as the first-level dependency relationship of the data to be monitored;

[0079] Continue to identify multiple second dependent data that each first dependent data depends on in the first-level dependency relationship of the data to be monitored, and determine the multiple second dependent data as the next level dependency relationship of the first-level dependency relationship of the data to be monitored;

[0080] Repeatedly execute the process of constructing the next level dependency relationship of the upper level dependency relationship of the data to be monitored based on the remaining dependency data that each dependency data in the upper level dependency relationship depends on, until all level dependency relationships are constructed, and obtain the global dependency relationship graph of the data to be monitored.

[0081] In this embodiment, the first dependent data refers to the data located in the first layer of the global relationship graph; the second dependent data refers to the data located in the second layer of the global relationship graph.

[0082] In this embodiment, by analyzing the multiple second information dependencies of multiple dependent data, a global dependency graph is constructed, which can intuitively determine the full-link task of the data to be monitored. By obtaining the full-link task, abnormal task nodes can be located in a timely manner during the monitoring process, thereby improving the efficiency and accuracy of data monitoring.

[0083] S14, monitor the operation of the entire link task and make a timeliness prediction to determine whether the task execution time of the entire link task meets the standard.

[0084] In this embodiment, timeliness pre-judgment refers to performing timeliness judgment on the entire chain task in advance to determine whether the timeliness of the entire chain task meets the standard.

[0085] In an optional embodiment, the monitoring of the operation of the entire chain task includes a timeliness pre-judgment, which includes determining whether the task execution timeliness of the entire chain task meets the standard.

[0086] Monitor the first moment of task execution of the entire link task;

[0087] When the first time is earlier than the second time in the first information of the data to be monitored, it is pre-judged that the task runtime efficiency of the entire link task meets the standard.

[0088] When the first time is later than the second time in the first information of the data to be monitored, it is pre-judged that the task runtime efficiency of the end-to-end task is not up to standard.

[0089] In this embodiment, based on the monitoring rules and task operation status, the end time of the last task in the entire chain task is monitored, and the end time of the last task in the entire chain task is determined as the first time. The second time refers to the expected timeliness of the data to be monitored.

[0090] In an optional embodiment, the monitoring of the operation of the entire chain task includes a timeliness pre-judgment, which includes determining whether the task execution timeliness of the entire chain task meets the standard.

[0091] The operation of the entire task chain is monitored based on timeliness anomaly rules;

[0092] When the operation of any task in the entire chain of tasks is monitored to meet the timeliness anomaly rule, the tag information of any task is identified;

[0093] When the label information of any one of the tasks is identified as the preset first level, an early warning is triggered; or, when the label information of any one of the tasks is identified as the preset second level, the operation of any one of the tasks is paused, the resource scheduling module is started, and resources are preferentially allocated from the full-link tasks to the task corresponding to the preset first level of the label information, the resources including: processor resources and memory resources;

[0094] Calculate the third time of task execution for the entire chain task;

[0095] When the third time is earlier than the second time in the first information of the data to be monitored, it is pre-judged that the task runtime efficiency of the entire link task meets the standard.

[0096] When the third time is later than the second time in the first information of the data to be monitored, it is pre-judged that the task execution efficiency of the entire link task is not up to standard.

[0097] In this embodiment, the preset first level is that the impact of the task's tag information on the data to be monitored is high; the preset second level is that the impact of the task's tag information on the data to be monitored is low.

[0098] In this embodiment, the third time refers to the time predicted based on the current running status of the entire task.

[0099] Furthermore, the calculation of the third time of task execution for the entire link task includes:

[0100] When the tag information of any task that meets the timeliness exception rule is a preset first level, the first duration of the task is obtained, and the historical runtime of the remaining tasks of the full-link task is obtained. The sum of the first duration and the historical runtime of the remaining tasks is determined as the second duration of the task running of the full-link task. Based on the second duration, the third duration of the task running of the full-link task is determined.

[0101] When the tag information of any task that meets the timeliness exception rule is a preset second level, the historical running time of the remaining tasks of the full-link task is obtained; the historical running time of the remaining tasks is determined as the third running time of the full-link task, and the third running time of the full-link task is determined based on the third running time.

[0102] In this embodiment, time-sensitive exception rules can be pre-created when monitoring the operation of the data to be monitored.

[0103] Specifically, the creation process of the timeliness exception rule includes:

[0104] Obtain historical runtime data of the entire chain of tasks for the data to be monitored within a preset time period;

[0105] Extract the target historical data of the entire chain where the task's runtime efficiency is not up to standard from the historical operation data;

[0106] Analyze the abnormal data in the target's historical data to obtain the analysis results;

[0107] Time-sensitive anomaly rules are created based on the analysis results.

[0108] In this embodiment, a preset time period can be set in advance. Specifically, the preset time period can be set to the 30 days before the current time or the two weeks before the current time.

[0109] In this embodiment, non-compliant target historical data is extracted from the target historical data, and each non-compliant target historical data is converted into anomaly rule statements according to a preset conversion method. The anomaly rule statements are determined as analysis results, and timeliness anomaly rules are created by combining the anomaly rule statements and the corresponding preset thresholds.

[0110] In this embodiment, the timeliness exception rules may include: the runtime of a single task increases for N consecutive days beyond the corresponding task runtime threshold, the number of tasks with a runtime exceeding M hours is greater than a preset task threshold, a task reports an error, or the task start time or task waiting time exceeds a preset duration threshold, etc.

[0111] In this embodiment, in order to reduce the impact of the final timeliness of the monitored data, when a task that reports an error or meets the timeliness exception rules has a high impact level on the monitored data, an early warning is triggered and sent to the technical person in charge for timely maintenance; when a task that reports an error or meets the timeliness exception rules has a low impact level on the monitored data, the task is suspended, and the resource scheduling module prioritizes allocating resources to tasks with a high impact level on the monitored data.

[0112] In this embodiment, when any task in the end-to-end task reports an error or meets the timeliness exception rule, the corresponding intervention strategy is determined according to the impact level of any task reporting an error or meeting the timeliness exception rule on the data to be monitored. Intervention is carried out based on the corresponding intervention strategy to reduce the impact on the timeliness of the data to be monitored and improve the monitoring efficiency of data timeliness monitoring.

[0113] S15, when it is pre-determined that the task runtime efficiency of the entire link task is not up to standard, an early warning is triggered.

[0114] In this embodiment, the runtime of the end-to-end task is the sum of the runtimes of each task. The end time of the last task in the end-to-end task is determined as the final timeliness of the data to be monitored. For example, if the end time of the last task of the report to be monitored is 6 p.m., then the final timeliness of the report to be monitored is 6 p.m.

[0115] In an optional embodiment, triggering an early warning when it is pre-determined that the task runtime efficiency of the entire link task is substandard includes:

[0116] Identify the target level of the data to be monitored;

[0117] Based on the target level of the data to be monitored, an early warning is triggered according to a preset warning method.

[0118] In this embodiment, a warning method can be preset. Specifically, the preset warning method may include one or more of the following combinations: email warning; telephone warning; SMS warning.

[0119] For example, the alert method can be selected based on the importance level of the data to be monitored. For data with a high importance level, telephone alerts and email alerts can be selected, while for data with a low importance level, email alerts can be selected.

[0120] In this embodiment, if the task execution efficiency of the entire link task is not up to standard as predicted, an early warning is triggered in a timely manner, which improves the early warning efficiency and accuracy of data information monitoring.

[0121] S16, when it is pre-determined that the task runtime efficiency of the entire link task meets the standard, target data is generated.

[0122] In this embodiment, when the task execution efficiency of the entire link task meets the standard, target data is generated based on the first information of the data to be monitored. The generated target data is the data that will eventually reach the user terminal, which improves the utilization rate and monitoring efficiency of data timeliness monitoring.

[0123] In summary, the data monitoring method described in this embodiment identifies the first information to obtain multiple second information pieces from multiple dependent data of the data to be monitored. Based on the multiple second information pieces from the multiple dependent data, the entire chain task of the data to be monitored is determined. During the monitoring process, abnormal task nodes can be located in a timely manner, improving the efficiency and accuracy of data monitoring. The method monitors the operation of the entire chain task and performs timeliness pre-judgment. When any task in the entire chain task reports an error or meets a timeliness anomaly rule, a corresponding intervention strategy is determined based on the impact level of any task reporting an error or meeting the timeliness anomaly rule on the data to be monitored. Intervention is then carried out based on the corresponding intervention strategy to reduce the impact on the timeliness of the data to be monitored, improving the monitoring efficiency of data timeliness monitoring. When the pre-judged task execution timeliness of the entire chain task is deemed unsatisfactory, an early warning is triggered, improving the accuracy of early warnings for data information monitoring.

[0124] Example 2

[0125] Figure 2 This is a structural diagram of the data information monitoring device provided in Embodiment 2 of the present invention.

[0126] In some embodiments, the data information monitoring device 20 may include multiple functional modules composed of program code segments. The program code of each program segment in the data information monitoring device 20 may be stored in the memory of an electronic device and executed by the at least one processor to perform (see details). Figure 1 (Description) The function of monitoring data information.

[0127] In this embodiment, the data information monitoring device 20 can be divided into multiple functional modules according to its functions. These functional modules may include: an acquisition module 201, an identification module 202, a determination module 203, a monitoring module 204, a triggering module 205, and a generation module 206. The module referred to in this invention is a series of computer-readable instruction segments that can be executed by at least one processor and perform a fixed function, stored in memory. In this embodiment, the functions of each module will be detailed in subsequent embodiments.

[0128] The acquisition module 201 is used to acquire the first information of the data to be monitored.

[0129] In this embodiment, the data to be monitored can be a report to be monitored, such as a financial report or a sales report. The first information is the basic information of the report, which is based on the data that will ultimately reach the user terminal. The basic information includes the report ID, report name, report directory, dependent data table name, expected timeframe, importance level, technical manager, and business manager of the report to be monitored.

[0130] In this embodiment, during data monitoring, in order to ensure that the monitoring results obtained from subsequent monitoring are the data required by the user terminal, the first information of the data to be monitored is obtained from the user terminal.

[0131] The identification module 202 is used to identify the first information and obtain multiple second information of multiple dependent data of the data to be monitored.

[0132] In this embodiment, the second information can be the task running information for each dependent data, wherein the task running information includes the dependent data table name, task name, task running type, task running status, task running start time, task running end time, task running duration, etc.

[0133] For example, if the data to be monitored is a report, the report contains multiple sub-reports, and there may be dependencies between the multiple sub-reports, with each sub-report containing second information.

[0134] In an optional embodiment, the identification module 202 identifies the first information and obtains multiple second pieces of information about the multiple dependent data of the data to be monitored, including:

[0135] Identify multiple report IDs in the first information;

[0136] Determine the corresponding dependent data name based on each report ID;

[0137] Traverse the database based on each dependency data name to obtain the task name for each dependency data;

[0138] The corresponding data interface is called based on the task name of each dependent data to obtain the second information of the corresponding dependent data.

[0139] In this embodiment, the largest and smallest report IDs are found in the database. The database is then divided into intervals based on the largest and smallest report IDs. Multiple threads are started to traverse the database simultaneously based on the interval division results to obtain the task name of each dependent data. Since multiple threads are started to traverse, the efficiency of obtaining the task name of each dependent data is improved.

[0140] For example, if the data to be monitored is a report, all report IDs in the first information corresponding to the report are identified, the corresponding dependent data table name is determined according to each report ID, the task name corresponding to the dependent data table name is found, and the corresponding task running information is obtained according to the task name.

[0141] The determination module 203 is used to determine the full-link task of the data to be monitored based on multiple second pieces of information of the multiple dependent data.

[0142] In this embodiment, the end-to-end task refers to all tasks related to the data to be monitored.

[0143] In an optional embodiment, the determining module 203 determines the end-to-end task of the data to be monitored based on multiple second pieces of information from the multiple dependent data, including:

[0144] A global dependency graph is constructed based on the dependency relationships of the multiple dependency data, wherein each node of the global dependency graph contains the second information of the corresponding dependency data;

[0145] The full-link tasks for the data to be monitored are determined based on the global dependency graph.

[0146] In an optional embodiment, constructing a global dependency graph based on the dependency relationships of the plurality of dependency data includes:

[0147] Identify multiple first dependent data that the data to be monitored depends on, and determine the multiple first dependent data as the first-level dependency relationship of the data to be monitored;

[0148] Continue to identify multiple second dependent data that each first dependent data depends on in the first-level dependency relationship of the data to be monitored, and determine the multiple second dependent data as the next level dependency relationship of the first-level dependency relationship of the data to be monitored;

[0149] Repeatedly execute the process of constructing the next level dependency relationship of the upper level dependency relationship of the data to be monitored based on the remaining dependency data that each dependency data in the upper level dependency relationship depends on, until all level dependency relationships are constructed, and obtain the global dependency relationship graph of the data to be monitored.

[0150] In this embodiment, the first dependent data refers to the data located in the first layer of the global relationship graph; the second dependent data refers to the data located in the second layer of the global relationship graph.

[0151] In this embodiment, by analyzing the multiple second information dependencies of multiple dependent data, a global dependency graph is constructed, which can intuitively determine the full-link task of the data to be monitored. By obtaining the full-link task, abnormal task nodes can be located in a timely manner during the monitoring process, thereby improving the efficiency and accuracy of data monitoring.

[0152] The monitoring module 204 is used to monitor the operation of the entire chain task and make a timeliness prediction to determine whether the task operation time of the entire chain task meets the standard.

[0153] In this embodiment, timeliness pre-judgment refers to performing timeliness judgment on the entire chain task in advance to determine whether the timeliness of the entire chain task meets the standard.

[0154] In an optional embodiment, the monitoring module 204 monitors the operation of the entire chain task and performs a timeliness pre-judgment. The pre-judgment of whether the task execution time of the entire chain task meets the standard includes:

[0155] Monitor the first moment of task execution of the entire link task;

[0156] When the first time is earlier than the second time in the first information of the data to be monitored, it is pre-judged that the task runtime efficiency of the entire link task meets the standard.

[0157] When the first time is later than the second time in the first information of the data to be monitored, it is pre-judged that the task runtime efficiency of the end-to-end task is not up to standard.

[0158] In this embodiment, based on the monitoring rules and task operation status, the end time of the last task in the entire chain task is monitored, and the end time of the last task in the entire chain task is determined as the first time. The second time refers to the expected timeliness of the data to be monitored.

[0159] In an optional embodiment, the monitoring module 204 monitors the operation of the entire chain task and performs a timeliness pre-judgment. The pre-judgment of whether the task execution time of the entire chain task meets the standard includes:

[0160] The operation of the entire task chain is monitored based on timeliness anomaly rules;

[0161] When the operation of any task in the entire chain of tasks is monitored to meet the timeliness anomaly rule, the tag information of any task is identified;

[0162] When the label information of any one of the tasks is identified as the preset first level, an early warning is triggered; or, when the label information of any one of the tasks is identified as the preset second level, the operation of any one of the tasks is paused, the resource scheduling module is started, and resources are preferentially allocated from the full-link tasks to the task corresponding to the preset first level of the label information, the resources including: processor resources and memory resources;

[0163] Calculate the third time of task execution for the entire chain task;

[0164] When the third time is earlier than the second time in the first information of the data to be monitored, it is pre-judged that the task runtime efficiency of the entire link task meets the standard.

[0165] When the third time is later than the second time in the first information of the data to be monitored, it is pre-judged that the task execution efficiency of the entire link task is not up to standard.

[0166] In this embodiment, the preset first level is that the impact of the task's tag information on the data to be monitored is high; the preset second level is that the impact of the task's tag information on the data to be monitored is low.

[0167] In this embodiment, the third time refers to the time predicted based on the current running status of the entire task.

[0168] Furthermore, the calculation of the third time of task execution for the entire link task includes:

[0169] When the tag information of any task that meets the timeliness exception rule is a preset first level, the first duration of the task is obtained, and the historical runtime of the remaining tasks of the full-link task is obtained. The sum of the first duration and the historical runtime of the remaining tasks is determined as the second duration of the task running of the full-link task. Based on the second duration, the third duration of the task running of the full-link task is determined.

[0170] When the tag information of any task that meets the timeliness exception rule is a preset second level, the historical running time of the remaining tasks of the full-link task is obtained; the historical running time of the remaining tasks is determined as the third running time of the full-link task, and the third running time of the full-link task is determined based on the third running time.

[0171] In this embodiment, time-sensitive exception rules can be pre-created when monitoring the operation of the data to be monitored.

[0172] Specifically, the creation process of the timeliness exception rule includes:

[0173] Obtain historical runtime data of the entire chain of tasks for the data to be monitored within a preset time period;

[0174] Extract the target historical data of the entire chain where the task's runtime efficiency is not up to standard from the historical operation data;

[0175] Analyze the abnormal data in the target's historical data to obtain the analysis results;

[0176] Time-sensitive anomaly rules are created based on the analysis results.

[0177] In this embodiment, a preset time period can be set in advance. Specifically, the preset time period can be set to the 30 days before the current time or the two weeks before the current time.

[0178] In this embodiment, non-compliant target historical data is extracted from the target historical data, and each non-compliant target historical data is converted into anomaly rule statements according to a preset conversion method. The anomaly rule statements are determined as analysis results, and timeliness anomaly rules are created by combining the anomaly rule statements and the corresponding preset thresholds.

[0179] In this embodiment, the timeliness exception rules may include: the runtime of a single task increases for N consecutive days beyond the corresponding task runtime threshold, the number of tasks with a runtime exceeding M hours is greater than a preset task threshold, a task reports an error, or the task start time or task waiting time exceeds a preset duration threshold, etc.

[0180] In this embodiment, in order to reduce the impact of the final timeliness of the monitored data, when a task that reports an error or meets the timeliness exception rules has a high impact level on the monitored data, an early warning is triggered and sent to the technical person in charge for timely maintenance; when a task that reports an error or meets the timeliness exception rules has a low impact level on the monitored data, the task is suspended, and the resource scheduling module prioritizes allocating resources to tasks with a high impact level on the monitored data.

[0181] In this embodiment, when any task in the end-to-end task reports an error or meets the timeliness exception rule, the corresponding intervention strategy is determined according to the impact level of any task reporting an error or meeting the timeliness exception rule on the data to be monitored. Intervention is carried out based on the corresponding intervention strategy to reduce the impact on the timeliness of the data to be monitored and improve the monitoring efficiency of data timeliness monitoring.

[0182] Trigger module 205 is used to trigger an early warning when it is pre-determined that the task runtime efficiency of the entire link task is not up to standard.

[0183] In this embodiment, the runtime of the end-to-end task is the sum of the runtimes of each task. The end time of the last task in the end-to-end task is determined as the final timeliness of the data to be monitored. For example, if the end time of the last task of the report to be monitored is 6 p.m., then the final timeliness of the report to be monitored is 6 p.m.

[0184] In an optional embodiment, when the triggering module 205 pre-determines that the task runtime efficiency of the end-to-end task is not up to standard, the triggering warning includes:

[0185] Identify the target level of the data to be monitored;

[0186] Based on the target level of the data to be monitored, an early warning is triggered according to a preset warning method.

[0187] In this embodiment, a warning method can be preset. Specifically, the preset warning method may include one or more of the following combinations: email warning; telephone warning; SMS warning.

[0188] For example, the alert method can be selected based on the importance level of the data to be monitored. For data with a high importance level, telephone alerts and email alerts can be selected, while for data with a low importance level, email alerts can be selected.

[0189] In this embodiment, if the task execution efficiency of the entire link task is not up to standard as predicted, an early warning is triggered in a timely manner, which improves the early warning efficiency and accuracy of data information monitoring.

[0190] The generation module 206 is used to generate target data when it is pre-determined that the task runtime efficiency of the full-link task meets the standard.

[0191] In this embodiment, when the task execution efficiency of the entire link task meets the standard, target data is generated based on the first information of the data to be monitored. The generated target data is the data that will eventually reach the user terminal, which improves the utilization rate and monitoring efficiency of data timeliness monitoring.

[0192] In summary, the data information monitoring device described in this embodiment identifies the first information to obtain multiple second information of multiple dependent data of the data to be monitored. Based on the multiple second information of the multiple dependent data, it determines the entire chain task of the data to be monitored. During the monitoring process, it can promptly locate abnormal task nodes, improving the efficiency and accuracy of data monitoring. It monitors the operation of the entire chain task and performs timeliness pre-judgment. When any task in the entire chain task reports an error or meets the timeliness anomaly rule, it determines the corresponding intervention strategy based on the impact level of any task reporting an error or meeting the timeliness anomaly rule on the data to be monitored. Intervention is then carried out based on the corresponding intervention strategy to reduce the impact on the timeliness of the data to be monitored, improving the monitoring efficiency of data timeliness monitoring. When the pre-judged task execution timeliness of the entire chain task is not up to standard, an early warning is triggered, improving the accuracy of early warning in data information monitoring.

[0193] Example 3

[0194] See Figure 3 The diagram shown is a structural schematic of an electronic device provided in Embodiment 3 of the present invention. In a preferred embodiment of the present invention, the electronic device 3 includes a memory 31, at least one processor 32, at least one communication bus 33, and a transceiver 34.

[0195] Those skilled in the art should understand that Figure 3 The structure of the electronic device shown does not constitute a limitation of the embodiments of the present invention. It can be a bus structure or a star structure. The electronic device 3 may also include more or fewer other hardware or software than shown, or different component arrangements.

[0196] In some embodiments, the electronic device 3 is an electronic device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), digital processors, and embedded devices. The electronic device 3 may also include client devices, including, but not limited to, any electronic product capable of human-computer interaction with a client via a keyboard, mouse, remote control, touchpad, or voice control device, such as personal computers, tablet computers, smartphones, and digital cameras.

[0197] It should be noted that the electronic device 3 is merely an example. Other existing or future electronic products that are suitable for this invention should also be included within the scope of protection of this invention and are incorporated herein by reference.

[0198] In some embodiments, the memory 31 is used to store program code and various data, such as the data information monitoring device 20 installed in the electronic device 3, and to achieve high-speed and automatic access to programs or data during the operation of the electronic device 3. The memory 31 includes read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0199] In some embodiments, the at least one processor 32 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The at least one processor 32 is the control unit of the electronic device 3, connecting various components of the entire electronic device 3 via various interfaces and lines. It executes programs or modules stored in the memory 31 and calls data stored in the memory 31 to perform various functions and process data of the electronic device 3.

[0200] In some embodiments, the at least one communication bus 33 is configured to enable communication between the memory 31 and the at least one processor 32, etc.

[0201] Although not shown, the electronic device 3 may also include a power supply (such as a battery) to power the various components. Optionally, the power supply may be logically connected to the at least one processor 32 via a power management device, thereby enabling functions such as charging, discharging, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 3 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0202] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0203] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) or processor to execute portions of the methods described in the various embodiments of the present invention.

[0204] In a further embodiment, combined with Figure 2 The at least one processor 32 can execute the operating device of the electronic device 3 and various installed applications (such as the data information monitoring device 20), program code, etc., for example, the various modules mentioned above.

[0205] The memory 31 stores program code, and the at least one processor 32 can call the program code stored in the memory 31 to execute related functions. For example, Figure 2 The modules described herein are program codes stored in the memory 31 and executed by the at least one processor 32, thereby realizing the functions of the modules to achieve the purpose of data information monitoring.

[0206] For example, the program code can be divided into one or more modules / units, which are stored in the memory 31 and executed by the processor 32 to complete this application. The one or more modules / units can be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the program code in the electronic device 3. For example, the program code can be divided into an acquisition module 201, an identification module 202, a determination module 203, a monitoring module 204, a triggering module 205, and a generation module 206.

[0207] In one embodiment of the present invention, the memory 31 stores a plurality of computer-readable instructions, which are executed by the at least one processor 32 to perform the function of monitoring data information.

[0208] Specifically, the specific implementation method of the above instructions by the at least one processor 32 can be found in [reference]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0209] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0210] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0211] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0212] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other elements, and the singular does not exclude the plural. Multiple elements or devices recited in the present invention may also be implemented by a single element or device in software or hardware. The terms "first," "second," etc., are used to denote names and do not indicate any particular order.

[0213] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A data information monitoring method, characterized in that, The method includes: Obtain the first information about the data to be monitored; The first information is identified to obtain multiple second information of multiple dependent data of the data to be monitored; The full-link task of the data to be monitored is determined based on multiple second pieces of information from the multiple dependent data. The system monitors the operation of the entire chain of tasks and performs timeliness pre-judgment to determine whether the task execution timeliness of the entire chain of tasks meets the standard. This includes: monitoring the operation of the entire chain of tasks based on timeliness anomaly rules; when the operation of any task in the entire chain of tasks is found to meet the timeliness anomaly rules, identifying the tag information of that task; when the tag information of any task is identified as a preset first level, triggering an early warning; or, when the tag information of any task is identified as a preset second level, pausing the operation of that task, starting the resource scheduling module, and prioritizing the allocation of resources from the entire chain of tasks whose tag information corresponds to the preset first level, the resources including processor resources and memory resources; calculating the third time of the task execution of the entire chain of tasks; when the third time is earlier than the second time in the first information of the data to be monitored, pre-judging that the task execution timeliness of the entire chain of tasks meets the standard; when the third time is later than the second time in the first information of the data to be monitored, pre-judging that the task execution timeliness of the entire chain of tasks does not meet the standard. When it is pre-determined that the task execution efficiency of the entire chain task will not meet the standard, an early warning will be triggered; or When the pre-determined task runtime efficiency of the entire chain task meets the standard, target data is generated.

2. The data information monitoring method as described in claim 1, characterized in that, The full-link task of determining the data to be monitored based on multiple second pieces of information from the multiple dependent data includes: A global dependency graph is constructed based on the dependency relationships of the multiple dependency data, wherein each node of the global dependency graph contains the second information of the corresponding dependency data; The full-link tasks for the data to be monitored are determined based on the global dependency graph.

3. The data information monitoring method as described in claim 2, characterized in that, The construction of a global dependency graph based on the dependency relationships of the multiple dependency data includes: Identify multiple first dependent data that the data to be monitored depends on, and determine the multiple first dependent data as the first-level dependency relationship of the data to be monitored; Continue to identify multiple second dependent data that each first dependent data depends on in the first-level dependency relationship of the data to be monitored, and determine the multiple second dependent data as the next level dependency relationship of the first-level dependency relationship of the data to be monitored; Repeatedly execute the process of constructing the next level dependency relationship of the upper level dependency relationship of the data to be monitored based on the remaining dependency data that each dependency data in the upper level dependency relationship depends on, until all level dependency relationships are constructed, and obtain the global dependency relationship graph of the data to be monitored.

4. The data information monitoring method as described in claim 1, characterized in that, The monitoring of the entire task's execution status includes a timeliness prediction, which determines whether the task's execution time meets the standard. Monitor the first moment of task execution of the entire link task; When the first time is earlier than the second time in the first information of the data to be monitored, it is pre-judged that the task runtime efficiency of the entire link task meets the standard. When the first time is later than the second time in the first information of the data to be monitored, it is pre-judged that the task runtime efficiency of the end-to-end task is not up to standard.

5. The data information monitoring method as described in claim 1, characterized in that, The calculation of the third time of task execution for the entire task includes: When the tag information of any task that meets the timeliness exception rule is a preset first level, the first duration of the task is obtained, and the historical runtime of the remaining tasks of the full-link task is obtained. The sum of the first duration and the historical runtime of the remaining tasks is determined as the second duration of the task running of the full-link task. Based on the second duration, the third duration of the task running of the full-link task is determined. When the tag information of any task that meets the timeliness exception rule is a preset second level, the historical running time of the remaining tasks of the full-link task is obtained; the historical running time of the remaining tasks is determined as the third running time of the full-link task, and the third running time of the full-link task is determined based on the third running time.

6. The data information monitoring method as described in claim 1, characterized in that, Before performing a timeliness pre-judgment on the monitoring of the entire chain task's operation status, and before pre-judging whether the task's runtime efficiency meets the standard, the method further includes: Obtain historical runtime data of the entire chain of tasks for the data to be monitored within a preset time period; Extract the target historical data of the entire chain where the task's runtime efficiency is not up to standard from the historical operation data; Analyze the abnormal data in the target's historical data to obtain the analysis results; Time-sensitive anomaly rules are created based on the analysis results.

7. A data information monitoring device, characterized in that, The device includes: The acquisition module is used to acquire the initial information of the data to be monitored. The identification module is used to identify the first information and obtain multiple second information of multiple dependent data of the data to be monitored; The determination module is used to determine the full-link task of the data to be monitored based on multiple second pieces of information of the multiple dependent data; The monitoring module is used to monitor the operation of the entire chain task and perform timeliness pre-judgment to determine whether the task execution time of the entire chain task meets the standard. This includes: monitoring the operation of the entire chain task based on timeliness anomaly rules; when the operation of any task in the entire chain task meets the timeliness anomaly rules, identifying the tag information of that task; when the tag information of any task is identified as a preset first level, triggering an early warning; or, when the tag information of any task is identified as a preset second level, pausing the operation of that task, starting the resource scheduling module, and prioritizing resource allocation from the entire chain task to the task corresponding to the preset first level tag information, the resources including processor resources and memory resources; calculating the third time of the task execution of the entire chain task; when the third time is earlier than the second time in the first information of the data to be monitored, pre-judging that the task execution time of the entire chain task meets the standard; when the third time is later than the second time in the first information of the data to be monitored, pre-judging that the task execution time of the entire chain task does not meet the standard. The triggering module is used to trigger an early warning when it is pre-determined that the runtime efficiency of the entire task is substandard; or The generation module is used to generate target data when the task runtime efficiency of the pre-determined end-to-end task meets the standard.

8. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the data information monitoring method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the data information monitoring method as described in any one of claims 1 to 6.

Images