Data monitoring method, data warehouse, electronic equipment, medium and product

Abstract

The invention provides a data monitoring method, a data warehouse, electronic equipment, a medium and a product, and relates to the technical field of data warehouses. Comprising the steps that a first monitoring task is executed on the task scheduling condition of a first data layer according to a first preset frequency, the first data layer is used for executing a first scheduling task according to the first preset frequency, and the first scheduling task is used for obtaining newly-added data of an external data source in a corresponding period and recording data obtaining time; and in response to the situation that the completion condition of the ith first scheduling task detected by the ith first monitoring task does not meet a first preset condition, updating a corresponding period of the (i + 1) th first scheduling task, so that the (i + 1) th first scheduling task obtains newly added data of the external data source in the updated period. The problem that the existing monitoring can only judge the abnormity according to the task state and cannot capture the data synchronization abnormity based on the task effect can be solved, the data accuracy is guaranteed, and the data synchronization delay is avoided.

Description

Technical Field

[0001] The present application relates to the technical field of data warehouses, and in particular to a data monitoring method, a data warehouse, an electronic device, a medium and a product. Background Art

[0002] With the development of business, massive amounts of data have emerged. The data warehouse is a structured data environment for decision support systems and online analytical application data sources. As a bridge between online production systems and downstream data application systems, corresponding adjustments are also needed to ensure data consistency and maintain upstream and downstream linkages to avoid stagnant production operations caused by abnormal data scheduling. Summary of the invention

[0003] Embodiments of the present application provide a data monitoring method, a data warehouse, an electronic device, a computer-readable storage medium, and a computer program product to alleviate or solve one or more technical problems existing in the prior art.

[0004] In a first aspect, an embodiment of the present application provides a data monitoring method, which is applied to a data warehouse, wherein the data warehouse includes a first data layer, and the method includes: executing a first monitoring task on a task scheduling situation of the first data layer according to a first preset frequency, wherein the first data layer is used to execute a first scheduling task according to the first preset frequency, and the first scheduling task is used to obtain new data from an external data source within a corresponding period and record the data acquisition time; wherein the start execution time of the first monitoring task is later than the first scheduling task;

[0005] In response to the i-th first monitoring task detecting that the completion status of the i-th first scheduling task does not meet the first preset condition, the corresponding period of the i+1-th first scheduling task is updated so that the i+1-th first scheduling task obtains new data from the external data source within the updated period, where i is a positive integer.

[0006] In a second aspect, an embodiment of the present application provides a data warehouse, which includes a first data layer, a second data layer and a third data layer in sequence according to the data flow direction, the first data layer is used to execute a first scheduling task according to a first preset frequency, the first scheduling task is used to obtain new data from an external data source within a corresponding period and record the data acquisition time; the second data layer is used to execute a second scheduling task according to a second preset frequency, the second scheduling task is used to synchronize the new data of the first data layer within the corresponding period and record the data synchronization time; the third data layer is used to execute a third scheduling task according to a third preset frequency, the third scheduling task is used to extract the new data of the second data layer within the corresponding period and record the data extraction time;

[0007] The first preset frequency is a multiple of the second preset frequency, the second preset frequency is a multiple of the third preset frequency, and the data warehouse is used for:

[0008] Execute a first monitoring task on the task scheduling of the first data layer according to a first preset frequency, and in response to the i-th first monitoring task detecting that the completion of the i-th first scheduling task does not meet the first preset condition, update the corresponding period of the i+1-th first scheduling task, so that the i+1-th first scheduling task obtains the newly added data of the external data source within the updated period; the start execution time of the first monitoring task is later than the first scheduling task, where i is a positive integer;

[0009] Execute a second monitoring task on the task scheduling of the second data layer according to a second preset frequency, and in response to the i-th second monitoring task detecting that the completion of the i-th second scheduling task does not meet the second preset condition, update the corresponding period of the i+1-th second scheduling task, so that the i+1-th second scheduling task synchronizes the newly added data of the first data layer in the period after the update; the start execution time of the second monitoring task is later than the second scheduling task;

[0010] A third monitoring task is executed on the task scheduling of the third data layer according to a third preset frequency, and in response to the i-th third monitoring task detecting that the completion status of the i-th third scheduling task does not meet the third preset condition, the corresponding period of the i+1-th third scheduling task is updated so that the i+1-th third scheduling task extracts the newly added data of the second data layer within the updated period; the start execution time of the third monitoring task is later than that of the third scheduling task.

[0011] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, and the processor implements any method of the embodiment of the present application when executing the computer program.

[0012] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method of any one of the embodiments of the present application is implemented.

[0013] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements any method of the embodiments of the present application when executed by a processor.

[0014] Based on the method of the embodiment of the present application, the scheduling of each data layer in the data warehouse can be planned and a synchronization monitoring mechanism can be configured, which can overcome the problem that the existing monitoring can only judge anomalies based on the task status but cannot capture data synchronization anomalies based on the task effect, thereby ensuring data accuracy and avoiding data synchronization delays.

[0015] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.

[0017] Figure 1 is a schematic diagram of the architecture of a data warehouse according to an embodiment of the present application;

[0018] Figure 2 is a flow chart of a data monitoring method according to an embodiment of the present application;

[0019] Figure 3 is a schematic diagram of the execution logic of the first scheduling task and the first monitoring task according to an embodiment of the present application;

[0020] Figure 4 is a schematic diagram of the execution logic of the second scheduling task and the second monitoring task according to an embodiment of the present application;

[0021] Figure 5 is a schematic diagram of the execution logic of the third scheduling task and the third monitoring task according to an embodiment of the present application;

[0022] Figure 6 This is a schematic diagram of ODS layer multi-task monitoring data flow according to an embodiment of the present application;

[0023] Figure 7 It is a block diagram of an electronic device used to implement the embodiment of the present application. DETAILED DESCRIPTION

[0024] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the concept or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary in nature and not restrictive.

[0025] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies of the embodiments of the present application. The following related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0026] Figure 1 It is a schematic diagram of the architecture of a data warehouse 10 according to an embodiment of the present application, which includes a first data layer 101, a second data layer 102 and a third data layer 103 in order according to data flow.

[0027] Here, the first data layer 101 may be an ODS (Operational Data Store) layer, which is a system for storing business data from multiple data sources and a transition from an external data source to a data warehouse. The data structure of the ODS is generally consistent with the data source and is a "subject-oriented, integrated, variable, current data value-reflecting and detailed data collection"; the second data layer 102102 may be a DW (Data Warehouse) layer, which is the destination of the data. The DW layer is the core part of the entire data warehouse architecture and is used to obtain all data from the ODS layer and store it for a long time, and the data will not be modified; the third data layer 103 may be a DM (Data Mart) layer, also called a data market, which is a subset of the data warehouse and focuses on a specific business field or department. It is for a specific application purpose, and a part of the data separated from the data warehouse can also be called subject data. Among them, the ODS layer, the DW layer and the DM layer are actually some logical layers in the data warehouse system. In the embodiment of the present application, a typical three-layer architecture is taken as an example, and in actual applications, the data warehouse can also be divided into more layers of architecture according to needs, and the present application is not limited to this.

[0028] ETL is the abbreviation of Extract, Transform and Load, which is used to describe the process of extracting, transforming and loading data from the source to the destination. Data extraction is to read data from the data source, data transformation is to convert the original data into the desired format and dimension, and data loading is to load the processed data to the target. The data warehouse in the prior art can use ETL tools to support email warnings for a single task to notify the operation and maintenance personnel of task operation abnormalities, but this warning can only be configured in a single task, and when the task itself has data synchronization delays, data synchronization pauses, data synchronization timeouts, etc., automatic warnings cannot be achieved, and when the lower data layer does not monitor the task synchronization delay, it will also cause the problem that the time range of data extraction in the upper database does not match the lower data layer, resulting in data synchronization delays or abnormalities in the upper database. Therefore, the task monitoring configured by the ETL tool of the data warehouse in the prior art cannot completely solve the problem of inaccurate data caused by data synchronization abnormalities.

[0029] Based on this, in order to solve the problem of inaccurate data caused by abnormal data synchronization in the data warehouse, firstly, the embodiment of the present application plans the data scheduling of each data layer in the data warehouse 10, specifically:

[0030] The first data layer 101 is used to execute the first scheduling task according to the first preset frequency, and the first scheduling task is used to obtain new data from the external data source within the corresponding period and record the data acquisition time; the second data layer 102 is used to execute the second scheduling task according to the second preset frequency, and the second scheduling task is used to synchronize the new data of the first data layer 101 within the corresponding period and record the data synchronization time; the third data layer 103 is used to execute the third scheduling task according to the third preset frequency, and the third scheduling task is used to extract the new data of the second data layer 102 within the corresponding period and record the data extraction time.

[0031] Here, the first preset frequency is a multiple of the second preset frequency, and the second preset frequency is a multiple of the third preset frequency. It can be understood that the data scheduling period of the second data layer 102 is an integer multiple of the data scheduling period of the first data layer 101, and the minimum setting of the data scheduling period of the second data layer 102 is equal to the data scheduling period of the first data layer 101. The data scheduling period of the third data layer 103 is an integer multiple of the data scheduling period of the second data layer 102, and the minimum setting of the data scheduling period of the third data layer 103 is equal to the data scheduling period of the second data layer 102. The data update frequency of the lower data layer can be made a multiple of the data update frequency of the upper data layer, and on this basis, the start time of the data scheduling of the upper data layer is configured to be appropriately later than the data scheduling of the lower data layer, so as to realize the regular synchronous update of each data layer in the data warehouse.

[0032] On this basis, in order to overcome the impact of scheduling task delays or anomalies on the current data layer and other data layers, the data warehouse is also configured as follows:

[0033] Execute a first monitoring task on the task scheduling of the first data layer according to a first preset frequency, and in response to the i-th first monitoring task detecting that the completion of the i-th first scheduling task does not meet the first preset condition, update the corresponding period of the i+1-th first scheduling task, so that the i+1-th first scheduling task obtains the newly added data of the external data source within the updated period; the start execution time of the first monitoring task is later than the first scheduling task, where i is a positive integer;

[0034] Execute a second monitoring task on the task scheduling of the second data layer according to a second preset frequency, and in response to the i-th second monitoring task detecting that the completion of the i-th second scheduling task does not meet the second preset condition, update the corresponding period of the i+1-th second scheduling task, so that the i+1-th second scheduling task synchronizes the newly added data of the first data layer in the period after the update; the start execution time of the second monitoring task is later than the second scheduling task;

[0035] A third monitoring task is executed on the task scheduling of the third data layer according to a third preset frequency, and in response to the i-th third monitoring task detecting that the completion status of the i-th third scheduling task does not meet the third preset condition, the corresponding period of the i+1-th third scheduling task is updated so that the i+1-th third scheduling task extracts the newly added data of the second data layer within the updated period; the start execution time of the third monitoring task is later than that of the third scheduling task.

[0036] Here, the embodiment of the present application configures a monitoring task for each data layer separately on the basis of configuring the data scheduling of each data layer in the data warehouse. By controlling the start execution time of the monitoring task to be later than the scheduling task of the corresponding layer and executing it at the same frequency as the scheduling task, the monitoring task can judge the completion status of the previous scheduling task during the execution interval of two adjacent scheduling tasks, and modify the original corresponding period of the next scheduling task when it is judged that the completion status of the scheduling task does not meet the conditions, so that the next scheduling task, when it needs to complete the task scheduling of its original period, can supplement the scheduling of the data within the unfinished time period of the previous scheduling task, thereby effectively monitoring the data scheduling status of the current data layer.

[0037] Specifically, the present application also proposes a data monitoring method to achieve Figure 1 Monitoring and early warning of data scheduling in data warehouses, Figure 2 : is a flow chart of a data monitoring method according to an embodiment of the present application, comprising:

[0038] Step S210, executing a first monitoring task on the task scheduling of the first data layer according to a first preset frequency, the first data layer is used to execute the first scheduling task according to the first preset frequency, the first scheduling task is used to obtain new data from the external data source within a corresponding period and record the data acquisition time; wherein the start execution time of the first monitoring task is later than the first scheduling task;

[0039] Step S220, in response to the i-th first monitoring task detecting that the completion status of the i-th first scheduling task does not meet the first preset condition, updates the corresponding period of the i+1-th first scheduling task so that the i+1-th first scheduling task obtains new data from the external data source within the updated period, where i is a positive integer.

[0040] Here, the first data layer is also the ODS layer, and the first preset condition includes that the first scheduling task is in a ready state and the recorded data acquisition time conforms to the corresponding period of the i-th first scheduling task.

[0041] Figure 3 FIG. 4 shows a schematic diagram of the execution logic of the first scheduling task and the first monitoring task in the ODS layer, such as Figure 3As shown, when the ODS layer executes the first scheduling task, three task results may be obtained. For example, assuming that the scheduling period corresponding to the first preset frequency is 5 minutes, the first scheduling task is configured to be executed every 5 minutes. If the first scheduling task needs to obtain new data from the external data source between 8:55 and 9:00, the first task result is that the task is successfully executed, and the time parameter of the new data from the external data source obtained by the task is the latest time, that is, after the first scheduling task is executed, the ODS layer has data with a time range of 8:55-9:00; the second task result is that the task is successfully executed, but the time parameter of the new data from the external data source obtained by the task is not the latest time, that is, after the first scheduling task is executed, the ODS layer does not have data with a time range of 8:55-9:00; the third task result is task failure or task execution exception. No matter which task result is, the ODS layer will record the result.

[0042] And about Figure 3 In the first monitoring task part, after the first scheduling task from 8:55 to 9:00 is completed, before the next round of the first scheduling task is executed (for example, 9:02), the first monitoring task will effectively monitor its scheduling process according to the task result of the first scheduling task, which specifically includes two rounds of judgment process. The first round of judgment process is to judge whether the first scheduling task is successfully executed. It can be judged according to the current state of the first scheduling task. If the first scheduling task is in the ready state, it means that the previous first scheduling task has been successfully executed and the second round of judgment can be carried out. If the first scheduling task is not in the ready state, it means that the previous round of the first scheduling task failed, and the remediation program can be directly entered. The second round of judgment process is to judge the execution effect of the previous round of the first scheduling task according to the data time parameter. If the data time parameter does not meet the corresponding period of the previous round of the first scheduling task, it means that although the task itself is executed normally, the data of the corresponding period is not effectively updated due to other factors, and the execution effect of the previous scheduling task still needs to be remedied.

[0043] Specifically, regarding the remedial procedure, updating the corresponding period of the (i+1)th first scheduled task in step S220 includes: modifying the start time of the corresponding period of the (i+1)th first scheduled task to the start time of the corresponding period of the (i)th first scheduled task.

[0044] Here, when the first monitoring task monitors that the completion status of the corresponding first scheduling task does not meet the first preset condition, the corresponding period of the next round of first scheduling tasks can be adjusted so that the next round of first scheduling tasks can complete their original tasks and perform additional data scheduling to remedy the previous round of first scheduling tasks. For example, if a first scheduling task needs to obtain new data from an external data source between 8:55 and 9:00, but the first monitoring task monitors that the first scheduling task fails or there is no data in the ODS layer within the time range of 8:55-9:00 after the execution of the first scheduling task, the corresponding period of the scheduling data required for the next round of first scheduling tasks will be modified from 9:00-9:05 to 8:55-9:05.

[0045] In addition, there is another scenario. If the server where the ETL is located is down due to power outages in the area where the server is located, for example, the server is down and not running from 9:00 to 10:00, and the abnormality is discovered at 10:00 and the ETL is restored, then the ODS monitoring will discover at, for example, 10:02 that the first scheduling task is still extracting data from 8:55 to 9:00 at 10:00. If no measures are taken, the task will extract data from 9:00 to 9:05 next time. At this time, the first monitoring task will compare the end time parameter of the first scheduling task (9:05) with the current system time rounded by the period of the first scheduling task, and find that the end time parameter of the first scheduling task should be 10:05. In this operation, the ODS monitoring will set the task running time to 9:00-10:05 to ensure that the next time the first scheduling task is executed (10:05), the data delay caused by the server abnormality can be well handled.

[0046] According to the method of this embodiment, the scheduling of the data layer in the data warehouse is planned and a synchronous monitoring mechanism is configured on this basis. This can overcome the problem that the existing monitoring can only judge anomalies based on the task status but cannot capture data synchronization anomalies based on the task effect, thereby ensuring data accuracy and avoiding data synchronization delays.

[0047] In some embodiments, after configuring the data scheduling and data monitoring of the first data layer of the bottom layer, it is also necessary to consider the data synchronization problem of other data layers. The data monitoring method of the embodiment of the present application further includes:

[0048] Execute a second monitoring task on the task scheduling of the second data layer according to the second preset frequency, the second data layer is used to execute the second scheduling task according to the second preset frequency, and the second scheduling task is used to synchronize the newly added data of the first data layer in the corresponding period and record the data synchronization time; wherein the start execution time of the second monitoring task is later than the second scheduling task, and the first preset frequency is a multiple of the second preset frequency;

[0049] In response to the i-th second monitoring task detecting that the completion status of the i-th second scheduling task does not meet the second preset condition, the corresponding period of the i+1-th second scheduling task is updated to synchronize the new data of the first data layer within the period after the i+1-th second scheduling task is updated.

[0050] Here, the second data layer can be the DW layer, that is, the upper data layer of the first data layer, which is used to synchronize the new data of the first data layer. The second preset condition is that the second scheduling task is in ready state and the recorded data synchronization time is consistent with the corresponding period of the i-th second scheduling task, and the corresponding period of the i+1-th second scheduling task is updated, including: modifying the start time of the corresponding period of the i+1-th second scheduling task to the start time of the corresponding period of the i-th second scheduling task.

[0051] In addition, for the second data layer, the second scheduling task records the data synchronization time in the way that the data synchronization time of the corresponding data is recorded according to the data acquisition time recorded by the first scheduling task. It can be understood that the data warehouse will record the lineage relationship of the data scheduled by different scheduling tasks, that is, for a certain type of second scheduling task in the DW layer, it is possible to query which first scheduling tasks in the ODS layer have scheduled the data required to be scheduled by the second scheduling task based on the data lineage relationship. Therefore, when the second scheduling task records the data synchronization time, assuming that the second scheduling task synchronizes the new data of the external data source obtained by the first data layer within the time range of 9:15-9:18 system time at 9:00-9:10, then the synchronization time recorded by the second scheduling task is based on the data lineage relationship of 9:00-9:10, rather than the actual synchronization process system time 9:15-9:18.

[0052] Figure 4 FIG. 4 shows a schematic diagram of the execution logic of the second scheduling task and the second monitoring task in the DW layer, such as Figure 4As shown, when the DW layer executes the second scheduling task, three types of task results may be obtained. For example, assuming that the scheduling period corresponding to the second preset frequency is 10 minutes, the second scheduling task is configured to be executed every 10 minutes. If a second scheduling task needs to synchronize the new data of the first data layer between 8:50 and 9:00, the first task result is that the task is successfully executed, and the time parameter of the new data of the first data layer obtained by the task is the latest data according to the data lineage relationship, that is, after the second scheduling task is executed, the DW layer has the data obtained by the first data layer between 8:50 and 9:00; the second task result is that the task is successfully executed, but the time parameter of the new data of the first data layer obtained by the task is not the latest data according to the data lineage relationship, that is, after the second scheduling task is executed, the DW layer does not have the data obtained by the first data layer between 8:50 and 9:00; the third task result is task failure or task execution exception. Regardless of which task result is, the DW layer will record the result.

[0053] And about Figure 4 In the second monitoring task part, after the second scheduling task from 8:50 to 9:00 is executed, before the next round of second scheduling tasks are executed, the second monitoring task will effectively monitor its scheduling process according to the task results of the second scheduling task, which specifically includes two rounds of judgment processes. The second round of judgment process is to judge whether the second scheduling task is successfully executed. It can be judged according to the current state of the second scheduling task. If the second scheduling task is in the ready state, it means that the previous second scheduling task has been successfully executed and the second round of judgment can be carried out. If the second scheduling task is not in the ready state, it means that the previous round of second scheduling tasks failed, and the remediation program can be directly entered. The second round of judgment process is to judge the execution effect of the previous round of second scheduling tasks according to the data time parameters. If the data time parameters do not meet the corresponding period of the previous round of second scheduling tasks, it means that although the task itself is executed normally, the data of the corresponding period is not effectively updated due to other factors, and the execution effect of the previous scheduling task still needs to be remedied.

[0054] The advantage of setting up the second scheduling task to record the data synchronization time according to the data lineage relationship is that it is possible to monitor whether the data extraction time range of the second scheduling task matches the scheduling time of the first scheduling task in its data source (first data layer) based on the task results recorded in the second data layer, thereby avoiding decoupling of data monitoring between related data layers, which is beneficial to forming a monitoring linkage of the data warehouse and ensuring timely data synchronization between data layers.

[0055] In some embodiments, after configuring the data scheduling and data monitoring of the first data layer and the second data layer, it is also necessary to consider the data synchronization problem of other data layers. The data monitoring method of the embodiment of the present application further includes:

[0056] Execute a third monitoring task on the task scheduling of the third data layer according to the third preset frequency, the third data layer is used to execute the third scheduling task according to the third preset frequency, and the third scheduling task is used to extract the newly added data of the second data layer in the corresponding period and record the data extraction time; wherein the start execution time of the third monitoring task is later than the third scheduling task, and the second preset frequency is a multiple of the third preset frequency;

[0057] In response to the i-th third monitoring task detecting that the completion status of the i-th third scheduling task does not meet the third preset condition, the corresponding period of the i+1-th third scheduling task is updated so that the i+1-th third scheduling task extracts the new data of the second data layer within the updated period.

[0058] Here, the third data layer can be the DM layer, that is, the upper data layer of the second data layer, used to synchronize the new data of the second data layer. The third preset condition is that the third scheduling task is in ready state and the recorded data extraction time is consistent with the corresponding period of the i-th third scheduling task, and the corresponding period of the i+1-th third scheduling task is updated, including: modifying the start time of the corresponding period of the i+1-th third scheduling task to the start time of the corresponding period of the i-th third scheduling task.

[0059] In addition, for the third data layer, the third scheduling task records the data synchronization time in the same way as the second scheduling task in the second data layer, that is, the data synchronization time of the corresponding data is recorded according to the data acquisition time recorded by the second scheduling task, that is, for a certain type of third scheduling task in the DM layer, it is possible to query which second scheduling tasks in the DW layer have scheduled the data required to be scheduled by the third scheduling task according to the data lineage relationship, so that when the third scheduling task records the data synchronization time, assuming that the third scheduling task synchronizes the newly added data of the first data layer obtained by the second data layer within the time range of 9:10-9:12 in the system time, then the synchronization time recorded by the third scheduling task is 8:40-9:00 based on the data lineage relationship, rather than the system time 9:10-9:12 of the actual synchronization process. On this basis, since the data synchronization time recorded by the second scheduling task in the second data layer is also determined based on the data lineage relationship, in fact, the data synchronization recorded by the third scheduling task is also the newly added data of the external data source obtained by the first data layer at 8:40-9:00, thereby realizing the synchronization monitoring of each data layer of the data warehouse.

[0060] Figure 5 FIG. 4 shows a schematic diagram of the execution logic of the third scheduling task and the third monitoring task in the DM layer, such as Figure 5As shown, when the DM layer executes the third scheduling task, three types of task results may be obtained. For example, assuming that the scheduling period corresponding to the third preset frequency is 20 minutes, the third scheduling task is configured to be executed every 20 minutes. If a third scheduling task needs to synchronize the new data of the second data layer between 8:40 and 9:00, the first task result is that the task is successfully executed, and the time parameter of the new data of the second data layer obtained by the task is the latest data according to the data lineage relationship, that is, after the third scheduling task is executed, the DM layer has the data obtained by the second data layer between 8:40 and 9:00; the second task result is that the task is successfully executed, but the time parameter of the new data of the second data layer obtained by the task is not the latest data according to the data lineage relationship, that is, after the third scheduling task is executed, the DM layer does not have the data obtained by the second data layer between 8:40 and 9:00; the third task result is task failure or task execution exception. Regardless of which task result is, the DM layer will record the result.

[0061] And about Figure 5 In the third monitoring task part, after the third scheduling task from 8:40 to 9:00 is executed, before the next round of the third scheduling task is executed, the third monitoring task will effectively monitor its scheduling process according to the task result of the third scheduling task, which specifically includes two rounds of judgment process. The third round of judgment process is to judge whether the third scheduling task is successfully executed. It can be judged according to the current state of the third scheduling task. If the third scheduling task is in the ready state, it means that the previous third scheduling task has been successfully executed and the third round of judgment can be carried out. If the third scheduling task is not in the ready state, it means that the previous round of the third scheduling task failed, and the remediation program can be directly entered. The third round of judgment process is to judge the execution effect of the previous round of the third scheduling task according to the data time parameter. If the data time parameter does not meet the corresponding period of the previous round of the third scheduling task, it means that although the task itself is executed normally, the data of the corresponding period is not effectively updated due to other factors, and the execution effect of the previous scheduling task still needs to be remedied.

[0062] By adopting the method of this embodiment, it is possible to ensure that the data scheduling time of all upper data layers matches the data scheduling time of the data layer that serves as its data source, effectively monitor data synchronization delays, and avoid situations where data synchronization anomalies of the data layer corresponding to the data source are difficult for the data layer to discover and effectively remedy. After monitoring the data synchronization anomaly, in addition to automatically updating the corresponding cycle of the next round of scheduling tasks, if the task itself has an error, it can also notify the operation and maintenance personnel to intervene and handle it. After the manual processing is completed, the time parameters of the task do not need human intervention to adjust, and the monitoring workflow can realize automatic parameter updates, thereby realizing automatic number replenishment for the entire process.

[0063] In some embodiments, it is understood that the data scheduling tasks of the data layer may include multiple types, and the execution cycles of the data scheduling tasks may be different. For such a situation, the maximum cycle of the various types of scheduling tasks in the layer can be taken as the judgment cycle of the monitoring task. Figure 6 Taking the ODS layer as an example, a schematic diagram of multi-task monitoring data flow is shown.

[0064] like Figure 6 As shown, assuming that there are three scheduling tasks with different execution cycles in the ODS layer, and their cycles are a1 minute, a2 minute and a3 minute respectively, then corresponding monitoring tasks can be configured for these three scheduling tasks respectively. According to the original judgment method, the corresponding monitoring tasks should wait for a1 / 2 minutes, a2 / 2 minutes and a3 / 2 minutes respectively to judge the completion of the corresponding scheduling tasks. The maximum value of a1 / 2 minutes, a2 / 2 minutes and a3 / 2 minutes can be taken as the judgment cycle of the monitoring task. When the judgment cycle arrives, the completion of various scheduling tasks of the ODS layer is judged respectively. If it is completed on time, it is judged that the scheduling task is successfully completed. If it is not completed on time, the corresponding cycle of the subsequent corresponding scheduling task can be adjusted according to the above-mentioned remedial method, so that the next round of scheduling tasks can supplement the previously unsuccessfully scheduled data. In addition, when it is judged that the reason for the unsuccessful completion of the task is that the scheduling task of the ODS layer itself is abnormal, the system will directly kill the abnormal process and send an email to the operation and maintenance personnel to notify manual processing, and the scheduling task will terminate the operation before the manual processing is completed.

[0065] The above describes the specific settings and implementation methods of the embodiments of the present application from different angles. Using the method provided in the above embodiments, the scheduling of each data layer in the data warehouse can be planned and a synchronization monitoring mechanism can be configured, which can overcome the problem that the existing monitoring can only judge the anomaly based on the task status but cannot capture the data synchronization anomaly based on the task effect, thereby ensuring data accuracy and avoiding data synchronization delays. In addition, it can also ensure that the data scheduling time of all upper data layers matches the data scheduling time of the data layer as its data source, effectively monitor data synchronization delays, and avoid the situation where the data synchronization anomaly of the data layer corresponding to the data source is difficult for the data layer to discover and effectively remedy.

[0066] According to an embodiment of the present application, the present application also provides an electronic device, a computer-readable storage medium and a computer program product.

[0067] Figure 7 FIG. 1 is a block diagram of an electronic device used to implement an embodiment of the present application. Figure 7As shown, the electronic device includes: a memory 701 and a processor 702. The memory 701 stores a computer program that can be run on the processor 702. When the processor 702 executes the computer program, the method in the above embodiment is implemented. The number of the memory 701 and the processor 702 can be one or more.

[0068] The electronic device also includes:

[0069] The communication interface 703 is used to communicate with external devices and perform data exchange transmission.

[0070] If the memory 701, the processor 702 and the communication interface 703 are implemented independently, the memory 701, the processor 702 and the communication interface 703 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0071] Optionally, in a specific implementation, if the memory 701, the processor 702 and the communication interface 703 are integrated on a chip, the memory 701, the processor 702 and the communication interface 703 can communicate with each other through an internal interface.

[0072] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.

[0073] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the present application.

[0074] It should be understood that the processor may be a CPU, or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), FPGAs or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0075] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of exemplary but not limiting description, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0076] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0077] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0078] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0079] Any process or method described in the flow chart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0080] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or used in combination with these instruction execution systems, devices or apparatuses.

[0081] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0082] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0083] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope recorded in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A data monitoring method, applied to a data warehouse, characterized in that: The data warehouse includes a first data layer, and the method includes: Execute a first monitoring task on the task scheduling of the first data layer according to a first preset frequency, the first data layer is used to execute the first scheduling task according to the first preset frequency, the first scheduling task is used to obtain new data from the external data source within a corresponding period and record the data acquisition time; wherein the start execution time of the first monitoring task is later than the first scheduling task; In response to the i-th first monitoring task detecting that the completion status of the i-th first scheduling task does not meet the first preset condition, the corresponding period of the i+1-th first scheduling task is updated so that the i+1-th first scheduling task obtains the new data from the external data source within the updated period, where i is a positive integer.

2. The method according to claim 1, characterized in that The first preset condition includes that the first scheduled task is in a ready state and the recorded data acquisition time conforms to the corresponding period of the i-th first scheduled task, and the updating of the corresponding period of the i+1-th first scheduled task includes: The start time of the corresponding period of the (i+1)th first scheduled task is modified to the start time of the corresponding period of the (i)th first scheduled task.

3. The method according to claim 1, characterized in that The data warehouse further includes a second data layer, and the method further includes: Execute a second monitoring task on the task scheduling of the second data layer according to a second preset frequency, the second data layer is used to execute the second scheduling task according to the second preset frequency, and the second scheduling task is used to synchronize the newly added data of the first data layer in the corresponding period and record the data synchronization time; wherein the start execution time of the second monitoring task is later than that of the second scheduling task, and the first preset frequency is a multiple of the second preset frequency; In response to the i-th second monitoring task detecting that the completion status of the i-th second scheduling task does not meet the second preset condition, the corresponding period of the i+1-th second scheduling task is updated to synchronize the new data of the first data layer within the period after the i+1-th second scheduling task is updated.

4. The method according to claim 3, characterized in that The second preset condition is that the second scheduled task is in a ready state and the recorded data synchronization time conforms to the corresponding period of the i-th second scheduled task, and the corresponding period of the i+1-th second scheduled task is updated, including: The start time of the corresponding period of the (i+1)th second scheduled task is modified to the start time of the corresponding period of the (i)th second scheduled task.

5. The method according to claim 3, characterized in that: The second scheduling task records the data synchronization time in a manner of recording the data synchronization time of the corresponding data according to the data acquisition time recorded by the first scheduling task.

6. The method according to claim 3, characterized in that The data warehouse further includes a third data layer, and the method further includes: Execute a third monitoring task on the task scheduling of the third data layer according to a third preset frequency, the third data layer is used to execute the third scheduling task according to the third preset frequency, and the third scheduling task is used to extract the newly added data of the second data layer in the corresponding period and record the data extraction time; wherein the start execution time of the third monitoring task is later than that of the third scheduling task, and the second preset frequency is a multiple of the third preset frequency; In response to the i-th third monitoring task detecting that the completion status of the i-th third scheduling task does not meet the third preset condition, the corresponding period of the i+1-th third scheduling task is updated so that the i+1-th third scheduling task extracts the new data of the second data layer within the updated period.

7. The method according to claim 6, characterized in that The third preset condition is that the third scheduling task is in a ready state and the recorded data extraction time conforms to the corresponding period of the i-th third scheduling task, and the updating of the corresponding period of the i+1-th third scheduling task includes: The start time of the corresponding period of the (i+1)th third scheduling task is modified to the start time of the corresponding period of the (i)th third scheduling task.

8. A data warehouse, characterized in that: According to the data flow direction, it includes the first data layer, the second data layer and the third data layer. The first data layer is used to execute a first scheduling task according to the first preset frequency, and the first scheduling task is used to obtain new data from an external data source within a corresponding period and record the data acquisition time; The second data layer is used to execute the second scheduling task according to the second preset frequency, and the second scheduling task is used to synchronize the newly added data of the first data layer in the corresponding period and record the data synchronization time; The third data layer is used to execute a third scheduling task according to the third preset frequency, and the third scheduling task is used to extract new data from the second data layer in a corresponding period and record the data extraction time; The first preset frequency is a multiple of the second preset frequency, the second preset frequency is a multiple of the third preset frequency, and the data warehouse is used for: Execute a first monitoring task on the task scheduling of the first data layer according to a first preset frequency, and in response to the i-th first monitoring task detecting that the completion status of the i-th first scheduling task does not meet the first preset condition, update the corresponding period of the i+1-th first scheduling task, so that the i+1-th first scheduling task obtains the newly added data of the external data source within the updated period; the start execution time of the first monitoring task is later than that of the first scheduling task, wherein i is a positive integer; Execute a second monitoring task on the task scheduling of the second data layer according to a second preset frequency, and in response to the i-th second monitoring task detecting that the completion of the i-th second scheduling task does not meet the second preset condition, update the corresponding period of the i+1-th second scheduling task, so that the newly added data of the first data layer in the period after the i+1-th second scheduling task is synchronously updated; the start execution time of the second monitoring task is later than that of the second scheduling task; A third monitoring task is executed on the task scheduling of the third data layer according to a third preset frequency, and in response to the i-th third monitoring task detecting that the completion status of the i-th third scheduling task does not meet the third preset condition, the corresponding period of the i+1-th third scheduling task is updated so that the i+1-th third scheduling task extracts the new data of the second data layer within the updated period; the start execution time of the third monitoring task is later than that of the third scheduling task.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images