Data monitoring method, device and equipment

Abstract

The invention provides a data monitoring method, device and equipment, and the method comprises the steps: starting a first timer, and monitoring first incremental data, newly added in a data warehouse table, of a data warehouse within a first preset time period; a second timer is started, second incremental data corresponding to a second preset time period is extracted from the first incremental data, and the second preset time period is smaller than the first preset time period; for each piece of data in the second incremental data, key column information is extracted, and the key column information is assembled based on remote dictionary service to obtain remote dictionary service query keywords; and based on the remote dictionary service query keyword, performing data query in the data storage device, and monitoring whether all the second incremental data exist in the data storage device.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data monitoring method, device and equipment. Background Art

[0002] With the development of computer networks, the massive data of various types of business systems can be managed by relying on data warehouses. Data warehouses are subject-oriented, integrated, relatively stable data sets that reflect historical changes and are used to support management decisions of downstream business systems. When using data warehouses to manage the massive data of a downstream business system, the massive data collected in the process of providing business is filled into various types of data warehouse tables for query and extraction.

[0003] It can be seen that pushing data from the data warehouse to the business system is a prerequisite for ensuring the normal operation of the business system. Therefore, how to ensure the normal pushing of data from the data warehouse to the business system and the normal consumption of data in the business system is a technical issue that technicians in this field need to pay attention to. Summary of the invention

[0004] In view of this, the present application provides a data monitoring method, device and equipment, the main purpose of which is to ensure the normal push and normal consumption of data between the data warehouse and the data storage device.

[0005] According to one aspect of the present application, a data monitoring method is provided for monitoring the data warehouse and the data storage device in a linkage manner during the process of transmitting data from the data warehouse to the target system via the data storage device. The method comprises:

[0006] Starting a first timer, and within a first predetermined time period, monitoring first incremental data newly added by the data warehouse in the data warehouse table;

[0007] Starting a second timer, and taking out second incremental data corresponding to a second predetermined time period from the first incremental data, wherein the second predetermined time period is shorter than the first predetermined time period;

[0008] For each piece of data in the second incremental data, key column information is extracted, and the key column information is assembled based on the remote dictionary service to obtain a remote dictionary service query keyword; based on the remote dictionary service query keyword, data query is performed in the data storage device, and the data storage device is monitored to see whether all the second incremental data exists.

[0009] In one implementation, the method further includes:

[0010] If the first incremental data is not monitored within the first predetermined time period, it is determined that an upstream non-push failure has occurred, and an alarm message corresponding to the upstream non-push failure is triggered.

[0011] In one implementation, the extracting, from the first incremental data, second incremental data corresponding to a second predetermined time period includes:

[0012] Read forward from the bottom of the data warehouse table, select incremental data corresponding to a second predetermined time period, and determine it as the second incremental data.

[0013] In one implementation, after monitoring whether the storage device has all the second incremental data, the method further includes:

[0014] If the data storage device does not contain all of the second incremental data, it is determined that a downstream unconsumed fault has occurred, and an alarm message corresponding to the downstream unconsumed fault is triggered.

[0015] In one implementation,

[0016] The data includes core data and non-core data;

[0017] The method further includes: for the core data and the non-core data, setting timers of different time periods to monitor the data in the data warehouse.

[0018] In one implementation, for core data and non-core data, setting timers of different time periods to monitor data in the data warehouse includes:

[0019] The first timer includes a first core timer and a first non-core timer, and a time period corresponding to the first core timer is smaller than a time period corresponding to the first non-core timer;

[0020] Starting a first core timer, and determining whether first core incremental data of the data warehouse is monitored within a time period corresponding to the first core timer;

[0021] A first non-core timer is started, and within a time period corresponding to the first non-core timer, it is determined whether first non-core incremental data of the data warehouse is monitored.

[0022] In one implementation, determining the second incremental data corresponding to the second predetermined time period corresponding to the second timer includes:

[0023] Only the second core incremental data corresponding to the second predetermined time period is determined from the first core incremental data.

[0024] In one implementation, determining whether the data storage device has all the second incremental data includes:

[0025] Only for the second core incremental data, it is queried in the storage device whether all the second core incremental data exist.

[0026] According to one aspect of the present application, a data monitoring device is provided, which is used to monitor the data warehouse and the data storage device in a linkage manner during the process of transmitting data from the data warehouse to the target system through the data storage device, and the device includes:

[0027] A data warehouse monitoring unit, configured to start a first timer and monitor first incremental data newly added by the data warehouse in a data warehouse table within a first predetermined time period;

[0028] a data range determination unit, configured to start a second timer, and retrieve second incremental data corresponding to a second predetermined time period from the first incremental data, wherein the second predetermined time period is smaller than the first predetermined time period;

[0029] The storage device monitoring unit is used to extract key column information for each piece of data in the second incremental data, assemble the key column information based on the remote dictionary service to obtain a remote dictionary service query keyword; based on the remote dictionary service query keyword, perform data query in the data storage device, and monitor whether the data storage device contains all the second incremental data.

[0030] According to one aspect of the present application, a device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the data monitoring method.

[0031] By means of the above technical scheme, the present application provides a data monitoring method, device and equipment, which are used for linkage monitoring of the data warehouse and the data storage device during the process of transmitting data from the data warehouse to the target system through the data storage device, wherein a first timer is started, and within a first predetermined time period, the first incremental data newly added in the data warehouse table by the data warehouse is monitored; a second timer is started, and the second incremental data corresponding to the second predetermined time period is taken out from the first incremental data, wherein the second predetermined time period is less than the first predetermined time period; for each piece of data in the second incremental data, key column information is extracted, and the key column information is assembled based on the remote dictionary service to obtain a remote dictionary service query keyword; based on the remote dictionary service query keyword, a data query is performed in the data storage device, and it is monitored whether the data storage device has all the second incremental data.

[0032] The data monitoring method proposed in the embodiment of the present application can solve the following problems: solve the problem that important data in the system is not displayed because it is not pushed upstream and the system is not aware of it, and solve the problem that the data cannot be displayed normally and the system is not aware of it because the consumption service is down in the system. And in the implementation process, different detection frequencies can be set for core data and non-core data, as well as for different situations of data push and data consumption. For example, for monitoring of incremental data in the data warehouse: a shorter monitoring period is set for core data, and a longer monitoring period is set for non-core data. For another example, only data consumption monitoring is performed on the data storage device for core data, so as to ensure the normal operation of the target system and reduce the processing pressure caused by non-core data monitoring.

[0033] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0035] Figure 1 A schematic diagram of an implementation scenario of a data monitoring method provided in an embodiment of the present application is shown;

[0036] Figure 2 A schematic diagram of a data monitoring method provided by an embodiment of the present application is shown;

[0037] Figure 3 A flow chart of a data monitoring method provided by an embodiment of the present application is shown;

[0038] Figure 4 An example flow chart of data push monitoring of a data monitoring method provided by an embodiment of the present application is shown;

[0039] Figure 5 An example flow chart of data consumption monitoring of a data monitoring method provided by an embodiment of the present application is shown;

[0040] Figure 6 A schematic diagram of the structure of a data monitoring device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only embodiments of a part of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0042] See also Figure 1 , showing a schematic diagram of an implementation scenario of a data monitoring method provided by an embodiment of the present application. The scenario includes several layers and a target system, wherein the data warehouse pushes data to the target system to ensure that the target system performs normal business according to the data. The target system can be any type of business system. For example, the target system can be an online teaching platform system. The data warehouse collects various types of data, including but not limited to learning data, supply chain data, transaction data, user data, etc. The data layers extract and clean these data, and then process, summarize and organize them through the system, and finally store them to obtain various data warehouse tables.

[0043] See also Figure 2 , shows a schematic diagram of an example scenario of a data monitoring method provided by an embodiment of the present application. In this example scenario, the data in the data warehouse is written into the data table with the shortest link to ensure data writing efficiency. Then, the data passes through a data storage device such as Kafka and Pika, and the target system reads data from the data storage device through an API interface.

[0044] During the research process, the inventors of the present application found that data delays mainly occur due to the following two reasons: first, the upstream data warehouse fails to push data to the data storage device of the target system in a timely manner, resulting in the problem of data not being pushed; second, the upstream data warehouse pushes data to the data storage device in a timely manner, but the target system does not normally consume the data of the data storage device, resulting in the problem of data not being consumed.

[0045] Based on the above analysis, an embodiment of the present application proposes a data monitoring method between a data warehouse and a target system data storage device to determine whether upstream data from the data warehouse stops being pushed or whether downstream data from the storage device stops being consumed.

[0046] See also Figure 3 , shows a flow chart of a data monitoring method provided by an embodiment of the present application. The data monitoring method is used to perform linkage monitoring on the data warehouse and the data storage device during the process of transmitting data from the data warehouse to the target system through the data storage device, and the method includes the following steps S301-S303.

[0047] S301: Start a first timer, and within a first predetermined time period, monitor the first incremental data newly added in the data warehouse table by the data warehouse.

[0048] Taking the data transmission link of "data warehouse-data storage device-target system" as an example, it is necessary to ensure that the upstream data warehouse operates normally, that is, the data warehouse has incremental data within a period of time. If the first incremental data is not monitored within the above-mentioned first predetermined time period, it is determined that the data warehouse stops generating new data, and it is considered that a data non-push failure has occurred upstream. At this time, the alarm mechanism can be triggered to trigger the alarm information corresponding to the upstream non-push failure. For example, set the identification code and field of the alarm information, and specifically mark the type of failure and the fault point.

[0049] Specifically, the data may be divided into core data and non-core data according to the data importance level of the target system, and timers of different time periods may be set for the core data and the non-core data to monitor the data warehouse.

[0050] In one implementation, for core data and non-core data, timers with different time periods are set to monitor data in a data warehouse, which may include the following steps: the first timer includes a first core timer and a first non-core timer, and the time period corresponding to the first core timer is smaller than the time period corresponding to the first non-core timer; the first core timer is started, and within the time period corresponding to the first core timer, it is determined whether the first core incremental data of the data warehouse is monitored; the first non-core timer is started, and within the time period corresponding to the first non-core timer, it is determined whether the first non-core incremental data of the data warehouse is monitored.

[0051] For example, for core data, because it is more important, a shorter time period timer can be set for monitoring, for example, monitoring every 4 hours to determine whether there is incremental data for core data; and for non-core data, because it is less important, a longer time period timer can be set for monitoring, for example, monitoring once a day (24 hours) to determine whether there is incremental data for non-core data. For example, taking an online teaching platform system as an example, core data includes but is not limited to teaching time data, homework submission data, course purchase data, class division data, telephone data, SMS data, etc., and non-core data includes remarks data, company introduction data, etc.

[0052] S302: Start a second timer, and extract second incremental data corresponding to a second predetermined time period from the first incremental data, wherein the second predetermined time period is shorter than the first predetermined time period.

[0053] The first incremental data is determined by monitoring the data warehouse table of the data warehouse, for example, by reading the data warehouse through a timestamp or data identifier to determine the incremental data. In the above step 301, by monitoring the data of the data warehouse, it can be determined whether an upstream non-push failure has occurred. When the push is normal, the first incremental data in the data warehouse can be determined; in step S302, a part of the data is taken out from the first incremental data of the data warehouse as the basis for whether the downstream consumption is normal in the subsequent step S303. Specifically, you can start reading from the bottom of the data warehouse table and select the incremental data corresponding to the second predetermined time period to determine it as the second incremental data.

[0054] For example, if the first scheduled time period of the first timer is set to 4 hours, then the second scheduled time period of the second timer is set to be shorter than 4 hours, for example, the second scheduled time period is set to 5 minutes, and the data of the most recent 5 minutes is taken from the bottom of the data warehouse table as the second incremental data.

[0055] S303: For each piece of data in the second incremental data, extract key column information, assemble the key column information based on the remote dictionary service to obtain a remote dictionary service query keyword; based on the remote dictionary service query keyword, perform data query in the data storage device, and monitor whether the data storage device contains all the second incremental data.

[0056] In order to query whether all the data of the second incremental data exists in the storage device, the key column information of the second incremental data can be extracted, and the key column information can be assembled into a remote dictionary service (redis) to obtain a remote dictionary service query keyword (redis key); based on the remote dictionary service query keyword, query whether all the second incremental data exists in the storage device. Figure 2 For example, pika is used as a data storage device in the online teaching system. It queries the incremental data in the data warehouse data table and assembles it into a redis key based on the incremental data. It uses the redis key as the keyword to query pika to determine whether pika has the incremental data added this time. If so, it is determined that the incremental data in the data warehouse has been stored normally in pika, which can ensure that the data is consumed normally.

[0057] If the data storage device does not have all the second incremental data, it is determined that a downstream non-consumption failure has occurred, and the alarm information corresponding to the downstream non-consumption failure is triggered. For example, if the query finds that part or all of the data is missing in pika for the second incremental data in the last 5 minutes, it is determined that the data has not reached the data storage device, that is, it is considered that a data non-consumption failure has occurred.

[0058] In specific operations, data monitoring of different frequencies can be performed on the data warehouse and data storage device according to the importance of different data in the target system. Taking the online teaching system as an example, the data can be divided into core data and non-core data. For example, core data includes but is not limited to teaching time data, homework submission data, course purchase data, class division data, telephone data, SMS data, etc. Non-core data includes notes data, company introduction data, etc. For core data, it is the key to ensure the normal operation of the online teaching system, and plays an important role in data display and teaching effect. Therefore, it is necessary to ensure that the core data is pushed to the data storage device normally by the data warehouse, and to ensure that the core data is consumed normally in the data storage device. For non-core data, it has little impact on teaching, so the frequency of monitoring and detecting non-core data can be reduced to reduce the pressure of data processing.

[0059] In addition, as analyzed above, whether the core data is consumed normally is the key to ensuring the normal operation of the target system. Therefore, in the process of monitoring and judging the data storage device, only the core data can be monitored. Therefore, in one implementation method, only the second core incremental data corresponding to the second predetermined time period is determined in the first core incremental data, and, only for the second core incremental data, it is queried in the storage device whether all of the second core incremental data exist.

[0060] At this time, the execution steps may be as follows.

[0061] First, perform upstream push monitoring. Set the first timer to include the first core timer and the first non-core timer. For example, set the time period corresponding to the first core timer to 4 hours, set the time period corresponding to the first non-core timer to 24 hours, and monitor the first core incremental data (such as core_data1, 2..., 100) within every 4 hours, and monitor the first non-core incremental data (such as non-core_data1, 2...50) within every 24 hours, so as to complete the upstream monitoring of the data warehouse.

[0062] Then, determine the downstream consumption monitoring range. For example, only perform downstream consumption monitoring on core data, by setting the time period corresponding to the second timer to 5 minutes, only extract the second core incremental data from the first core incremental data, for example, extract the second core incremental data within the last 5 minutes, such as core_data90, 91..., 100, from the first core incremental data core_data1, 2..., 100.

[0063] Finally, perform downstream consumption monitoring. Based on the second core incremental data, assemble each piece of data into a redis key, use the redis key as the keyword to query pika, and determine whether pika has all the second core incremental data added this time. If so, it is determined that the incremental data of the data warehouse has been stored normally in pika, which can ensure that the data is consumed normally. For example, concatenate the above core_data90, 91..., 100 with the key of the redis query according to the field protocol, and execute the redis query in pika in batches with the redis key to determine whether pika has all of this part of the data.

[0064] In actual operation, data monitoring of data warehouses and data storage devices can be implemented by pre-writing a data monitoring script. The data monitoring script can be temporarily called and executed by the data warehouse or storage device. The data monitoring script monitors the new data insertion operation in the data warehouse or data storage device to achieve the purpose of monitoring the newly added data. Specifically, the steps to implement the data monitoring script may include: writing the connection configuration of the data warehouse or storage device, connecting the data warehouse or storage device according to the connection configuration, and creating an SQL trigger to monitor the insertion operation of the data warehouse or storage device in the data table to monitor the latest data written to the data warehouse or storage device.

[0065] The following is a code example that shows how to connect to the database and query incremental data.

[0066] #Database connection configuration

[0067] config={

[0068] 'host':'your_database_host',

[0069] 'user':'your_username',

[0070] 'password':'your_password',

[0071] 'db':'your_database',

[0072] 'charset':'utf8mb4',

[0073] 'cursorclass':pymysql.cursors.DictCursor

[0074] }

[0075] #Connect to database

[0076] connection=pymysql.connect(**config)

[0077] try:

[0078] with connection.cursor()as cursor:

[0079] #Write SQL query statements to check the health of the data warehouse

[0080] sql="SELECT COUNT(*)FROM your_main_fact_table"

[0081] cursor.execute(sql)

[0082] result = cursor.fetchone()

[0083] print(f"Fact Table Row Count:{result[0]}")

[0084] finally:

[0085] connection.close()

[0086] In the above script, the pymysql library is used to connect to the MySQL database and a simple query is executed to get the number of rows in the fact table. In addition, you can modify this query as needed or add more complex queries to monitor other aspects of the data warehouse, such as checking the integrity of the data, performance indicators, etc.

[0087] To monitor the incremental data in the data warehouse, you can use the database's change data capture (CDC) function or triggers. The following is an example of using a SQL Server trigger, which is activated when data is inserted into a table and records these inserts into a log table.

[0088] First, suppose there is a data warehouse table DimProduct and you need to monitor the insert operations of this table.

[0089] --Assume that this is the DimProduct table in the data warehouse and its insert operations need to be monitored

[0090] CREATE TABLE DimProduct(

[0091] ProductID INT PRIMARY KEY,

[0092] ProductName NVARCHAR(50) );

[0094] --Create a log table to record changes

[0095] CREATE TABLE ProductChangeLog(

[0096] ChangeLogID INT IDENTITY(1,1)PRIMARY KEY,

[0097] ProductID INT,

[0098] ProductName NVARCHAR(50),

[0099] ChangeTypeNVARCHAR(10),

[0100] ChangeDate DATETIME DEFAULT(GETDATE()) );

[0102] Next, create a trigger to monitor insert operations on the DimProduct table:

[0103] --Create a trigger to record new data in the ProductChangeLog table when it is inserted into DimProduct

[0104] CREATE TRIGGER trg_DimProduct_Insert

[0105] ON DimProduct

[0106] AFTER INSERT

[0107] AS

[0108] BEGIN

[0109] INSERT INTO ProductChangeLog(ProductID,ProductName,ChangeType)

[0110] SELECT ProductID,ProductName,'Insert'

[0111] FROM Inserted

[0112] END;

[0113] Therefore, every time new data is inserted into the DimProduct table, the relevant changes will be recorded in the ProductChangeLog table.

[0114] It should be noted that this example is for the SQL Server database. Different database systems (such as Oracle, PostgreSQL, MySQL, etc.) may use different specific implementation methods.

[0115] As analyzed above, data delay mainly includes two situations: data is not pushed and data is not consumed. Therefore, by regularly monitoring the incremental data of the data warehouse, it is possible to determine whether the data is pushed normally. That is, if the incremental data of the data warehouse is monitored within a period of time, it is determined that the data is pushed normally. If no incremental data is monitored, it is determined that the data is not pushed normally. At this time, alarm measures need to be taken, such as prompting system maintenance personnel to check or process the abnormality through alarm information.

[0116] On the premise that the upstream data is pushed normally, it is necessary to further monitor whether the data is consumed normally by the downstream. In the embodiment of the present application, by monitoring whether the data storage device has all the incremental data of the aforementioned monitored data warehouse, it is judged whether the data is consumed normally. That is, if the data pushed by the data warehouse is transmitted normally to the data storage device, it is considered that the data is consumed normally, otherwise, it is determined that the downstream data stops being consumed. At this time, the cluster takes an alarm operation, such as notifying the system maintenance personnel of an abnormality through an alarm message, prompting them to check or process it.

[0117] As analyzed above, data delay generally occurs in two situations: one is when the upstream (data warehouse) stops pushing data, and the other is when the downstream system stops consuming data. The following examples illustrate these two situations.

[0118] See also Figure 4 , shows an example flow chart of data push monitoring of a data monitoring method provided in an embodiment of the present application.

[0119] 1. Start;

[0120] 2. Whether the timer has reached the detection time;

[0121] 3. If yes, query the incremental data of the data warehouse table within the specified time;

[0122] 4. Determine whether there is incremental data. If so, return to step 2.

[0123] 5. If there is no incremental data, push the alarm information for alarm processing.

[0124] For example, different detection times are set for data of different importance. For example, core data is in class every day, so it is generally set to a few hours, for example, 4 hours. If there is no data within 4 hours, it can be judged that the upstream has not pushed data. Among them, an example of incremental data is: if there are 100 data at 8 o'clock, and 150 data at 9 o'clock, these 50 data can be called incremental data.

[0125] The above timer refers to the first timer or the second timer. For example, a first timer of 4 hours may be set for core data, or a second timer of 24 hours may be set for non-core data.

[0126] See also Figure 5 , shows an example flow chart of data consumption monitoring of a data monitoring method provided in an embodiment of the present application.

[0127] 1. Start;

[0128] 2. Whether the timer has reached the detection time;

[0129] 3. If the detection time has arrived, query the incremental data in the data warehouse data table within the timer, for example, query the incremental data within 5 minutes;

[0130] 4. Extract key column data from all data tables;

[0131] 5. Assemble redis query key based on key column data;

[0132] 6. Batch query pika to see whether there is incremental data added this time;

[0133] 7. Are the results consistent? If so, return to step 2;

[0134] 8. Push alarm information.

[0135] The timer is a third timer with a shorter time period, for example, the corresponding detection time period is 5 minutes.

[0136] As mentioned above, data monitoring and judgment can be implemented through a script task. The specific execution steps of executing the data consumption monitoring task through the script include:

[0137] Step 1: Scheduled execution of tasks: Set the timer to execute the task regularly, and set the timer to 1 minute. The task is executed every 1 minute. Each execution is to determine whether the task in execution is completed. If it is still in execution, the current task is abandoned and the judgment is repeated again when the next 1 minute arrives until the task in execution is completed. The current task reverses the state and starts execution.

[0138] Step 2, data monitoring scope: Considering the actual business situation, each task takes the incremental data in the data base table in the past 5 minutes, and splices the key of the redis query according to the field protocol based on these incremental data.

[0139] Step 3: Data query and comparison: Execute Redis query in batches using the Redis key obtained in Step 2, record the query results, and compare them with the specific field data of the incremental source data obtained from the base table in Step 2.

[0140] Step 4: Alarm push: If there is any inconsistency in the comparison results in step 3, combine the inconsistent fields, inconsistent quantities and other information and push them to the designated person in charge through the self-developed alarm channel.

[0141] It can be seen that the embodiment of the present application provides a data monitoring method, which is mainly used in the downstream system of the data warehouse to quickly perceive whether the data inflow is normal, whether the data consumption link is delayed, and give an alarm at the moment of data delay. The data monitoring method proposed in the embodiment of the present application can solve the following problems: solve the problem that important data in the system is not displayed because the upstream is not pushed and the system is not aware of it, and solve the problem that the data cannot be displayed normally and the system is not aware of it because the consumption service is hung up in the system. And in the implementation process, different detection frequencies can be set for core data and non-core data, as well as different situations of data push and data consumption. For example, a shorter monitoring cycle is set for the data warehouse for core data, and a longer monitoring cycle is set for non-core data. For another example, only data consumption monitoring is performed on the data storage device for core data, so as to ensure the normal operation of the target system, and reduce the processing pressure brought by non-core data monitoring.

[0142] See also Figure 6 , shows a schematic diagram of the structure of a data monitoring device provided by an embodiment of the present application. The data monitoring device is used to monitor the data warehouse and the data storage device in a linkage manner during the process of transmitting data from the data warehouse to the target system through the data storage device, and the device includes:

[0143] The data warehouse monitoring unit 601 is used to start a first timer and monitor the first incremental data newly added in the data warehouse table by the data warehouse within a first predetermined time period;

[0144] The data range determination unit 602 is configured to start a second timer to retrieve second incremental data corresponding to a second predetermined time period from the first incremental data, wherein the second predetermined time period is smaller than the first predetermined time period;

[0145] The storage device monitoring unit 603 is used to extract key column information for each piece of data in the second incremental data, assemble the key column information based on the remote dictionary service to obtain a remote dictionary service query keyword; based on the remote dictionary service query keyword, perform data query in the data storage device, and monitor whether the data storage device contains all the second incremental data.

[0146] In one implementation,

[0147] The data warehouse monitoring unit 601 is also used to determine that an upstream non-push failure has occurred if the first incremental data is not monitored within the first predetermined time period, and trigger an alarm message corresponding to the upstream non-push failure.

[0148] In one implementation,

[0149] The data range determination unit 602 is specifically configured to read forward from the bottom of the data warehouse table, select incremental data corresponding to a second predetermined time period, and determine the incremental data as the second incremental data.

[0150] In one implementation,

[0151] The storage device monitoring unit 603 is specifically used to determine that a downstream non-consumption fault has occurred if all of the second incremental data does not exist in the data storage device, and trigger an alarm message corresponding to the downstream non-consumption fault.

[0152] In one implementation,

[0153] The data includes core data and non-core data;

[0154] The device is also used for: for core data and non-core data, setting timers of different time periods to monitor data in the data warehouse.

[0155] In one implementation,

[0156] The first timer includes a first core timer and a first non-core timer, and a time period corresponding to the first core timer is smaller than a time period corresponding to the first non-core timer;

[0157] The data warehouse monitoring unit 601 is specifically used to start the first core timer, and determine whether the first core incremental data of the data warehouse is monitored within the time period corresponding to the first core timer; and start the first non-core timer, and determine whether the first non-core incremental data of the data warehouse is monitored within the time period corresponding to the first non-core timer.

[0158] In one implementation,

[0159] The data range determination unit 602 is specifically configured to determine only the second core incremental data corresponding to the second predetermined time period in the first core incremental data.

[0160] In one implementation,

[0161] The storage device monitoring unit 603 is specifically used to query whether all the second core incremental data exist in the storage device only for the second core incremental data.

[0162] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0163] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0164] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0165] Starting a first timer, and within a first predetermined time period, monitoring first incremental data newly added by the data warehouse in the data warehouse table;

[0166] Starting a second timer, and taking out second incremental data corresponding to a second predetermined time period from the first incremental data, wherein the second predetermined time period is shorter than the first predetermined time period;

[0167] For each piece of data in the second incremental data, key column information is extracted, and the key column information is assembled based on the remote dictionary service to obtain a remote dictionary service query keyword; based on the remote dictionary service query keyword, data query is performed in the data storage device, and the data storage device is monitored to see whether all the second incremental data exists.

[0168] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0169] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0170] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0171] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0172] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0173] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0174] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0175] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data monitoring method, characterized in that: In the process of transmitting data from a data warehouse to a target system via a data storage device, the data warehouse and the data storage device are monitored in linkage, the method comprising: Starting a first timer, and within a first predetermined time period, monitoring first incremental data newly added by the data warehouse in the data warehouse table; Starting a second timer, and taking out second incremental data corresponding to a second predetermined time period from the first incremental data, wherein the second predetermined time period is shorter than the first predetermined time period; For each piece of data in the second incremental data, key column information is extracted, and the key column information is assembled based on the remote dictionary service to obtain a remote dictionary service query keyword; based on the remote dictionary service query keyword, data query is performed in the data storage device, and the data storage device is monitored to see whether all the second incremental data exists.

2. The method according to claim 1, characterized in that Also includes: If the first incremental data is not monitored within the first predetermined time period, it is determined that an upstream non-push failure has occurred, and an alarm message corresponding to the upstream non-push failure is triggered.

3. The method according to claim 1, characterized in that The extracting the second incremental data corresponding to the second predetermined time period from the first incremental data includes: Read forward from the bottom of the data warehouse table, select incremental data corresponding to a second predetermined time period, and determine it as the second incremental data.

4. The method according to claim 1, characterized in that After monitoring whether the storage device has all the second incremental data, the method further includes: If the data storage device does not contain all of the second incremental data, it is determined that a downstream unconsumed fault has occurred, and an alarm message corresponding to the downstream unconsumed fault is triggered.

5. The method according to any one of claims 1 to 4, characterized in that: The data includes core data and non-core data; The method further comprises: for the core data and the non-core data, setting timers of different time periods to monitor the data in the data warehouse.

6. The method according to claim 5, characterized in that For core data and non-core data, setting timers of different time periods to monitor data in the data warehouse includes: The first timer includes a first core timer and a first non-core timer, and a time period corresponding to the first core timer is smaller than a time period corresponding to the first non-core timer; Starting a first core timer, and determining whether first core incremental data of the data warehouse is monitored within a time period corresponding to the first core timer; A first non-core timer is started, and within a time period corresponding to the first non-core timer, it is determined whether first non-core incremental data of the data warehouse is monitored.

7. The method according to claim 6, characterized in that The determining of second incremental data corresponding to a second predetermined time period corresponding to a second timer includes: Only the second core incremental data corresponding to the second predetermined time period is determined from the first core incremental data.

8. The method according to claim 7, characterized in that The determining whether the data storage device stores all the second incremental data includes: Only for the second core incremental data, it is queried in the storage device whether all the second core incremental data exist.

9. A data monitoring device, characterized in that: Used for monitoring the data warehouse and the data storage device in a linkage manner during data transmission from the data warehouse to the target system via the data storage device, the device comprising: A data warehouse monitoring unit, configured to start a first timer and monitor first incremental data newly added by the data warehouse in a data warehouse table within a first predetermined time period; a data range determination unit, configured to start a second timer, and retrieve second incremental data corresponding to a second predetermined time period from the first incremental data, wherein the second predetermined time period is smaller than the first predetermined time period; The storage device monitoring unit is used to extract key column information for each piece of data in the second incremental data, assemble the key column information based on the remote dictionary service to obtain a remote dictionary service query keyword; based on the remote dictionary service query keyword, perform data query in the data storage device, and monitor whether the data storage device contains all the second incremental data.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 8.

Images