Data monitoring method, device and equipment of data warehouse and storage medium

Abstract

The application relates to the technical field of big data, and discloses a data monitoring method and device for a data warehouse, equipment and a storage medium, wherein the method comprises the following steps: in response to a data synchronization request, updating a data amount statistical table according to the number of source table incremental data to be stored and the number of source table full amount data to be stored of each to-be-uploaded data table in a target business system; in response to the data synchronization request, updating the data warehouse according to each to-be-uploaded data table, updating the data amount statistical table and a data abnormal value checking table according to the data updated into the data warehouse; performing abnormality judgment on the data amount according to a target time range and the data amount statistical table, obtaining a data amount monitoring judgment result; and performing abnormal value judgment on the data according to the target time range and the data abnormal value checking table, obtaining a data monitoring judgment result. The data amount and abnormal value of the data synchronized to the data warehouse are automatically monitored in real time, and the timeliness of data monitoring is improved.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a data monitoring method, apparatus, equipment and storage medium for a data warehouse. Background Technology

[0002] Currently, the data monitoring system of the data warehouse of the big data platform of various projects is relatively lacking. The number of data entries synchronized every day and the size of the data are not monitored. It is unknown how much of the synchronized data meets the standards. Furthermore, there are no statistics on how many tables have incremental data each day and how many tasks are being executed overall. As a result, it is impossible to scientifically evaluate the resource usage of each project, and the data synchronization status is also in a state of ambiguity.

[0003] Traditional data statistics involve accessing Hive (a data warehouse tool) tables and using counting functions (COUNT) to count the incremental, full, and dirty data volumes for the day. However, counting functions are relatively slow in both Hive SQL (Structured Query Language) and Spark SQL (Spark is a fast and general-purpose computing engine designed for large-scale data processing). This is because Hive SQL is based on MapReduce (data mining), while Spark SQL triggers Actions (running Spark jobs and actually performing transformation operations). Performing counting operations on every table results in thousands of counting function statements, leading to slow execution and low timeliness of data monitoring. Summary of the Invention

[0004] The main purpose of this application is to provide a data monitoring method, apparatus, device, and storage medium for data warehouses, aiming to solve the technical problem that existing technologies use counting functions to count the incremental data volume, total data volume, and dirty data volume of the day for anomaly monitoring of data warehouses, which has the disadvantage of slow counting function speed and low data monitoring timeliness.

[0005] To achieve the aforementioned objectives, this application proposes a data monitoring method for a data warehouse, the method comprising:

[0006] Obtain the data synchronization request from the target business system;

[0007] In response to the data synchronization request, update the data volume statistics table according to the number of incremental data rows to be stored and the number of full data rows to be stored in each data table to be uploaded in the target business system.

[0008] In response to the data synchronization request, update the data warehouse according to each of the data tables to be uploaded, and update the data volume statistics table and the data outlier check table according to the data updated in the data warehouse;

[0009] Obtain an anomaly assessment request, wherein the anomaly assessment request carries a target time range;

[0010] Based on the target time range and the data volume statistics table, anomalies in data volume are determined to obtain data volume monitoring and judgment results.

[0011] Based on the target time range and the data outlier checklist, outlier values ​​are determined in the data to obtain the data monitoring judgment result.

[0012] Furthermore, the step of updating the data volume statistics table based on the number of incremental data rows to be stored and the number of full data rows to be stored in each data table to be uploaded in the target business system includes:

[0013] Obtain the latest historical data synchronization time corresponding to the target business system, and determine the target filtering conditions based on the latest historical data synchronization time and the generation time of the data synchronization request;

[0014] Based on the target filtering conditions, the number of incremental data rows corresponding to each data table to be uploaded is obtained as the number of incremental data rows of the source table to be stored;

[0015] Based on the generation time, the number of full data rows corresponding to each data table to be uploaded is obtained as the number of full data rows of the source table to be stored;

[0016] The table name of the data table to be uploaded, the number of incremental data rows of the source table to be stored, and the total number of data rows of the source table to be stored are added as associated data to the data volume statistics table.

[0017] Furthermore, the step of updating the data volume statistics table and the data outlier check table based on the data updated in the data warehouse includes:

[0018] Update the data volume statistics table according to the Hive metadata in the data warehouse corresponding to each of the data tables to be uploaded;

[0019] The data anomaly check table is updated based on the data cache layer table, data source layer table, and data detail layer table in the data warehouse corresponding to each of the data tables to be uploaded.

[0020] Furthermore, the step of updating the data volume statistics table based on the Hive metadata corresponding to each of the data tables to be uploaded in the data warehouse includes:

[0021] Obtain the table names corresponding to each of the data tables to be uploaded from the Hive metadata in the data warehouse to obtain the set of table names to be processed;

[0022] Obtain one table name from the set of table names to be processed as the table name to be processed;

[0023] Based on the table name to be processed, obtain the following from the big data platform management data table: the name of the data cache layer library table to be stored, the number of incremental data rows in the data cache layer library table to be stored, the name of the data source layer library table to be stored, the number of full data rows in the data source layer library table to be stored, the number of full data rows in the data detail layer library table to be stored, and the number of dirty data rows to be stored.

[0024] The data volume statistics table is updated with the associated data: the name of the data cache layer table to be stored, the number of incremental data rows in the data cache layer table to be stored, the name of the data source layer table to be stored, the number of full data rows in the data source layer table to be stored, the number of full data rows in the data detail layer table to be stored, and the number of dirty data rows to be stored.

[0025] Furthermore, the step of updating the data outlier check table based on the data cache layer table, data source layer table, and data detail layer table corresponding to each of the data tables to be uploaded in the data warehouse includes:

[0026] Obtain the table names corresponding to each of the data tables to be uploaded from the Hive metadata in the data warehouse to obtain the set of table names to be analyzed;

[0027] Obtain any one of the table names in the set of table names to be analyzed as the table name to be analyzed;

[0028] Based on the name of the table to be analyzed, the business primary key is found from the newly added data of the data cache layer library table corresponding to each of the data tables to be uploaded on the same day, and the target business primary key is obtained.

[0029] When the primary key of the target business is an empty string or an empty object, the primary key exception judgment result is determined to be abnormal; otherwise, the primary key exception judgment result is determined to be normal.

[0030] Based on the name of the table to be analyzed, the creation time is found from the newly added data of the data cache layer library table corresponding to each of the data tables to be uploaded on the same day, and the first creation time is obtained;

[0031] When the first creation time is an empty string or an empty object, the time anomaly judgment result is determined to be abnormal; otherwise, the time anomaly judgment result is determined to be normal.

[0032] Determine whether the name of the table to be analyzed exists as a business primary key in the data source layer library table corresponding to each of the data tables to be uploaded, and obtain the business primary key lookup result;

[0033] When the business primary key lookup result is found to exist, the creation time is found from the newly added data of the data source layer database table corresponding to each of the data tables to be uploaded, based on the name of the table to be analyzed, to obtain the second creation time;

[0034] When the first creation time and the second creation time are the same, the primary key duplication determination result is determined to be non-duplicate; otherwise, the primary key duplication determination result is determined to be duplicate.

[0035] The data anomaly check table is updated based on the name of the table to be analyzed, the primary key anomaly judgment result, the time anomaly judgment result, and the primary key duplicate judgment result.

[0036] Furthermore, the step of determining data volume anomalies based on the target time range and the data volume statistics table to obtain data volume monitoring and judgment results includes:

[0037] Based on the target time range, obtain the relevant data to be analyzed from the data volume statistics table;

[0038] Take any one of the aforementioned related data to be analyzed as the target related data;

[0039] When the number of incremental data rows in the source table of the target associated data and the number of incremental data rows in the data cache layer database table are different, the incremental judgment result corresponding to the target associated data is determined to be abnormal; otherwise, the incremental judgment result is determined to be normal.

[0040] When the sum of the total number of data entries and the number of dirty data entries in the data detail layer database table of the target associated data is different from the total number of data entries in the data source layer database table of the target associated data, the total judgment result corresponding to the target associated data is determined to be abnormal; otherwise, the total judgment result is determined to be normal.

[0041] An incremental trend chart is generated based on the number of incremental data entries in the data cache layer tables of each of the related data to be analyzed;

[0042] The slope of the incremental trend graph is calculated to obtain the target slope;

[0043] When the target slope is outside the preset slope range, the incremental trend judgment result is determined to be abnormal; otherwise, the incremental trend judgment result is determined to be normal.

[0044] When the incremental judgment result, the full judgment result, and the incremental trend judgment result are all normal, the data volume judgment result of the data volume monitoring judgment result is determined to be normal; otherwise, the data volume judgment result is determined to be abnormal. Based on the abnormal incremental judgment result, the full judgment result, and the incremental trend judgment result, the cause of the abnormal data volume of the data volume monitoring judgment result is determined.

[0045] Furthermore, the step of judging outliers in the data based on the target time range and the data outlier checklist to obtain the data monitoring judgment result includes:

[0046] Based on the target time range, obtain each piece of related data to be processed from the data outlier check table;

[0047] Take any one of the aforementioned associated data to be processed as the associated data to be judged;

[0048] If the results of the primary key anomaly judgment, the time anomaly judgment, and the primary key duplicate anomaly judgment in the data to be judged are all normal, the data monitoring judgment result is determined to be normal; otherwise, the data monitoring judgment result is determined to be abnormal.

[0049] This application also proposes a data monitoring device for a data warehouse, the device comprising:

[0050] The first request acquisition module is used to acquire data synchronization requests from the target business system.

[0051] The first update module is used to respond to the data synchronization request and update the data volume statistics table according to the number of incremental data rows to be stored and the number of full data rows to be stored in each data table to be uploaded in the target business system.

[0052] The second update module is used to respond to the data synchronization request, update the data warehouse according to each of the data tables to be uploaded, and update the data volume statistics table and the data outlier check table according to the data updated in the data warehouse.

[0053] The second request acquisition module is used to acquire an anomaly judgment request, the anomaly judgment request carrying a target time range;

[0054] The data volume monitoring judgment result determination module is used to make anomaly judgments on data volume based on the target time range and the data volume statistics table, and obtain the data volume monitoring judgment result.

[0055] The data monitoring judgment result determination module is used to judge the outliers of the data based on the target time range and the data outlier check table, and obtain the data monitoring judgment result.

[0056] This application also proposes a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described above.

[0057] This application also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0058] This application discloses a data warehouse data monitoring method, apparatus, device, and storage medium. The method first updates a data volume statistics table in response to a data synchronization request, based on the number of incremental and full data rows in the source tables to be stored for each data table to be uploaded in the target business system. Then, in response to the data synchronization request, it updates the data warehouse based on each of the data tables to be uploaded, and updates the data volume statistics table and data outlier check table based on the data updated in the data warehouse. Finally, it performs anomaly judgment on data volume based on a target time range and the data volume statistics table, obtaining a data volume monitoring judgment result. It also performs anomaly judgment on data based on the target time range and the data outlier check table, obtaining a data monitoring judgment result. This achieves real-time automated monitoring of the data volume and outliers of data synchronized to the data warehouse, avoiding the need to access Hive tables and use counting functions to count the incremental, full, and dirty data volumes for the day for data warehouse anomaly monitoring, thus improving the timeliness of data monitoring. Attached Figure Description

[0059] Figure 1 This is a flowchart illustrating a data monitoring method for a data warehouse according to an embodiment of this application;

[0060] Figure 2 This is a schematic block diagram of the structure of a data monitoring device for a data warehouse according to an embodiment of this application;

[0061] Figure 3 This is a schematic block diagram of the structure of a computer device according to an embodiment of this application.

[0062] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0064] Reference Figure 1This application provides a data monitoring method for a data warehouse, the method comprising:

[0065] S1: Obtain the data synchronization request from the target business system;

[0066] S2: In response to the data synchronization request, update the data volume statistics table according to the number of incremental data rows to be stored and the number of full data rows to be stored in each data table to be uploaded in the target business system.

[0067] S3: In response to the data synchronization request, update the data warehouse according to each of the data tables to be uploaded, and update the data volume statistics table and the data outlier check table according to the data updated in the data warehouse;

[0068] S4: Obtain an anomaly assessment request, wherein the anomaly assessment request carries a target time range;

[0069] S5: Based on the target time range and the data volume statistics table, make an anomaly judgment on the data volume and obtain the data volume monitoring judgment result;

[0070] S6: Based on the target time range and the data anomaly check table, perform anomaly judgment on the data to obtain the data monitoring judgment result.

[0071] This embodiment first updates the data volume statistics table in response to the data synchronization request, based on the number of incremental data rows and the number of full data rows in the source tables to be stored for each data table to be uploaded in the target business system. Then, in response to the data synchronization request, it updates the data warehouse based on each of the data tables to be uploaded, and updates the data volume statistics table and the data outlier check table based on the data updated in the data warehouse. Finally, it performs anomaly detection based on the target time range and the data volume statistics table to obtain a data volume monitoring judgment result, and performs anomaly detection based on the target time range and the data outlier check table to obtain a data monitoring judgment result. This achieves real-time automated monitoring of the data volume and outliers of the data synchronized to the data warehouse, avoiding the need to access Hive tables and use counting functions to count the incremental, full, and dirty data volumes for the day for data warehouse anomaly monitoring, thus improving the timeliness of data monitoring.

[0072] S1 can be a data synchronization request sent by the target business system according to preset conditions, or a data synchronization request generated by the big data platform according to preset conditions. For example, the preset conditions are preset time intervals, which are not specifically limited here.

[0073] A data synchronization request is a request to synchronize data from various tables in the database of a target business system to the data warehouse of a big data platform.

[0074] The target business system is an application system that provides business services.

[0075] Understandably, the target business system has a database for storing data, which includes multiple data tables.

[0076] For S2, in response to the data synchronization request, the database of the target business system is accessed, and each data table in the database of the target business system is taken as a data table to be uploaded. The incremental data count of each data table to be uploaded (that is, the incremental data count of the source table) is obtained from the database of the target business system. Each obtained incremental data count is taken as the incremental data count of the source table to be stored. The total data count of each data table to be uploaded (that is, the total data count of the source table) is obtained from the database of the target business system. Each obtained total data count is taken as the total data count of the source table to be stored. The incremental data count and the total data count of the source table to be stored are updated to the corresponding fields in the data volume statistics table.

[0077] The data volume statistics table includes: auto-increment key, source table name, number of incremental data rows in the source table, number of full data rows in the source table, data cache layer database table name, number of incremental data rows in the data cache layer database table, data paste source layer database table name, number of full data rows in the data paste source layer database table, number of full data rows in the data detail layer database table, number of dirty data rows, subsystem code, source database name, difference in the number of incremental data rows, and difference in the number of full data rows. Among them, the auto-increment key is the primary key, and the field name of the auto-increment key is id; the source table name refers to the table name of the data table in the business system; the incremental data count of the source table is the number of incremental data records in the source table (the data table in the business system); the total data count of the source table is the total number of data records in the source table (the data table in the business system); the data cache layer table name is the name of the data cache layer table; the incremental data count of the data cache layer table is the number of incremental data records in the data cache layer table; the source table name of the data paste is the name of the source table of the data paste; the total data count of the data paste source layer is... The total number of data rows in the source layer database table; the total number of data rows in the detail layer database table (which is the total number of data rows in the detail layer database table); the number of dirty data rows (which is the number of data rows that do not conform to the data table specifications, and this data can be obtained from the data management tables of the big data platform of the data warehouse); the subsystem code (which is the business system identifier); the source database name (which is the business system identifier); the incremental data row difference (which is the difference between the incremental data row count in the source table and the incremental data row count in the data cache layer database table); and the total data row difference (which is the difference between the total data row count in the source table and the total data row count in the data pasting source layer database table).

[0078] The data in the big data platform's data warehouse originates from business systems (such as a pharmacy's purchasing system). Each table in the business system has a corresponding table in the big data platform's data cache layer (STG), data source layer (ODS), and data detail layer (DWD) databases. The naming convention for these three tables is source table name + database name. The data cache layer tables primarily record daily changes or additions within the business system. The data source layer tables aggregate the daily data from the data cache layer tables, removing duplicates. The data detail layer tables filter out non-standard data based on the data source layer tables. Since these three tables filter data during processing, to prevent data loss and detect potential issues, the number of records in each table is recorded. This allows for comparison of differences between the tables to determine if data loss has occurred.

[0079] All information about the data tables created on the big data platform is recorded in the big data platform management data tables. In the open-source big data platform's metadata management database, there are tables named `tables` (recording table names and table IDs), `partitions` (recording table IDs, partition IDs, and partition names), and `partition_params` (recording partition IDs and the number of rows in each partition). When querying the data volume of a data cache layer table, simply find the table name ID in the `tables` table based on the data cache layer table name. Since partition names in the data cache layer table are date-based, find the partition ID in the `partitions` table based on the date and table name ID, and then find the number of rows in that partition (i.e., the incremental data row count) based on the partition ID. When querying the number of rows in a data source layer table, simply find the table name ID in the `tables` table based on the data source layer table name. Since the data source layer table is always... To query the total number of rows in a data source layer table, you can retrieve all partition IDs from the `partitions` table based on the table name ID of the source layer table. Then, you can find the number of rows corresponding to each partition ID and sum them up to obtain the total number of rows in the source layer table (i.e., the full number of rows). Similarly, to query the total number of rows in a detail layer table, you only need to find the table name ID in the `tables` table based on the detail layer table name. Since the detail layer table query only requires a total row count, you can retrieve all partition IDs from the `partitions` table based on the table name ID of the detail layer table. Then, you can find the number of rows corresponding to each partition ID and sum them up to obtain the total number of rows in the detail layer table (i.e., the full number of rows).

[0080] The number of incremental data rows in the source table and the number of full data rows in the source table are obtained by accessing the database of the business system. The names of the data cache layer tables, the number of incremental data rows in the data cache layer tables, the names of the data paste source layer tables, the number of full data rows in the data paste source layer tables, and the number of full data rows in the data detail layer tables are obtained through the Hive metadata of the data warehouse.

[0081] Hive metadata describes the data. Hive metadata includes: data synchronization time, table name, number of incremental data rows, and number of full data rows.

[0082] For S3, in response to the data synchronization request, the data in each of the data tables to be uploaded is updated to the data warehouse, and the data volume statistics table and the data outlier check table are updated respectively according to the data updated to the data warehouse.

[0083] The data outlier checklist includes: auto-increment key, data cache layer table name, primary key anomaly, time anomaly, and primary key duplication. Specifically, primary key anomaly refers to whether the business primary key in the data cache layer table is null; time anomaly refers to whether the data creation time in the data cache layer table is null; and primary key duplication refers to whether the business primary key exists in the data source layer table but its creation time is consistent with that of the business primary key in the data cache layer table.

[0084] A business primary key refers to a primary key composed of the business logic relationships of the tables within the business system itself.

[0085] For S4, the big data platform generates anomaly detection requests based on preset time intervals, or the anomaly detection requests can be sent by third-party application systems.

[0086] An exception detection request is a request to perform anomaly detection on the data in the data warehouse.

[0087] The target time range includes the period between the start time and the end time.

[0088] For S5, the data volume monitoring judgment result includes: data volume judgment result and data volume anomaly reason. The data volume judgment result has two possible values: one is abnormal, and the other is normal. When the data volume judgment result is abnormal, the data volume anomaly reason includes data; when the data volume judgment result is normal, the data volume anomaly reason does not include data.

[0089] Specifically, a first set of evaluation indicators is used to determine the abnormality of data volume based on the data in the data volume statistics table whose creation time is within the target time range. When the data in the data volume statistics table whose creation time is within the target time range meets all the first evaluation indicators in the first set of evaluation indicators, the data volume judgment result of the data volume monitoring and judgment result is determined to be normal; otherwise, the data volume judgment result of the data volume monitoring and judgment result is determined to be abnormal.

[0090] The first set of evaluation indicators includes one or more evaluation indicators.

[0091] For S6, a second set of evaluation indicators is used. Based on the data in the data anomaly check table whose creation time is within the target time range, anomaly judgment is made on the data. When the data in the data anomaly check table whose creation time is within the target time range meets all the second evaluation indicators in the second set of evaluation indicators, the data monitoring judgment result is determined to be normal; otherwise, the data monitoring judgment result is determined to be abnormal.

[0092] In one embodiment, the step of updating the data volume statistics table based on the number of incremental data rows to be stored and the number of full data rows to be stored in each data table to be uploaded in the target business system includes:

[0093] S21: Obtain the latest historical synchronization data time corresponding to the target business system, and determine the target filtering conditions based on the latest historical synchronization data time and the generation time of the data synchronization request;

[0094] S22: Based on the target filtering conditions, obtain the number of incremental data rows corresponding to each data table to be uploaded as the number of incremental data rows of the source table to be stored;

[0095] S23: Based on the generation time, obtain the number of full data rows corresponding to each data table to be uploaded as the number of full data rows of the source table to be stored;

[0096] S24: Add the table name of the data table to be uploaded, the number of incremental data rows of the source table to be stored, and the number of full data rows of the source table to be stored as associated data to the data volume statistics table.

[0097] This embodiment first determines the target filtering criteria based on the latest historical data synchronization time and the generation time of the current data synchronization request. Then, it obtains the number of incremental data rows to be stored in the source table based on the target filtering criteria, and obtains the number of full data rows to be stored in the source table based on the generation time of the current data synchronization request. Finally, it adds the table name of the data table to be uploaded, the number of incremental data rows to be stored in the source table, and the number of full data rows to be stored in the source table as associated data to the data volume statistics table. This realizes the addition of the data volume synchronized to the data warehouse by the target business system each time to the data volume statistics table, providing a basis for subsequent anomaly judgment of data volume.

[0098] For S21, the latest historical data synchronization time corresponding to the target business system can be obtained from the database. The latest historical data synchronization time is the time when the target business system last synchronized data to the data warehouse.

[0099] Among them, preset filtering conditions are adopted, and filtering conditions are generated based on the latest historical synchronized data time and the generation time of the data synchronization request, and the generated filtering conditions are used as target filtering conditions.

[0100] Optionally, the preset filtering condition is the time period between the latest historical data synchronization time and the generation time of the data synchronization request as the statistical time period, which does not include the latest historical data synchronization time but includes the generation time of the data synchronization request.

[0101] For S22, using the target filtering conditions, the number of incremental data rows (i.e., the number of data rows added to the data table) corresponding to each data table to be uploaded is obtained as the number of incremental data rows to be stored in the source table. In other words, each data table to be uploaded corresponds to one number of incremental data rows to be stored in the source table.

[0102] For example, when the preset filtering condition is the time period between the latest historical synchronized data time and the generation time of the data synchronization request, the number of incremental data rows of each data table to be uploaded within this time period is obtained as the number of incremental data rows of the source table to be stored. This example is not specifically limited.

[0103] For S23, the total number of data rows in each of the data tables to be uploaded at the generation time is obtained, and the obtained total number of data rows (that is, the total number of data rows in the data table) is used as the total number of data rows in the source table to be stored. In other words, each data table to be uploaded corresponds to one total number of data rows in the source table to be stored.

[0104] For S24, the table name of the data table to be uploaded, the number of incremental data rows of the source table to be stored, and the number of full data rows of the source table to be stored are used as associated data. This associated data is added to the data volume statistics table, so that the associated data is used as a record in the data volume statistics table.

[0105] Optionally, the generation time of the data synchronization request is used as the data synchronization time; the data synchronization time, the table name of the data table to be uploaded, the number of incremental data rows of the source table to be stored, and the number of full data rows of the source table to be stored are added as associated data to the data volume statistics table.

[0106] Specifically, the table name of the data table to be uploaded is inserted into the source table name in the data volume statistics table; the number of incremental data rows of the source table to be stored is inserted into the number of incremental data rows of the source table in the data volume statistics table; and the number of full data rows of the source table to be stored is inserted into the number of full data rows of the source table in the data volume statistics table.

[0107] In one embodiment, the steps of updating the data volume statistics table and the data outlier check table based on the data updated to the data warehouse include:

[0108] S31: Update the data volume statistics table according to the Hive metadata in the data warehouse corresponding to each of the data tables to be uploaded;

[0109] S32: Update the data anomaly check table according to the data cache layer table, data source layer table, and data detail layer table corresponding to each of the data tables to be uploaded in the data warehouse.

[0110] This embodiment updates the data volume statistics table using Hive metadata in the data warehouse, and updates the data outlier check table using the data cache layer table, data source layer table, and data detail layer table in the data warehouse, providing a basis for subsequent data volume anomaly judgment and data outlier judgment.

[0111] For S31, based on the Hive metadata in the data warehouse corresponding to each of the data tables to be uploaded, update the data volume statistics table with the data cache layer table name, the incremental data count of the data cache layer table, the data source layer table name, the total data count of the data source layer table, the total data count of the data detail layer table, and the dirty data count.

[0112] For S32, based on the data cache layer table name, data source layer table, and data detail layer table corresponding to each of the data tables to be uploaded in the data warehouse, update the data cache layer table name, primary key anomaly status, time anomaly status, primary key duplication status, data synchronization time, data creation time, and data update time of the data anomaly check table.

[0113] In one embodiment, the step of updating the data volume statistics table based on the Hive metadata corresponding to each of the data tables to be uploaded in the data warehouse includes:

[0114] S311: Obtain the table names corresponding to each of the data tables to be uploaded from the Hive metadata in the data warehouse to obtain the set of table names to be processed;

[0115] S312: Obtain a table name from the set of table names to be processed as the table name to be processed;

[0116] S313: Based on the table name to be processed, obtain from the big data platform management data table the name of the data cache layer library to be stored, the number of incremental data rows of the data cache layer library to be stored, the name of the data source layer library to be stored, the number of full data rows of the data source layer library to be stored, the number of full data rows of the data detail layer library to be stored, and the number of dirty data rows to be stored.

[0117] S314: Update the data volume statistics table with the name of the data cache layer table to be stored, the number of incremental data rows in the data cache layer table to be stored, the name of the data source layer table to be stored, the number of full data rows in the data source layer table to be stored, the number of full data rows in the data detail layer table to be stored, and the number of dirty data rows to be stored as associated data.

[0118] This embodiment first obtains the table names corresponding to each of the data tables to be uploaded from the Hive metadata in the data warehouse, and then obtains data from the big data platform management data table according to the table names to update the data volume statistics table, providing a basis for subsequent judgment of data volume anomalies.

[0119] For S311, the generation time of the data synchronization request is searched in the various data synchronization times in the Hive metadata of the data warehouse, and the table names corresponding to the data synchronization times found in the Hive metadata (the table names corresponding to the various data tables to be uploaded) are used as the set of table names to be processed.

[0120] For S312, one table name from the set of table names to be processed is obtained sequentially as the table name to be processed.

[0121] For S313, the following steps are taken: First, the name of the data cache layer table to be stored is found in the big data platform management data table. Second, the number of incremental data rows in the data cache layer table is found as the number of incremental data rows in the data cache layer table to be stored. Third, the name of the data source layer table is found as the name of the data source layer table to be stored. Fourth, the total number of data rows in the data source layer table is found as the total number of data rows in the data source layer table to be stored. Fifth, the total number of data rows in the data detail layer table is found as the total number of data rows in the data detail layer table to be stored. Sixth, the number of dirty data rows is found as the number of dirty data rows to be stored.

[0122] For S314, the name of the data cache layer table to be stored, the number of incremental data rows in the data cache layer table to be stored, the name of the data source layer table to be stored, the number of full data rows in the data source layer table to be stored, the number of full data rows in the data detail layer table to be stored, and the number of dirty data rows to be stored are used as associated data, and the associated data is used as a record to update each field in the data volume statistics table.

[0123] Optionally, the data synchronization time (i.e., the generation time of the data synchronization request), the name of the data cache layer table to be stored, the number of incremental data rows in the data cache layer table to be stored, the name of the data source layer table to be stored, the total number of data rows in the data source layer table to be stored, the total number of data rows in the data detail layer table to be stored, and the number of dirty data rows to be stored are used as associated data to update the data volume statistics table.

[0124] In one embodiment, the step of updating the data outlier check table based on the data cache layer table, data source layer table, and data detail layer table corresponding to each of the data tables to be uploaded in the data warehouse includes:

[0125] S321: Obtain the table names corresponding to each of the data tables to be uploaded from the Hive metadata in the data warehouse to obtain the set of table names to be analyzed;

[0126] S322: Obtain any one of the table names in the set of table names to be analyzed as the table name to be analyzed;

[0127] S323: Based on the name of the table to be analyzed, search for the business primary key from the newly added data of the data cache layer library table corresponding to each of the data tables to be uploaded, and obtain the target business primary key;

[0128] S324: When the target business primary key is an empty string or an empty object, the primary key exception judgment result is determined to be abnormal; otherwise, the primary key exception judgment result is determined to be normal.

[0129] S325: Based on the name of the table to be analyzed, find the creation time from the newly added data of the data cache layer library table corresponding to each of the data tables to be uploaded on the same day, and obtain the first creation time;

[0130] S326: When the first creation time is an empty string or an empty object, the time anomaly judgment result is determined to be abnormal; otherwise, the time anomaly judgment result is determined to be normal.

[0131] S327: Determine whether the name of the table to be analyzed exists as a business primary key in the data source layer library table corresponding to each of the data tables to be uploaded, and obtain the business primary key lookup result;

[0132] S328: When the business primary key lookup result is found to exist, according to the name of the table to be analyzed, the creation time is found from the newly added data of the data source layer database table corresponding to each of the data tables to be uploaded on the same day, and the second creation time is obtained;

[0133] S329: When the first creation time and the second creation time are the same, determine that the primary key duplication judgment result is not duplicated; otherwise, determine that the primary key duplication judgment result is duplicated.

[0134] S3210: Update the data anomaly check table according to the name of the table to be analyzed, the primary key anomaly judgment result, the time anomaly judgment result, and the primary key duplicate judgment result.

[0135] This embodiment implements the updating of the data outlier check table based on the data cache layer table name, primary key anomaly status, time anomaly status, primary key duplication status, data synchronization time, data creation time, and data update time, according to the data cache layer table name, data source layer table, and data detail layer table corresponding to each of the data tables to be uploaded in the data warehouse. This provides a foundation for subsequent outlier judgment of the data.

[0136] For S321, the generation time of the data synchronization request is searched in the various data synchronization times in the Hive metadata of the data warehouse, and the table names corresponding to the data synchronization times found in the Hive metadata (the table names corresponding to the various data tables to be uploaded) are used as the set of table names to be analyzed.

[0137] For S322, one table name from the set of table names to be analyzed is obtained sequentially as the table name to be analyzed.

[0138] For S323, the business primary key of the table to be analyzed is found in the newly added data of the data cache layer table corresponding to each data table to be uploaded on the same day, and the found business primary key is used as the target business primary key.

[0139] For S324, when the target business primary key is an empty string or an empty object, it means that the business primary key is abnormal, so the primary key abnormality judgment result can be determined to be abnormal; when the target business primary key is neither an empty string nor an empty object, it means that the business primary key is not abnormal, so the primary key abnormality judgment result can be determined to be normal.

[0140] For S325, the creation time of the table to be analyzed is found in the newly added data of the data cache layer table corresponding to each table to be uploaded on the same day, and the found time is used as the first creation time.

[0141] Creation time refers to the time the data was created within the data warehouse.

[0142] For S326, when the first creation time is an empty string or an empty object, it means that the creation time update is abnormal, so the time abnormality judgment result can be determined to be abnormal; when the first creation time is neither an empty string nor an empty object, it means that the creation time update is not abnormal, so the time abnormality judgment result can be determined to be normal.

[0143] For S327, the business primary key of the table to be analyzed is searched in the data source layer library table corresponding to each data table to be uploaded. When the business primary key is found, the business primary key search result is determined to exist. When the business primary key is not found, the business primary key search result is determined to not exist.

[0144] For S328, when the business primary key lookup result is found to exist, the creation time of the table to be analyzed is found in the daily new data of the data source layer database table corresponding to each data table to be uploaded, and the found creation time is used as the second creation time.

[0145] For S329, when the first creation time and the second creation time are the same, the primary key duplication judgment result is determined to be non-duplicate; when the first creation time and the second creation time are not the same, the primary key duplication judgment result is determined to be duplicated.

[0146] For S3210, the name of the table to be analyzed, the result of the primary key anomaly judgment, the result of the time anomaly judgment, and the result of the primary key duplicate judgment are used as associated data, and the associated data is used to update the data anomaly check table.

[0147] In one embodiment, the step of determining data volume anomalies based on the target time range and the data volume statistics table to obtain data volume monitoring and judgment results includes:

[0148] S51: Based on the target time range, obtain the relevant data to be analyzed from the data volume statistics table;

[0149] S52: Take any one of the aforementioned related data to be analyzed as the target related data;

[0150] S53: When the number of incremental data rows in the source table of the target associated data and the number of incremental data rows in the data cache layer database table are different, the incremental judgment result corresponding to the target associated data is determined to be abnormal; otherwise, the incremental judgment result is determined to be normal.

[0151] S54: When the sum of the total number of data entries and the number of dirty data entries in the data detail layer database table of the target associated data is different from the total number of data entries in the data source layer database table of the target associated data, the total judgment result corresponding to the target associated data is determined to be abnormal; otherwise, the total judgment result is determined to be normal.

[0152] S55: Generate an incremental trend chart based on the number of incremental data entries in the data cache layer table of each of the related data to be analyzed;

[0153] S56: Calculate the slope of the incremental trend graph to obtain the target slope;

[0154] S57: When the target slope is outside the preset slope range, the incremental trend judgment result is determined to be abnormal; otherwise, the incremental trend judgment result is determined to be normal.

[0155] S58: When the incremental judgment result, the full judgment result, and the incremental trend judgment result are all normal, the data volume judgment result of the data volume monitoring judgment result is determined to be normal; otherwise, the data volume judgment result is determined to be abnormal. Based on the abnormal incremental judgment result, the full judgment result, and the incremental trend judgment result, the cause of the abnormal data volume of the data volume monitoring judgment result is determined.

[0156] This embodiment uses incremental judgment results, full judgment results, and incremental trend judgment results as evaluation indicators to judge the abnormality of data volume. It realizes automated monitoring of the data volume of data synchronized to the data warehouse, avoids accessing Hive tables and using counting functions to count the incremental data volume, full data volume, and dirty data volume of the day for data warehouse anomaly monitoring, and improves the timeliness of data monitoring.

[0157] For S51, each related data within the target time range is obtained from the data volume statistics table, and each obtained related data is used as a related data to be analyzed.

[0158] For S52, any one of the various related data to be analyzed is taken as the target related data.

[0159] For S53, when the number of incremental data rows in the source table of the target associated data and the number of incremental data rows in the data cache layer table are different, the incremental judgment result corresponding to the target associated data is determined to be abnormal; when the number of incremental data rows in the source table of the target associated data and the number of incremental data rows in the data cache layer table are the same, the incremental judgment result is determined to be normal.

[0160] For S54, when the sum of the total number of data entries and the number of dirty data entries in the data detail layer table of the target associated data is different from the total number of data entries in the data source layer table of the target associated data, the total judgment result corresponding to the target associated data is determined to be abnormal; when the sum of the total number of data entries and the number of dirty data entries in the data detail layer table of the target associated data is the same as the total number of data entries in the data source layer table of the target associated data, the total judgment result is determined to be normal.

[0161] For S55, when generating the incremental trend chart, the number of incremental data records in the data cache layer table of each of the related data to be analyzed is used as the y-axis, and the generation time of the number of incremental data records in each of the data cache layer table is used as the x-axis.

[0162] For S56, the slope corresponding to the number of incremental data entries in each of the data cache layer tables is calculated for the incremental trend graph, and each calculated slope is used as a target slope.

[0163] For S57, when the target slope is outside the range of the preset slope obtained, the incremental trend judgment result is determined to be abnormal; when the target slope is within the range of the preset slope obtained, the incremental trend judgment result is determined to be normal.

[0164] For S58, the first set of evaluation indicators includes: incremental judgment results, full judgment results, and incremental trend judgment results.

[0165] Specifically, when the incremental judgment result, the full judgment result, and the incremental trend judgment result are all normal, it means that the data in the data volume statistics table whose update time is within the target time range meets all the first evaluation indicators in the first evaluation indicator set. Therefore, the data volume judgment result of the data volume monitoring judgment result is determined to be normal. When any one of the incremental judgment result, the full judgment result, and the incremental trend judgment result is abnormal, it means that the data in the data volume statistics table whose update time is within the target time range does not meet all the first evaluation indicators in the first evaluation indicator set. Therefore, the data volume judgment result is determined to be abnormal. When the data volume judgment result is abnormal, the abnormal incremental judgment result, the abnormal full judgment result, and the abnormal incremental trend judgment result are taken as the reasons for the abnormal data volume in the data volume monitoring judgment result. This facilitates the rapid identification of the cause of the abnormality.

[0166] In one embodiment, the step of judging outliers in the data based on the target time range and the data outlier check table to obtain the data monitoring judgment result includes:

[0167] S61: Based on the target time range, obtain each piece of related data to be processed from the data anomaly check table;

[0168] S62: Take any one of the aforementioned associated data to be processed as the associated data to be judged;

[0169] S63: When the primary key anomaly judgment result, time anomaly judgment result, and primary key duplicate anomaly judgment result in the data to be judged are all normal, the data monitoring judgment result is determined to be normal; otherwise, the data monitoring judgment result is determined to be abnormal.

[0170] This embodiment uses the results of primary key anomaly detection, time anomaly detection, and primary key duplicate anomaly detection as evaluation indicators to determine outliers in the data, thereby achieving automated monitoring of outliers in the data synchronized to the data warehouse and improving the timeliness of data monitoring.

[0171] For S61, each associated data within the target time range is obtained from the data outlier check table, and each obtained associated data is treated as an associated data to be processed.

[0172] For S62, any one of the related data to be processed is taken as the related data to be judged.

[0173] For S63, when the primary key anomaly judgment result, time anomaly judgment result, and primary key duplication anomaly judgment result in the data to be judged are all normal, the data monitoring judgment result is determined to be normal; when any one of the primary key anomaly judgment result, time anomaly judgment result, and primary key duplication anomaly judgment result in the data to be judged is abnormal, the data monitoring judgment result is determined to be abnormal.

[0174] Reference Figure 2 This application also proposes a data monitoring device for a data warehouse, the device comprising:

[0175] The first request acquisition module 100 is used to acquire data synchronization requests from the target business system.

[0176] The first update module 200 is used to respond to the data synchronization request and update the data volume statistics table according to the number of incremental data rows to be stored and the number of full data rows to be stored in each data table to be uploaded in the target business system.

[0177] The second update module 300 is used to respond to the data synchronization request, update the data warehouse according to each of the data tables to be uploaded, and update the data volume statistics table and the data outlier check table according to the data updated in the data warehouse.

[0178] The second request acquisition module 400 is used to acquire an anomaly judgment request, wherein the anomaly judgment request carries a target time range.

[0179] The data volume monitoring judgment result determination module 500 is used to make anomaly judgments on data volume based on the target time range and the data volume statistics table, and obtain the data volume monitoring judgment result.

[0180] The data monitoring judgment result determination module 600 is used to judge the outliers of the data based on the target time range and the data outlier check table, and obtain the data monitoring judgment result.

[0181] This embodiment first updates the data volume statistics table in response to the data synchronization request, based on the number of incremental data rows and the number of full data rows in the source tables to be stored for each data table to be uploaded in the target business system. Then, in response to the data synchronization request, it updates the data warehouse based on each of the data tables to be uploaded, and updates the data volume statistics table and the data outlier check table based on the data updated in the data warehouse. Finally, it performs anomaly detection based on the target time range and the data volume statistics table to obtain a data volume monitoring judgment result, and performs anomaly detection based on the target time range and the data outlier check table to obtain a data monitoring judgment result. This achieves real-time automated monitoring of the data volume and outliers of the data synchronized to the data warehouse, avoiding the need to access Hive tables and use counting functions to count the incremental, full, and dirty data volumes for the day for data warehouse anomaly monitoring, thus improving the timeliness of data monitoring.

[0182] Reference Figure 3 This application also provides a computer device, which may be a server, and its internal structure may be as follows: Figure 3 As shown, this computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data such as data monitoring methods for a data warehouse. The network interface allows communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a data monitoring method for a data warehouse. The data monitoring method for the data warehouse includes: acquiring a data synchronization request from a target business system; responding to the data synchronization request by updating a data volume statistics table based on the number of incremental data rows to be stored and the number of full data rows to be stored in each data table to be uploaded in the target business system; responding to the data synchronization request by updating the data warehouse based on each of the data tables to be uploaded, and updating the data volume statistics table and the data outlier check table based on the data updated in the data warehouse; acquiring an anomaly judgment request, the anomaly judgment request carrying a target time range; performing anomaly judgment on the data volume based on the target time range and the data volume statistics table to obtain a data volume monitoring judgment result; and performing anomaly judgment on the data based on the target time range and the data outlier check table to obtain a data monitoring judgment result.

[0183] This embodiment first updates the data volume statistics table in response to the data synchronization request, based on the number of incremental data rows and the number of full data rows in the source tables to be stored for each data table to be uploaded in the target business system. Then, in response to the data synchronization request, it updates the data warehouse based on each of the data tables to be uploaded, and updates the data volume statistics table and the data outlier check table based on the data updated in the data warehouse. Finally, it performs anomaly detection based on the target time range and the data volume statistics table to obtain a data volume monitoring judgment result, and performs anomaly detection based on the target time range and the data outlier check table to obtain a data monitoring judgment result. This achieves real-time automated monitoring of the data volume and outliers of the data synchronized to the data warehouse, avoiding the need to access Hive tables and use counting functions to count the incremental, full, and dirty data volumes for the day for data warehouse anomaly monitoring, thus improving the timeliness of data monitoring.

[0184] One embodiment of this application also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a data monitoring method for a data warehouse, comprising the steps of: obtaining a data synchronization request from a target business system; responding to the data synchronization request by updating a data volume statistics table based on the number of incremental data rows and the total number of data rows in the source tables to be stored for each data table to be uploaded in the target business system; responding to the data synchronization request by updating the data warehouse based on each of the data tables to be uploaded, and updating the data volume statistics table and the data outlier check table based on the data updated in the data warehouse; obtaining an anomaly judgment request, the anomaly judgment request carrying a target time range; performing anomaly judgment on the data volume based on the target time range and the data volume statistics table to obtain a data volume monitoring judgment result; and performing anomaly judgment on the data based on the target time range and the data outlier check table to obtain a data monitoring judgment result.

[0185] The data warehouse monitoring method described above first updates the data volume statistics table in response to the data synchronization request, based on the number of incremental and full data rows in the source tables to be stored for each data table to be uploaded in the target business system. Then, in response to the data synchronization request, it updates the data warehouse based on each of the data tables to be uploaded, and updates the data volume statistics table and the data outlier check table based on the data updated in the data warehouse. Finally, it performs anomaly detection based on the target time range and the data volume statistics table to obtain a data volume monitoring judgment result, and performs anomaly detection based on the target time range and the data outlier check table to obtain a data monitoring judgment result. This achieves real-time automated monitoring of the data volume and outliers of the data synchronized to the data warehouse, avoiding the need to access Hive tables and use counting functions to count the incremental, full, and dirty data volumes for the day for data warehouse anomaly monitoring, thus improving the timeliness of data monitoring.

[0186] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media provided in this application and in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0187] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0188] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A data monitoring method of a data warehouse, characterized by, The method comprises: obtaining a data synchronization request of a target business system; in response to the data synchronization request, updating a data volume statistics table according to the number of source table incremental data to be stored and the number of source table full data to be stored corresponding to each of the to-be-uploaded data tables in the target business system; in response to the data synchronization request, updating a data warehouse according to each of the to-be-uploaded data tables, and updating the data volume statistics table and a data outlier check table according to the data updated into the data warehouse; obtaining an exception judgment request, the exception judgment request carrying a target time range; using a first evaluation index set to perform data volume exception judgment according to the target time range and the data volume statistics table, and obtaining a data volume monitoring judgment result; the first evaluation index set includes an incremental judgment result, a full data judgment result, and an incremental trend judgment result; when the data in the data volume statistics table with a creation time within the target time range meets all the first evaluation indexes in the first evaluation index set, it is determined that the data volume judgment result of the data volume monitoring judgment result is normal, otherwise, it is determined that the data volume judgment result of the data volume monitoring judgment result is abnormal; performing data outlier judgment according to the target time range and the data outlier check table, and obtaining a data monitoring judgment result; the step of performing data volume exception judgment according to the target time range and the data volume statistics table to obtain a data volume monitoring judgment result comprises: obtaining each of the to-be-analyzed associated data from the data volume statistics table according to the target time range; taking any one of the to-be-analyzed associated data as target associated data; when the source table incremental data volume and the data cache layer library table incremental data volume of the target associated data are different, it is determined that the incremental judgment result corresponding to the target associated data is abnormal, otherwise, it is determined that the incremental judgment result is normal; when the sum of the data detail layer library table full data volume and the dirty data volume of the target associated data is different from the data source layer library table full data volume of the target associated data, it is determined that the full data judgment result corresponding to the target associated data is abnormal, otherwise, it is determined that the full data judgment result is normal; generating an incremental trend graph according to the data cache layer library table incremental data volume of each of the to-be-analyzed associated data; when generating the incremental trend graph, taking the data cache layer library table incremental data volume of each of the to-be-analyzed associated data as the y-axis, and taking the generation time of each of the data cache layer library table incremental data volume as the x-axis; performing slope calculation on the incremental trend graph corresponding to each of the data cache layer library table incremental data volume to obtain a target slope; when the target slope is outside the obtained preset slope range, it is determined that the incremental trend judgment result is abnormal, otherwise, it is determined that the incremental trend judgment result is normal; When the incremental judgment result, the full-amount judgment result and the incremental trend judgment result are all normal, determining that a data amount judgment result of the data amount monitoring judgment result is normal, otherwise, determining that the data amount judgment result is abnormal, and determining a data amount abnormal reason of the data amount monitoring judgment result according to the incremental judgment result, the full-amount judgment result and the incremental trend judgment result which are abnormal.

2. The data monitoring method of a data warehouse according to claim 1, wherein, The step of updating the data amount statistical table according to the source table incremental data amount and the source table full-amount data amount corresponding to each of the to-be-uploaded data tables in the target business system comprises: obtaining a historical latest synchronization data time corresponding to the target business system, and determining a target screening condition according to the historical latest synchronization data time and a generation time of the data synchronization request; obtaining the source table incremental data amount corresponding to each of the to-be-uploaded data tables as the source table incremental data amount according to the target screening condition; obtaining the source table full-amount data amount corresponding to each of the to-be-uploaded data tables as the source table full-amount data amount according to the generation time; adding the table name of the to-be-uploaded data table, the source table incremental data amount and the source table full-amount data amount as associated data into the data amount statistical table.

3. The data monitoring method of a data warehouse according to claim 1, wherein, The step of updating the data amount statistical table and the data abnormal value checking table according to the data updated into the data warehouse comprises: updating the data amount statistical table according to the Hive metadata corresponding to each of the to-be-uploaded data tables in the data warehouse; updating the data abnormal value checking table according to the data cache layer database table, the data source layer database table and the data detail layer database table corresponding to each of the to-be-uploaded data tables in the data warehouse.

4. The data monitoring method of a data warehouse according to claim 3, wherein, The step of updating the data amount statistical table according to the Hive metadata corresponding to each of the to-be-uploaded data tables in the data warehouse comprises: obtaining the table name corresponding to each of the to-be-uploaded data tables from the Hive metadata in the data warehouse to obtain a set of to-be-processed table names; obtaining one table name in the set of to-be-processed table names as a to-be-processed table name; obtaining the data cache layer database table name, the data cache layer database table incremental data amount, the data source layer database table name, the data source layer database table full-amount data amount, the data detail layer database table full-amount data amount and the dirty data amount to be stored according to the to-be-processed table name from the big data platform management data table; updating the data cache layer database table name, the data cache layer database table incremental data amount, the data source layer database table name, the data source layer database table full-amount data amount, the data detail layer database table full-amount data amount and the dirty data amount to be stored as associated data into the data amount statistical table.

5. The data monitoring method of a data warehouse according to claim 3, wherein, The step of updating the data abnormal value checking table according to the data cache layer database table, the data source layer database table and the data detail layer database table corresponding to each of the to-be-uploaded data tables in the data warehouse comprises: obtaining a table name corresponding to each of the to-be-uploaded data tables from Hive metadata in the data warehouse, to obtain a set of table names to be analyzed; obtaining any table name in the set of table names to be analyzed as a table name to be analyzed; finding a target business primary key from daily incremental data of the data cache layer table corresponding to each of the to-be-uploaded data tables according to the table name to be analyzed, to obtain the target business primary key; when the target business primary key is an empty string or an empty object, determining that a primary key exception judgment result is abnormal, otherwise, determining that the primary key exception judgment result is normal; finding a first creation time from daily incremental data of the data cache layer table corresponding to each of the to-be-uploaded data tables according to the table name to be analyzed, to obtain the first creation time; when the first creation time is an empty string or an empty object, determining that a time exception judgment result is abnormal, otherwise, determining that the time exception judgment result is normal; determining whether a business primary key exists in the data source layer table corresponding to each of the to-be-uploaded data tables according to the table name to be analyzed, to obtain a business primary key search result; when the business primary key search result is that the business primary key exists, finding a second creation time from daily incremental data of the data source layer table corresponding to each of the to-be-uploaded data tables according to the table name to be analyzed; when the first creation time and the second creation time are the same, determining that a primary key duplication judgment result is not duplicated, otherwise, determining that the primary key duplication judgment result is duplicated; updating the data exception value check table according to the table name to be analyzed, the primary key exception judgment result, the time exception judgment result, and the primary key duplication judgment result.

6. The data monitoring method of a data warehouse according to claim 1, wherein, The step of performing data exception value judgment according to the target time range and the data exception value check table to obtain a data monitoring judgment result, comprising: obtaining each to-be-processed associated data from the data exception value check table according to the target time range; taking any to-be-processed associated data as to-be-judged associated data; when the primary key exception judgment result, the time exception judgment result, and the primary key duplication exception judgment result in the to-be-judged associated data are all normal, determining that the data monitoring judgment result is normal, otherwise, determining that the data monitoring judgment result is abnormal.

7. A data monitoring apparatus of a data warehouse, characterized by comprising: The apparatus comprises: a first request obtaining module configured to obtain a data synchronization request of a target business system; a first updating module configured to update a data volume statistical table according to a number of to-be-stored source table incremental data and a number of to-be-stored source table full data of each to-be-uploaded data table in the target business system in response to the data synchronization request; a second updating module configured to update a data warehouse according to each to-be-uploaded data table in response to the data synchronization request, and update the data volume statistical table and a data exception value check table according to data updated into the data warehouse; a second request obtaining module configured to obtain an exception judgment request, the exception judgment request carrying a target time range; The data volume monitoring judgment result determination module is configured to determine data volume monitoring judgment results by using a first evaluation index set and performing data volume anomaly judgment according to the target time range and the data volume statistics table; the first evaluation index set includes an increment judgment result, a total volume judgment result, and an increment trend judgment result; when data in the data volume statistics table whose creation time is within the target time range meets all first evaluation indexes in the first evaluation index set, it is determined that the data volume judgment result of the data volume monitoring judgment result is normal; otherwise, it is determined that the data volume judgment result of the data volume monitoring judgment result is abnormal. The data monitoring judgment result determination module is configured to perform data anomaly value judgment according to the target time range and the data anomaly value inspection table to obtain data monitoring judgment results. The data volume monitoring judgment result determination module is configured to determine data volume monitoring judgment results by using a first evaluation index set and performing data volume anomaly judgment according to the target time range and the data volume statistics table; the first evaluation index set includes an increment judgment result, a total volume judgment result, and an increment trend judgment result; when data in the data volume statistics table whose creation time is within the target time range meets all first evaluation indexes in the first evaluation index set, it is determined that the data volume judgment result of the data volume monitoring judgment result is normal; otherwise, it is determined that the data volume judgment result of the data volume monitoring judgment result is abnormal. The data volume monitoring judgment result determination module is configured to determine data volume monitoring judgment results by using a first evaluation index set and performing data volume anomaly judgment according to the target time range and the data volume statistics table; the first evaluation index set includes an increment judgment result, a total volume judgment result, and an increment trend judgment result; when data in the data volume statistics table whose creation time is within the target time range meets all first evaluation indexes in the first evaluation index set, it is determined that the data volume judgment result of the data volume monitoring judgment result is normal; otherwise, it is determined that the data volume judgment result of the data volume monitoring judgment result is abnormal. The data volume monitoring judgment result determination module is configured to determine data volume monitoring judgment results by using a first evaluation index set and performing data volume anomaly judgment according to the target time range and the data volume statistics table; the first evaluation index set includes an increment judgment result, a total volume judgment result, and an increment trend judgment result; when data in the data volume statistics table whose creation time is within the target time range meets all first evaluation indexes in the first evaluation index set, it is determined that the data volume judgment result of the data volume monitoring judgment result is normal; otherwise, it is determined that the data volume judgment result of the data volume monitoring judgment result is abnormal. The data volume monitoring judgment result determination module is configured to determine data volume monitoring judgment results by using a first evaluation index set and performing data volume anomaly judgment according to the target time range and the data volume statistics table; the first evaluation index set includes an increment judgment result, a total volume judgment result, and an increment trend judgment result; when data in the data volume statistics table whose creation time is within the target time range meets all first evaluation indexes in the first evaluation index set, it is determined that the data volume judgment result of the data volume monitoring judgment result is normal; otherwise, it is determined that the data volume judgment result of the data volume monitoring judgment result is abnormal. The data volume monitoring judgment result determination module is configured to determine data volume monitoring judgment results by using a first evaluation index set and performing data volume anomaly judgment according to the target time range and the data volume statistics table; the first evaluation index set includes an increment judgment result, a total volume judgment result, and an increment trend judgment result; when data in the data volume statistics table whose creation time is within the target time range meets all first evaluation indexes in the first evaluation index set, it is determined that the data volume judgment result of the data volume monitoring judgment result is normal; otherwise, it is determined that the data volume judgment result of the data volume monitoring judgment result is abnormal. The data volume monitoring judgment result determination module is configured to determine data volume monitoring judgment results by using a first evaluation index set and performing data volume anomaly judgment according to the target time range and the data volume statistics table; the first evaluation index set includes an increment judgment result, a total volume judgment result, and an increment trend judgment result; when data in the data volume statistics table whose creation time is within the target time range meets all first evaluation indexes in the first evaluation index set, it is determined that the data volume judgment result of the data volume monitoring judgment result is normal; otherwise, it is determined that the data volume judgment result of the data volume monitoring judgment result is abnormal. The data volume monitoring judgment result determination module is configured to determine data volume monitoring judgment results by using a first evaluation index set and performing data volume anomaly judgment according to the target time range and the data volume statistics table; the first evaluation index set includes an increment judgment result, a total volume judgment result, and an increment trend judgment result; when data in the data volume statistics table whose creation time is within the target time range meets all first evaluation indexes in the first evaluation index set, it is determined that the data volume judgment result of the data volume monitoring judgment result is normal; otherwise, it is determined that the data volume judgment result of the data volume monitoring judgment result is abnormal. The data volume monitoring judgment result determination module is configured to determine data volume monitoring judgment results by using a first evaluation index set and performing data volume anomaly judgment according to the target time range and the data volume statistics table; the first evaluation index set includes an increment judgment result, a total volume judgment result, and an increment trend judgment result; when data in the data volume statistics table whose creation time is within the target time range meets all first evaluation indexes in the first evaluation index set, it is determined that the data volume judgment result of the data volume monitoring judgment result is normal; otherwise, it is determined that the data volume judgment result of the data volume monitoring judgment result is abnormal. The data volume monitoring judgment result determination module is configured to determine data volume monitoring judgment results by using a first evaluation index set and performing data volume anomaly judgment according to the target time range and the data volume statistics table; the first evaluation index set includes an increment judgment result, a total volume judgment result, and an increment trend judgment result; when data in the data volume statistics table whose creation time is within the target time range meets all first evaluation indexes in the first evaluation index set, it is determined that the data volume judgment result of the data volume monitoring judgment result is normal; otherwise, it is determined that the data volume judgment result of the data volume monitoring judgment result is abnormal.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The data volume monitoring judgment result determination module is configured to determine data volume monitoring judgment results by using a first evaluation index set and performing data volume anomaly judgment according to the target time range and the data volume statistics table; the first evaluation index set includes an increment judgment result, a total volume judgment result, and an increment trend judgment result; when data in the data volume statistics table whose creation time is within the target time range meets all first evaluation indexes in the first evaluation index set, it is determined that the data volume judgment result of the data volume monitoring judgment result is normal; otherwise, it is determined that the data volume judgment result of the data volume monitoring judgment result is abnormal.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The data volume monitoring judgment result determination module is configured to determine data volume monitoring judgment results by using a first evaluation index set and performing data volume anomaly judgment according to the target time range and the data volume statistics table; the first evaluation index set includes an increment judgment result, a total volume judgment result, and an increment trend judgment result; when data in the data volume statistics table whose creation time is within the target time range meets all first evaluation indexes in the first evaluation index set, it is determined that the data volume judgment result of the data volume monitoring judgment result is normal; otherwise, it is determined that the data volume judgment result of the data volume monitoring judgment result is abnormal. The data volume monitoring judgment result determination module is configured to determine data volume monitoring judgment results by using a first evaluation index set and performing data volume anomaly judgment according to the target time range and the data volume statistics table; the first evaluation index set includes an increment judgment result, a total volume judgment result, and an increment trend judgment result; when data in the data volume statistics table whose creation time is within the target time range meets all first evaluation indexes in the first evaluation index set, it is determined that the data volume judgment result of the data volume monitoring judgment result is normal; otherwise, it is determined that the data volume judgment result of the data volume monitoring judgment result is abnormal. The data volume monitoring judgment result determination module is configured to determine data volume monitoring judgment results by using a first evaluation index set and performing data volume anomaly judgment according to the target time range and the data volume statistics table; the first evaluation index set includes an increment judgment result, a total volume judgment result, and an increment trend judgment result; when data in the data volume statistics table whose creation time is within the target time range meets all first evaluation indexes in the first evaluation index set, it is determined that the data volume judgment result of the data volume monitoring judgment result is normal; otherwise, it is determined that the data volume judgment result of the data volume monitoring judgment result is abnormal. The data volume monitoring judgment result determination module is configured to determine data volume monitoring judgment results by using a first evaluation index set and performing data volume anomaly judgment according to the target time range and the data volume statistics table; the first evaluation index set includes an increment judgment result, a total volume judgment result, and an increment trend judgment result; when data in the data volume statistics table whose creation time is within the target time range meets all first evaluation indexes in the first evaluation index set, it is determined that the data volume judgment result of the data volume

Images