Data exception monitoring method and device, equipment, storage medium and product

Abstract

The embodiment of the invention provides a data exception monitoring method and device, equipment, a storage medium and a product. According to the technical scheme provided by the embodiment of the invention, metadata information of a plurality of to-be-monitored tables is acquired, and to-be-analyzed data size information and historical data size information of each flow table are determined according to table monitoring metadata; determining to-be-analyzed data index information and historical data index information of each service table according to the index monitoring metadata, determining a data size confidence interval according to the historical data size information, and determining a data index confidence interval according to the historical data index information; the flow monitoring result of the flow table is determined according to the to-be-analyzed data size information and the data size confidence interval, and the index monitoring result of the service table is determined according to the to-be-analyzed data index information and the data index confidence interval, so that the dependence of data exception monitoring on a predefined exception threshold is effectively reduced, and the data exception monitoring efficiency is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular, to a data anomaly monitoring method, device, equipment, storage medium and product. Background Art

[0002] With the rapid growth of enterprise data scale, the big data technology ecosystem is becoming increasingly complex. Data flows between multiple components and data sources with different structures. Big data anomaly monitoring must not only run through the entire data pipeline, but also involve massive data scenarios with multiple sources, heterogeneity, and probabilistic changes in metadata.

[0003] In the current data anomaly monitoring solution, the data in the set table to be monitored is generally compared with the set anomaly threshold, and the data anomaly is judged based on the comparison result between the real-time data and the anomaly threshold. However, this data anomaly monitoring solution has a strong reliance on predefined anomaly thresholds, requires a lot of manual configuration of anomaly thresholds, is difficult to adapt to the rapidly changing data environment, and has low data anomaly monitoring efficiency. Summary of the invention

[0004] The embodiments of the present application provide a data anomaly monitoring method, apparatus, device, storage medium and product to solve the technical problems in the related art that the data anomaly monitoring scheme strongly relies on predefined anomaly thresholds, is difficult to adapt to the rapidly changing data environment, and has low data anomaly monitoring efficiency. It can effectively reduce the dependence of data anomaly monitoring on predefined anomaly thresholds and improve the efficiency of data anomaly monitoring.

[0005] In a first aspect, an embodiment of the present application provides a data anomaly monitoring method, comprising:

[0006] Obtain metadata information of multiple tables to be monitored, the metadata information includes table monitoring metadata and indicator monitoring metadata, the tables to be monitored include flow tables corresponding to upstream services and business tables corresponding to midstream and downstream services;

[0007] Determine the data size information to be analyzed and the historical data size information of each of the flow tables according to the table monitoring metadata, and determine the data index information to be analyzed and the historical data index information of each of the business tables according to the index monitoring metadata;

[0008] Determining a data size confidence interval according to the historical data size information, and determining a data indicator confidence interval according to the historical data indicator information;

[0009] The flow monitoring result of the flow table is determined according to the data size information to be analyzed and the data size confidence interval, and the indicator monitoring result of the business table is determined according to the data indicator information to be analyzed and the data indicator confidence interval.

[0010] In a second aspect, an embodiment of the present application provides a data anomaly monitoring device, including a data acquisition module, a data processing module, a data analysis module and a result determination module, wherein:

[0011] The data acquisition module is configured to acquire metadata information of multiple tables to be monitored, wherein the metadata information includes table monitoring metadata and indicator monitoring metadata, and the tables to be monitored include flow tables corresponding to upstream services and business tables corresponding to midstream and downstream services;

[0012] The data processing module is configured to determine the size information of the data to be analyzed and the size information of the historical data of each of the flow tables according to the table monitoring metadata, and to determine the index information of the data to be analyzed and the index information of the historical data of each of the business tables according to the index monitoring metadata;

[0013] The data analysis module is configured to determine a data size confidence interval based on the historical data size information, and to determine a data indicator confidence interval based on the historical data indicator information;

[0014] The result determination module is configured to determine the flow monitoring result of the flow table according to the data size information to be analyzed and the data size confidence interval, and to determine the indicator monitoring result of the business table according to the data indicator information to be analyzed and the data indicator confidence interval.

[0015] In a third aspect, an embodiment of the present application provides a data anomaly monitoring device, including: a memory and one or more processors;

[0016] The memory is used to store one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the data anomaly monitoring method as described in the first aspect.

[0018] In a fourth aspect, an embodiment of the present application provides a non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute the data anomaly monitoring method as described in the first aspect.

[0019] In the fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program, which is stored in a computer-readable storage medium, and at least one processor of the device reads and executes the computer program from the computer-readable storage medium, so that the device performs the data anomaly monitoring method as described in the first aspect.

[0020] The embodiment of the present application obtains metadata information of multiple tables to be monitored, determines the data size information to be analyzed and the historical data size information of each flow table according to the table monitoring metadata, and determines the data indicator information to be analyzed and the historical data indicator information of each business table according to the indicator monitoring metadata, determines the data size confidence interval according to the historical data size information, and determines the data indicator confidence interval according to the historical data indicator information, determines the flow monitoring result of the flow table according to the data size information to be analyzed and the data size confidence interval, and determines the indicator monitoring result of the business table according to the data indicator information to be analyzed and the data indicator confidence interval, determines the data confidence intervals of the upstream flow table and the midstream and downstream business tables respectively through historical data, and determines the monitoring results corresponding to the flow table and the business table according to the data confidence interval, does not require a large amount of manual configuration of predefined thresholds, effectively reduces the dependence of data anomaly monitoring on predefined anomaly thresholds, and improves the efficiency of data anomaly monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of a data anomaly monitoring method provided by an embodiment of the present application;

[0022] Figure 2 is a flow chart of another data anomaly monitoring method provided by an embodiment of the present application;

[0023] Figure 3 It is a structural schematic diagram of a data anomaly monitoring device provided in an embodiment of the present application;

[0024] Figure 4 It is a structural diagram of a data anomaly monitoring device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The above process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The above process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0026] The data anomaly monitoring method provided in this application can be applied to the full life cycle quality monitoring scenarios of multi-source heterogeneous big data. It aims to determine the data confidence intervals of the upstream flow table and the mid- and downstream business tables respectively through historical data, and determine the monitoring results corresponding to the flow table and the business table based on the data confidence intervals. It does not require a large amount of manual configuration of predefined thresholds, effectively reducing the dependence of data anomaly monitoring on predefined anomaly thresholds and improving the efficiency of data anomaly monitoring.

[0027] With the rapid growth of enterprise data, the big data technology ecosystem is becoming increasingly complex. Data flows between multiple components and different structured data sources, such as Hadoop (an open source framework that can be used to efficiently store and process large data sets from GB to PB), Spark (an analytical engine for large-scale data processing), Kafka (a distributed data storage), Hive (a non-relational database management system), Clickhouse (an open source column-based database for online analytical processing), etc. Big data anomaly monitoring must not only run through the entire data pipeline, but also involve massive data scenarios with multi-source heterogeneity and probabilistic changes in metadata. However, existing data anomaly monitoring solutions strongly rely on predefined business rules and thresholds, require a lot of manual configuration, and are difficult to adapt to rapidly changing data environments. They generally use statistical methods to detect outliers, and have limited recognition capabilities for nonlinear trends and complex patterns. The automatic learning of data patterns using machine learning algorithms requires a large amount of high-quality training data, and model tuning is complex. Existing data anomaly monitoring solutions generally only target downstream business reports, ignoring other stages of the data life cycle. Data anomaly monitoring relies too much on manual experience and lacks adaptive capabilities.

[0028] Based on this, a data anomaly monitoring method is provided in an embodiment of the present application to solve the technical problems that the existing data anomaly monitoring solutions strongly rely on predefined anomaly thresholds, are difficult to adapt to rapidly changing data environments, and have low data anomaly monitoring efficiency. The method can integrate many cutting-edge technologies such as metadata management, data governance, machine learning, and visualization. Through machine learning, the data patterns of the table and indicators are learned from historical sample data, providing smarter and more efficient quality analysis capabilities, and then realizing dynamic quality monitoring based on analysis, achieving timely, accurate, and efficient monitoring effects, ensuring the security, integrity, and consistency of enterprise data assets, and providing reliable guarantees for maximizing data value.

[0029] Figure 1 A flow chart of a data anomaly monitoring method provided in an embodiment of the present application is given. The data anomaly monitoring method provided in an embodiment of the present application can be executed by a data anomaly monitoring device, which can be implemented by hardware and / or software and integrated in the data anomaly monitoring device.

[0030] The following description is made by taking the data anomaly monitoring device executing the data anomaly monitoring method as an example. Figure 1 , the data anomaly monitoring method includes:

[0031] S110: Obtain metadata information of multiple tables to be monitored, where the metadata information includes table monitoring metadata and indicator monitoring metadata. The tables to be monitored include flow tables corresponding to upstream services and business tables corresponding to midstream and downstream services.

[0032] The tables to be monitored provided by this solution include the flow table corresponding to the upstream business and the business table corresponding to the mid- and downstream business. The flow table corresponding to the upstream business can be the original flow data on the production side in the enterprise data, and the business table corresponding to the mid- and downstream business can be the analyzable data on the consumption side in the enterprise data. The flow table corresponding to the upstream business is mainly the original data table on the production side that has not been processed by ETL (extraction, transformation and loading). The data volume is huge and the types are diverse. It has not been cleaned by the data warehouse, such as error correction, deduplication, and aggregation. Since the flow table is located upstream of the data, if the data is interrupted, dropped suddenly, or double-written due to system failure or misoperation, the downstream cannot perceive it in time, which will easily cause large-scale data pollution. This solution can monitor the upstream data from the macro distributed file system file (such as HDFS file). For the mid- and downstream business tables, the business tables directly or indirectly correspond to the report. The business tables are analyzable data processed by the data warehouse, and the business tables can be monitored at a more fine-grained field level.

[0033] Exemplarily, metadata information of multiple pre-set tables to be monitored can be obtained in response to data anomaly monitoring operations, or according to set time intervals or set monitoring times. Optionally, the tables to be monitored may be all or part of the data tables in the database, such as obtaining metadata for a full data table. Among them, metadata includes table monitoring metadata and indicator monitoring metadata, such as obtaining table monitoring metadata for a traffic table corresponding to the upstream of the business, and obtaining indicator monitoring metadata for a business table corresponding to the downstream of the business. Optionally, the tables to be monitored and metadata information provided by this solution can be saved in a preset data table (such as a Hive table). In one embodiment, the metadata information may also include auxiliary metadata corresponding to the table to be monitored.

[0034] In one embodiment, metadata may be a combination of one or more of file path information (e.g., HDFS path information), partition information, indicator name, indicator type, indicator annotation, table size, partition size, table heat, lineage, and owner. For example, table monitoring metadata may include file path information and partition information, indicator monitoring metadata may include indicator name, indicator type, and indicator annotation, and auxiliary metadata may include table size, partition size, table heat, lineage, and owner. Optionally, after obtaining metadata information of multiple tables to be monitored, the metadata information of multiple tables to be monitored may be saved in a Hive table for subsequent data anomaly monitoring process to call.

[0035] In a possible embodiment, after obtaining metadata information of multiple tables to be monitored, the data anomaly monitoring method provided by the present solution may also include: determining data anomaly reports in each table to be monitored based on the metadata information, and screening out data anomaly reports from multiple tables to be monitored, wherein the report anomaly types of the data anomaly reports include one or more combinations of empty tables, stopped tables, interrupted tables, fluctuating tables, and low-heat tables.

[0036] Exemplarily, after obtaining metadata information of each table to be monitored, it is determined whether each table to be monitored is a data anomaly report based on the metadata information, and these data anomaly reports are filtered out. The table to be monitored corresponding to the non-data anomaly report will enter the next step of the data anomaly monitoring process.

[0037] In one embodiment, the report anomaly type of the data anomaly report provided by this solution includes a combination of one or more of an empty table, a stop table, a disconnect table, a fluctuation table, and a low heat table. For example, an empty table is a report whose data storage size is 0 or less than a set size (for example, the size of the table to be monitored is determined based on metadata); a stop table is a table whose corresponding scheduling task has been stopped but retains historical data, and whether it is a stop table can be determined based on the stop date or the latest update date of the report; a disconnect table is a report whose corresponding scheduling task is operating normally, has the latest date partition, but the data storage size in some partitions is 0 or less than the set size; a fluctuation table is a report whose scheduling task is operating normally, has the latest date partition, but belongs to a new business line, or is in the process of migration, and the data is in an unstable transition period; a low heat table is a report whose corresponding scheduling task is operating normally, has the latest date partition, but the number of valid accesses in the last set number of days is less than the set number. This solution reduces the monitoring of data anomaly reports with low monitoring value by screening out data anomaly reports in the table to be monitored, reduces the data processing volume of data anomaly monitoring, and improves the efficiency of data anomaly monitoring.

[0038] S120: Determine the size information of the data to be analyzed and the size information of the historical data of each flow table according to the table monitoring metadata, and determine the index information of the data to be analyzed and the index information of the historical data of each business table according to the index monitoring metadata.

[0039] Exemplarily, the data size information to be analyzed of the traffic table corresponding to each table monitoring metadata (i.e., the data size information of the traffic table corresponding to the current time window) and the historical data size information (i.e., the data size information of the traffic table corresponding to one or more time windows before the current time window) are determined, and the data indicator information to be analyzed of the business table corresponding to each indicator monitoring metadata (i.e., the data indicator information to be analyzed of the business table corresponding to the current time window) and the historical data indicator information (i.e., the data indicator information corresponding to the business table in one or more time windows before the current time window) are determined.

[0040] In one embodiment, the data indicator information provided by the scheme may be set indicator information and / or numerical information. The type corresponding to the indicator information may be non-monitoring types such as date, predefined field, supplementary field and complex structure field, or core id types such as user id, device id, order id and entry id, or core dimension types such as country, region, operating system and business line, or core attribute types such as version, brand, manufacturer, user name, channel and CPU core number, or numerical core indicator types such as DAU (Daily Active User), MAU (Monthly Active Users), PV (Page View), UV (Unique Visitor) and Retention. The numerical information may be the number of valid values, the sum of valid values, the mean of valid values, the proportion of extreme values, the proportion of abnormal values, the number of default values ​​(default values), the proportion of default values, the number of rows in the statistical table, the number of rows in the dimension, etc.

[0041] S130: Determine a data size confidence interval based on historical data size information, and determine a data indicator confidence interval based on historical data indicator information.

[0042] Exemplarily, the data size confidence interval is determined based on the historical data size information obtained above, and the data indicator confidence interval is determined based on the historical data indicator information. Wherein, when the flow table is in a normal state, the corresponding data size information to be analyzed should be within the corresponding data size confidence interval, and when the business table is in a normal state, the corresponding data indicator information to be analyzed should be within the corresponding data indicator confidence interval.

[0043] In a possible embodiment, the data anomaly monitoring method provided by the present scheme further includes at least one of the following before determining the data size confidence interval based on the historical data size information and determining the data indicator confidence interval based on the historical data indicator information: determining the monitored tables in each monitored table that exist in a preset whitelist, and filtering out the monitored tables in the preset whitelist from multiple monitored tables; determining the monitored tables in each monitored table that have undergone data migration within a set time period, and filtering out the monitored tables in which data migration has occurred from multiple monitored tables; determining the monitored tables in each monitored table whose historical data volume is less than the set data volume, and filtering out the monitored tables in which the historical data volume is less than the set data volume from multiple monitored tables.

[0044] Exemplarily, the preset whitelist provided by the present solution can be used to record reports that do not need to be monitored (for example, reports whose anomalies are known in advance and no longer need to be monitored, or reports that are not worth monitoring). Reports that do not need to be monitored can be determined in advance and recorded in the preset whitelist. After obtaining the data size information corresponding to the flow table and the data indicator information corresponding to the business table, it can be determined whether each table to be monitored exists in the preset whitelist, and the tables to be monitored that exist in the preset whitelist can be screened out from multiple tables to be monitored. These screened monitoring tables will not participate in the subsequent data anomaly monitoring process, reducing the processing of reports that do not need to be monitored and improving the efficiency of data anomaly monitoring.

[0045] In one embodiment, it can be determined whether there is a monitored table in which data migration has occurred within a set time period among the multiple monitored tables determined above, and the monitored tables in which data migration has occurred can be screened out from the multiple monitored tables. These screened-out monitoring tables will not participate in the subsequent data anomaly monitoring process, thereby reducing the processing of reports on data migration, improving the efficiency of data anomaly monitoring, and effectively reducing the situation of false alarms of data anomalies due to the instability of the monitored tables in which data migration has occurred.

[0046] In one embodiment, the amount of historical data of each table to be monitored is determined, the table to be monitored whose historical data amount is less than the set data amount is determined, and the table to be monitored whose historical data amount is less than the set data amount is screened out from multiple tables to be monitored. For example, if the amount of historical data for N days is less than N / n, for example, n=6, or the amount of data per day / number of rows in a dimension>i*j (for example, i*j=100*1024), it is considered that the amount of historical data is less than the set data amount, and these tables to be monitored are indicators corresponding to ultra-small dimensions, and these data are often missing, and these reports may not be monitored, thereby reducing the processing of reports that do not need to be monitored and improving the efficiency of data anomaly monitoring.

[0047] S140: Determine the flow monitoring result of the flow table according to the data size information to be analyzed and the data size confidence interval, and determine the indicator monitoring result of the business table according to the data indicator information to be analyzed and the data indicator confidence interval.

[0048] Exemplarily, after determining the data size confidence interval corresponding to each flow table and the data indicator confidence interval corresponding to each business table, the monitoring results corresponding to each flow table and business table are determined according to the data size confidence interval and the data indicator confidence interval respectively.

[0049] In one embodiment, for each flow meter, it is determined whether the data size information to be analyzed is within the corresponding data size confidence interval. If the data size information to be analyzed is within the corresponding data size confidence interval, a flow monitoring result indicating that the flow meter is normal is generated. If the data size information to be analyzed is not within the corresponding data size confidence interval, a flow monitoring result indicating that the flow meter is abnormal is generated.

[0050] In one embodiment, for each business table, it is determined whether the data indicator information to be analyzed is within the corresponding data indicator confidence interval. If the data indicator information to be analyzed is within the corresponding data indicator confidence interval, a business monitoring result indicating that the business table is normal is generated. If the data indicator information to be analyzed is not within the corresponding data indicator confidence interval, a business monitoring result indicating that the business table is abnormal is generated.

[0051] In the above, by obtaining metadata information of multiple tables to be monitored, determining the data size information to be analyzed and the historical data size information of each flow table according to the table monitoring metadata, and determining the data indicator information to be analyzed and the historical data indicator information of each business table according to the indicator monitoring metadata, determining the data size confidence interval according to the historical data size information, and determining the data indicator confidence interval according to the historical data indicator information, determining the flow monitoring result of the flow table according to the data size information to be analyzed and the data size confidence interval, and determining the indicator monitoring result of the business table according to the data indicator information to be analyzed and the data indicator confidence interval, determining the data confidence interval of the upstream flow table and the midstream and downstream business tables respectively through historical data, and determining the monitoring results corresponding to the flow table and the business table according to the data confidence interval, there is no need for a large amount of manual configuration of predefined thresholds, effectively reducing the dependence of data anomaly monitoring on predefined anomaly thresholds, and improving the efficiency of data anomaly monitoring.

[0052] Based on the above embodiments, Figure 2 A flowchart of another data anomaly monitoring method provided in an embodiment of the present application is given, and the data anomaly monitoring method is a specific embodiment of the above-mentioned data anomaly monitoring method. Figure 2 , the data anomaly monitoring method includes:

[0053] S210: Obtain metadata information of multiple tables to be monitored, where the metadata information includes table monitoring metadata and indicator monitoring metadata. The tables to be monitored include flow tables corresponding to upstream services and business tables corresponding to midstream and downstream services.

[0054] S220: Determine the size information of the data to be analyzed and the size information of the historical data of each flow table according to the table monitoring metadata, and determine the index information of the data to be analyzed and the index information of the historical data of each business table according to the index monitoring metadata.

[0055] Optionally, a task for monitoring traffic or dimension indicators can be constructed based on table monitoring metadata and / or indicator monitoring metadata, with reports (including traffic tables and business tables) as units, and submitted to the process pool for concurrent execution. By executing each task, the data size information and historical data size information to be analyzed of each traffic table, as well as the data indicator information and historical data indicator information to be analyzed of each business table are obtained. For example, in a distributed scheduling system platform for executing a data anomaly monitoring method, the check nodes in the workflow are configured according to the data kinship relationship of the report. When all upstream tasks are completed, a process pool is created, and the process pool performs balanced processing of task management tasks on each task. Among them, the corresponding data processing result format of each traffic table can be kept consistent, and the indicator statistics (such as sql format data) can level the differences between reports, ensure the consistency of the output data format, and enable the data to be uniformly stored or read. In addition to the storage of the task execution results, the execution status of each task can also be recorded and stored to reduce repeated execution when the task fails and is retried, saving computing resources.

[0056] In one possible embodiment, the data anomaly monitoring method provided by the present solution determines the data size information to be analyzed and the historical data size information of each flow table based on the table monitoring metadata, which can be: determining the data size information to be analyzed and the historical data size information of each flow table based on the table monitoring metadata and the partition type of each flow table, and the partition type includes a combination of one or more of the full table, daily partition table, hourly partition table, event partition table, and file system path data.

[0057] Exemplarily, for each flow table, the partition type corresponding to the flow table is determined, and the data size information of the flow table is determined according to the table monitoring metadata and the partition type. The size information of the data to be analyzed in the current time window and the size information of the historical data in the historical time window can be determined according to the data update time. For example, taking the flow table as a unit, the metadata corresponding to each flow table is read, and a data size acquisition command (such as a hadoop command) is constructed to calculate the corresponding size information of the data to be analyzed under a specific file path (such as an HDFS path) stored on the disk, and the historical data size information is obtained by accessing the hive table.

[0058] The partition type can be understood as the form of recording data in the flow table. For example, the partition type includes a combination of one or more of the full table, daily partition table, hourly partition table, event partition table, and file system path data. For different partition types, the corresponding data processing strategies can be: for the full table, the data size under the entire file path (such as the HDFS path) can be obtained; for the day partition table, the data size under the day partition path can be obtained. If the day partition table is calculated at the hourly level, the data size of the file generation time under the day partition path is calculated as a certain hour; for the hourly partition table, the data size under the day partition-hour partition path can be obtained; for the event partition table, the data size under the day partition-event partition path can be obtained. If the event partition table is calculated at the hourly level, the data size under the day partition-event partition-hour partition path is calculated; for pure file path (pure HDFS path) data, you can first explore whether the flow table has partitions. If there are partitions, the data size under the day partition path is counted. If it is calculated at the hourly level, the data size of the file generation time under the day partition path is calculated as a certain hour. If the flow table has no partitions, the flow table is treated as a full table for processing to obtain the data size under the entire file path. This solution accurately determines the size information of the data to be analyzed and the size information of the historical data of each flow table based on the table monitoring metadata and the partition type of each flow table, thereby improving the processing efficiency of the flow table and improving the accuracy and processing efficiency of data anomaly monitoring.

[0059] In one possible embodiment, the data anomaly monitoring method provided by the present solution determines the data indicator information to be analyzed and the historical data indicator information of each business table according to the indicator monitoring metadata, which can be: determining the statistical strategy corresponding to each business table according to the indicator type corresponding to the indicator monitoring metadata, the indicator type includes a combination of one or more of a non-monitoring type, a core identification type, a core dimension type, a core attribute type and a numerical core indicator type, and the statistical strategy includes a combination of one or more of an accuracy statistical strategy, a validity statistical strategy, an integrity statistical strategy and a data volume statistical strategy; determining the data indicator information to be analyzed and the historical data indicator information of each business table based on the statistical strategy.

[0060] Exemplarily, for each business table, one or more statistical strategies corresponding to the business table corresponding to the flow table are determined, and the indicator information of each business table is determined according to the statistical strategies. The indicator information of the data to be analyzed in the current time window and the indicator information of the historical data in the historical time window can be determined according to the indicator information update time. For example, taking the business table as a unit, the metadata corresponding to each business table is read, the indicator name, type and annotation information corresponding to the business table are identified through the metadata, and one or more statistical strategies are determined according to the indicator name, type and annotation information.

[0061] In one embodiment, the statistical strategy provided by this solution includes one or more combinations of accuracy statistical strategy, effectiveness statistical strategy, integrity statistical strategy and data volume statistical strategy. Among them, the accuracy statistical strategy can be to count the number of valid values, the sum of valid values ​​and the mean of valid values ​​in the corresponding business table, the effectiveness statistical strategy can be to count the extreme values ​​and the proportion of abnormal values ​​corresponding to the values ​​in the corresponding business table, the integrity statistical strategy can be to count the number of default values ​​and the proportion of default values ​​in the corresponding business table, and the data volume statistical strategy can be to count the number of rows in the statistical table and the number of rows in the dimension in the corresponding business table. This solution determines the statistical strategy corresponding to each business table according to the indicator type corresponding to the indicator monitoring metadata, and accurately determines the data indicator information to be analyzed and the historical data indicator information of each business table based on the statistical strategy, thereby improving the processing efficiency of the business table and the accuracy and processing efficiency of data anomaly monitoring.

[0062] S230: Determine a data size confidence interval based on historical data size information, and determine a data indicator confidence interval based on historical data indicator information.

[0063] In one possible embodiment, for each flow meter, the historical data size information can be processed based on normal distribution and / or linear prediction to determine the data size confidence interval corresponding to each flow meter (the corresponding data size confidence interval includes a first data size confidence interval and / or a second data size confidence interval). Based on this, the data anomaly monitoring method provided by the present scheme determines the data size confidence interval based on the historical data size information, which can be: determining the data size expected value and the data size standard deviation based on the historical data size information; determining the first error threshold based on the product of the data size standard deviation and a preset first probability adjustment parameter, and determining the first data size confidence interval of each flow meter based on the data size expected value and the first error threshold; and / or, performing linear prediction based on the historical data size information to obtain a size prediction value; determining the size error ratio based on a preset first linear adjustment parameter, and determining the second data size confidence interval of each flow meter based on the size prediction value and the size error ratio.

[0064] Exemplarily, for determining the data size confidence interval corresponding to each flow meter based on the normal distribution method, the data size expected value (mean1) and the data size standard deviation (std1) can be determined according to the historical data size information, the first error threshold can be determined according to the product of the data size standard deviation and a preset first probability adjustment parameter, and the first data size confidence interval of each flow meter can be determined according to the data size expected value and the first error threshold.

[0065] For example, the first data size confidence interval can be expressed as: [mean1-std1*k1, mean1+std1*k1], k1 is the first probability adjustment parameter, wherein, when k1 is 1, the probability that the data size information to be analyzed falls into the first data size confidence interval is 68.27%, when k1 is 2, the probability is 95.45, when k1 is 3, the probability is 99.73%, and optionally, the default value of the first probability adjustment parameter is 2.5. Wherein, when the flow monitoring result of the flow table is determined according to the data size information to be analyzed and the data size confidence interval, when the data size information to be analyzed is within the first data size confidence interval, the flow monitoring result is that the flow table is normal, and when the data size information to be analyzed is outside the first data size confidence interval, the flow monitoring result is that the flow table is abnormal. This scheme improves the accuracy of abnormal judgment of the flow table and improves the quality of data anomaly detection by accurately determining the data size confidence interval corresponding to each flow table based on the normal distribution method.

[0066] In one embodiment, for determining the data size confidence interval corresponding to each flow meter based on a linear prediction method, a linear prediction can be performed based on the historical data size information to obtain a size prediction value (for example, an expected value mean2 corresponding to the current time window of the execution prediction based on the historical data size information simulation), and the size error ratio can be determined based on the preset first linear adjustment parameter, and the second data size confidence interval of each flow meter can be determined based on the size prediction value and the size error ratio. For example, the sum of the preset reference ratio (for example, 1) and the first linear adjustment parameter and the difference between the preset reference ratio and the first linear adjustment parameter are respectively used as the first size error ratio and the second size error ratio, and the product of the first size error ratio and the size prediction value, and the range corresponding to the product of the second size error ratio and the size prediction value are used as the second data size confidence interval of the table to be monitored.

[0067] Optionally, the second data size confidence interval can be expressed as: [mean2*(1-k2), mean2*(1+k2)], where k2 is the first linear adjustment parameter, and the default value is 0.3, that is, the data size fluctuation of the flow meter is less than 30%, which is normal. Among them, when the flow monitoring result of the flow meter is determined according to the data size information to be analyzed and the data size confidence interval, when the data size information to be analyzed is within the second data size confidence interval, the flow monitoring result is that the flow meter is normal, and when the data size information to be analyzed is outside the second data size confidence interval, the flow monitoring result is that the flow meter is abnormal. This scheme improves the accuracy of abnormal judgment of the flow meter and improves the quality of data anomaly detection by accurately determining the data size confidence interval corresponding to each flow meter based on linear prediction.

[0068] In one embodiment, the flow monitoring results of the flow meter can be determined by combining the normal distribution and linear prediction methods. For example, the data size confidence interval corresponding to each flow meter is first accurately determined based on the normal distribution method, and it is determined whether the data size information to be analyzed of the flow meter is normal under the normal distribution. The normal distribution is more suitable for the determination of random distribution probability. For the new online / offline scenario, that is, the scenario of continuous growth / decrease of data, the data misjudgment rate based on the normal distribution is high. This solution is supplemented by the linear prediction algorithm, and further accurately determines the data size confidence interval corresponding to each flow meter based on the linear prediction method, and determines whether the data size information to be analyzed of the flow meter is normal under the linear prediction. Optionally, when the data size information to be analyzed of the flow meter based on the linear prediction method and the linear prediction is abnormal, the flow meter is considered abnormal, thereby improving the accuracy of abnormal judgment of the flow meter and improving the quality of data anomaly detection.

[0069] In one possible embodiment, for each flow table, the historical data size information can be processed based on normal distribution and / or linear prediction to determine the data indicator confidence interval corresponding to each business table (the corresponding data indicator confidence interval includes the first data indicator confidence interval and / or the second data indicator confidence interval). Based on this, the data anomaly monitoring method provided by this solution is characterized in that the data indicator confidence interval is determined based on the historical data indicator information, including:

[0070] Determine the expected value and standard deviation of the data indicator based on the historical data indicator information; determine the second error threshold based on the product of the data indicator standard deviation and a preset second probability adjustment parameter, and determine the first data indicator confidence interval of each business table based on the data indicator expected value and the second error threshold; and / or, perform linear prediction based on the historical data indicator information to obtain the indicator prediction value; determine the indicator error ratio based on a preset second linear adjustment parameter, and determine the second data indicator confidence interval of each business table based on the indicator prediction value and the indicator error ratio.

[0071] Exemplarily, for determining the confidence interval of the data indicator corresponding to each business table based on the normal distribution method, the data indicator expected value (mean3) and the data indicator standard deviation (std2) can be determined according to the historical data indicator information, the second error threshold can be determined according to the product of the data indicator standard deviation and a preset second probability adjustment parameter, and the second data indicator confidence interval of each business table can be determined according to the data indicator expected value and the second error threshold.

[0072] For example, the second data indicator confidence interval can be expressed as: [mean3-std2*k3, mean3+std2*k3], k3 is the second probability adjustment parameter, wherein, when k3 is 1, the probability that the data indicator information to be analyzed falls into the second data indicator confidence interval is 68.27%, when k3 is 2, the probability is 95.45, when k3 is 3, the probability is 99.73%, and optionally, the default value of the second probability adjustment parameter is 2.5. Wherein, when determining the indicator monitoring result of the business table according to the data indicator information to be analyzed and the data indicator confidence interval, when the data indicator information to be analyzed is within the second data indicator confidence interval, the indicator monitoring result is that the business table is normal, and when the data indicator information to be analyzed is outside the second data indicator confidence interval, the indicator monitoring result is that the business table is abnormal. This scheme accurately determines the data indicator confidence interval corresponding to each business table based on the normal distribution method, improves the accuracy of abnormal judgment of the business table, and improves the quality of data anomaly detection.

[0073] In one embodiment, for determining the confidence interval of the data indicator corresponding to each business table based on the linear prediction method, a linear prediction can be performed based on the historical data indicator information to obtain the indicator prediction value (for example, the expected value mean4 corresponding to the current time window of the execution prediction based on the historical data indicator information simulation), the indicator error ratio can be determined based on the preset second linear adjustment parameter, and the second data indicator confidence interval of each business table can be determined based on the indicator prediction value and the indicator error ratio. For example, the sum of the preset reference ratio (for example, 1) and the second linear adjustment parameter and the difference between the preset reference ratio and the second linear adjustment parameter are respectively used as the first indicator error ratio and the second indicator error ratio, and the product of the first indicator error ratio and the indicator prediction value, and the range corresponding to the product of the second indicator error ratio and the indicator prediction value are used as the second data indicator confidence interval of the table to be monitored.

[0074] Optionally, the second data indicator confidence interval can be expressed as: [mean4*(1-k4), mean4*(1+k4)], where k4 is the second linear adjustment parameter, and the default value is 0.3, that is, the data indicator fluctuation of the business table is less than 30% and the difference is normal. Among them, when determining the indicator monitoring result of the business table based on the data indicator information to be analyzed and the data indicator confidence interval, it can be that when the data indicator information to be analyzed is within the second data indicator confidence interval, the indicator monitoring result is that the business table is normal, and when the data indicator information to be analyzed is outside the second data indicator confidence interval, the indicator monitoring result is that the business table is abnormal. This scheme improves the accuracy of abnormal judgment of the business table and improves the quality of data anomaly detection by accurately determining the data indicator confidence interval corresponding to each business table based on a linear prediction method.

[0075] In one embodiment, the indicator monitoring results of the business table can be determined by combining the normal distribution and linear prediction methods. For example, the confidence interval of the data indicators corresponding to each business table is first accurately determined based on the normal distribution method, and whether the data indicator information to be analyzed of the business table is normal under the normal distribution. The normal distribution is more suitable for the determination of random distribution probability. For the scenario of new online / offline, that is, the scenario of continuous growth / decline of data, the data misjudgment rate based on the normal distribution is high. This solution is supplemented by the linear prediction algorithm, and further accurately determines the confidence interval of the data indicators corresponding to each business table based on the linear prediction method, and determines whether the data indicator information to be analyzed of the business table is normal under the linear prediction. Optionally, when the data indicator information to be analyzed of the business table based on the linear prediction method and the linear prediction is abnormal, the business table is considered abnormal, thereby improving the accuracy of abnormal judgment of the business table and improving the quality of data anomaly detection.

[0076] S240: Determine the flow monitoring result of the flow table according to the data size information to be analyzed and the data size confidence interval, and determine the indicator monitoring result of the business table according to the data indicator information to be analyzed and the data indicator confidence interval.

[0077] S250: Determine an abnormal flow table from multiple flow tables according to the flow monitoring result, and determine an abnormal business table from multiple business tables according to the indicator monitoring result.

[0078] S260: Sending the abnormal flow table and the abnormal service table to the user terminal for visual display processing based on the abnormal flow table and the abnormal service table.

[0079] In one embodiment, after determining the traffic monitoring results of the traffic table based on the data size information to be analyzed and the data size confidence interval, and determining the indicator monitoring results of the business table based on the data indicator information to be analyzed and the data indicator confidence interval, an abnormal traffic table can be determined from multiple traffic tables based on the traffic monitoring results, and an abnormal business table can be determined from multiple business tables based on the indicator monitoring results.

[0080] In one embodiment, after determining the abnormal flow table and the abnormal business table, the abnormal flow table and the abnormal business table can be sent to the user end. After receiving the abnormal flow table and the abnormal business table, the user end can perform visual display processing on the abnormal flow table and the abnormal business table so that the staff can observe the data abnormality. Optionally, in order to improve the efficiency of manual processing of abnormal data, an abnormal propagation diagram for displaying abnormal data can be constructed to track the source and impact range of the abnormality. At the same time, the abnormal propagation diagram can provide a timeline playback function to review the historical data quality change trend. The staff directly clicks on the pushed URL link to jump to the corresponding visualization page for multi-dimensional display and interactive exploration of the data. This solution improves the timeliness of data monitoring by sending the abnormal flow table and the abnormal business table to the user end for visual display processing, effectively ensures the timeliness of data quality and governance, provides reliable data quality insights for the data management layer, and effectively supports data-driven decision making.

[0081] In one embodiment, the abnormal flow table and abnormal business table are visualized, which can be the overall abnormal display of the abnormal report, the fluctuation abnormal display and the title (topic) flow display. Among them, the overall abnormal display can be performed for the abnormal flow table, and the overall abnormal display can be based on the data abnormal report determined according to the metadata information. The report source, abnormal type (such as stop, interruption, empty table, etc.) and table name and other filter items can be provided, and the abnormal data is displayed in the linkage table. The fluctuation abnormal display can be performed for the abnormal business table, and the report source, table / dimension / indicator name, day / hour filter items can be provided, and the abnormal data is displayed in the linkage numerical fluctuation curve. The title flow (such as the topic flow of kafka / pulsar) display is performed for the data of the upstream of the hive flow table, which is kafka or pulsar. If the fluctuation curve is consistent with the consumption curve of the title, the data is normal at this time, which is a common false alarm. Otherwise, relevant staff need to intervene and handle it, and its business line, cluster information, title and other parameters can be provided in the summary information, which is convenient for staff to quickly locate and confirm.

[0082] In a possible embodiment, the metadata information also includes auxiliary metadata. Based on this, the data anomaly monitoring method provided by the present solution determines an abnormal flow table from multiple flow tables according to the flow monitoring results, including: determining a candidate abnormal flow table from multiple flow tables according to the flow monitoring results; determining value information of each candidate abnormal flow table according to the auxiliary metadata corresponding to each candidate abnormal flow table, and determining the abnormal flow table from the candidate abnormal flow table according to the value information, the auxiliary metadata including one or more combinations of business source information, table size information and number of accesses.

[0083] Exemplarily, after obtaining the corresponding flow monitoring results of the flow table, a candidate abnormal flow table (for example, a flow table whose data size information to be analyzed is outside the data size confidence interval) is determined from multiple flow tables, and the value information of each candidate abnormal flow table is determined based on the auxiliary class metadata corresponding to each candidate abnormal flow table, and the abnormal flow table whose value information reaches the set value threshold is determined from multiple candidate abnormal flow tables.

[0084] In one embodiment, the auxiliary metadata provided by this solution includes one or more combinations of business source information, table size information, and the number of visits. The value information of the flow table can be calculated based on the combination of one or more of the business source information, table size information, and the number of visits. For example, the value values ​​corresponding to different business source information, table size information, and the number of visits are pre-set, and the sum, product, or weighted sum of the value values ​​corresponding to the business source information, table size information, and the number of visits is used as the value information corresponding to the flow table. This solution determines the value information of each candidate abnormal flow table based on the auxiliary metadata corresponding to each candidate abnormal flow table, pushes higher-value abnormal reports to the user end, reduces the push of low-value abnormal reports, and improves data monitoring quality.

[0085] In a possible embodiment, the data anomaly monitoring method provided by the present solution determines an abnormal business table from multiple business tables according to the indicator monitoring result, which may be:

[0086] Determine candidate abnormal business tables from multiple business tables based on the indicator monitoring results;

[0087] Determine weight information of each candidate abnormal business table according to the value and / or number of rows corresponding to each candidate abnormal business table, and determine the abnormal business table from the candidate abnormal business tables according to the weight information; and / or,

[0088] According to the abnormal continuity information corresponding to each candidate abnormal service table, and according to the abnormal continuity information, the abnormal service table is determined from the candidate abnormal service table.

[0089] Exemplarily, according to the above-determined indicator monitoring result of the business table, a candidate abnormal business table (eg, a business table whose data indicator information to be analyzed is outside the data indicator confidence interval) is determined from multiple business tables.

[0090] In one embodiment, after determining the candidate abnormal business tables, the weight information of each candidate abnormal business table can be determined according to the numerical values ​​and / or number of rows corresponding to each candidate abnormal business table (for example, the average value, median value, etc. corresponding to the numerical values ​​and / or number of rows), and the abnormal business table can be determined from the candidate abnormal business tables according to the weight information. For example, the candidate abnormal business table whose weight information is greater than the set weight threshold is determined as the abnormal business table, and abnormal reports with higher weights are pushed to the user end, reducing the push of low-weight abnormal reports, thereby improving data monitoring quality.

[0091] In one embodiment, after determining the candidate abnormal business table, the abnormal continuity information corresponding to each candidate abnormal business table can be determined (i.e., the number of consecutive times determined as a candidate abnormal business table, or the duration of the data indicator information to be analyzed being outside the data indicator confidence interval), and the candidate abnormal business table whose abnormal continuity information reaches a set number of times (for example, 3 times) or a set duration (for example, 3 hours) is used as the abnormal business table, and abnormal reports with a large number of abnormal times or a long duration of abnormalities are pushed to the user end, reducing the push of abnormal reports with a small number of abnormal times or a short duration of abnormalities under common abnormalities such as sudden increases or decreases in traffic, thereby improving data monitoring quality.

[0092] In the above, by obtaining metadata information of multiple tables to be monitored, determining the data size information and historical data size information to be analyzed of each flow table according to the table monitoring metadata, determining the data indicator information and historical data indicator information to be analyzed of each business table according to the indicator monitoring metadata, determining the data size confidence interval according to the historical data size information, and determining the data indicator confidence interval according to the historical data indicator information, determining the flow monitoring result of the flow table according to the data size information to be analyzed and the data size confidence interval, and determining the indicator monitoring result of the business table according to the data indicator information to be analyzed and the data indicator confidence interval, respectively determining the data confidence interval of the upstream flow table and the midstream and downstream business table through historical data, and determining the monitoring results corresponding to the flow table and the business table according to the data confidence interval, there is no need to manually configure a large number of predefined thresholds, effectively reducing the dependence of data anomaly monitoring on predefined anomaly thresholds, and improving the efficiency of data anomaly monitoring. At the same time, by sending abnormal flow tables and abnormal business tables to the user end for visual display processing by the user end, the timeliness of data monitoring is improved, the timeliness of data quality and governance is effectively guaranteed, and reliable data quality insights are provided to the data management layer, effectively supporting data-driven decision-making. This solution also provides full life cycle monitoring of data, realizes automated, intelligent, and visual monitoring of data, improves the coverage and accuracy of data monitoring, greatly reduces user time costs, and intuitively reflects the trends of various abnormal data in a visual way, assists in the rapid discovery and diagnosis of problems, and realizes an efficient data governance closed loop. Through the full life cycle monitoring of data, the data monitoring coverage rate is improved, covering the entire process from raw data collection to the final analysis report, and realizing comprehensive monitoring of all data assets of the enterprise. Through automated metadata management, the use of automation and intelligent technology significantly reduces manual configuration and maintenance work, and real-time perception of the report's online / offline and changes in the data structure in the table, and dynamically adapts the monitoring strategy, evaluates data quality from multiple angles such as data volume, completeness, accuracy, and consistency, and realizes intelligent anomaly identification. It also provides intuitive data quality trends and anomaly displays through multi-dimensional visualization, supports interactive exploration of data, and quickly locates and diagnoses data anomalies, effectively improving data monitoring quality and efficiency.

[0093] Figure 3 Schematic diagram of a data anomaly monitoring device provided by an embodiment of the present application. Figure 3 The data anomaly monitoring device includes a data acquisition module 31, a data processing module 32, a data analysis module 33 and a result determination module 34.

[0094] Among them, the data acquisition module 31 is configured to obtain metadata information of multiple tables to be monitored, the metadata information includes table monitoring metadata and indicator monitoring metadata, and the tables to be monitored include flow tables corresponding to the upstream business and business tables corresponding to the midstream and downstream business; the data processing module 32 is configured to determine the data size information to be analyzed and the historical data size information of each flow table according to the table monitoring metadata, and to determine the data indicator information to be analyzed and the historical data indicator information of each business table according to the indicator monitoring metadata; the data analysis module 33 is configured to determine the data size confidence interval according to the historical data size information, and to determine the data indicator confidence interval according to the historical data indicator information; the result determination module 34 is configured to determine the flow monitoring result of the flow table according to the data size information to be analyzed and the data size confidence interval, and to determine the indicator monitoring result of the business table according to the data indicator information to be analyzed and the data indicator confidence interval.

[0095] In the above, by obtaining metadata information of multiple tables to be monitored, determining the data size information to be analyzed and the historical data size information of each flow table according to the table monitoring metadata, and determining the data indicator information to be analyzed and the historical data indicator information of each business table according to the indicator monitoring metadata, determining the data size confidence interval according to the historical data size information, and determining the data indicator confidence interval according to the historical data indicator information, determining the flow monitoring result of the flow table according to the data size information to be analyzed and the data size confidence interval, and determining the indicator monitoring result of the business table according to the data indicator information to be analyzed and the data indicator confidence interval, determining the data confidence interval of the upstream flow table and the midstream and downstream business tables respectively through historical data, and determining the monitoring results corresponding to the flow table and the business table according to the data confidence interval, there is no need for a large amount of manual configuration of predefined thresholds, effectively reducing the dependence of data anomaly monitoring on predefined anomaly thresholds, and improving the efficiency of data anomaly monitoring.

[0096] In one possible embodiment, the data processing module 32 determines the size information of the data to be analyzed and the size information of the historical data of each flow table based on the table monitoring metadata, and is configured as follows: according to the table monitoring metadata and the partition type of each flow table, the size information of the data to be analyzed and the size information of the historical data of each flow table is determined, and the partition type includes a combination of one or more of the full table, the day partition table, the hour partition table, the event partition table, and the file system path data.

[0097] In a possible embodiment, the data processing module 32 determines the to-be-analyzed data indicator information and the historical data indicator information of each business table according to the indicator monitoring metadata, and is configured as follows:

[0098] Determine the statistical strategy corresponding to each business table according to the indicator type corresponding to the indicator monitoring metadata, the indicator type includes one or more combinations of non-monitoring type, core identification type, core dimension type, core attribute type and numerical core indicator type, and the statistical strategy includes one or more combinations of accuracy statistical strategy, effectiveness statistical strategy, integrity statistical strategy and data volume statistical strategy;

[0099] Determine the data indicator information to be analyzed and the historical data indicator information of each business table based on the statistical strategy.

[0100] In a possible embodiment, the data analysis module 33 determines the data size confidence interval according to the historical data size information, and is configured as follows:

[0101] Determine the expected data size and the data size standard deviation according to the historical data size information; determine the first error threshold according to the product of the data size standard deviation and a preset first probability adjustment parameter, and determine the first data size confidence interval of each flow meter according to the expected data size and the first error threshold; and / or

[0102] Linear prediction is performed based on historical data size information to obtain a size prediction value; the size error ratio is determined based on a preset first linear adjustment parameter, and the second data size confidence interval of each flow meter is determined based on the size prediction value and the size error ratio.

[0103] In a possible embodiment, the data analysis module 33 determines the data indicator confidence interval according to the historical data indicator information, and is configured as follows:

[0104] Determine the expected value of the data indicator and the standard deviation of the data indicator according to the historical data indicator information; determine the second error threshold according to the product of the standard deviation of the data indicator and a preset second probability adjustment parameter, and determine the confidence interval of the first data indicator of each business table according to the expected value of the data indicator and the second error threshold; and / or

[0105] Linear prediction is performed based on historical data indicator information to obtain indicator prediction values; the indicator error ratio is determined based on a preset second linear adjustment parameter, and the second data indicator confidence interval of each business table is determined based on the indicator prediction value and the indicator error ratio.

[0106] In a possible embodiment, the data anomaly monitoring device also includes an anomaly push module, which is configured to: determine an abnormal flow table from multiple flow tables based on the flow monitoring results, and determine an abnormal business table from multiple business tables based on the indicator monitoring results; send the abnormal flow table and the abnormal business table to the user end for visual display processing based on the abnormal flow table and the abnormal business table.

[0107] In a possible embodiment, the metadata information also includes auxiliary metadata, and the abnormal push module determines the abnormal flow table from multiple flow tables according to the flow monitoring result, and is configured as follows:

[0108] Determine a candidate abnormal flow table from multiple flow tables according to the flow monitoring result;

[0109] The value information of each candidate abnormal flow table is determined according to the auxiliary metadata corresponding to each candidate abnormal flow table, and the abnormal flow table is determined from the candidate abnormal flow tables according to the value information. The auxiliary metadata includes one or more combinations of business source information, table size information and number of access times.

[0110] In a possible embodiment, the abnormal push module determines the abnormal business table from multiple business tables according to the indicator monitoring result, and is configured as follows:

[0111] Determine candidate abnormal business tables from multiple business tables based on the indicator monitoring results;

[0112] Determine weight information of each candidate abnormal business table according to the value and / or number of rows corresponding to each candidate abnormal business table, and determine the abnormal business table from the candidate abnormal business tables according to the weight information; and / or

[0113] According to the abnormal continuity information corresponding to each candidate abnormal service table, and according to the abnormal continuity information, the abnormal service table is determined from the candidate abnormal service table.

[0114] In a possible embodiment, the data anomaly monitoring device further includes a data screening module, and the data screening module is configured as at least one of the following:

[0115] Determine the to-be-monitored tables that exist in the preset white list among the various to-be-monitored tables, and filter out the to-be-monitored tables that exist in the preset white list from the multiple to-be-monitored tables;

[0116] Determine the monitored tables in which data migration has occurred within a set time period among the monitored tables, and filter out the monitored tables in which data migration has occurred from the multiple monitored tables;

[0117] Determine the table to be monitored whose historical data volume is less than the set data volume among the tables to be monitored, and screen out the table to be monitored whose historical data volume is less than the set data volume from the multiple tables to be monitored.

[0118] In a possible embodiment, the data anomaly monitoring device also includes a data screening module, which is configured to: determine the data anomaly reports in each table to be monitored based on metadata information, and screen out the data anomaly reports from multiple tables to be monitored, and the report anomaly types of the data anomaly reports include one or more combinations of empty tables, stopped tables, interrupted tables, fluctuation tables and low heat tables.

[0119] It is worth noting that in the embodiment of the above-mentioned data anomaly monitoring device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present application.

[0120] The embodiment of the present application also provides a data anomaly monitoring device, which can integrate the data anomaly monitoring apparatus provided in the embodiment of the present application. Figure 4 Schematic diagram of a data anomaly monitoring device provided by an embodiment of the present application. Figure 4 The data anomaly monitoring device includes: an input device 43, an output device 44, a memory 42 and one or more processors 41; the memory 42 is used to store one or more programs; when one or more programs are executed by one or more processors 41, the one or more processors 41 implement the data anomaly monitoring method provided in the above embodiment. The above-mentioned data anomaly monitoring device, equipment and computer can be used to execute the data anomaly monitoring method provided in any of the above embodiments, and have corresponding functions and beneficial effects.

[0121] The embodiments of the present application also provide a non-volatile storage medium storing computer executable instructions, which are used to execute the data anomaly monitoring method provided in the above embodiments when executed by a computer processor. Of course, the non-volatile storage medium storing computer executable instructions provided in the embodiments of the present application, whose computer executable instructions are not limited to the data anomaly monitoring method provided above, can also execute the related operations in the data anomaly monitoring method provided in any embodiment of the present application. The data anomaly monitoring device, equipment and storage medium provided in the above embodiments can execute the data anomaly monitoring method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the data anomaly monitoring method provided in any embodiment of the present application.

[0122] On the basis of the above embodiments, the embodiments of the present application also provide a computer program product. The essence of the technical solution of the present application or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer program product is stored in a storage medium, including a number of instructions for enabling a computer device, a mobile terminal or a processor therein to execute all or part of the steps of the data anomaly monitoring method provided in each embodiment of the present application.

Claims

1. A data anomaly monitoring method, characterized in that: include: Obtain metadata information of multiple tables to be monitored, the metadata information includes table monitoring metadata and indicator monitoring metadata, the tables to be monitored include flow tables corresponding to upstream services and business tables corresponding to midstream and downstream services; Determine the data size information to be analyzed and the historical data size information of each of the flow tables according to the table monitoring metadata, and determine the data index information to be analyzed and the historical data index information of each of the business tables according to the index monitoring metadata; Determining a data size confidence interval according to the historical data size information, and determining a data indicator confidence interval according to the historical data indicator information; The flow monitoring result of the flow table is determined according to the data size information to be analyzed and the data size confidence interval, and the indicator monitoring result of the business table is determined according to the data indicator information to be analyzed and the data indicator confidence interval.

2. The data anomaly monitoring method according to claim 1, characterized in that: The determining the to-be-analyzed data size information and the historical data size information of each of the flow tables according to the table monitoring metadata includes: According to the table monitoring metadata and the partition type of each of the traffic tables, determine the data size information to be analyzed and the historical data size information of each of the traffic tables. The partition type includes one or more combinations of the full table, daily partition table, hourly partition table, event partition table, and file system path data.

3. The data anomaly monitoring method according to claim 1, characterized in that: The determining the data indicator information to be analyzed and the historical data indicator information of each of the business tables according to the indicator monitoring metadata includes: Determine the statistical strategy corresponding to each of the business tables according to the indicator type corresponding to the indicator monitoring metadata, the indicator type includes one or more combinations of non-monitoring type, core identification type, core dimension type, core attribute type and numerical core indicator type, and the statistical strategy includes one or more combinations of accuracy statistical strategy, effectiveness statistical strategy, integrity statistical strategy and data volume statistical strategy; The data indicator information to be analyzed and the historical data indicator information of each of the business tables are determined based on the statistical strategy.

4. The data anomaly monitoring method according to claim 1, characterized in that: Determining the data size confidence interval according to the historical data size information includes: Determine the expected data size value and the data size standard deviation according to the historical data size information; determine the first error threshold according to the product of the data size standard deviation and a preset first probability adjustment parameter, and determine the first data size confidence interval of each flow meter according to the expected data size value and the first error threshold; and / or Linear prediction is performed based on the historical data size information to obtain a size prediction value; the size error ratio is determined based on a preset first linear adjustment parameter, and the second data size confidence interval of each flow table is determined based on the size prediction value and the size error ratio.

5. The data anomaly monitoring method according to claim 1, characterized in that: Determining the data indicator confidence interval according to the historical data indicator information includes: Determine the data indicator expected value and the data indicator standard deviation according to the historical data indicator information; determine the second error threshold according to the product of the data indicator standard deviation and a preset second probability adjustment parameter, and determine the first data indicator confidence interval of each of the business tables according to the data indicator expected value and the second error threshold; and / or Linear prediction is performed based on the historical data indicator information to obtain indicator prediction values; the indicator error ratio is determined based on a preset second linear adjustment parameter, and the second data indicator confidence interval of each business table is determined based on the indicator prediction value and the indicator error ratio.

6. The data anomaly monitoring method according to claim 1, characterized in that: After determining the flow monitoring result of the flow table according to the data size information to be analyzed and the data size confidence interval, and determining the indicator monitoring result of the business table according to the data indicator information to be analyzed and the data indicator confidence interval, the method further includes: Determine an abnormal flow table from the plurality of flow tables according to the flow monitoring result, and determine an abnormal service table from the plurality of service tables according to the indicator monitoring result; The abnormal flow table and the abnormal service table are sent to the user terminal for performing visual display processing based on the abnormal flow table and the abnormal service table.

7. The data anomaly monitoring method according to claim 6, characterized in that: The metadata information also includes auxiliary metadata, and the determining of an abnormal flow table from the plurality of flow tables according to the flow monitoring result includes: Determine a candidate abnormal flow table from the plurality of flow tables according to the flow monitoring result; The value information of each candidate abnormal flow table is determined according to the auxiliary class metadata corresponding to each candidate abnormal flow table, and the abnormal flow table is determined from the candidate abnormal flow tables according to the value information, and the auxiliary class metadata includes one or more combinations of business source information, table size information and number of access times.

8. The data anomaly monitoring method according to claim 7, characterized in that: Determining an abnormal business table from the plurality of business tables according to the indicator monitoring result includes: Determine a candidate abnormal business table from the plurality of business tables according to the indicator monitoring result; Determine weight information of each candidate abnormal business table according to the value and / or number of rows corresponding to each candidate abnormal business table, and determine the abnormal business table from the candidate abnormal business tables according to the weight information; and / or According to the abnormal continuity information corresponding to each of the candidate abnormal service tables, and according to the abnormal continuity information, an abnormal service table is determined from the candidate abnormal service tables.

9. The data anomaly monitoring method according to claim 1, characterized in that: Before determining the data size confidence interval according to the historical data size information and determining the data index confidence interval according to the historical data index information, at least one of the following is further included: Determine the to-be-monitored tables that exist in the preset white list among the to-be-monitored tables, and filter out the to-be-monitored tables that exist in the preset white list from the multiple to-be-monitored tables; Determine the to-be-monitored tables in which data migration has occurred within a set time period, and screen out the to-be-monitored tables in which data migration has occurred from the multiple to-be-monitored tables; Determine the to-be-monitored tables whose historical data volume is less than the set data volume among the to-be-monitored tables, and screen out the to-be-monitored tables whose historical data volume is less than the set data volume from the multiple to-be-monitored tables.

10. The data anomaly monitoring method according to claim 1, characterized in that: After obtaining metadata information of multiple tables to be monitored, the method further includes: According to the metadata information, data anomaly reports in each of the tables to be monitored are determined, and the data anomaly reports are screened out from the multiple tables to be monitored, wherein the report anomaly types of the data anomaly reports include one or more combinations of empty tables, stopped tables, interrupted tables, fluctuating tables, and low-heat tables.

11. A data anomaly monitoring device, characterized in that: It includes a data acquisition module, a data processing module, a data analysis module and a result determination module, wherein: The data acquisition module is configured to acquire metadata information of multiple tables to be monitored, wherein the metadata information includes table monitoring metadata and indicator monitoring metadata, and the tables to be monitored include flow tables corresponding to upstream services and business tables corresponding to midstream and downstream services; The data processing module is configured to determine the size information of the data to be analyzed and the size information of the historical data of each of the flow tables according to the table monitoring metadata, and to determine the index information of the data to be analyzed and the index information of the historical data of each of the business tables according to the index monitoring metadata; The data analysis module is configured to determine a data size confidence interval based on the historical data size information, and to determine a data indicator confidence interval based on the historical data indicator information; The result determination module is configured to determine the flow monitoring result of the flow table according to the data size information to be analyzed and the data size confidence interval, and to determine the indicator monitoring result of the business table according to the data indicator information to be analyzed and the data indicator confidence interval.

12. A data anomaly monitoring device, characterized in that: include: memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the data anomaly monitoring method as described in any one of claims 1-10.

13. A non-volatile storage medium storing computer executable instructions, characterized in that: The computer executable instructions are used to execute the data anomaly monitoring method as described in any one of claims 1 to 10 when executed by a computer processor.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the data anomaly monitoring method according to any one of claims 1 to 10 is implemented.

Images