Data quality monitoring method and system, electronic equipment and storage medium

Abstract

The invention provides a data quality monitoring method and system, electronic equipment and a storage medium. The method comprises the following steps: acquiring a service data table; wherein the business data table comprises a plurality of fields to be monitored; according to a preset monitoring rule corresponding to the to-be-monitored field, respectively detecting data corresponding to the to-be-monitored field in each detection period to obtain a corresponding detection result; obtaining a quality monitoring result of the data in the quality monitoring period according to the detection result; wherein the quality monitoring period comprises one or more detection periods. According to the method and the device, the corresponding preset monitoring rule can be configured for the to-be-monitored field, the quality of the data corresponding to the to-be-monitored field is detected according to the preset monitoring rule, and the corresponding quality monitoring result is generated according to the detection result, so that automatic data quality detection and monitoring are realized.

Description

Technical Field

[0001] The present disclosure relates to the technical field of data quality monitoring, and particularly to a data quality monitoring method, system, electronic device, and storage medium. Background Art

[0002] Due to a large number of problems in the business side, such as irregular processes, few, missing, or incomplete data monitoring standards, there are a large number of incomplete, inaccurate, and unreasonable data in the business data.

[0003] Currently, in the management and monitoring of data quality, for the collected business data, detection rules are usually manually configured, and the business data is regularly detected according to the detection rules to obtain corresponding detection results, and corresponding alarm information is sent by analyzing the detection results, which requires a large amount of manpower, high cost, and low monitoring efficiency. Summary of the Invention

[0004] To solve the above technical problems, the present disclosure provides a data quality monitoring method, system, electronic device, and storage medium.

[0005] The present disclosure solves the above technical problems through the following technical solutions:

[0006] In a first aspect, the present disclosure provides a data quality monitoring method, the method including:

[0007] Obtain a business data table; wherein, the business data table includes a plurality of fields to be monitored;

[0008] According to the preset monitoring rules corresponding to the fields to be monitored, respectively detect the data corresponding to the fields to be monitored in each detection cycle to obtain corresponding detection results;

[0009] Obtain the quality monitoring result of the data in the quality monitoring cycle according to the detection results; wherein, the quality monitoring cycle includes one or more of the detection cycles.

[0010] In this solution, the corresponding preset monitoring rules can be configured for the fields to be monitored, the data corresponding to the fields to be monitored can be quality detected according to the preset monitoring rules, and the corresponding quality monitoring results can be generated according to the detection results, realizing automatic data quality detection and monitoring.

[0011] Optionally, after the step of obtaining the quality monitoring result of the data in the quality monitoring cycle according to the detection results, the method further includes:

[0012] For different types of the quality monitoring results in the quality monitoring cycle, respectively monitor them using matching preset alarm thresholds;

[0013] When it is detected that the quality monitoring result exceeds its corresponding preset warning threshold, a quality assessment result indicating that the quality of the data is unqualified is generated.

[0014] In this solution, according to the quality monitoring result, an automated threshold monitoring process is adopted for quality assessment, so that users can quickly and intuitively understand the quality assessment result of the business data table.

[0015] Optionally, the step of respectively using matching preset warning thresholds for monitoring different types of the quality monitoring results within the quality monitoring period includes:

[0016] Determine the corresponding first sample mean according to the quality monitoring result;

[0017] Respectively determine whether different types of the quality monitoring results conform to the normal distribution;

[0018] If it conforms to the normal distribution, determine the first upper limit value of the corresponding first confidence interval according to the quality monitoring result, and generate the quality assessment result when the first sample mean exceeds the corresponding first upper limit value;

[0019] If it does not conform to the normal distribution, determine the corresponding second confidence interval according to the quality monitoring result, and determine the quality assessment result based on the second confidence interval and the first sample mean; wherein, the statistical range of the first confidence interval is larger than the statistical range of the second confidence interval.

[0020] In this solution, by judging whether the quality monitoring result, such as the statistical result of the quality monitoring parameter within the detection period, conforms to the normal distribution throughout the quality monitoring period, different methods are adaptively adopted to generate corresponding quality assessment results, so as to improve the accuracy of the quality assessment result.

[0021] Optionally, the step of determining the quality assessment result based on the second confidence interval and the quality monitoring result includes:

[0022] If the first sample mean is not within the second confidence interval, determine the first abnormal data volume in the quality monitoring result, and generate the quality assessment result when the first abnormal data volume exceeds the corresponding first warning threshold;

[0023] If the first sample mean is within the second confidence interval, according to the quality monitoring result, determine the corresponding second sample mean, determine the second abnormal data volume in the quality monitoring result based on the first sample mean and the second sample mean, and generate the quality assessment result when the second abnormal data volume exceeds the corresponding second warning threshold; wherein, the data volume used for the second sample mean is less than the data volume used for the first sample mean.

[0024] In this solution, for the situation where the quality monitoring result does not conform to the normal distribution, according to the comparison result between the overall distribution of the quantity and the corresponding confidence interval, adaptively adopt different methods to determine the abnormal data volume in the quality monitoring result, and use different warning thresholds for different abnormal data volumes for monitoring to improve the accuracy of the further quality assessment result.

[0025] Optionally, the step of determining the first abnormal data volume in the quality monitoring result includes:

[0026] If the quality monitoring result is less than the lower limit value of the second confidence interval, the first abnormal data volume is the data volume in the quality monitoring result that is less than the first abnormal lower limit value;

[0027] If the quality monitoring result exceeds the upper limit value of the second confidence interval, the first abnormal data volume is the data volume in the quality monitoring result when it exceeds the first abnormal upper limit value.

[0028] Optionally, the step of determining the second abnormal data volume in the quality monitoring result based on the first sample mean and the second sample mean includes:

[0029] If the first sample mean exceeds the second sample mean, the second abnormal data volume is the data volume in the quality monitoring result that exceeds the second abnormal upper limit value;

[0030] If the first sample mean does not exceed the second sample mean, the second abnormal data volume is the data volume in the quality monitoring result that is less than the second abnormal lower limit value.

[0031] Optionally, the first warning threshold and the second warning threshold are determined according to the quality monitoring result within the quality monitoring period.

[0032] In this solution, the corresponding confidence interval and warning threshold can be dynamically adjusted according to the quality monitoring result, so as to flexibly conduct quality assessment and make the quality assessment result more accurate.

[0033] Optionally, the method further includes:

[0034] Generate a data quality monitoring table according to the quality monitoring results and the corresponding quality evaluation results.

[0035] In this solution, the data quality monitoring table can clearly and intuitively reflect the quality monitoring results and quality evaluation results of the fields to be monitored during the quality monitoring period.

[0036] Optionally, the quality monitoring results include the statistical results of at least one of the following quality monitoring parameters during the detection period:

[0037] The first data volume that does not conform to the preset monitoring rules, the second data volume that conforms to the preset monitoring rules, the proportion of the first data volume that does not conform to the preset monitoring rules, and the proportion of the second data volume that conforms to the preset monitoring rules.

[0038] In this solution, the above quality monitoring parameters for each detection period are statistically analyzed according to the detection results, so that users can intuitively understand the quality of business data in each detection period based on the values of the data volume and the proportion of the data volume.

[0039] Optionally, the quality monitoring results further include a trend evaluation index of the quality monitoring parameters during the quality monitoring period;

[0040] Among them, the trend evaluation index includes at least one of a central tendency index, a dispersion tendency index, a distribution tendency index, and a historical comparison tendency index.

[0041] In this solution, the above trend evaluation indexes are statistically analyzed according to the detection results during the quality monitoring period, so that users can intuitively understand the trend of the quality of business data during the quality monitoring period based on different types of trend evaluation indexes.

[0042] In a second aspect, the present disclosure provides a data quality monitoring system to implement the data quality monitoring method described in the first aspect.

[0043] The system includes:

[0044] A data acquisition module for acquiring a business data table; wherein, the business data table includes a plurality of fields to be monitored;

[0045] A data detection module for respectively detecting the data corresponding to the fields to be monitored in each detection period according to the preset monitoring rules corresponding to the fields to be monitored, so as to obtain corresponding detection results;

[0046] A quality monitoring module for obtaining the quality monitoring results of the data during the quality monitoring period according to the detection results; wherein, the quality monitoring period includes one or more of the detection periods.

[0047] In this solution, corresponding preset monitoring rules can be configured for the fields to be monitored, the data corresponding to the fields to be monitored can be subjected to quality detection according to the preset monitoring rules, and corresponding quality monitoring results can be generated according to the detection results, so as to realize automatic data quality detection and monitoring.

[0048] Optionally, the system further includes:

[0049] A quality evaluation module, configured to monitor different types of the quality monitoring results during the quality monitoring period respectively by using matching preset alarm thresholds; when it is detected that the quality monitoring results exceed their corresponding preset alarm thresholds, generate a quality evaluation result indicating that the quality of the data is unqualified.

[0050] In this solution, according to the quality monitoring results, an automatic threshold monitoring process is adopted for quality evaluation, so that users can quickly and intuitively understand the quality evaluation results of the business data table.

[0051] Optionally, the quality evaluation module is used for:

[0052] Determine a corresponding first sample mean according to the quality monitoring results;

[0053] Respectively determine whether different types of the quality monitoring results conform to the normal distribution;

[0054] If it conforms to the normal distribution, determine a first upper limit value of a corresponding first confidence interval according to the quality monitoring results, and generate the quality evaluation result when the first sample mean exceeds the corresponding first upper limit value;

[0055] If it does not conform to the normal distribution, determine a corresponding second confidence interval according to the quality monitoring results, and determine the quality evaluation result based on the second confidence interval and the first sample mean; wherein, the statistical range of the first confidence interval is larger than the statistical range of the second confidence interval.

[0056] In this solution, by judging whether the quality monitoring results, such as the statistical results of quality monitoring parameters during the detection period, conform to the normal distribution throughout the quality monitoring period, different methods are adaptively adopted to generate corresponding quality evaluation results to improve the accuracy of the quality evaluation results.

[0057] Optionally, the quality evaluation module is specifically used for:

[0058] If the first sample mean is not within the second confidence interval, determine a first abnormal data volume in the quality monitoring results, and generate the quality evaluation result when the first abnormal data volume exceeds the corresponding first alarm threshold;

[0059] If the first sample mean is within the second confidence interval, based on the quality monitoring result, determine the corresponding second sample mean, determine the second abnormal data volume in the quality monitoring result according to the first sample mean and the second sample mean, and generate the quality evaluation result when the second abnormal data volume exceeds the corresponding second warning threshold; wherein, the data volume used for the second sample mean is less than the data volume used for the first sample mean.

[0060] In this solution, for the case where the quality monitoring result does not conform to the normal distribution, according to the comparison result between the overall distribution of the quantity and the corresponding confidence interval, adaptively adopt different methods to determine the abnormal data volume in the quality monitoring result, and use different warning thresholds for different abnormal data volumes for monitoring to improve the accuracy of the further quality evaluation result.

[0061] Optionally, the quality evaluation module is specifically used for:

[0062] If the quality monitoring result is less than the lower limit value of the second confidence interval, the first abnormal data volume is the data volume in the quality monitoring result that is less than the first abnormal lower limit value;

[0063] If the quality monitoring result exceeds the upper limit value of the second confidence interval, the first abnormal data volume is the data volume in the quality monitoring result when it exceeds the first abnormal upper limit value.

[0064] Optionally, the quality evaluation module is specifically used for:

[0065] If the first sample mean exceeds the second sample mean, the second abnormal data volume is the data volume in the quality monitoring result that exceeds the second abnormal upper limit value;

[0066] If the first sample mean does not exceed the second sample mean, the second abnormal data volume is the data volume in the quality monitoring result that is less than the second abnormal lower limit value.

[0067] Optionally, the first warning threshold and the second warning threshold are determined according to the quality monitoring result during the quality monitoring period.

[0068] In this solution, the corresponding confidence interval and warning threshold can be dynamically adjusted according to the quality monitoring result, so as to flexibly conduct quality evaluation and make the quality evaluation result more accurate.

[0069] Optionally, the quality evaluation module is further used for:

[0070] Generate a data quality monitoring table according to the quality monitoring result and the corresponding quality evaluation result.

[0071] In this solution, the data quality monitoring table can clearly and intuitively reflect the quality monitoring results and quality assessment results of the fields to be monitored during the quality monitoring period.

[0072] Optionally, the quality monitoring results include the statistical results of at least one of the following quality monitoring parameters during the detection period:

[0073] The first data volume that does not conform to the preset monitoring rules, the second data volume that conforms to the preset monitoring rules, the proportion of the first data volume that does not conform to the preset monitoring rules, and the proportion of the second data volume that conforms to the preset monitoring rules.

[0074] In this solution, the above quality monitoring parameters for each detection period are statistically analyzed according to the detection results, so that users can intuitively understand the quality of business data in each detection period based on the values of the data volume and the proportion of the data volume.

[0075] Optionally, the quality monitoring results further include a trend evaluation index of the quality monitoring parameters during the quality monitoring period;

[0076] Wherein, the trend evaluation index includes at least one of a central tendency index, a dispersion trend index, a distribution trend index, and a historical comparison trend index.

[0077] In this solution, the above trend evaluation indexes are statistically analyzed according to the detection results during the quality monitoring period, so that users can intuitively understand the trend of the quality of business data during the quality monitoring period based on different types of trend evaluation indexes.

[0078] In a third aspect, the present disclosure provides a data management platform for a battery swapping station, and the data management platform for the battery swapping station includes the data quality monitoring system described in the second aspect.

[0079] In a fourth aspect, the present disclosure provides an electronic device, including a memory, a processor, and a computer program stored on the memory and used to run on the processor. When the processor executes the computer program, the data quality monitoring method described in the first aspect is implemented.

[0080] In a fifth aspect, the present disclosure provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the data quality monitoring method described in the first aspect is implemented.

[0081] The positive and progressive effects of the present disclosure are as follows: By configuring corresponding preset monitoring rules for the fields to be monitored, the data corresponding to the fields to be monitored can be quality-tested according to the preset monitoring rules, and corresponding quality monitoring results can be generated according to the test results, thereby realizing automated data quality detection and monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 Schematic flowchart of a data quality monitoring method provided by an embodiment of the present disclosure;

[0083] Figure 2 Schematic diagram of a type of quality rule provided by an embodiment of the present disclosure;

[0084] Figure 3 Schematic diagram of a data quality monitoring form provided by an embodiment of the present disclosure;

[0085] Figure 4 Schematic actual flowchart of data quality monitoring provided by an embodiment of the present disclosure;

[0086] Figure 5 Schematic diagram of modules of a data quality monitoring system provided by an embodiment of the present disclosure;

[0087] Figure 6 Schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Detailed implementation manners

[0088] The present disclosure will be further described below by way of embodiments, but the present disclosure is not limited to the scope of the described embodiments.

[0089] It should be noted that if there are descriptions involving "first", "second", etc. in the embodiments of the present disclosure, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the quantity of the indicated technical features. Thus, features defined with "first", "second" may explicitly or implicitly include at least one of such features.

[0090] In addition, the technical solutions between various embodiments can be combined with each other, but it must be based on the fact that those skilled in the art can implement them. When the combination of technical solutions results in contradictions or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present disclosure.

[0091] The data quality monitoring method provided by the present disclosure can perform quality detection on the data corresponding to the field to be monitored by configuring corresponding preset monitoring rules for the field to be monitored, and generate corresponding quality monitoring results according to the detection results, so as to achieve automated data quality detection and monitoring.

[0092] Figure 1 Shows a data quality monitoring method provided by an embodiment of the present disclosure. Refer to Figure 1 , the method includes steps S101 - S103:

[0093] S101. Obtain the business data table; wherein, the business data table includes several fields to be monitored.

[0094] The business data table is usually a structured way to store and organize business data in a database. In a database, data is presented in the form of a table. Each business data table contains specific types of data, each column in the table corresponds to a different data field, and each row records the data corresponding to each data field.

[0095] According to actual needs, several data fields from the business data table can be selected as the fields to be monitored.

[0096] S102. According to the preset monitoring rules corresponding to the fields to be monitored, respectively detect the data corresponding to the fields to be monitored in each detection cycle to obtain corresponding detection results.

[0097] The main types of data quality problems include data incompleteness, data non-standardization, data redundancy, data inconsistency, etc.

[0098] Exemplarily, around 6 data quality characteristics and combined with 4 scenario types, the data quality assessment dimension can be split into 15 types of quality rule types as shown in Figure 2 shown.

[0099] According to the above quality rule types, corresponding preset monitoring rules can be formulated for each field to be monitored according to business requirements.

[0100] S103. Obtain the quality monitoring result of the data in the quality monitoring cycle according to the detection result; wherein, the quality monitoring cycle includes one or more detection cycles.

[0101] In one embodiment, the quality monitoring result includes at least one of the following statistical results of quality monitoring parameters in the detection cycle:

[0102] The amount of data that does not conform to the preset monitoring rules, the amount of data that conforms to the preset monitoring rules, the proportion of the amount of data that does not conform to the preset monitoring rules, the proportion of the amount of data that conforms to the preset monitoring rules.

[0103] In this embodiment, the above quality monitoring parameters of each detection cycle are respectively statistically analyzed, so that users can intuitively understand the quality of business data in each detection cycle according to the values of the data volume and the proportion of the data volume.

[0104] In another embodiment, the quality monitoring result further includes a trend evaluation index of the statistical result in the quality monitoring cycle; wherein, the trend evaluation index includes at least one of a central tendency index, a dispersion tendency index, a distribution tendency index, and a historical comparison tendency index.

[0105] Exemplarily, the central tendency indicators include at least one of the arithmetic mean, weighted mean, trimmed mean, mode, and median of the statistical results within the quality monitoring period.

[0106] The dispersion tendency indicators include at least one of the range, quartile / percentile, variance, standard deviation, and coefficient of variation of the statistical results within the quality monitoring period.

[0107] The distribution tendency indicators include at least one of the kurtosis and skewness of the statistical results within the quality monitoring period.

[0108] The historical comparison tendency indicators include at least one of the year-on-year and month-on-month of the statistical results within the quality monitoring period.

[0109] In this embodiment, the above-mentioned trend evaluation indicators are statistically analyzed according to the detection results within the quality monitoring period, so that users can intuitively understand the trend of the quality of business data within the quality monitoring period through different types of trend evaluation indicators.

[0110] For the above-mentioned quality monitoring results, a quality evaluation result of the data can also be automatically generated according to the set preset warning threshold.

[0111] In one embodiment, after step S103, the above method further includes:

[0112] S104. For different types of quality monitoring results within the quality monitoring period, use the matching preset warning thresholds for monitoring respectively;

[0113] S105. When it is detected that the quality monitoring result exceeds its corresponding preset warning threshold, generate a quality evaluation result indicating that the quality of the data is unqualified.

[0114] In this embodiment, a quality evaluation is performed on the quality monitoring results using an automated threshold monitoring process, so that users can quickly and intuitively understand the quality evaluation results of the business data table.

[0115] In one embodiment, step S1041 specifically includes:

[0116] Determine the corresponding first sample mean according to the quality monitoring results;

[0117] Judge whether different types of quality monitoring results conform to the normal distribution respectively;

[0118] If it conforms to the normal distribution, determine the first upper limit value of the corresponding first confidence interval according to the quality monitoring results, and generate a quality evaluation result when the first sample mean exceeds the corresponding first upper limit value;

[0119] If it does not conform to the normal distribution, determine the corresponding second confidence interval according to the quality monitoring result, and determine the quality evaluation result based on the second confidence interval and the first sample mean; wherein, the statistical range of the first confidence interval is larger than that of the second confidence interval.

[0120] In this embodiment, by judging whether the quality monitoring result, such as the statistical result of the quality monitoring parameter within the detection period, conforms to the normal distribution throughout the quality monitoring cycle, adaptively generate corresponding quality evaluation results according to different confidence intervals to improve the accuracy of the quality evaluation result.

[0121] Specifically, the steps of determining the quality evaluation result based on the second confidence interval and the first sample mean specifically include:

[0122] If the first sample mean is not within the second confidence interval, determine the first abnormal data volume in the quality monitoring result, and generate a quality evaluation result when the first abnormal data volume exceeds the corresponding first warning threshold;

[0123] If the first sample mean is within the second confidence interval, determine the corresponding second sample mean according to the quality monitoring result, determine the second abnormal data volume in the quality monitoring result according to the first sample mean and the second sample mean, and generate a quality evaluation result when the second abnormal data volume exceeds the corresponding second warning threshold; wherein, the data volume used for the second sample mean is less than that used for the first sample mean.

[0124] In this embodiment, for the case where the quality monitoring result does not conform to the normal distribution, adaptively adopt different methods to determine the abnormal data volume in the quality monitoring result according to the comparison result between the overall distribution of the quantity and the corresponding confidence interval, and use different warning thresholds for different abnormal data volumes for monitoring to improve the accuracy of the further quality evaluation result.

[0125] In one embodiment, the steps of determining the first abnormal data volume of the quality monitoring result include:

[0126] If the quality monitoring result is less than the lower limit value of the second confidence interval, the first abnormal data volume is the data volume in the quality monitoring result that is less than the first abnormal lower limit value;

[0127] If the quality monitoring result exceeds the upper limit value of the second confidence interval, the first abnormal data volume is the data volume when the quality monitoring result exceeds the first abnormal upper limit value.

[0128] In another embodiment, the steps of determining the second abnormal data volume in the quality monitoring result according to the first sample mean and the second sample mean include:

[0129] If the first sample mean exceeds the second sample mean, the second amount of abnormal data is the amount of data in the quality monitoring result that exceeds the second upper limit value of abnormality;

[0130] If the first sample mean does not exceed the second sample mean, the second amount of abnormal data is the amount of data in the quality monitoring result that is less than the second lower limit value of abnormality.

[0131] Among them, the first warning threshold and the second warning threshold are determined according to the quality monitoring results within the quality monitoring period. In this way, the corresponding confidence interval and warning threshold can be dynamically adjusted according to the quality monitoring results, so as to flexibly conduct quality assessment and make the quality assessment results more accurate.

[0132] In one embodiment, step S105 further includes: when it is monitored that the quality monitoring result does not exceed its corresponding preset warning threshold, generating a quality assessment result indicating that the quality of the data is qualified.

[0133] In another embodiment, after step S105, the above method further includes:

[0134] Step S106, generating a data quality monitoring table according to the quality monitoring result and the corresponding quality assessment result.

[0135] In this embodiment, through the data quality monitoring table, the quality monitoring result and the quality assessment result of the field to be monitored within the quality monitoring period can be clearly and intuitively reflected.

[0136] Exemplarily, referring to Figure 3 the data quality monitoring table shown, for the field to be monitored A, the field to be monitored B, and the field to be monitored C, the quality monitoring results include the statistical results of the quality monitoring parameters "number of records meeting the rules", "number of records not meeting the rules", and "proportion of records not meeting the rules" within the quality monitoring period respectively.

[0137] Among them, for the "number of records not meeting the rules", the matching preset warning threshold is 1200. For the "proportion of records not meeting the rules", the matching preset warning threshold is 0.03.

[0138] Since both the "number of records not meeting the rules" and the "proportion of records not meeting the rules" corresponding to the field to be monitored B exceed the corresponding preset warning thresholds, an alarm flag is generated in the "whether to alarm" column to indicate that the quality of the data of the field to be monitored B is unqualified.

[0139] The following is a further illustration of the data quality monitoring process through a specific example as Figure 4 shown:

[0140] For the business data table that records the status of the charging pile positions, first determine the fields to be monitored and their corresponding preset monitoring rules, set the detection period to be daily, and the quality monitoring period to be the last 30 days.

[0141] Within the last 30 days, according to the preset monitoring rules, the data corresponding to the fields to be monitored is detected every day. Based on the detection results, the amount of data that does not meet the rules each day is counted, that is, the amount of data that does not meet the rules each day within only 30 days is obtained as the quality monitoring result.

[0142] Then, automated threshold monitoring is performed on the amount of data that does not meet the rules each day within the last 30 days. Specifically, first judge whether there is an unacceptable threshold. If so, when the amount of data that does not meet the rules on any day exceeds the unacceptable threshold, a quality assessment result indicating that the quality of the data corresponding to the field to be monitored is unqualified within the quality monitoring period needs to be generated, that is, an alarm is issued; if not, calculate the average value of the amount of data that does not meet the rules each day within the last 30 days as the first sample mean, and perform a K-S normality test on the amount of data that does not meet the rules each day within the last 30 days to judge whether the amount of data that does not meet the rules each day within the last 30 days conforms to a normal distribution.

[0143] If it conforms to a normal distribution, perform a T-test on the amount of data that does not meet the rules each day within the last 30 days to obtain the upper limit value of the 95% confidence interval. If the first sample mean exceeds the upper limit value of the 95% confidence interval, an alarm is issued.

[0144] If it does not conform to a normal distribution, determine the 25% - 75% confidence interval according to the amount of data that does not meet the rules each day within the last 30 days, and judge whether the first sample mean is within the 25% - 75% confidence interval.

[0145] If the first sample mean is not within the 25% - 75% confidence interval:

[0146] When the first sample mean is within the 25% confidence interval, it means that there are relatively many small data amounts in the amount of data that does not meet the rules each day within the last 30 days, or it means that there is a particularly small data amount in the amount of data that does not meet the rules each day within the last 30 days, that is, the amount of data that does not meet the rules is small and the data quality performance is good. The number of anomalies where the amount of data that does not meet the rules each day within the last 30 days is less than the first anomaly lower limit value can be counted. If the number of anomalies is greater than the 75% quantile value of the amount of data that does not meet the rules each day within the last 30 days, an alarm is issued.

[0147] When the first sample mean is above the 75% confidence interval, it indicates that there is a relatively large amount of data with relatively large values in the amount of data that does not meet the rules every day in the past 30 days, or it indicates that there is a particularly large amount of data in the amount of data that does not meet the rules every day in the past 30 days. That is, the amount of data that does not meet the rules is relatively large, and the data quality performance is poor. The number of anomalies where the amount of data that does not meet the rules every day in the past 30 days is greater than the first abnormal upper limit value can be counted, and the number of data that does not meet the rules every day in the past 30 days and is greater than the first sample mean can be counted as the corresponding alarm threshold. If the number of anomalies is greater than the corresponding alarm threshold, an alarm is issued.

[0148] If the first sample mean is within the 25% - 75% confidence interval, first calculate the mean of the 25% - 75% quantiles of the amount of data that does not meet the rules every day in the past 30 days as the second sample mean, and compare the first sample mean with the second sample mean:

[0149] If the first sample mean exceeds the second sample mean, it indicates that there is a relatively large amount of data with relatively large values in the amount of data that does not meet the rules every day in the past 30 days, and the overall data quality performance is poor. The number of anomalies where the amount of data that does not meet the rules every day in the past 30 days is greater than the second abnormal upper limit value can be counted, and the alarm threshold can be adaptively adjusted to a larger value. That is, the corresponding alarm threshold can be set to the 95% quantile value of the amount of data that does not meet the rules every day in the past 30 days. If the number of anomalies exceeds the 95% quantile value, an alarm is issued.

[0150] If the first sample mean does not exceed the second sample mean, it indicates that there is a relatively large amount of data with relatively small values in the amount of data that does not meet the rules every day in the past 30 days, and the overall data quality performance is good. The number of anomalies where the amount of data that does not meet the rules every day in the past 30 days is greater than the second abnormal lower limit value can be counted, and the alarm threshold can be adaptively adjusted to a smaller value. That is, the corresponding alarm threshold can be set to the 90% quantile value of the amount of data that does not meet the rules every day in the past 30 days. If the number of anomalies exceeds the 90% quantile value, an alarm is issued.

[0151] The embodiments of the present disclosure further provide a data quality monitoring system for implementing the above data quality monitoring method.

[0152] Figure 5 Illustrates a data quality monitoring system provided by an embodiment of the present disclosure. Refer to Figure 5 and this system includes:

[0153] A data acquisition module 201 for acquiring a service data table; wherein, the service data table includes a number of fields to be monitored;

[0154] A data detection module 202, configured to detect the data corresponding to the to-be-monitored field in each detection period according to a preset monitoring rule corresponding to the to-be-monitored field, so as to obtain corresponding detection results;

[0155] A quality monitoring module 203, configured to obtain a quality monitoring result of the data in a quality monitoring period according to the detection result; wherein, the quality monitoring period includes one or more of the detection periods.

[0156] Among them, a business data table is usually a structured way to store and organize business data in a database. In a database, data is presented in the form of a table. Each business data table contains specific types of data. Each column in the table corresponds to a different data field, and each row records the data corresponding to each data field.

[0157] According to actual needs, several data fields in the business data table can be selected as the to-be-monitored fields.

[0158] Generally, the main types of data quality problems include data incompleteness, data non-standardization, data redundancy, data inconsistency, etc.

[0159] Exemplarily, around 6 data quality characteristics and combined with 4 scenario types, the data quality evaluation dimension can be split into Figure 2 15 types of quality rule types as shown.

[0160] According to the above quality rule types, corresponding preset monitoring rules can be formulated for each to-be-monitored field according to business requirements.

[0161] In one embodiment, the quality monitoring result includes at least one of the following statistical results of quality monitoring parameters in a detection period:

[0162] The amount of data that does not conform to the preset monitoring rule, the amount of data that conforms to the preset monitoring rule, the proportion of the amount of data that does not conform to the preset monitoring rule, the proportion of the amount of data that conforms to the preset monitoring rule.

[0163] In this embodiment, the above quality monitoring parameters of each detection period are respectively statistically counted, so that the user can intuitively understand the quality of business data in each detection period according to the values of the amount of data and the proportion of the amount of data.

[0164] In another embodiment, the quality monitoring result further includes a trend evaluation index of the statistical result in the quality monitoring period; wherein, the trend evaluation index includes at least one of a central tendency index, a dispersion tendency index, a distribution tendency index, and a historical comparison tendency index.

[0165] Exemplarily, the central tendency indicators include at least one of the arithmetic mean, weighted mean, trimmed mean, mode, and median of the statistical results within the quality monitoring period.

[0166] The dispersion trend indicators include at least one of the range, quartile / percentile, variance, standard deviation, and coefficient of variation of the statistical results within the quality monitoring period.

[0167] The distribution trend indicators include at least one of the kurtosis and skewness of the statistical results within the quality monitoring period.

[0168] The historical comparison trend indicators include at least one of the year-on-year and month-on-month comparisons of the statistical results within the quality monitoring period.

[0169] In this embodiment, the above-mentioned trend evaluation indicators are statistically analyzed based on the detection results within the quality monitoring period, so that users can intuitively understand the trend of the quality of business data during the quality monitoring period through different types of trend evaluation indicators.

[0170] For the above-mentioned quality monitoring results, a quality evaluation result of the data can also be automatically generated according to the set preset warning threshold.

[0171] In one embodiment, the above system further includes:

[0172] A quality evaluation module, which is used to monitor different types of quality monitoring results within the quality monitoring period respectively by using matching preset warning thresholds; when it is detected that the quality monitoring result exceeds its corresponding preset warning threshold, a quality evaluation result indicating that the quality of the data is unqualified is generated.

[0173] In this embodiment, quality evaluation is performed according to the quality monitoring results using an automated threshold monitoring process, so that users can quickly and intuitively understand the quality evaluation results of the business data table.

[0174] In one embodiment, the quality evaluation module is used for:

[0175] Determine the corresponding first sample mean according to the quality monitoring results;

[0176] Respectively determine whether different types of quality monitoring results conform to the normal distribution;

[0177] If it conforms to the normal distribution, determine the first upper limit value of the corresponding first confidence interval according to the quality monitoring results, and generate a quality evaluation result when the first sample mean exceeds the corresponding first upper limit value;

[0178] If it does not conform to the normal distribution, determine the corresponding second confidence interval according to the quality monitoring result, and determine the quality assessment result based on the second confidence interval and the first sample mean; wherein, the statistical range of the first confidence interval is greater than the statistical range of the second confidence interval.

[0179] In this embodiment, by judging the quality monitoring result, such as the statistical result of the quality monitoring parameter within the detection period, whether it conforms to the normal distribution throughout the quality monitoring period, adaptively generate the corresponding quality assessment result according to different confidence intervals to improve the accuracy of the quality assessment result.

[0180] Specifically, the quality assessment module is specifically used for:

[0181] If the first sample mean is not within the second confidence interval, determine the first abnormal data volume in the quality monitoring result, and generate a quality assessment result when the first abnormal data volume exceeds the corresponding first warning threshold;

[0182] If the first sample mean is within the second confidence interval, determine the corresponding second sample mean according to the quality monitoring result, determine the second abnormal data volume in the quality monitoring result based on the first sample mean and the second sample mean, and generate a quality assessment result when the second abnormal data volume exceeds the corresponding second warning threshold; wherein, the data volume used for the second sample mean is less than the data volume used for the first sample mean.

[0183] In this embodiment, for the situation where the quality monitoring result does not conform to the normal distribution, according to the comparison result between the overall distribution of the quantity and the corresponding confidence interval, adaptively adopt different methods to determine the abnormal data volume in the quality monitoring result, and adopt different warning thresholds for different abnormal data volumes for monitoring to improve the accuracy of the further quality assessment result.

[0184] In one embodiment, the quality assessment module is specifically used for:

[0185] If the quality monitoring result is less than the lower limit value of the second confidence interval, the first abnormal data volume is the data volume in the quality monitoring result that is less than the first abnormal lower limit value;

[0186] If the quality monitoring result exceeds the upper limit value of the second confidence interval, the first abnormal data volume is the data volume in the quality monitoring result when it exceeds the first abnormal upper limit value.

[0187] In another embodiment, the quality assessment module is specifically used for:

[0188] If the first sample mean exceeds the second sample mean, the second abnormal data volume is the data volume in the quality monitoring result that exceeds the second abnormal upper limit value;

[0189] If the first sample mean does not exceed the second sample mean, the second abnormal data volume is the data volume in the quality monitoring result that is less than the second abnormal lower limit value.

[0190] Among them, the first warning threshold and the second warning threshold are determined according to the quality monitoring results within the quality monitoring period. In this way, the corresponding confidence interval and warning threshold can be dynamically adjusted according to the quality monitoring results, so as to flexibly perform quality evaluation and make the quality evaluation results more accurate.

[0191] In one embodiment, the quality evaluation module is further configured to: when it is monitored that the quality monitoring result does not exceed its corresponding preset warning threshold, generate a quality evaluation result indicating that the quality of the data is qualified.

[0192] In another embodiment, the quality evaluation module is further configured to:

[0193] Generate a data quality monitoring table according to the quality monitoring results and the corresponding quality evaluation results.

[0194] In this embodiment, through the data quality monitoring table, the quality monitoring results and quality evaluation results of the fields to be monitored within the quality monitoring period can be clearly and intuitively reflected.

[0195] Exemplarily, refer to Figure 3 the data quality monitoring table shown. For the fields to be monitored A, B, and C, the quality monitoring results include the statistical results of the quality monitoring parameters "number of records meeting the rules", "number of records not meeting the rules", and "proportion of records not meeting the rules" within the quality monitoring period respectively.

[0196] Among them, for the "number of records not meeting the rules", the matching preset warning threshold is 1200. For the "proportion of records not meeting the rules", the matching preset warning threshold is 0.03.

[0197] Since both the "number of records not meeting the rules" and the "proportion of records not meeting the rules" corresponding to the field to be monitored B exceed the corresponding preset warning thresholds, an alarm flag is generated in the "whether to alarm" column to indicate that the quality of the data of the field to be monitored B is unqualified.

[0198] The embodiments of the present disclosure also provide a data management platform for a battery swapping station. The data management platform for the battery swapping station includes the above data quality monitoring system. Based on the above quality monitoring system, the data management platform for the battery swapping station can configure corresponding preset monitoring rules for the fields to be monitored, perform quality detection on the data corresponding to the fields to be monitored according to the preset monitoring rules, and generate corresponding quality monitoring results according to the detection results, realizing automatic data quality detection and monitoring, and thus making data management more convenient and efficient.

[0199] Figure 6 The structure of one kind of electronic device of the present disclosure is shown. The electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the above-mentioned data quality monitoring method is implemented. Figure 6 The displayed electronic device 30 is merely an example and should not impose any limitation on the functions and usage scope of the embodiments of the present disclosure.

[0200] As Figure 6 shown, the electronic device 30 may also be presented in the form of a general-purpose computing device. For example, it may be a server device. The components of the electronic device 30 may include but are not limited to: the above-mentioned at least one processor 31, the above-mentioned at least one memory 32, and a bus 33 connecting different system components (including the memory 32 and the processor 31).

[0201] The bus 33 includes a data bus, an address bus, and a control bus.

[0202] The memory 32 may include volatile memory, such as a random access memory (RAM) 321 and / or a cache memory 322, and may further include a read-only memory (ROM) 323.

[0203] The memory 32 may further include a program / utility 325 having a set (at least one) of program modules 324. Such program modules 324 include but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment.

[0204] The processor 31 executes various functional applications and data processing by running the computer program stored in the memory 32, such as the above-mentioned data quality monitoring method of the present disclosure.

[0205] The electronic device 30 may also communicate with one or more external devices 34 (such as a keyboard, a pointing device, etc.). Such communication may be carried out through an input / output (I / O) interface 35. And, the device 30 for model generation may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 36. As Figure 6 shown, the network adapter 36 communicates with other modules of the device 30 for model generation through the bus 33. It should be understood that although not shown in the figure, other hardware and / or software modules may be used in combination with the device 30 for model generation, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems, etc.

[0206] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0207] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the above data quality monitoring method is implemented.

[0208] Among them, the more specific computer-readable storage medium that can be adopted may include, but is not limited to: portable disks, hard disks, random access memories, read-only memories, erasable programmable read-only memories, optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0209] In a possible implementation manner, the present disclosure can also be implemented in the form of a program product, which includes program code. When the program product runs on a terminal device, the program code is used to cause the terminal device to execute and implement the above data quality monitoring method.

[0210] Among them, the program code for executing the present disclosure can be written in any combination of one or more programming languages. The program code can be executed entirely on the user device, partially on the user device, executed as an independent software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0211] Although the specific implementation manners of the present disclosure are described above, those skilled in the art should understand that this is only an example. The protection scope of the present disclosure is defined by the appended claims. Without departing from the principles and essence of the present disclosure, those skilled in the art can make various changes or modifications to these implementation manners, but these changes and modifications all fall within the protection scope of the present disclosure.

Claims

1. A data quality monitoring method, characterized in that, The method includes: Obtaining a business data table; wherein, the business data table includes a number of fields to be monitored; Detecting the data corresponding to each of the fields to be monitored in each detection period according to the preset monitoring rules corresponding to the fields to be monitored, so as to obtain corresponding detection results; Obtaining a quality monitoring result of the data in a quality monitoring period according to the detection results; wherein, the quality monitoring period includes one or more of the detection periods.

2. The data quality monitoring method according to claim 1, wherein After the step of obtaining the quality monitoring result of the data in the quality monitoring period according to the detection results, the method further includes: For different types of the quality monitoring results in the quality monitoring period, respectively monitoring them with matching preset alarm thresholds; When it is detected that the quality monitoring result exceeds its corresponding preset alarm threshold, generating a quality evaluation result indicating that the quality of the data is unqualified.

3. The data quality monitoring method according to claim 2, characterized in that The step of respectively monitoring different types of the quality monitoring results in the quality monitoring period with matching preset alarm thresholds includes: Determining a corresponding first sample mean according to the quality monitoring result; Respectively determining whether different types of the quality monitoring results conform to a normal distribution; If it conforms to a normal distribution, determining a first upper limit value of a corresponding first confidence interval according to the quality monitoring result, and generating the quality evaluation result when the first sample mean exceeds the corresponding first upper limit value; If it does not conform to a normal distribution, determining a corresponding second confidence interval according to the quality monitoring result, and determining the quality evaluation result based on the second confidence interval and the first sample mean; wherein, the statistical range of the first confidence interval is larger than the statistical range of the second confidence interval.

4. The data quality monitoring method according to claim 3, wherein The step of determining the quality evaluation result based on the second confidence interval and the quality monitoring result includes: If the first sample mean is not within the second confidence interval, determining a first abnormal data volume in the quality monitoring result, and generating the quality evaluation result when the first abnormal data volume exceeds a corresponding first alarm threshold; If the first sample mean is within the second confidence interval, determining a corresponding second sample mean according to the quality monitoring result, determining a second abnormal data volume in the quality monitoring result according to the first sample mean and the second sample mean, and generating the quality evaluation result when the second abnormal data volume exceeds a corresponding second alarm threshold; wherein, the data volume used for the second sample mean is smaller than the data volume used for the first sample mean.

5. The data quality monitoring method according to claim 4, wherein The step of determining the first abnormal data volume in the quality monitoring result includes: If the quality monitoring result is less than the lower limit value of the second confidence interval, the first abnormal data volume is the data volume in the quality monitoring result that is less than a first abnormal lower limit value; If the quality monitoring result exceeds the upper limit value of the second confidence interval, the first abnormal data volume is the data volume in the quality monitoring result when it exceeds a first abnormal upper limit value; and / or The step of determining the second amount of abnormal data in the quality monitoring result according to the first sample mean and the second sample mean includes: If the first sample mean exceeds the second sample mean, the second amount of abnormal data is the amount of data in the quality monitoring result that exceeds the second abnormal upper limit value; If the first sample mean does not exceed the second sample mean, the second amount of abnormal data is the amount of data in the quality monitoring result that is less than the second abnormal lower limit value; And / or, The first alarm threshold and the second alarm threshold are determined according to the quality monitoring result within the quality monitoring period.

6. The data quality monitoring method according to claim 2, wherein The method further includes: Generating a data quality monitoring table according to the quality monitoring result and the corresponding quality evaluation result.

7. The data quality monitoring method according to any one of claims 1-6, characterized in that, The quality monitoring result includes at least one of the following statistical results of the quality monitoring parameters within the detection period: The first amount of data that does not conform to the preset monitoring rule, the second amount of data that conforms to the preset monitoring rule, the proportion of the first amount of data that does not conform to the preset monitoring rule, the proportion of the second amount of data that conforms to the preset monitoring rule; The quality monitoring result further includes a trend evaluation index of the quality monitoring parameter within the quality monitoring period; Wherein, the trend evaluation index includes at least one of a central tendency index, a dispersion tendency index, a distribution tendency index, and a historical comparison tendency index.

8. A data quality monitoring system, characterized in that, The system includes: A data acquisition module for acquiring a service data table; wherein, the service data table includes a plurality of fields to be monitored; A data detection module for respectively detecting the data corresponding to each monitored field in each detection period according to the preset monitoring rule corresponding to the monitored field to obtain corresponding detection results; A quality monitoring module for obtaining the quality monitoring result of the data within the quality monitoring period according to the detection result; wherein, the quality monitoring period includes one or more of the detection periods.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and adapted to run on the processor, characterized in that, When the processor executes the computer program, it implements the data quality monitoring method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data quality monitoring method according to any one of claims 1-7.

Images