Long-term control method, system and equipment for data quality and storage medium

Abstract

The invention provides a long-term control method, system and device for data quality and a storage medium. The long-term control method for the data quality comprises the steps of receiving to-be-evaluated data; monitoring the to-be-assessed data in real time, when the to-be-assessed data meets an early warning condition, sending an early warning signal to the outside, and optimizing and updating the to-be-assessed data until the to-be-assessed data does not meet the early warning condition; obtaining an evaluation target according to a preset period, performing quality evaluation on the to-be-evaluated data based on a pre-constructed data quality evaluation system corresponding to the evaluation target, and outputting a quality evaluation result; if the quality evaluation is not passed, optimizing and updating the to-be-evaluated data according to a quality evaluation result, and / or adjusting a data quality evaluation system until the quality evaluation is passed; outputting the optimized and updated data to be evaluated; the method has the beneficial effects that the newly generated abnormal data problem can be found in time, and the quality is kept stable after long-time operation; the method is suitable for the data quality control field.

Description

Technical Field

[0001] The present application relates to the technical field of data quality control, and in particular to a method, system, device, and storage medium for long-term control of data quality. Background Art

[0002] In the information age, data quality plays a vital role in corporate decision-making, business operations and market competition. Currently, there are many data quality control methods on the market, but most of them focus on short-term data verification and cleaning. Although they can improve data quality to a certain extent, they have the following disadvantages:

[0003] 1. Failure to achieve continuous data quality control, resulting in a decrease in data quality after long-term operation or processing;

[0004] 2. The control of data quality is mostly limited to post-processing, and abnormal data cannot be discovered in time;

[0005] 3. When data anomalies occur, the focus is mostly on processing the data itself, but there is a lack of systematic improvement methods, and there is no guarantee that the same abnormal data will be generated in the future. Summary of the invention

[0006] In order to solve one of the above technical defects, the present application provides a long-term control method, system and device, and storage medium for data quality.

[0007] According to a first aspect of the present application, a long-term control method for data quality is provided, comprising:

[0008] receiving data to be evaluated;

[0009] Monitor the data to be evaluated in real time. When the data to be evaluated meets the warning conditions, send a warning signal to the outside world and optimize and update the data to be evaluated until the data to be evaluated does not meet the warning conditions.

[0010] Acquire the assessment target according to the preset cycle, conduct quality assessment on the data to be assessed based on the pre-built data quality assessment system corresponding to the assessment target, and output the quality assessment results;

[0011] If the quality assessment fails, the data to be assessed will be optimized and updated according to the quality assessment results, and / or the data quality assessment system will be adjusted until the quality assessment passes;

[0012] Output the optimized and updated data to be evaluated.

[0013] Preferably, if the quality assessment fails, the data to be assessed is optimized and updated according to the quality assessment result, and / or the data quality assessment system is adjusted until the quality assessment passes, specifically including:

[0014] Conduct quality inspection on the data to be evaluated that have failed the quality assessment, and optimize and update the data to be evaluated that have failed the quality inspection until the quality assessment or quality inspection passes;

[0015] When the quality inspection of the data to be evaluated or the optimized and updated data to be evaluated passes, the data quality evaluation system is adjusted, and the quality of the current data to be evaluated is re-evaluated through the adjusted data quality evaluation system until the quality evaluation passes.

[0016] More preferably, the construction of a data quality assessment system corresponding to the assessment target specifically includes:

[0017] According to the preset update frequency, data information related to the evaluation target is obtained from multiple data sources, and a multi-source data set is generated;

[0018] Identify key data with high relevance to the assessment objectives from multi-source data sets;

[0019] Match the corresponding quality inspection methods for key data so that the quality inspection methods can be used to inspect the quality of the data to be evaluated;

[0020] Based on the characteristics of key data, a data quality assessment system is constructed, including:

[0021] Based on the characteristics of key data, determine the evaluation dimensions corresponding to the key data, including: completeness, accuracy, consistency, availability, timeliness and security;

[0022] Set a corresponding evaluation indicator for each evaluation dimension, and set a corresponding indicator calculation method, indicator threshold and indicator weight for the evaluation indicator, so that the quality of the data to be evaluated can be evaluated through the indicator calculation method;

[0023] A data quality assessment system is constructed based on key data, corresponding quality inspection methods, corresponding assessment dimensions, assessment indicators corresponding to the assessment dimensions, indicator calculation methods corresponding to the evaluation indicators, indicator thresholds and indicator weights.

[0024] Preferably, identifying key data with a high correlation with the evaluation target from the multi-source data set specifically includes:

[0025] Calculate the correlation between data information in multi-source data sets and identify data information with high correlation with the evaluation target;

[0026] Obtain external feedback on modifications to highly relevant data information;

[0027] Analyze the impact of different highly relevant data on the evaluation target, sort the data based on the impact, and select a set number of data as key data according to the sorting;

[0028] Perform reliability verification and sensitivity verification on key data, and optimize and adjust the key data based on the reliability verification results and sensitivity verification results.

[0029] Preferably, the method for adjusting the data quality assessment system specifically includes:

[0030] Conduct validity tests on current assessment dimensions;

[0031] Delete the evaluation dimensions that have not passed the validity test, add new evaluation dimensions and corresponding new evaluation indicators based on the current data to be evaluated and the current evaluation objectives, set the corresponding indicator calculation method, indicator threshold and indicator weight for the new evaluation indicators, and update them to the data quality evaluation system;

[0032] Conduct validity tests on the evaluation indicators corresponding to the evaluation dimensions that have passed the validity test; delete the evaluation indicators that have not passed the validity test; adjust the corresponding indicator calculation methods, indicator thresholds and indicator weights of the evaluation indicators that have passed the validity test, and update them to the data quality assessment system;

[0033] The corresponding indicator calculation methods of the evaluation indicators that pass the validity test are subject to validity testing, and the indicator calculation methods that fail the validity test are adjusted and updated to the data quality assessment system.

[0034] Preferably, the optimizing and updating of the data to be evaluated that fails the quality inspection specifically includes:

[0035] Identify the problematic data sources corresponding to the data to be evaluated that have failed the quality inspection, replace the problematic data sources by replacing the data sources, and update the multi-source data set;

[0036] Identify erroneous data items in the data to be evaluated that have failed the quality inspection, correct the erroneous data items through the correction strategy, and update the data to be evaluated.

[0037] Preferably, the data to be evaluated is monitored in real time, specifically including:

[0038] Obtain early warning targets;

[0039] Based on the warning target, the monitoring objects are extracted from the multi-source data set;

[0040] Set a corresponding warning threshold for each monitoring object, where the warning threshold includes a static warning threshold, a dynamic warning threshold and a trend warning threshold;

[0041] The data information corresponding to the monitored object in the real-time collected data to be evaluated is processed through the stream processing framework and then compared with the warning threshold.

[0042] According to a second aspect of the present application, a long-term data quality control system is provided, comprising a module for implementing the long-term data quality control method as described in any one of the above items.

[0043] According to a third aspect of the present application, a device is provided, comprising:

[0044] Memory;

[0045] Processor; and

[0046] Computer programs;

[0047] The computer program is stored in the memory and is configured to be executed by the processor to implement the long-term control method for data quality as described in any one of the above items.

[0048] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored; the computer program is executed by a processor to implement the long-term control method for data quality as described in any one of the above items.

[0049] The present application provides a long-term control method for data quality, which can identify abnormal data through data quality assessment, and then process the identified abnormal data. According to the use requirements, the preset period of data quality assessment can be once a week or a month. Through real-time monitoring, newly generated abnormal data can be discovered in time, which is convenient for timely processing of abnormal data, ensuring the stability of data quality, so that the quality of data remains stable after long-term operation, and avoiding the problem of data information quality degradation after long-term operation or processing in the prior art. In this application, the assessment target is obtained according to the preset period, and the quality of the data to be assessed is assessed based on the assessment target. When the quality assessment fails, the quality problem of the data to be assessed itself and the problem of inapplicability of the data quality assessment system are taken into account, so that the quality assessment is more systematic and complete, avoiding the occurrence of repeated problems, and making the quality assessment results more in line with the assessment requirements of the assessment target, more stable and accurate. It provides strong data support for the decision-making, business operations and market competition of enterprises, thereby enhancing the market competitiveness and business benefits of enterprises.

[0050] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the contents indicated in the written description and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0052] Figure 1 A schematic diagram of a flow chart of a long-term data quality control method provided in an embodiment of the present application;

[0053] Figure 2 for Figure 1 A flowchart of the steps to be performed when the quality assessment fails in the embodiment;

[0054] Figure 3 for Figure 1 A schematic diagram of a process for constructing a data quality assessment system corresponding to an assessment target in the embodiment;

[0055] Figure 4 for Figure 3 A schematic diagram of a process for identifying key data with a high correlation with an evaluation target from a multi-source data set in the embodiment;

[0056] Figure 5 for Figure 1 A schematic diagram of a process for real-time monitoring of the data to be evaluated in the embodiment;

[0057] Figure 6 A functional structure diagram of a long-term data quality control system provided in an embodiment of the present application;

[0058] Figure 7 for Figure 6 A schematic diagram of the functional structure of the optimization and adjustment unit in the embodiment;

[0059] Figure 8 A schematic diagram of the functional structure of the evaluation system building module provided in the embodiment of the present application;

[0060] Fig. 9 for Figure 8 A schematic diagram of the functional structure of the key data identification unit in the embodiment;

[0061] Fig.10 for Figure 7 A functional structure diagram of the evaluation system adjustment unit in the embodiment;

[0062] Fig.11 for Figure 6 A functional structure diagram of the real-time monitoring module in the embodiment;

[0063] In the figure:

[0064] 10 is a data receiving module, 20 is a real-time monitoring module, 30 is a warning signal output module, 40 is an optimization and update module, 50 is a quality assessment module, 60 is a result output unit, 70 is an optimization and adjustment unit, 80 is a data output module, 90 is an evaluation system construction module, 201 is a warning target acquisition unit, 202 is a monitoring object extraction unit, 203 is a warning condition setting unit, 204 is a warning comparison unit, 701 is a quality inspection unit, 702 is a data update unit, 703 is an evaluation system adjustment unit, 704 is a quality assessment unit, 901 is a multi-source data set generation unit, 902 is a key data identification unit, 903 is a quality inspection unit, The method matching unit is 904, which is the evaluation system building unit. The evaluation dimension validity verification unit is 7031, which is the evaluation dimension adjustment unit. The evaluation indicator validity verification unit is 7034, which is the evaluation indicator adjustment unit. The evaluation indicator calculation method validity verification unit is 7035, which is the indicator calculation method adjustment unit. The evaluation system generation unit is 9041, which is the evaluation dimension determination unit. The evaluation indicator setting unit is 9042, which is the evaluation indicator setting unit. The evaluation system generation unit is 9043. DETAILED DESCRIPTION

[0065] In order to make the technical solutions and advantages in the embodiments of the present application more clearly understood, the exemplary embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than an exhaustive list of all the embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0066] Figure 1 A flow chart of a long-term data quality control method provided in an embodiment of the present application is shown as follows: Figure 1 As shown, in order to solve the above problems, a method for long-term control of data quality is provided in an embodiment of the present application, including:

[0067] receiving data to be evaluated;

[0068] Monitor the data to be evaluated in real time. When the data to be evaluated meets the warning conditions, send a warning signal to the outside world and optimize and update the data to be evaluated until the data to be evaluated does not meet the warning conditions.

[0069] Obtain assessment targets according to preset cycles;

[0070] Preprocess the data to be evaluated, including: abnormal data cleaning, data standardization and normalization conversion, and data integration;

[0071] Based on the pre-built data quality assessment system corresponding to the assessment objectives, the quality of the data to be assessed is assessed, and the quality assessment results are output;

[0072] If the quality assessment fails, the data to be assessed is optimized and updated according to the quality assessment results, and / or the data quality assessment system is adjusted until the quality assessment passes, and then the quality assessment success information is output;

[0073] Output the optimized and updated data to be evaluated.

[0074] The present application provides a long-term control method for data quality, which can identify abnormal data through data quality assessment, and then process the identified abnormal data. According to the use requirements, the preset period of data quality assessment can be once a week or a month. Through real-time monitoring, newly generated abnormal data can be discovered in time, which is convenient for timely processing of abnormal data, ensuring the stability of data quality, so that the quality of data remains stable after long-term operation, and avoiding the problem of data information quality degradation after long-term operation or processing in the prior art. In this application, the assessment target is obtained according to the preset period, and the quality of the data to be assessed is assessed based on the assessment target. When the quality assessment fails, the quality problem of the data to be assessed itself and the problem of inapplicability of the data quality assessment system are taken into account, so that the quality assessment is more systematic and complete, avoiding the occurrence of repeated problems, and making the quality assessment results more in line with the assessment requirements of the assessment target, more stable and accurate. It provides strong data support for the decision-making, business operations and market competition of enterprises, thereby enhancing the market competitiveness and business benefits of enterprises.

[0075] Figure 2 for Figure 1 A flow chart of the steps to be performed when the quality assessment fails in the embodiment is as follows: Figure 2 As shown, further, if the quality assessment fails, the data to be assessed is optimized and updated according to the quality assessment results, and / or the data quality assessment system is adjusted until the quality assessment passes, specifically including:

[0076] Conduct quality inspection on the data to be evaluated that have failed the quality assessment, and optimize and update the data to be evaluated that have failed the quality inspection until the quality assessment or quality inspection passes;

[0077] When the quality inspection of the data to be evaluated or the optimized and updated data to be evaluated passes, the data quality evaluation system is adjusted, and the quality of the current data to be evaluated is re-evaluated through the adjusted data quality evaluation system until the quality evaluation passes.

[0078] In this application, when the quality assessment fails, the data is first quality-checked to check the quality of the data itself; when the quality assessment fails but the quality inspection passes, it means that there is no problem with the quality of the data itself, then it is necessary to consider whether the currently used data quality assessment system cannot meet the usage requirements, and to adjust the data quality assessment system adaptively. Consideration and adjustment are made from two dimensions: the quality of the data itself and the data quality assessment system, which further ensures the reliability of the data quality assessment and enables better long-term control of the data.

[0079] Figure 3 for Figure 1 A flow chart of the construction of a data quality assessment system corresponding to the assessment target in the embodiment is as follows: Figure 3 As shown, further, the construction of the data quality assessment system corresponding to the assessment objectives specifically includes:

[0080] According to the preset update frequency, data information related to the evaluation target is obtained from multiple data sources, and a multi-source data set is generated; the data sources include databases, customer relationship management systems (CRM systems), enterprise resource planning systems (ERP systems), market research reports, etc.; key data with a high degree of correlation with the evaluation target is identified from the multi-source data set;

[0081] Match the corresponding quality inspection methods for key data so that the quality inspection methods can be used to inspect the quality of the data to be evaluated;

[0082] Based on the characteristics of key data, a data quality assessment system is constructed, including:

[0083] Based on the characteristics of key data, determine the evaluation dimensions corresponding to the key data, including: completeness, accuracy, consistency, availability, timeliness and security;

[0084] Set a corresponding evaluation indicator for each evaluation dimension, and set a corresponding indicator calculation method, indicator threshold and indicator weight for the evaluation indicator, so that the quality of the data to be evaluated can be evaluated through the indicator calculation method;

[0085] A data quality assessment system is constructed based on key data, corresponding quality inspection methods, corresponding assessment dimensions, assessment indicators corresponding to the assessment dimensions, indicator calculation methods corresponding to the evaluation indicators, indicator thresholds and indicator weights.

[0086] In this application, based on the evaluation objectives, key data are identified, and then the data quality evaluation system is specifically constructed according to the characteristics of the key data, so that the constructed data quality evaluation system can meet the use requirements of the evaluation objectives, with better matching and better data quality evaluation effect.

[0087] Specifically, after the multi-source data set is generated, the data information in the multi-source data set is security preprocessed, including:

[0088] The source, type and purpose of the data information obtained;

[0089] Divide data information into structured information and unstructured information;

[0090] Clean the data information to remove duplicate data, invalid data and abnormal data;

[0091] Safely back up data information to external hard drives, cloud storage servers, etc.

[0092] Verify the accuracy and consistency of data information based on business rules and data standards;

[0093] Set corresponding access rights for data information in multi-source data sets and match corresponding access objects; verify access rights at intervals. Specifically, set corresponding access rights for data information in multi-source data sets and match corresponding access objects, including:

[0094] Based on the sensitivity and importance of the data information, the data information in the multi-source data set is classified; for example, it can be classified into general data and sensitive data. Sensitive data may include core business data, sensitive customer data, etc.

[0095] Encrypt sensitive data;

[0096] Based on the principle of minimizing permissions, different access permissions and corresponding access objects are configured for different categories of data information.

[0097] More specifically, when an access object accesses data information in a multi-source data set, identity authentication and authorization verification are performed on the access object;

[0098] During the process of the access object accessing data information in the multi-source data set, the access object is continuously verified.

[0099] Adopting zero-trust security rules, all access objects are authenticated and authorized during access and during the access process, ensuring that the identity and permissions of the access objects are always valid, regardless of whether the visitor is from inside or outside the enterprise, ensuring data security and stability.

[0100] In this application, the corresponding permissions can be set for the data information in the multi-source data set, and the principle of minimizing permissions is to grant users the minimum permissions required to complete their work, thereby reducing the risk of data leakage. The setting of permissions can, to a certain extent, prevent the generation of bad data from the source of the data and prevent bad data from spreading downstream.

[0101] Specifically, quality inspection and quality assessment include:

[0102] Match the data to be evaluated with the key data in the data quality evaluation system, or select representative sample data from the data to be evaluated and match them with the key data in the data quality evaluation system; the method of selecting representative sample data can be simple random sampling, systematic sampling, cluster sampling, stratified sampling, etc.;

[0103] After the matching is successful, the quality inspection method corresponding to the key data is used to inspect the quality of the data to be evaluated, and the quality of the data to be evaluated is evaluated through the evaluation dimensions corresponding to the key data, the evaluation indicators corresponding to the evaluation dimensions, the indicator calculation methods corresponding to the evaluation indicators, the indicator thresholds and the indicator weights.

[0104] In order to achieve efficient processing in this application, representative sample data can be selected from the data to be evaluated, which effectively reduces the time and cost used for quality evaluation and can ensure the accuracy of quality evaluation when the amount of data information is large.

[0105] Specifically, the characteristics of key data include:

[0106] Data scale, including the number of records and fields of data information corresponding to key data and the amount of growth;

[0107] Data diversity, including different data types corresponding to different key data, standardization and consistency of different formats; different formats, such as date format, currency format, etc.;

[0108] Data velocity, including the frequency of updating of data information corresponding to key data;

[0109] Data authenticity includes accuracy and reliability. Accuracy includes the data precision of multiple data sources, and reliability includes the system performance and data quality of data information.

[0110] Correspondingly, the evaluation dimensions include: completeness, accuracy, consistency, availability, timeliness and security, among which:

[0111] Integrity can evaluate whether the data information corresponding to the key data is complete and without missing, that is, whether all necessary records and fields are included;

[0112] Accuracy can assess whether the data information corresponding to the key data accurately reflects the actual situation and whether there is any erroneous or misleading information;

[0113] Consistency can evaluate whether the data information corresponding to key data is consistent at different time points, such as whether the data information has the same name and meaning, accuracy, and processing logic.

[0114] Availability can evaluate whether the data information corresponding to key data can be accessed, obtained and processed by appropriate users or systems;

[0115] Timeliness can evaluate the update frequency and real-time performance of the data information corresponding to key data, and whether it meets the user's demand for fresh data.

[0116] Security can assess whether the data information corresponding to critical data is protected to prevent unauthorized access and modification.

[0117] Figure 4 for Figure 3 A schematic diagram of a process for identifying key data with a high correlation with an evaluation target from a multi-source data set in the embodiment is shown in FIG. Figure 4 As shown, further, identifying key data with a high correlation with the evaluation target from the multi-source data set specifically includes:

[0118] By calculating the correlation between data information in multi-source data sets, data information with high correlation with the evaluation target can be identified; the correlation between data information in multi-source data sets can be calculated by using statistical analysis to obtain the distribution, trend and correlation of data information, or by using statistical models or machine learning algorithms to predict the correlation of data information, or by using data mining methods such as association rule learning, cluster analysis, and classification prediction to mine the correlation of data information, or by using visualization tools such as charts and dashboards to visualize the data information to intuitively obtain the correlation of data information;

[0119] Obtain external feedback on modifications to highly relevant data information;

[0120] Analyze the impact of different highly relevant data on the evaluation target, sort the data based on the impact, and select a set number of data as key data according to the sorting;

[0121] Perform reliability verification and sensitivity verification on key data, and optimize and adjust the key data based on the reliability verification results and sensitivity verification results.

[0122] In this application, highly correlated data information can be obtained based on the correlation between data information and external feedback, and then key data can be selected from it. The accuracy of the selected key data can be ensured through reliability verification and sensitivity verification, which is convenient for subsequent use in quality assessment.

[0123] Specifically, reliability verification of key data includes: using different multi-source data sets or data information of different time periods in multi-source data sets to verify the reliability of key data; sensitivity verification of key data includes: analyzing the sensitivity of key data to the assessment target, and comparing the sensitivity verification results with the sensitivity threshold.

[0124] Furthermore, the method for adjusting the data quality assessment system specifically includes:

[0125] Conduct validity tests on current assessment dimensions;

[0126] Delete the evaluation dimensions that have not passed the validity test, add new evaluation dimensions and corresponding new evaluation indicators based on the current data to be evaluated and the current evaluation objectives, set the corresponding indicator calculation method, indicator threshold and indicator weight for the new evaluation indicators, and update them to the data quality evaluation system;

[0127] Conduct validity tests on the evaluation indicators corresponding to the evaluation dimensions that have passed the validity test; delete the evaluation indicators that have not passed the validity test; adjust the corresponding indicator calculation methods, indicator thresholds and indicator weights of the evaluation indicators that have passed the validity test, and update them to the data quality assessment system;

[0128] The corresponding indicator calculation methods of the evaluation indicators that pass the validity test are subject to validity testing, and the indicator calculation methods that fail the validity test are adjusted and updated to the data quality assessment system.

[0129] In this application, the data quality assessment system was comprehensively tested and adjusted through validity verification, and the data quality assessment system was updated to meet the requirements for the use of current assessment indicators and ensure the reliability and accuracy of the quality assessment results.

[0130] Furthermore, the optimizing and updating of the data to be evaluated that have failed the quality inspection specifically includes:

[0131] Identify the problematic data sources corresponding to the data to be evaluated that have failed the quality inspection, replace the problematic data sources by replacing the data sources, and update the multi-source data set;

[0132] Identify erroneous data items in the data to be evaluated that have failed the quality inspection, correct the erroneous data items through the correction strategy, and update the data to be evaluated.

[0133] Specifically, optimizing and updating the data to be evaluated that fails the quality inspection may also include: updating the preprocessing method of the data to be evaluated.

[0134] In this application, the problematic data source is identified and replaced, the erroneous data item is identified and corrected, and all systems that use the problematic data source or erroneous data item are updated to ensure the stability of data quality. The preprocessing method of the data to be evaluated is updated to avoid damage to the data to be evaluated during the preprocessing process.

[0135] Figure 5 for Figure 1 A schematic diagram of a process for real-time monitoring of the data to be evaluated in the embodiment is as follows: Figure 5 As shown, further, real-time monitoring of the data to be evaluated is performed, specifically including:

[0136] Obtain early warning targets;

[0137] Based on the warning target, the monitoring objects are extracted from the multi-source data set;

[0138] Set a corresponding warning threshold for each monitoring object, where the warning threshold includes a static warning threshold, a dynamic warning threshold and a trend warning threshold;

[0139] The data information corresponding to the monitoring object in the real-time collected data to be evaluated is processed through a stream processing framework, and then compared with the warning threshold. The stream processing framework can be Kafka (Apache Kafka) or Flink (Apache Flink).

[0140] Specifically, the static warning threshold is a fixed threshold that can be directly compared; the dynamic warning threshold is a threshold that is dynamically adjusted according to the fluctuation of historical data; the trend warning threshold is the changing trend of historical data, and a warning signal is issued when the trend is abnormal.

[0141] In this application, the early warning target can be to prevent potential risks, facilitate timely response to abnormal situations, optimize business processes, etc. The monitoring objects extracted from the multi-source data set can include sales data, etc. When the early warning conditions are met, the early warning signal can be sent to the outside world through email, SMS, system notification, etc. The early warning signal includes data problems, possible causes, recommended solutions, etc.

[0142] The present application also provides a long-term data quality control system, including a module for implementing the long-term data quality control method as described in any one of the above items.

[0143] Figure 6 A functional structure diagram of a long-term control system for data quality provided in an embodiment of the present application, such as Figure 6 Specifically, a long-term control system for data quality includes:

[0144] A data receiving module 10, used for receiving data to be evaluated;

[0145] A real-time monitoring module 20, used for real-time monitoring of the data to be evaluated;

[0146] The warning signal output module 30 is used to send a warning signal to the outside world when the data to be evaluated meets the warning conditions;

[0147] The optimization and updating module 40 is used to optimize and update the data to be evaluated when the data to be evaluated meets the warning condition, until the data to be evaluated does not meet the warning condition;

[0148] The quality assessment module 50 is used to obtain the assessment target according to a preset period, and perform quality assessment on the data to be assessed based on a pre-built data quality assessment system corresponding to the assessment target;

[0149] A result output unit 60, used to output the quality assessment result;

[0150] The optimization and adjustment unit 70 is used to optimize and update the data to be evaluated according to the quality evaluation result, and / or adjust the data quality evaluation system until the quality evaluation passes if the quality evaluation fails;

[0151] The data output module 80 is used to output the optimized and updated data to be evaluated.

[0152] Figure 7 for Figure 6 The functional structure diagram of the optimization and adjustment unit in the embodiment is as follows: Figure 7 As shown, specifically, the optimization adjustment unit 70 includes:

[0153] The quality inspection unit 701 is used to perform quality inspection on the data to be evaluated that fails the quality evaluation;

[0154] The data updating unit 702 is used to optimize and update the data to be evaluated that has not passed the quality inspection until the quality evaluation or quality inspection passes;

[0155] An evaluation system adjustment unit 703, used to adjust the data quality evaluation system when the quality inspection of the data to be evaluated or the optimized and updated data to be evaluated passes;

[0156] The quality evaluation unit 704 is used to re-evaluate the quality of the current data to be evaluated by using the adjusted data quality evaluation system until the quality evaluation passes.

[0157] Figure 8 The functional structure diagram of the evaluation system building module provided in the embodiment of the present application is as follows: Figure 8 As shown, specifically, a long-term control system for data quality also includes:

[0158] An evaluation system construction module 90, used for constructing a data quality evaluation system corresponding to the evaluation target;

[0159] The multi-source data set generation unit 901 is used to obtain data information related to the evaluation target from multiple data sources according to a preset update frequency, and generate a multi-source data set

[0160] A key data identification unit 902 is used to identify key data with a high correlation with the evaluation target from the multi-source data set;

[0161] A quality inspection method matching unit 903 is used to match a corresponding quality inspection method for key data, so that the quality inspection method can be used to perform quality inspection on the data to be evaluated;

[0162] An evaluation system construction unit 904, used to construct a data quality evaluation system based on the characteristics of key data;

[0163] An evaluation dimension determination unit 9041 is used to determine the evaluation dimensions corresponding to the key data based on the characteristics of the key data, and the evaluation dimensions include: completeness, accuracy, consistency, availability, timeliness and security;

[0164] An evaluation index setting unit 9042 is used to set a corresponding evaluation index for each evaluation dimension, and to set a corresponding index calculation method, index threshold and index weight for the evaluation index, so that the quality of the data to be evaluated can be evaluated through the index calculation method;

[0165] The evaluation system generation unit 9043 is used to construct a data quality evaluation system based on key data, corresponding quality inspection methods, corresponding evaluation dimensions, evaluation indicators corresponding to the evaluation dimensions, indicator calculation methods corresponding to the evaluation indicators, indicator thresholds and indicator weights.

[0166] Fig. 9 for Figure 8 The functional structure diagram of the key data identification unit in the embodiment is as follows: Fig. 9 As shown, specifically, the key data identification unit 902 includes:

[0167] A calculation and identification unit 9021 is used to calculate the correlation between data information in the multi-source data set and identify data information with high correlation with the evaluation target;

[0168] The feedback acquisition unit 9022 is used to acquire modification feedback of the highly relevant data information from the outside world;

[0169] The key data selection unit 9023 is used to analyze the influence of different highly correlated data information on the evaluation target, sort the data information based on the influence degree, and select a set number of data information as key data according to the sorting;

[0170] A key data verification unit 9024 is used to verify the reliability and sensitivity of key data;

[0171] The key data adjustment unit 9025 is used to optimize and adjust the key data according to the reliability verification results and the sensitivity verification results.

[0172] Fig.10 for Figure 7 The functional structure diagram of the evaluation system adjustment unit in the embodiment is as follows: Fig.10 As shown, specifically, the evaluation system adjustment unit 703 includes:

[0173] The evaluation dimension validity verification unit 7031 is used to verify the validity of the current evaluation dimension;

[0174] The evaluation dimension adjustment unit 7032 is used to delete the evaluation dimensions that have not passed the validity test; to add new evaluation dimensions based on the current data to be evaluated and the current evaluation target, and to update them to the data quality evaluation system;

[0175] An evaluation indicator validity test unit 7033 is used to test the validity of the evaluation indicators corresponding to the evaluation dimensions that have passed the validity test;

[0176] The evaluation indicator adjustment unit 7034 is used to add corresponding new evaluation indicators according to the new evaluation dimensions, and set corresponding indicator calculation methods, indicator thresholds and indicator weights for the new evaluation indicators; to delete the evaluation indicators that have not passed the validity test; to adjust the corresponding indicator calculation methods, indicator thresholds and indicator weights of the evaluation indicators that have passed the validity test, and update them to the data quality evaluation system;

[0177] The indicator calculation method validity test unit 7035 is used to test the validity of the indicator calculation method corresponding to the evaluation indicator that has passed the validity test;

[0178] The indicator calculation method adjustment unit 7036 is used to adjust the indicator calculation method that has not passed the validity test and update it to the data quality assessment system.

[0179] Fig.11 for Figure 6 The functional structure diagram of the real-time monitoring module in the embodiment is as follows: Fig.11 As shown, specifically, the real-time monitoring module 20 includes:

[0180] The warning target acquisition unit 201 is used to acquire the warning target;

[0181] A monitoring object extraction unit 202 is used to extract monitoring objects from multi-source data sets based on early warning targets;

[0182] The warning condition setting unit 203 is used to set a corresponding warning threshold for each monitored object, wherein the warning threshold includes a static warning threshold, a dynamic warning threshold and a trend warning threshold;

[0183] The early warning comparison unit 204 processes the data information corresponding to the monitoring object in the real-time collected data to be evaluated through a stream processing framework, and then compares it with the early warning threshold.

[0184] The present application also provides a device, comprising:

[0185] Memory;

[0186] Processor; and

[0187] Computer programs;

[0188] The computer program is stored in the memory and is configured to be executed by the processor to implement the long-term control method for data quality as described in any one of the above items.

[0189] The present application also provides a computer-readable storage medium having a computer program stored thereon; the computer program is executed by a processor to implement the long-term control method for data quality as described in any one of the above items.

[0190] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, C language, VHDL language, Verilog language, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0191] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0192] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0193] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0194] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0195] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A long-term control method for data quality, characterized in that: include: receiving data to be evaluated; Monitor the data to be evaluated in real time. When the data to be evaluated meets the warning conditions, send a warning signal to the outside world and optimize and update the data to be evaluated until the data to be evaluated does not meet the warning conditions. Acquire the assessment target according to the preset cycle, conduct quality assessment on the data to be assessed based on the pre-built data quality assessment system corresponding to the assessment target, and output the quality assessment results; If the quality assessment fails, the data to be assessed will be optimized and updated according to the quality assessment results, and / or the data quality assessment system will be adjusted until the quality assessment passes; Output the optimized and updated data to be evaluated.

2. The long-term control method for data quality according to claim 1, characterized in that: If the quality assessment fails, the data to be assessed will be optimized and updated according to the quality assessment results, and / or the data quality assessment system will be adjusted until the quality assessment passes, including: Conduct quality inspection on the data to be evaluated that have failed the quality assessment, and optimize and update the data to be evaluated that have failed the quality inspection until the quality assessment or quality inspection passes; When the quality inspection of the data to be evaluated or the optimized and updated data to be evaluated passes, the data quality evaluation system is adjusted, and the quality of the current data to be evaluated is re-evaluated through the adjusted data quality evaluation system until the quality evaluation passes.

3. The long-term control method for data quality according to claim 2, characterized in that: The construction of a data quality assessment system corresponding to the assessment objectives includes: According to the preset update frequency, data information related to the evaluation target is obtained from multiple data sources, and a multi-source data set is generated; Identify key data with high relevance to the assessment objectives from multi-source data sets; Match the corresponding quality inspection methods for key data so that the quality inspection methods can be used to inspect the quality of the data to be evaluated; Based on the characteristics of key data, a data quality assessment system is constructed, including: Based on the characteristics of key data, determine the evaluation dimensions corresponding to the key data, including: completeness, accuracy, consistency, availability, timeliness and security; Set a corresponding evaluation indicator for each evaluation dimension, and set a corresponding indicator calculation method, indicator threshold and indicator weight for the evaluation indicator, so that the quality of the data to be evaluated can be evaluated through the indicator calculation method; A data quality assessment system is constructed based on key data, corresponding quality inspection methods, corresponding assessment dimensions, assessment indicators corresponding to the assessment dimensions, indicator calculation methods corresponding to the evaluation indicators, indicator thresholds and indicator weights.

4. The long-term control method for data quality according to claim 3, characterized in that: The step of identifying key data with a high degree of relevance to the evaluation target from the multi-source data set specifically includes: Calculate the correlation between data information in multi-source data sets and identify data information with high correlation with the evaluation target; Obtain external feedback on modifications to highly relevant data information; Analyze the impact of different highly relevant data on the evaluation target, sort the data based on the impact, and select a set number of data as key data according to the sorting; Perform reliability verification and sensitivity verification on key data, and optimize and adjust the key data based on the reliability verification results and sensitivity verification results.

5. The long-term control method for data quality according to claim 3, characterized in that: The method for adjusting the data quality assessment system specifically includes: Conduct validity tests on current assessment dimensions; Delete the evaluation dimensions that have not passed the validity test, add new evaluation dimensions and corresponding new evaluation indicators based on the current data to be evaluated and the current evaluation objectives, set the corresponding indicator calculation method, indicator threshold and indicator weight for the new evaluation indicators, and update them to the data quality evaluation system; Conduct validity tests on the evaluation indicators corresponding to the evaluation dimensions that have passed the validity test; delete the evaluation indicators that have not passed the validity test; adjust the corresponding indicator calculation methods, indicator thresholds and indicator weights of the evaluation indicators that have passed the validity test, and update them to the data quality assessment system; The corresponding indicator calculation methods of the evaluation indicators that pass the validity test are subject to validity testing, and the indicator calculation methods that fail the validity test are adjusted and updated to the data quality assessment system.

6. The long-term control method for data quality according to claim 3, characterized in that: The optimization and updating of the data to be evaluated that have failed the quality inspection specifically includes: Identify the problematic data sources corresponding to the data to be evaluated that have failed the quality inspection, replace the problematic data sources by replacing the data sources, and update the multi-source data set; Identify erroneous data items in the data to be evaluated that have failed the quality inspection, correct the erroneous data items through the correction strategy, and update the data to be evaluated.

7. The long-term control method for data quality according to claim 3, characterized in that: Real-time monitoring of the data to be evaluated, including: Obtain early warning targets; Based on the warning target, the monitoring objects are extracted from the multi-source data set; Set a corresponding warning threshold for each monitoring object, where the warning threshold includes a static warning threshold, a dynamic warning threshold and a trend warning threshold; The data information corresponding to the monitored object in the real-time collected data to be evaluated is processed through the stream processing framework and then compared with the warning threshold.

8. A long-term control system for data quality, characterized in that: The method comprises a module for implementing the long-term control method of data quality as claimed in any one of claims 1 to 7.

9. A device, characterized in that: include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the long-term control method for data quality according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon; the computer program is executed by a processor to implement the long-term control method for data quality as claimed in any one of claims 1 to 7.

Images