Data quality monitoring and repairing system

Abstract

The invention provides a data quality monitoring and repairing system, and belongs to the technical field of data repairing, and the system comprises an identification module which obtains data of a plurality of data sources, and automatically identifies a plurality of data quality problems existing in the data; the first determination module is used for determining a root cause and an influence path of a data quality problem based on a causal relationship reasoning technology; the evaluation module is used for evaluating the severity and the influence range of each data quality problem; the selection module is used for distributing priority for each data quality problem on the basis of a predefined rule according to the severity and the influence range, determining a processing sequence and selecting an optimal repair strategy for automatic repair; and the automatic repairing module is used for continuously monitoring the repaired data in real time, detecting a new data quality problem by utilizing a stream processing technology and a real-time analysis algorithm, and automatically repairing the data. The problems that data is inaccurate due to low identification efficiency and automatic and real-time processing of the data quality problem cannot be guaranteed are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data repair, and in particular to a data quality monitoring and repair system. Background Art

[0002] As business data continues to grow and change, enterprises are facing increasingly complex data management challenges.

[0003] Traditional data quality monitoring and repair systems require manual identification of various problems in the data, and low identification efficiency leads to inaccurate data. At the same time, once data quality problems are identified, they cannot be classified, which reduces the assessment of the severity and scope of the problem. As a result, it is difficult to judge the priority and processing order of the problems in the process of handling data problems, and it is impossible to ensure the automated, intelligent and real-time processing of data quality problems.

[0004] Therefore, the present invention proposes a data quality monitoring and repairing system. Summary of the invention

[0005] The present invention provides a data quality monitoring and repair system to solve the various problems in the prior art that require manual data identification, inaccurate data caused by low identification efficiency, and the inability to classify problems, which reduces the assessment of the severity and impact of the problems. As a result, it is difficult to judge the priority and processing order of the problems in the process of processing data problems, and it is impossible to ensure the automated, intelligent and real-time processing of data quality problems.

[0006] In one aspect, the present invention provides a data quality monitoring and repair system, comprising: Identification module: obtains data from multiple data sources and uses the data quality rule base and machine learning algorithms to automatically identify multiple data quality issues in the data; A first determination module: determining the root cause and impact path of the data quality problem based on the causal relationship reasoning technology according to the multiple data quality problems; Assessment module: Evaluate the severity and impact of each data quality issue based on the root causes and impact paths; Selection module: assigns a priority to each data quality issue based on the severity and impact scope based on predefined rules, determines the processing order based on the priority, and selects the best repair strategy to automatically repair each data quality issue; Automatic repair module: Real-time and continuous monitoring of repaired data, using stream processing technology and real-time analysis algorithms to detect new data quality issues and perform automatic repairs.

[0007] According to a data quality monitoring and repair system provided by the present invention, an identification module includes: The first acquisition unit: acquires data from different data sources through data extraction tools; Detection unit: pre-processes the data and detects errors and anomalies in the data according to a data quality rule base; Conversion unit: converts data into useful features by adjusting feature weights and checks the quality of features according to the data quality rule base; Prediction unit: Use the preprocessed data and feature-engineered data to train the machine learning model, and predict the quality score of each data based on the trained machine learning model; The second acquisition unit: sorts and classifies the data according to the quality scores, and obtains multiple data quality problems existing in the data according to the sorting and classification results.

[0008] According to a data quality monitoring and repairing system provided by the present invention, a first determination module includes: A first determination unit: obtaining problem types and data properties of the multiple data quality problems, and determining corresponding causal relationship models according to the problem types and data properties; The first adjustment unit: Use cross-validation technology to evaluate the performance and generalization ability of the model, and adjust the model structure and hyperparameters according to the performance and generalization ability of the model; Drawing unit: predict the relationship between dependent variables through the adjusted model output and draw a causal diagram; Inference unit: inferring the causal effect of the data quality problem according to the causal graph; The second determination unit: determines the root cause and impact path of the data quality problem according to the causal effect.

[0009] According to a data quality monitoring and repair system provided by the present invention, the evaluation module includes: A third acquisition unit: determining data error points and error ratios according to the root causes, determining data quality problem types according to the data error points and error ratios, and acquiring a distribution of data quality problem types; A third determining unit: determining the severity of each data quality problem according to the distribution of the data quality problem types; The fourth determining unit is: obtaining branches and lengths of impact paths of data quality problems, and determining the impact scope of each data quality problem according to the branches and lengths of the impact paths.

[0010] According to a data quality monitoring and repair system provided by the present invention, a selection module includes: Fifth determination unit: determining the urgency and risk level of each data quality issue according to the severity and impact scope; Assignment unit: assigns a priority to each data quality issue based on predefined rules according to the urgency and risk level; Sorting unit: assigns a unique identifier and descriptive label to each data quality issue according to priority, sorts multiple data quality issues, and determines the processing order based on the sorting; Selection unit: obtaining the characteristics of each data quality problem according to the processing order, and selecting the best repair strategy according to the characteristics; Automatic repair unit: automatically repairs each data quality problem according to the optimal repair strategy.

[0011] According to a data quality monitoring and repair system provided by the present invention, the automatic repair module includes: Establishment unit: Establish a real-time monitoring mechanism and obtain the repaired real-time data stream through data pipelines and data storage services; Fourth acquisition unit: using processing technology and real-time analysis algorithm to analyze the real-time data stream, and obtaining new data quality problems existing in the real-time data stream according to the analysis results; The second adjustment unit: adjusts algorithm parameters and improves data processing procedures in real time according to newly emerging data quality issues, and triggers the repair process for automatic repair.

[0012] According to a data quality monitoring and repairing system provided by the present invention, a fifth determining unit includes: The first determination subunit: Determine the business decision impact weight and business operation impact weight of each data quality issue based on severity and impact scope; Second determination subunit: determine the urgency of each data quality issue based on the business decision impact weight and business operation impact weight; Acquisition subunit: acquires the data hidden danger factor of each data quality problem, and determines the risk hidden danger description parameter of each data quality problem according to the data hidden danger factor; The third determination subunit: determines the risk level of each data quality issue based on the risk description parameters.

[0013] According to a data quality monitoring and repairing system provided by the present invention, after automatically identifying multiple data quality problems existing in the data, the system further includes: Construction module: determining the current data attributes of the data according to the existing multiple data quality problems and constructing the attribute vector string according to the current data attributes; Acquisition module: determines the included data attributes and missing data attributes of the data based on the attribute vector string, and obtains the data object of each missing data attribute; The second determination module: determines the data format state and data display state corresponding to the data object, and models the data object according to the data format state and data display state; The third determination module: determines the failure characteristics of the data according to the data object model, and determines the current data flow parameters of the data based on the failure characteristics; Fourth determination module: determining current configuration item parameters of data according to current data flow parameters, and determining equivalent mapping elements of data based on the current configuration item parameters; A fifth determination module: determining basic eligibility of the data according to the equivalent mapping elements and the equivalent mapping target function of the data, and determining a data label of the data based on the basic eligibility; The sixth determination module: determines the specific prime number corresponding to the data label, and determines the gradient parameter of the data through a preset calculation method according to the specific prime number; Upload module: Determine the global characteristics of the data based on the gradient parameters, determine the causes of the data quality problems based on the global characteristics and the problem attributes of each data quality problem, and upload the causes of the data quality problems.

[0014] Compared with the prior art, the present invention has the following beneficial effects: By identifying problems in data from multiple data sources, determining the root causes and impact paths of the problems, assigning priorities to each data quality problem, and using the best repair strategy to repair data quality problems, the identification efficiency can be improved. At the same time, the assessment of the severity and scope of the problem is improved, and the priority and processing order of the problem can be quickly and accurately judged, ensuring the automation, intelligent and real-time processing of data quality problems, and improving the credibility and value of the data center. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 is a structural diagram of a data quality monitoring and repair system provided by an embodiment of the present invention; Figure 2 It is a structural diagram of the identification module provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Embodiment 1: The embodiment of the present invention provides a data quality monitoring and repair system, such as Figure 1 As shown, the system mainly includes the following modules: Identification module: obtains data from multiple data sources and uses the data quality rule base and machine learning algorithms to automatically identify multiple data quality issues in the data; A first determination module: determining the root cause and impact path of the data quality problem based on the causal relationship reasoning technology according to the multiple data quality problems; Assessment module: Evaluate the severity and impact of each data quality issue based on the root causes and impact paths; Selection module: assigns a priority to each data quality issue based on the severity and impact scope based on predefined rules, determines the processing order based on the priority, and selects the best repair strategy to automatically repair each data quality issue; Automatic repair module: Real-time and continuous monitoring of repaired data, using stream processing technology and real-time analysis algorithms to detect new data quality issues and perform automatic repairs.

[0019] In this embodiment, the data source refers to the source of data from which information is obtained or operations are performed. The data source can be internal data, such as records in a database, file system, or application, or external data, such as data from a website, social media platform, or other sources.

[0020] In this embodiment, the data quality rule library is a software tool for storing data quality rules, which can organize the definition, maintenance and management of data quality rules, and automatically detect and correct data quality problems according to these rules.

[0021] In this embodiment, the machine learning algorithm is an algorithm that allows a computer to automatically "learn" how to complete a specific task. The machine learning algorithm uses a large amount of data and algorithms to infer patterns and rules, thereby achieving tasks such as prediction, classification, and clustering, such as: regression algorithm, classification algorithm, neural network algorithm, random forest algorithm.

[0022] In this embodiment, the various problems existing in the data may be: missing values, erroneous values, abnormal values, redundant values, and inconsistency.

[0023] In this embodiment, the causal relationship inference technology is a statistical method for inferring causal relationships from observed phenomena, and the causal relationship between variables is inferred by analyzing the correlation between them.

[0024] In this embodiment, the influence path is a conceptual diagram used to describe how one variable causes changes in another variable. It includes a series of relationships between variables and describes how these variables affect the final result through a series of interacting steps. The influence path is usually represented as a directed line or arrow, and the direction of the arrow points to the "source" or "cause" of the influence path. For example, if the data has an incorrect value, it may be an input error or a formula error.

[0025] In this embodiment, the predefined rules refer to rules that are set in advance. For example, when assigning priorities to problems, data with erroneous values ​​are solved first than data with redundant values.

[0026] In this embodiment, data stream processing technology is a technology for processing large amounts of data and can be used to identify and fix data quality issues by analyzing the source of the data and delivering the data to a specified destination for further processing or reporting.

[0027] In this embodiment, the real-time analysis algorithm is a method for processing real-time data, which can analyze the data immediately after it is generated.

[0028] In this embodiment, the causal relationship reasoning technology may be:

[0029] in, is the outcome variable, is the cause variable, is the effect of the cause variable on the outcome variable, is the error term and is the outcome variable A random perturbation of , the introduction of error terms can distinguish causal effects from purely accidental effects.

[0030] The beneficial effects of the above technical solution are: by identifying problems existing in data from multiple data sources, determining the root causes and impact paths of the problems, assigning priorities to each data quality problem, and using the best repair strategy to repair data quality problems, it can improve identification efficiency. At the same time, it improves the assessment of the severity and scope of the problem, quickly and accurately judges the priority and processing order of the problem, ensures the automation, intelligent and real-time processing of data quality problems, and improves the credibility and value of the data center.

[0031] Embodiment 2: Based on the first embodiment, the identification module of the embodiment of the present invention includes: The first acquisition unit: acquires data from different data sources through data extraction tools; Detection unit: pre-processes the data and detects errors and anomalies in the data according to a data quality rule base; Conversion unit: converts data into useful features by adjusting feature weights and checks the quality of features according to the data quality rule base; Prediction unit: Use the preprocessed data and feature-engineered data to train the machine learning model, and predict the quality score of each data based on the trained machine learning model; The second acquisition unit: sorts and classifies the data according to the quality scores, and obtains multiple data quality problems existing in the data according to the sorting and classification results.

[0032] In this embodiment, the data extraction tool is a software application for obtaining data from a database or other data source. These tools can help users automatically or manually extract data from various data sources and store the results in local files or other data storage media.

[0033] In this embodiment, preprocessing refers to the process of cleaning, converting, merging, etc. of the original data. In this embodiment, adjusting feature weights refers to assigning different weight values ​​to different features in order to more accurately evaluate the data. Weight is a metric used to measure the contribution of each feature to the result. For example, if a feature is considered critical, then it should get a higher weight value, and if a feature is not particularly important, then it should get a lower weight value. The methods for adjusting feature weights can be: mean adjustment, variance adjustment, and weighted adjustment.

[0034] In this embodiment, converting data into useful features refers to converting the original data into a more useful form. For example, if you want to understand the sales of a product, you may collect a lot of information about the product, such as price, description, tags, etc. This information can be regarded as raw data, but they are not necessarily useful features.

[0035] In this embodiment, the quality of a feature refers to the effectiveness and reliability of a feature.

[0036] The beneficial effect of the above technical solution is that by preprocessing and transforming the data in the data source, the data quality can be improved. Furthermore, the machine model is trained to predict the quality score of each data point, and multiple problems existing in the data are determined based on the quality score, which can improve the accuracy and speed of obtaining problems.

[0037] Embodiment 3: Based on Example 2, the first determination module in this embodiment of the present invention includes: A first determination unit: obtaining problem types and data properties of the multiple data quality problems, and determining corresponding causal relationship models according to the problem types and data properties; The first adjustment unit: Use cross-validation technology to evaluate the performance and generalization ability of the model, and adjust the model structure and hyperparameters according to the performance and generalization ability of the model; Drawing unit: predict the relationship between dependent variables through the adjusted model output and draw a causal diagram; Inference unit: inferring the causal effect of the data quality problem according to the causal graph; The second determination unit: determines the root cause and impact path of the data quality problem according to the causal effect.

[0038] In this embodiment, the types of data problems include: Inaccurate data: refers to situations where there are errors or biases in the data.

[0039] Missing data: refers to the situation where there are missing values ​​or missing time periods in the data.

[0040] Data duplication: refers to the situation where there are duplicate values ​​in the data.

[0041] Data inconsistency: refers to the presence of inconsistencies or inconsistencies in the data.

[0042] Data noise: refers to the noise or interference information present in the data.

[0043] In this embodiment, data properties refer to characteristics or attributes of data, such as: numerical data, categorical data, sequential data, time data, text data, and image data.

[0044] In this embodiment, the causal relationship model is a statistical model used to describe the causal relationship between variables, which assumes that some variables are dependent variables and other variables are outcome variables.

[0045] In this embodiment, the cross-validation technique is a method for evaluating the performance of a prediction model. It checks the generalization ability of the model by dividing the training data set into multiple parts and using each part as a test set. Specifically, the cross-validation technique divides the training data set into k subsets according to a certain ratio, where k is a positive integer. Then, the k subsets are evenly divided into two subsets, one as a training set and the other as a test set. During model training, the training set is used to train the model, and the test is performed on the test set to evaluate the performance of the model.

[0046] In this embodiment, the model structure refers to the relationship and composition between the various components of the machine learning model.

[0047] In this embodiment, model hyperparameters refer to parameters used to adjust model behavior in machine learning, such as regularization coefficients and learning rates.

[0048] In this embodiment, the causal diagram is a visualization tool for displaying the causal relationship between variables. The causal diagram is usually composed of a series of nodes, which represent different variables or factors. The nodes are connected by arrows to represent the relationship between these variables.

[0049] In this embodiment, the causal effect of the data quality problem refers to the impact of the data quality problem on business decisions.

[0050] In this embodiment, the quality problem affecting the path, such as data, is that there are missing values, so the path may have input errors and illegal inputs during the calculation process.

[0051] The beneficial effects of the above technical solution are: determining the corresponding causal relationship model through the problem type and data properties, adjusting the parameters, outputting the relationship between the predicted dependent variables, and drawing a causal diagram, so as to determine the root cause and impact path of the data quality problem, accurately find out the essence of the problem, and make corresponding solutions, thereby improving data repair efficiency.

[0052] Embodiment 4: Based on Example 3, the evaluation module of the embodiment of the present invention includes: A third acquisition unit: determining data error points and error ratios according to the root causes, determining data quality problem types according to the data error points and error ratios, and acquiring a distribution of data quality problem types; A third determining unit: determining the severity of each data quality problem according to the distribution of the data quality problem types; The fourth determining unit is: obtaining branches and lengths of impact paths of data quality problems, and determining the impact scope of each data quality problem according to the branches and lengths of the impact paths.

[0053] In this embodiment, the branch of the impact path means that the data quality problem may cause a series of events. These events can be called branches of the impact path. These events may occur in different times and spaces and may be independent of each other.

[0054] The beneficial effects of the above technical solution are: obtaining the distribution of data quality problem types through root causes, thereby determining the severity of each data quality problem, which can help better manage data quality problems and ensure data consistency and accuracy. At the same time, the impact scope of each data quality problem is determined according to the branches and length of the impact path, and the impact of each data quality problem on the entire data can be intuitively determined, so as to make corresponding solution decisions.

[0055] Embodiment 5: Based on the fourth embodiment, the selection module of the embodiment of the present invention includes: Fifth determination unit: determining the urgency and risk level of each data quality issue according to the severity and impact scope; Assignment unit: assigns a priority to each data quality issue based on predefined rules according to the urgency and risk level; Sorting unit: assigns a unique identifier and descriptive label to each data quality issue according to priority, sorts multiple data quality issues, and determines the processing order based on the sorting; Selection unit: obtaining the characteristics of each data quality problem according to the processing order, and selecting the best repair strategy according to the characteristics; Automatic repair unit: automatically repairs each data quality problem according to the optimal repair strategy.

[0056] In this embodiment, assigning a unique identifier to each data quality issue means assigning a unique code or mark to each issue. The unique identifier may include any alphanumeric combination, or even a combination of other characters, such as underscores and dots. Common unique identifiers include: sequential numbers (such as a sequence of numbers or letters arranged in chronological order), random numbers (random numbers or letter sequences generated by a computer), names or descriptions (such as the name of the issue or a brief description).

[0057] In this embodiment, the descriptive label refers to a set of character strings or other information used to classify, group or describe an object. These labels are usually determined according to the attributes or characteristics of the object.

[0058] In this embodiment, the characteristics of the data quality problem refer to the unique attributes of the problem, such as: data inconsistency, data missing, and data error.

[0059] The beneficial effects of the above technical solution are: determining the priority and processing order of each data quality problem according to the urgency and risk level of each data quality problem, and selecting the best repair strategy for repair according to the characteristics of each data quality problem, which can improve the judgment of problem priority and avoid dealing with non-urgent problems first, which will lead to risks caused by serious data quality problems.

[0060] Embodiment 6: Based on Example 5, the automatic repair module of the embodiment of the present invention includes: Establishment unit: Establish a real-time monitoring mechanism and obtain the repaired real-time data stream through data pipelines and data storage services; Fourth acquisition unit: using processing technology and real-time analysis algorithm to analyze the real-time data stream, and obtaining new data quality problems existing in the real-time data stream according to the analysis results; The second adjustment unit: adjusts algorithm parameters and improves data processing procedures in real time according to newly emerging data quality issues, and triggers the repair process for automatic repair.

[0061] In this embodiment, the real-time monitoring mechanism is a technology for monitoring data.

[0062] In this embodiment, a data pipeline is a process that transfers data from one or more sources to another or more target systems.

[0063] In this embodiment, the data storage service is a technology and method for storing and managing a large amount of data.

[0064] In this embodiment, the real-time data stream refers to data that flows continuously in a fast and continuous manner.

[0065] In this embodiment, the repair process of automatic repair is: when a new data quality problem occurs, the data quality problem is checked and classified, the cause of the problem is determined, and then the problem is repaired and replaced according to manual intervention or rule engine methods.

[0066] In this embodiment, not all data can be quantified, and some types of data require other methods to evaluate their quality and effectiveness, such as text analysis and speech recognition.

[0067] In this embodiment, before starting the repair, it is necessary to collect the original data and the repaired data. The original data should contain all known errors and inconsistencies, and the repaired data should be generated after modification according to the repair plan. For the repaired data, the proportion of samples with correctly identified quality problems to the total number of samples is obtained. For example: there are 10 errors and inconsistencies in the original data, 9 problems are correctly identified, and 8 problems are correctly repaired, then the repair accuracy is 80%, and the repair strategy with a repair accuracy lower than 90% needs to be modified.

[0068] The beneficial effects of the above technical solution are: obtaining and analyzing real-time data streams through data pipelines and data storage services, obtaining data quality problems existing in real-time data streams, and repairing them by adjusting algorithm parameters in real time and improving data processing processes. It can quickly obtain existing real-time new problems and obtain corresponding strategies for repair, thereby ensuring the speed and pertinence of data repair. At the same time, it improves the efficiency of data processing, reduces the need for manual intervention, and helps to ensure the stability and consistency of the data processing process.

[0069] Embodiment 7: Based on Embodiment 6, a fifth determining unit in Embodiment 5 of the present invention includes: The first determination subunit: Determine the business decision impact weight and business operation impact weight of each data quality issue based on severity and impact scope; Second determination subunit: determine the urgency of each data quality issue based on the business decision impact weight and business operation impact weight; Acquisition subunit: acquires the data hidden danger factor of each data quality problem, and determines the risk hidden danger description parameter of each data quality problem according to the data hidden danger factor; The third determination subunit: determines the risk level of each data quality issue based on the risk description parameters.

[0070] In this embodiment, the business decision impact weight of the data quality problem refers to the degree of impact of the data quality problem on the decision result that needs to be considered when making a business decision.

[0071] In this embodiment, the business operation impact weight refers to a relative weight used when evaluating the impact of a decision or activity on business operations.

[0072] In this embodiment, data hidden danger factors refer to potential problems or factors that may affect the data analysis results, such as incomplete data, inconsistent data, and data pollution.

[0073] In this embodiment, the risk hidden danger description parameter is a set of information that describes the risk hidden danger in detail, such as: hidden danger name, hidden danger classification, and hidden danger location.

[0074] The beneficial effects of the above technical solution are: by determining the urgency of the problem through the business decision impact weight and business operation impact weight of each data quality problem, and at the same time, determining the risk level of the problem according to the data hidden danger factor of each data quality problem, it is easier to understand and grasp the characteristics and nature of the risk hidden dangers of the problem, more effectively assess the risk, and thus make targeted solutions and responses to improve data repair efficiency.

[0075] Embodiment 8: Based on Example 7, after automatically identifying multiple data quality issues existing in the data, the embodiment of the present invention further includes: Construction module: determining the current data attributes of the data according to the existing multiple data quality problems and constructing the attribute vector string according to the current data attributes; Acquisition module: determines the included data attributes and missing data attributes of the data based on the attribute vector string, and obtains the data object of each missing data attribute; The second determination module: determines the data format state and data display state corresponding to the data object, and models the data object according to the data format state and data display state; The third determination module: determines the failure characteristics of the data according to the data object model, and determines the current data flow parameters of the data based on the failure characteristics; Fourth determination module: determining current configuration item parameters of data according to current data flow parameters, and determining equivalent mapping elements of data based on the current configuration item parameters; A fifth determination module: determining basic eligibility of the data according to the equivalent mapping elements and the equivalent mapping target function of the data, and determining a data label of the data based on the basic eligibility; The sixth determination module: determines the specific prime number corresponding to the data label, and determines the gradient parameter of the data through a preset calculation method according to the specific prime number; Upload module: Determine the global characteristics of the data based on the gradient parameters, determine the causes of the data quality problems based on the global characteristics and the problem attributes of each data quality problem, and upload the causes of the data quality problems.

[0076] In this embodiment, data attributes refer to information possessed by a data object that describes its characteristics and status. This information may include numerical values, text, date and time, etc., and is used to describe the nature and meaning of the data.

[0077] In this embodiment, the attribute vector string is a coding method for representing data attributes, which combines multiple data types and values ​​to form a single data structure.

[0078] In this embodiment, the data format state is information describing different stages or states of data during the processing process. The data format state is used to record the location of the data during the processing process, changes, and possible problems that may occur.

[0079] In this embodiment, the data display state refers to a description of the current state of data presented on the display in a computer application, such as: preparation state, input state, processing state, and analysis state.

[0080] In this embodiment, modeling of data objects is to abstract entities in the real world into models that can be recognized and processed by computer programs, such as: Relational database model: This is a data model used to store and organize data in which data is organized into tables, where each table contains a set of related columns, each column representing a specific attribute of the data.

[0081] NoSQL database model: NoSQL databases usually do not use relational models, but instead use non-relational models such as tree and graph.

[0082] In this embodiment, the data failure feature refers to the situation in which the training data set is faulty or inaccurate in the machine learning or artificial intelligence system, such as: Class imbalance: refers to the situation where the number of instances in one or more categories is much larger than that in other categories.

[0083] Missing values: refers to the fact that some attribute values ​​in the data set are empty or unknown.

[0084] Noisy data: refers to the presence of some erroneous or misleading data points in the data set, such as duplicate values ​​and outliers.

[0085] In this embodiment, the data flow parameter is a method of defining the input or output of data flow in computer programming, and describes the input and output data structure and format when the program processes data.

[0086] In this embodiment, the current configuration item parameters of the data refer to the settings and parameters related to a specific task in a data processing process, such as: file path, data format, data cleaning rules.

[0087] In this embodiment, the equivalent mapping element of data refers to the mapping relationship between data types and data structures in different data systems. The equivalent mapping element can more easily convert data from the source system format to the target system format when converting data from one system to another, such as: data type, data length, and data boundary.

[0088] In this embodiment, the data equivalent mapping objective function is a function that maps input data to expected output, and it is necessary to consider various constraints and limitations in the data system, such as data range, format requirements, security and integrity rules, to ensure that the input data can be correctly converted into the expected output results.

[0089] In this embodiment, the data label is information used to describe the characteristics or categories of each sample in the data set. These labels are usually discrete values, such as integers or categorical variables, or they can be continuous values, such as measurement values.

[0090] In this embodiment, the gradient parameters of the data refer to a set of values ​​used to calculate model parameters in machine learning. The values ​​of these parameters are usually calculated by forward propagation of the training samples and then by the chain rule.

[0091] In this embodiment, the global feature refers to a certain regularity or information existing in the entire data set. For example, in natural language processing, the global feature usually refers to the features common to all texts, such as word frequency, syntax, etc.

[0092] The beneficial effects of the above technical solution are: determining the causes of data quality problems based on global characteristics and the problem attributes of each data quality problem can quickly discover the main problem areas in the data set, such as high error rates in certain categories or obvious noise, which can help us perform data preprocessing and cleaning more specifically, improve data quality and the efficiency and accuracy of detecting problems.

[0093] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data quality monitoring and repair system, characterized in that: include: Identification module: obtains data from multiple data sources and uses the data quality rule base and machine learning algorithms to automatically identify multiple data quality issues in the data; A first determination module: determining the root cause and impact path of the data quality problem based on the causal relationship reasoning technology according to the multiple data quality problems; Assessment module: Evaluate the severity and impact of each data quality issue based on the root causes and impact paths; Selection module: assigns a priority to each data quality issue based on the severity and impact scope based on predefined rules, determines the processing order based on the priority, and selects the best repair strategy to automatically repair each data quality issue; Automatic repair module: Real-time and continuous monitoring of repaired data, using stream processing technology and real-time analysis algorithms to detect new data quality issues and perform automatic repairs.

2. The data quality monitoring and repair system according to claim 1, characterized in that: Identification module, including: The first acquisition unit: acquires data from different data sources through data extraction tools; Detection unit: pre-processes the data and detects errors and anomalies in the data according to a data quality rule base; Conversion unit: converts data into useful features by adjusting feature weights and checks the quality of features according to the data quality rule base; Prediction unit: Use the preprocessed data and feature-engineered data to train the machine learning model, and predict the quality score of each data based on the trained machine learning model; The second acquisition unit: sorts and classifies the data according to the quality scores, and obtains multiple data quality problems existing in the data according to the sorting and classification results.

3. The data quality monitoring and repair system according to claim 1, characterized in that: The first determination module includes: A first determination unit: obtaining problem types and data properties of the multiple data quality problems, and determining corresponding causal relationship models according to the problem types and data properties; The first adjustment unit: Use cross-validation technology to evaluate the performance and generalization ability of the model, and adjust the model structure and hyperparameters according to the performance and generalization ability of the model; Drawing unit: predict the relationship between dependent variables through the adjusted model output and draw a causal diagram; Inference unit: inferring the causal effect of the data quality problem according to the causal graph; The second determination unit: determines the root cause and impact path of the data quality problem according to the causal effect.

4. The data quality monitoring and repair system according to claim 1, characterized in that: Assessment modules include: A third acquisition unit: determining data error points and error ratios according to the root causes, determining data quality problem types according to the data error points and error ratios, and acquiring a distribution of data quality problem types; A third determining unit: determining the severity of each data quality problem according to the distribution of the data quality problem types; The fourth determining unit is: obtaining branches and lengths of impact paths of data quality problems, and determining the impact scope of each data quality problem according to the branches and lengths of the impact paths.

5. The data quality monitoring and repair system according to claim 1, characterized in that: Select modules including: Fifth determination unit: determining the urgency and risk level of each data quality issue according to the severity and impact scope; Assignment unit: assigns a priority to each data quality issue based on predefined rules according to the urgency and risk level; Sorting unit: assigns a unique identifier and descriptive label to each data quality issue according to priority, sorts multiple data quality issues, and determines the processing order based on the sorting; Selection unit: obtaining the characteristics of each data quality problem according to the processing order, and selecting the best repair strategy according to the characteristics; Automatic repair unit: automatically repairs each data quality problem according to the optimal repair strategy.

6. The data quality monitoring and repair system according to claim 1, characterized in that: Automatic repair modules, including: Establishment unit: Establish a real-time monitoring mechanism and obtain the repaired real-time data stream through data pipelines and data storage services; Fourth acquisition unit: using processing technology and real-time analysis algorithm to analyze the real-time data stream, and obtaining new data quality problems existing in the real-time data stream according to the analysis results; The second adjustment unit: adjusts algorithm parameters and improves data processing procedures in real time according to newly emerging data quality issues, and triggers the repair process for automatic repair.

7. The data quality monitoring and repair system according to claim 5, characterized in that: The fifth determining unit includes: The first determination subunit: Determine the business decision impact weight and business operation impact weight of each data quality issue based on severity and impact scope; Second determination subunit: determine the urgency of each data quality issue based on the business decision impact weight and business operation impact weight; Acquisition subunit: acquires the data hidden danger factor of each data quality problem, and determines the risk hidden danger description parameter of each data quality problem according to the data hidden danger factor; The third determination subunit: determines the risk level of each data quality issue based on the risk description parameters.

8. The data quality monitoring and repair system according to claim 1, characterized in that: After automatically identifying multiple data quality issues in the data, it also includes: Construction module: determining the current data attributes of the data according to the existing multiple data quality problems and constructing the attribute vector string according to the current data attributes; Acquisition module: determines the included data attributes and missing data attributes of the data based on the attribute vector string, and obtains the data object of each missing data attribute; The second determination module: determines the data format state and data display state corresponding to the data object, and models the data object according to the data format state and data display state; The third determination module: determines the failure characteristics of the data according to the data object model, and determines the current data flow parameters of the data based on the failure characteristics; Fourth determination module: determining current configuration item parameters of data according to current data flow parameters, and determining equivalent mapping elements of data based on the current configuration item parameters; A fifth determination module: determining basic eligibility of the data according to the equivalent mapping elements and the equivalent mapping target function of the data, and determining a data label of the data based on the basic eligibility; The sixth determination module: determines the specific prime number corresponding to the data label, and determines the gradient parameter of the data through a preset calculation method according to the specific prime number; Upload module: Determine the global characteristics of the data based on the gradient parameters, determine the causes of the data quality problems based on the global characteristics and the problem attributes of each data quality problem, and upload the causes of the data quality problems.

Images