Dynamic discovery and correction of data quality issues

A modular system for dynamic data quality assessment and correction addresses inefficiencies in processing IoT data by iteratively improving data quality, enhancing computational efficiency and accuracy for AI applications in Industry 4.0.

JP7590087B2Active Publication Date: 2024-11-26INTERNATIONAL BUSINESS MACHINE CORPORATION
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2022562935
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-10-20
Filing Date
2021-04-07
Publication Date
2024-11-26
Estimated Expiration
2041-04-07

AI Technical Summary

Technical Problem

Current data processing architectures face challenges in efficiently handling large volumes of heterogeneous and unstructured data from IoT sensors, leading to inefficiencies in computational resources and accuracy due to issues like noise, missing values, and data irregularity, with existing methods lacking comprehensive data cleaning and preprocessing tools.

Method used

A modular and flexible system for dynamic data quality assessment and correction, utilizing iterative validation nodes with data quality checks and remediation actions, including schema inference, graph-based validation, and intelligent data transformations to improve data quality without wasting resources.

Benefits of technology

Enhances computational efficiency and accuracy by effectively processing large datasets with reduced resource consumption, ensuring high-quality data for AI applications in Industry 4.0 environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007590087000001
    Figure 0007590087000001
  • Figure 0007590087000002
    Figure 0007590087000002
  • Figure 0007590087000003
    Figure 0007590087000003
Patent Text Reader

Abstract

A device, method, and system are provided for improving data quality without wasting computational resources. A computing device receives a raw dataset. One or more data quality metric targets corresponding to the received raw dataset are received. A schema of the dataset is determined. An initial set of validation nodes is identified based on the schema of the dataset. The initial set of validation nodes is executed. A next set of validation nodes is iteratively expanded and executed based on the schema of the dataset until a termination criterion is reached. A modified dataset of the raw dataset is provided based on the iterative execution of the initial and next sets of validation nodes.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates generally to computer systems, and more particularly to automatic dynamic data quality assessment and correction thereof. [Background technology]

[0002] From the First Industrial Revolution, which brought about mechanization through water and steam power, to the Second Industrial Revolution, which enabled mass production and assembly lines through electricity, the Fourth Industrial Revolution is now transforming what began in the Third Industrial Revolution, which introduced computers and automation, and enhancing it with smart autonomous systems driven by big data and machine learning. As the amount of available data increases, the quality of the large amount of information becomes a key consideration for computers that use this data for machine learning.

[0003] In recent years, with the increasing adoption of Industry 4.0, many industrial production processes are closely monitored in real time by many sensors. The quality of the data used by a computing device has a significant impact on its efficiency, the accuracy of the calculations, and the amount of computational resources used. Such specifically configured computing devices can be used to build data that guides artificial intelligence (AI)-based solutions in Industry 4.0, for example, for machine failures, anomaly detection, survival analysis, etc. However, current architectures face various challenges in processing the enormous amount of data from real Internet of Things (IoT) sensors. For example, the data can be very large (e.g., big data) and require a large amount of computational resources, including processing power, memory, network bandwidth, and time, of the computing device. The data can be noisy, the data can have missing values, the data samples can be irregular, etc. Furthermore, the data processed by the computing device can be heterogeneous and unstructured, collectively referred to herein as heterogeneous (e.g., can include time series, text, images, audio, different sample rates, etc.).

[0004] By at least one estimate, during the AI ​​development stage, this data pre-processing stage takes up 80% of the time in the cycle of developing an AI model. In other words, the majority of the time involved in analyzing data involves preparing the data so that it is in a suitable format from which an AI model can be prepared. Some challenges of using automation in the data pre-processing stage include the resources required to analyze the potential uniqueness and characteristics of each data set. Existing methods currently do not provide tools for thorough data cleaning and pre-processing, thus negatively impacting the efficiency, accuracy, and amount of resources used by the computing device.

[0005] The AI ​​model development stage can have many frameworks and end-to-end solutions defined, including machine learning platforms such as sklearn, tensorflow, Tpot, H2O, etc. However, currently there is no known end-to-end framework or library or both that can efficiently clean data.

[0006] Thus, provided herein is a method and a computerized system of the dynamic discovery and correction of data quality problems. In one aspect, the teachings herein provide automation in the data pre-processing stage. In one embodiment, an iterative process is managed in that there is interaction and collaboration from AI practitioners that allows for quality checks of heterogeneous IoT data. By using installable function libraries, a wide variety of checks are provided. The data quality assessment and validation discussed herein can be highly customized. In one aspect, the system described herein is modular, allowing for a flexible process for data quality assessment. For example, there is flexibility to include various checks and yet chain them together in a way that suits the data being processed. Summary of the Invention

[0007] Various embodiments provide a computing device, a non-transitory computer-readable storage medium, and a computer-implemented method, as well as a system for improving data quality so as not to waste computational resources. Raw data is received. One or more data quality metric objectives corresponding to the received raw data set are received. A schema of the data set is determined. An initial set of validation nodes is determined based on the schema of the data set. The initial set of validation nodes is executed. A next set of validation nodes is iteratively expanded and executed based on the schema of the data set until a termination criterion is reached. A modified data set of the raw data set is provided based on the iterative execution of the initial and next sets of validation nodes.

[0008] In one embodiment, each validation node includes a data quality check and one or more remediation actions.

[0009] In one embodiment, execution of the validation nodes of the initial set of validation nodes includes identifying all possible improvement actions for every data quality check, transforming the data with each possible improvement action, and calculating a number of data quality metrics (DQMs) to evaluate the transformations.

[0010] In one embodiment, the execution of the validation node includes a first stage including performing logical checks on the raw data set by a validator object to detect one or more anomalies in the raw data set, and performing different data transformations on the raw data set by corresponding operators to correct the one or more detected anomalies, and creating a new version of the data for each data transformation.

[0011] In one embodiment, the validation node includes a second stage that includes generating a Data Quality Metric (DQM) by an Internal Quality Evaluator (IQE) module for each new version of the data created and generating a DQM for the raw data set.

[0012] In one embodiment, each DQM of the second stage includes at least one of: (i) a summary of features in multiple dimensions of the corresponding new version of the data created from the raw dataset; or (ii) gain or change information of the corresponding new version of the data created from the raw dataset.

[0013] In one embodiment, execution of the validation node includes a third stage which includes performing a comparison of each DQM of each new version of data created and the DQM of the raw data against the raw dataset and assessing the improvement from the raw dataset.

[0014] In one embodiment, execution of the validation node includes a fourth stage that includes selecting the operator for the new version of the created data that best meets the data quality metric goal.

[0015] In one embodiment, the operator selected has a maximum gap between its corresponding DQM and the DQM of the raw data that is less than a predefined threshold to maintain similarity to the original data.

[0016] In one embodiment, expanding the next set of validation nodes includes at least one of: determining validation nodes that best achieve one or more received quality metric goals; and determining the validation nodes based on mining a performance information repository to find all validation nodes that typically occur together.

[0017] According to one embodiment, the system includes a schema inference module configured to receive an unprocessed dataset and determine a schema of the unprocessed dataset. A data synthesizer module is coupled to the schema inference module and configured to receive one or more data quality metric objectives from a knowledge base corresponding to the received unprocessed dataset. The data synthesizer module includes an initiater module configured to identify an initial set of validation nodes based on the schema of the dataset. The data synthesizer module further includes an executor module coupled to the initiater module and configured to execute the initial set of validation nodes. The data synthesizer module further includes an expander module coupled to the executor module and configured to iteratively expand and execute a next set of validation nodes based on the schema of the dataset until a termination criterion is reached. The data synthesizer module includes a terminator module coupled to the expander module and configured to iteratively determine a next set of validation nodes to consider by the expander module and determine when to terminate the iterative determination. There is an information repository coupled to the executor and terminator modules and configured to provide a modified dataset of the raw dataset based on the iterative execution of the initial set and the next set of validating nodes.

[0018] In one embodiment, each validation node includes a data quality check and one or more remediation actions.

[0019] In one embodiment, an initial set of validation nodes is configured to identify all possible improvement actions for every data quality check, transform the data with each possible improvement action, and calculate a number of data quality metrics (DQMs) to evaluate the transformations.

[0020] In one embodiment, each validation node includes a first stage configured to perform logical checks on the raw dataset by a validator object to detect one or more anomalies in the raw dataset, perform different data transformations on the raw dataset by corresponding operators to correct the one or more anomalies, and create a new version of the data for each data transformation.

[0021] In one embodiment, for each new version of data created, there is a second stage configured to generate Data Quality Metrics (DQM) by an Internal Quality Evaluator (IQE) module and generate DQMs for the raw data set.

[0022] In one embodiment, for each DQM of each new version of data created, there is a third stage configured to perform a comparison against the raw dataset to assess the improvement from the raw dataset.

[0023] In one embodiment, there is a fourth stage configured to select an operator for the new version of the created data that best meets the data quality metric goal.

[0024] The concepts discussed herein provide technical improvements that increase the computational efficiency and accuracy of computing devices that process large amounts of data. These and other features will become apparent from the following detailed description of illustrative embodiments, which is to be read in connection with the accompanying drawings. [Brief description of the drawings]

[0025] The drawings are exemplary embodiments. They do not depict all embodiments. Other embodiments may additionally or alternatively be used. Obvious or unnecessary details may be omitted to save space or for a more efficient illustration. Some embodiments may be practiced with additional components or steps and / or without all components or steps shown. When the same numerals appear in different drawings, they refer to the same or similar components or steps.

[0026] [Figure 1] 1 is a block diagram of an example architecture of a graph-based data quality synthesizer system, consistent with an example embodiment. [Diagram 2] 1 provides a high-level block diagram of a validator node, consistent with an illustrative embodiment. [Diagram 3] 3 provides a block diagram of Stage 1 of FIG. 2, consistent with an illustrative embodiment. [Figure 4] 3 provides a block diagram of a Stage 2 path of FIG. 2 , consistent with an illustrative embodiment; [Diagram 5] 3 provides a block diagram of a path of Stage 3 of FIG. 2, consistent with an illustrative embodiment. [Figure 6] 3 is a block diagram of stage 4 of FIG. 2, consistent with an illustrative embodiment. [Figure 7] 1 illustrates an example process for improving data quality in a manner that conserves computing resources, consistent with an example embodiment. [Figure 8] Provided is a functional block diagram of a computer hardware platform that can be used to implement a computing device specifically configured to host a dynamic discovery and modification engine. [Figure 9] Consistent with an illustrative embodiment, a cloud computing environment is provided. [Figure 10]10 is a set of functional abstraction layers according to the cloud computing environment of FIG. 9 , consistent with an illustrative embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] Overview In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the relevant teachings. However, it should be apparent that the present teachings may be practiced without those details. In other instances, well-known methods, procedures, components, and / or circuits have been described at a relatively high-level, without detail, in order to avoid unnecessarily obscuring aspects of the present teachings.

[0028] The present disclosure generally relates to systems and computerized methods for dynamic discovery and correction of data quality problems to improve computational efficiency and accuracy. The concepts discussed herein provide for improved computational efficiency and accuracy for computing devices that process large amounts of data. Reference is now made in detail to the examples illustrated in the accompanying drawings and discussed below.

[0029] FIG. 1 is a block diagram of an example architecture 100 of a graph-based data quality synthesizer system consistent with one example embodiment. There is an input dataset 102, which may represent raw data stored in a data repository representing big data. In various embodiments, the input dataset may be homogenous (e.g., of similar type) or heterogeneous in that it originates from different sources and includes different formats. The input dataset is received by a schema inference module configured to infer a schema. As a non-limiting example, the schema may include classifying the data type of each field. For example, column 1 of a table is a numeric column, column 2 of a table is a Boolean column, etc. Understanding the schema may help identify checks that are applicable to that column.

[0030] In one embodiment, the schema inference module 104 can determine what each row or column or both in the input data set means. In various embodiments, different methods can be used. In one example, a machine learning model in the schema inference module 104 learns the schema and automatically provides it to the data quality synthesizer 110 along with the input data set. In another embodiment, there is a "human in the loop" in that a computing device in the architecture 100 receives the schema from an administrator of the input database 102.

[0031] The data quality synthesizer module 110 includes a graph initializer 112, an executor 114, a terminator 116, an expander 118, and an execution information repository 120. The graph initializer 112 is configured to initialize a first set of nodes in a graph to perform validation on the data (e.g., the input database 102). As used herein, a node is a set of checks to be performed on the data. Each node in the graph represents a data quality check and a corresponding improvement action(s) to transform the input data set, which are discussed in more detail below.

[0032] The input to the graph initializer 112 is the input dataset 102 with annotated schema from the schema inference module 104. The output of the graph initializer 112 is a list of validation nodes that act as a first layer in the data quality graph. A set of checks (e.g., domain-independent checks, e.g., null value check, infinity value check, column-wise constant value check, number of zeros in a column, column-wise unique value check, duplicate row check, duplicate column name, duplicate column value check, low variance column check, etc.) are performed, which can be based on the schema information. For example, different types of checks are performed based on the annotated schema 104 of the input dataset 102.

[0033] The Executor module 114 receives the output from the graph initializer 112 and executes the operations defined by the nodes in the graph. In other words, different checks defined by those nodes are performed on the data. In one embodiment, single-node vs. multi-node execution is based on the size of the dataset. In one embodiment, as shown in FIG. 2, stage 1 can be visualized as a graph where there are two types of nodes: validators (checks) and operations (improvement functions). Each validator node can be connected to one or more operator nodes. While the creation of the graph is done in the graph initializer, the execution of the graph is done by the Executor module (any graph traversal strategy can be used, such as depth-first search, by traversing from the start node to each validator and operator as defined by the graph).

[0034] In one embodiment, if a problem is identified (e.g., the output information from the executor module does not yield results that fall within a predetermined criteria), a notification is sent to the expander module 118 via the terminator module 116. The expander module 118 is configured to select the next set of nodes to be added to the graph based on the current output of the terminator from the executor module 114. The output of the terminator module 116 determines the next set of nodes to be considered. Thus, the expander module 118 receives the output of the terminator. For example, if the data generated by the executor module 114 is insufficient (i.e., the generated data does not meet the termination criteria of the terminator module), the graph terminator 116 asks the expander module 118 to provide additional nodes for the graph. Finally, the expander module 118 can also ask for a validation library 140 that provides various validation checks via a validation check provisioner module 142. The validation checks may include general checks 144 , AI checks 146 , chronology checks 148 , and language (eg, domain) checks 150 .

[0035] For example, generic checks generally look at the structural features and numerical consistency of data, not its content (e.g., number of rows, number of columns, similarity between rows and / or columns, missing data, zero values, negative values, etc. Generic checks do not apply to a specific data type, but can be used across many domains and data modalities. As for linguistic checks, they concern checks that are applicable to a particular domain or AI problem, given data.

[0036] AI checks 146 are nuanced in that they look at the content of the data and determine if it fits the field of operation. Any outliers are therefore identified. For example, operators for AI checks can be used to remove or process the outliers.

[0037] With respect to time series check 148, modalities of the data, such as seasonality or migration trends for a particular domain, are determined. For example, a time series as discussed herein may relate to time-ordered data where each data value in a row is associated with a valid timestamp.

[0038] Linguistic checks 150, sometimes referred to herein as domain checks, filter out checks that are not relevant to a particular domain, thereby significantly reducing the number of checks performed. Linguistic checks 150 provide a tiered selection of checks tailored to the particular field of the operation. One or more of these checks 144-150 can be selected based on the schema of the input data set.

[0039] In one embodiment, each iteration of execution by the executor module 114 is stored in the execution information repository 120. The execution information repository 120 represents a storage of all data quality checks and actions taken for each node in the graph. For example, this repository can be used to automatically learn the functions of graph expanders and graph terminators.

[0040] In this manner, the execution information repository is dynamically updated after each execution of the set of nodes. The terminator module 116 is configured to determine when to terminate this iterative process of adding new nodes to the graph. The iterative process continues until the terminator 116 considers that the information provided by the executor module 114 meets a predefined criterion. For example, the number of successful tests reaches a predefined threshold.

[0041]

[0023] Referring now to Figure 2, a high level block diagram of a validation node 200 is provided, consistent with one exemplary embodiment. In the example of Figure 2, validation node 200 includes four stages: stage 1 (210), stage 2 (230), stage 3 (240), and stage 4 (250). Node 200 represents a self-contained module that detects and operates on an input data set 202(D). The input to validation node 200 is raw data from input data set 202(D) that has quality issues. The output is a corrected data set, represented by block 260(D'). The architecture of the system of Figure 2 is discussed in more detail below.

[0042] In stage 1 (210), the raw data set is received from the input data set 202 by a validator object 212 where logical checks are performed. As used herein, logical checks include the system checking for a given condition and if the condition is met, will return a result according to the logic. There are multiple operators O1, O2, O3 where different types of data transformations can be performed to create new versions of the data represented by D1, D2, and D3. Although three operators are illustrated as an example, it should be understood that any number of operators can be used based on the desired resolution.

[0043] Reference is now made to FIG. 3, which provides a block diagram of stage 1 of FIG. 2, consistent with one exemplary embodiment. The primary logical check of the raw data is performed by a validator block 310. The input to the validator block 310 is raw data 302, which may contain potential errors. The validator block 310 is configured to detect anomalies in the raw data. If anomalies are detected in the data, the validator 310 may perform corrective operations on the received raw data 302.

[0044] The validators can choose different correction actions, referred to herein as operators (e.g., 320(1)-320(3)). For example, validator 310 can be a detector, such as a missing value detector, a duplicate value detector, etc. The corresponding operators 320(1)-320(3) are configured to provide the appropriate correction action, e.g., removing rows, removing columns, and / or performing data imputation.

[0045] The output of each of these correction operations 320(1)-320(3) is a new version of the dataset with potential problems removed, represented by D1-D3 (330(1)-330(3)). Referring again to FIG. 2, one notable aspect of the Data Quality Advisor (DQA) is that validators and possible operators are linked. DQA refers to the entire system described herein. Each validator has a pointer to a set of callable operators O1-O3 that can be invoked if validation fails. This allows for an automated process of corrections once an anomaly is detected.

[0046] In stage 2 (230), for each new version of data generated, a data quality metric (DQM) (234(1)-234(4)) for each of them is generated using a corresponding internal quality evaluator (IQE) (232(1)-232(4)). In various embodiments, the IQE can be understood as a common block or multiple blocks. The parameters can be different for each path. The metrics can also be user recommended. Each IQE is named to indicate the comparison that is made, for example, IQE 232(1) is a comparison of data sets D1 and D, whereas 232(2) is a comparison of data sets D2 and D, etc. The content of each DQM provides a summary of various characteristics of the data in multiple dimensions. The content can have multiple "quality scores", for example, automatic modeling fit, stability detector, cohort analyzer, similarity detector, etc. The content of each DQM may further include gain or change information of data from the previous version. Note that the raw data from the input data set 202(D) is also passed through its corresponding IQE 232(4) and IQE 234(4) for later comparison. In one embodiment, the amount of data may be reduced if the operator deletes all rows where missing values ​​exist to improve the missing values. In that case, V is the missing value check, O is the deletion of all rows, and the IQE provides a comparison of how much data reduction has occurred.

[0047] Referring now to FIG. 4, a block diagram of the path of stage 2 of FIG. 2 is provided, consistent with one exemplary embodiment. Quality metrics of incoming data 402 and new data are generated by an internal quality evaluator (IQE) 410. The IQE 410 is used to find an intrinsic quality metric(s) for a given data set 402. As shown in FIG. 4, any data set D402 can go to the IQE 410 for outputting a data quality metric (DQM) 420. In various embodiments, the IQE 410 can use different techniques, either independently or together, to make an assessment of different quality metrics. For example, the IQE 410 can use an automatic modeling fit to make an assessment of the ability of the data to fit a regression model or a classification model or both. The IQE 410 can use a stability detector to detect how statistically stable the data 402 is. Additionally, a cohort analyzer can be used to determine how many clusters exist in the data 402. Symmetry detection can be used to determine whether the distribution of data around the mean is symmetric. Additionally, similarity detection and other techniques can be used.

[0048] The output of the IQE 410 is a Data Quality Metric (DQM) 420, which in one embodiment provides two main types of information. First, the DQM 420 provides a summary of various characteristics of the data in multiple dimensions (e.g., multiple statistics or multiple types of quality metrics discussed herein). The IQE 410 has multiple quality scorers, such as, without limitation, an auto-model fit, a stability detector, a cohort analyzer, a similarity detector, etc., as shown in FIG. 4. Second, the DQM 420 can provide gain or change information of the data from a previous version.

[0049] Referring again to FIG. 2, in stage 3 (240), each of the transformed data from its corresponding DQM is aggregated (e.g., compared) by its corresponding metric aggregator block 242(1) and 242(2), respectively, with the original raw data from input data set 202(D) that was processed by IQE 232(4) and DQM 234(4). For example, various data quality metrics (DQMs) are compared to see how much change is observed in the newly generated data from the operator versus the original data. The purpose of doing this is to evaluate the effectiveness of the checks and operators. Because the DQMs are arrays, the values ​​are aggregated before being compared.

[0050] In stage 4, the aggregated scores are sent to a decision module 252 to select an appropriate operator to use. This decision can be based on various criteria, which can include, without limitation, administrator selected, recommendation based, and / or automated. In one embodiment, each metric aggregator block 242 provides a measure of the gap between the data provided by the path from stage 2 and the data provided by the output of DQM 234(4). In this manner, each metric aggregator can determine which data set to use (e.g., D1, D2, or D3). The larger the gap between that path and the output from DQM 234(4), the more the original raw data D has been modified, i.e., improved. In this regard, the stage 4 decision module 252 selects a data set (e.g., D1, D2, or D3) based on the scores and decision criteria. In one embodiment, the larger the gap between the modified data and the original data set measured by the metric aggregator in each path of the data, the more desirable the data set is. However, in one embodiment, if the gap exceeds a predefined threshold, the modified data set is disqualified because it may be based on errors. Therefore, the judgment module 252 can select the data set (D1, D2, or D3) based on the largest gap but smaller than the predefined threshold. In other words, the most appropriate gap is selected. The gap should be sufficient in the positive direction. Secondly, the gap should not be too large so that the data is not similar to the original data based on the predefined criteria. Therefore, multiple DQMs (sequences of statistics) are compared using sequence comparison and gap calculation methods. From these, knowledge can then be obtained about the gap between the new data and the raw data, because the DQMs are representative of the data on which they are calculated. As mentioned above, the gap between the two data sets should be appropriate.

[0051] Referring now to FIG. 5, a block diagram of the pathway of stage 3 of FIG. 2 is provided, consistent with one exemplary embodiment. Metrics for the new dataset 520(1) and the old dataset 520(2) are compared by a metric aggregator block 530 to determine which operation is more effective for improving the dataset. The larger the gap between the new dataset 520(1) and the old dataset 520(2), the higher the score. Such a comparison of data metrics between the two datasets can be performed by aggregation. Techniques used in aggregation can include, without limitation, averaging, maximization, weighted averaging, threshold sum, custom aggregation, average of the maximum, maximum of the average, feature bagging, etc. In various embodiments, the result can be an aggregated metric 540 or a single similarity / distance score 550, or both, to characterize the improvement of the data.

[0052] Figure 6 is a block diagram of stage 4 of Figure 2, consistent with an example embodiment. Decision module 630 is configured to select between data sets 620(1) and 620(2) based on the scores and predefined decision criteria discussed in the context of Figure 5. In various embodiments, the defined criteria can be (i) automated (e.g., aggregated metrics for each operation are calculated in stage 3 and used to make a decision at the node that the operator ultimately uses), (ii) administrator-selected (e.g., letting the administrator make the decision on which operator to select), or (iii) recommendation-based (e.g., learning over time about which operations work best for given data), or all of these.

[0053] Example Process With the foregoing overview of the exemplary architecture 100 and the validator node 200, it may now be useful to consider a high-level discussion of an exemplary process. To that end, FIG. 7 illustrates an exemplary process for improving data quality so as not to waste computational resources. The process 700 is illustrated as a collection of blocks in a logical flow chart, which represent a sequence of operations that may be implemented in hardware, software, or a combination thereof. In the context of software, the blocks represent computer-executable instructions that, when executed by one or more processors, perform the recited operations. In general, computer-executable instructions may include routines, programs, objects, components, data structures, etc., that perform the functions or means of an abstract data type. In each process, the order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks may be combined in any order and / or the process may be performed in parallel. For purposes of illustration, the process 700 is described with reference to the architecture 200 of FIG. 2.

[0054] At block 702 , the computing device receives the raw data set 202 .

[0055] At block 704, one or more data quality metric targets corresponding to the received raw data set are received. As used herein, the term "data quality metric target" refers to a combination of a metric and a setting, which may be provided by a user prior to running the system. This includes not only data quality metrics, but also termination criteria.

[0056] At block 708, the schema of the data set is determined.

[0057] At block 710, an initial set of validation nodes is identified based on the schema of the dataset.

[0058] At block 712, an initial set of validation nodes is executed.

[0059] At block 714, a next set of validation nodes are iteratively expanded and executed based on the schema of the database until a termination criterion is reached. If it is determined that the termination criterion has not been reached (i.e., "no" at decision block 716), the iterative process continues. However, if it is determined that the termination criterion has been reached (i.e., "yes" at decision block 716), the process proceeds to block 718, where a modified dataset of the raw dataset is provided based on the iterative execution of the initial and next sets of validation nodes.

[0060] An exemplary computer platform. As discussed above, functions related to automatically adapting tasks to relationships may be performed using one or more computing devices connected for data communication via wireless or wired communication as shown in FIG. 1 and in accordance with process 400 of FIG. 4. FIG. 8 provides an illustrative functional block diagram of a computer hardware platform 800 that may be used to implement a specially configured computing device capable of hosting a dynamic discovery and modification engine 840. Thus, the computer hardware platform 800 may communicate with various databases, knowledge bases, validation libraries, etc., as discussed herein. In particular, FIG. 8 illustrates a network or host computer platform 800 that may be used to implement an appropriately configured server.

[0061] The computer platform 800 may include a central processing unit (CPU) 804, a hard disk drive (HDD) 806, a random access memory (RAM) or read only memory (ROM) or both 808, a keyboard 810, a mouse 812, a display 814, and a communication interface 816, which are connected to the system bus 802.

[0062] In one embodiment, HDD 806 has capabilities including storing programs capable of performing various processes in the manner described herein, such as dynamic discovery and modification engine 840. Dynamic discovery and modification engine 840 can have various modules configured to perform different functions.

[0063] For example, there may be a schema inference module 842 that operates to receive the raw data set and infer its schema. There may be an initializer module 844 that operates to initialize a first set of nodes in a graph to perform validation on the raw data. There may be an executor module that operates to receive an output from the initializer 844 and execute operations defined by the nodes in the graph provided by the initializer 844. There may be a terminator module 850 that operates to determine the next set of nodes to be considered for execution and provide a termination signal when a criterion is met, as discussed herein. There may be an expander module that operates to select the next set of nodes to be added to the graph from the executor module 848 based on the current output of the terminator 850. There may be a validation module 856 that operates to provide various validation checks, including, without limitation, general checks, AI checks, time series checks, and language checks, as discussed herein.

[0064] Exemplary Cloud Platforms As discussed above, functionality related to matching tasks to relationships can include cloud. Although this disclosure includes detailed descriptions of cloud computing, it should be understood that implementation of the teachings described herein is not limited to a cloud computing environment. Rather, embodiments of the present disclosure can be implemented in conjunction with any other type of computing environment now known or later developed.

[0065] Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with a service provider. The cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0066] Its features are as follows.

[0067] On-Demand Self-Service: Cloud consumers can automatically and unilaterally provision computing capacity, such as server time and network storage, as they need it, without the need for human interaction with the provider of the service.

[0068] Pervasive Network Access: Functionality is available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin- or thick-client platforms (e.g., cell phones, laptops, and PDAs).

[0069] Resource Pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated on demand. Consumers are location independent in that they generally have no control or knowledge of the exact location of the resources provided to them, although location may be specified at a higher level of abstraction (e.g., country, state, or data center).

[0070] Rapid Elasticity: Capabilities can be rapidly and elastically provisioned and rapidly scaled out, and rapidly released and rapidly scaled in, sometimes automatically. To the consumer, the capacity available for provisioning often appears unlimited, and can be purchased in any quantity at any time.

[0071] Metering of Services: Cloud systems automatically control and optimize resource usage by using metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services being used.

[0072] The service model is as follows:

[0073] Software as a Service (SaaS): The functionality provided to the consumer is the use of the provider's applications running on a cloud infrastructure. These applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or individual application capabilities, with the expected exception of limited user-specific application configuration settings.

[0074] Platform as a Service (PaaS): The functionality offered to the consumer is the deployment onto a cloud infrastructure of applications the consumer creates or acquires, written using programming languages ​​and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does control the deployed applications and, in some cases, the configuration of the environment that hosts the applications.

[0075] Infrastructure as a Service (IaaS): The functionality provided to a consumer is to provision processing, storage, network, and other basic computing resources onto which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating systems, storage, deployed applications, and possibly limited control over the selection of network components (e.g., host firewalls).

[0076] The deployment model is as follows:

[0077] Private Cloud: The cloud infrastructure is operated exclusively for an organization. It can be managed by the organization or a third party and can exist on-premise or off-premise.

[0078] Community Cloud: Cloud infrastructure is shared by several organizations to support a specific community with common concerns (e.g., mission, security requirements, policies, and compliance considerations). It can be managed by those organizations or by a third party and can reside on-premise or off-premise.

[0079] Public Cloud: The cloud infrastructure is available to the general public or large industry groups and is owned by organizations that sell cloud services.

[0080] Hybrid Cloud: A cloud infrastructure is a hybrid of two or more clouds (private, community, or public) that remain unique entities but are tied together by standardized or proprietary technologies (e.g., cloud bursting for load balancing between clouds) that enable data and application portability.

[0081] Cloud computing environments are service-oriented and focus on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0082] 9, an exemplary cloud computing environment 950 is illustrated. As illustrated, the cloud computing environment 950 includes one or more cloud computing nodes 910 with which a local computing device used by a cloud consumer can communicate, such as, for example, a personal digital assistant (PDA) or cellular phone 954A, a desktop computer 954B, a laptop computer 954C, or an automobile computer system 954N, or a combination thereof. The nodes 910 can communicate with each other. The nodes 910 can be physically or virtually grouped in one or more networks, such as a private cloud, a community cloud, a public cloud, or a hybrid cloud, or a combination thereof, as described above (not shown). This enables the cloud computing environment 950 to provide Infrastructure as a Service, Platform as a Service, or Software as a Service, or a combination thereof, without the cloud consumer having to maintain resources on a local computing device. It will be understood that the types of computing devices 954A-N shown in FIG. 9 are intended to be merely exemplary, and that the computing node 910 and cloud computing environment 950 are capable of communicating with any type of computerized device (e.g., using a web browser) via any type of network or network-addressable connection or both.

[0083] 10, an exemplary set of functional abstraction layers provided by cloud computing environment 950 (FIG. 9) is shown. It should be understood in advance that the components, layers, and functions shown in FIG. 10 are intended to be merely exemplary, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0084] Hardware and software layer 1060 includes hardware and software components. Examples of hardware components include mainframe 1061, RISC (Reduced Instruction Set Computer) architecture based servers 1062, servers 1063, blade servers 1064, storage devices 1065, and networks and networking components 1066. In some embodiments, software components include network application server software 1067 and database software 1068.

[0085] The virtualization layer 1070 provides an abstraction layer that can provide the following examples of virtual entities: virtual servers 1071, virtual storage 1072, virtual networks including virtual private networks 1073, virtual applications and operating systems 1074, and virtual clients 1075.

[0086] In one example, the management layer 1080 can provide the functionality described below. Resource provisioning 1081 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 1082 provides cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources. In one example, these resources can include application software licenses. Security provides identity verification for cloud consumers and tasks, and protection for data and other resources. User portal 1083 provides access to the cloud computing environment for consumers and system administrators. Microservice level management 1084 provides allocation and management of cloud computing resources such that required microservice levels are met. Microservice level agreement (SLA) planning and fulfillment 1085 provides pre-positioning and procurement of cloud computing resources for predicted future needs according to SLAs.

[0087] The workload layer 1090 provides examples of functionality that can utilize a cloud computing environment. Examples of workloads and functionality that can be provided from this layer include mapping and navigation 1091, software development and lifecycle management 1092, virtual classroom instructional delivery 1093, data analytics processing 1094, transaction processing 1095, and a dynamic discovery and remediation engine 1096 as described herein. conclusion

[0088] The description of various embodiments of the present teachings is presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications, or technical improvements over the art found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0089] While the above describes what is believed to be the best mode or other examples, or both, it is understood that various modifications may be made therein, that the subject matter disclosed herein can be implemented in various forms and examples, and that the teachings can be applied to numerous applications, only some of which are described herein. It is intended by the following claims to claim any and all applications, modifications, and variations that fall within the true scope of the present teachings.

[0090] The components, steps, features, objects, benefits and advantages described herein are merely illustrative. None of them, or any description related thereto, is intended to limit the scope of protection. Although various advantages are described herein, it will be understood that not all embodiments necessarily include all advantages. Unless otherwise specified, it will be understood that all measurements, values, ratings, positions, dimensions, sizes and other specifications described herein, including within the scope of the following claims, are approximate and not exact. They are intended to have a reasonable range consistent with the functions to which they relate and which are customary in the art to which they pertain.

[0091] Numerous other embodiments are contemplated, including embodiments having fewer, additional, and / or different components, steps, features, objects, benefits and advantages, including embodiments in which the components and / or steps are arranged and / or ordered differently.

[0092] Aspects of the present disclosure are described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0093] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, whereby the instructions executed by the processor of the computer or other programmable data processing apparatus create means for performing the functions / acts specified in one or more blocks of the flowcharts or block diagrams, or both. These computer readable program instructions may also be stored in a computer readable medium that can direct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, whereby the computer readable storage medium having instructions stored thereon includes an article of manufacture including instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts or block diagrams, or both.

[0094] The computer program instructions may be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to generate a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device perform the functions / operations specified in one or more blocks of the flowcharts or block diagrams, or both.

[0095] The call flows, flowcharts, and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions shown in the blocks may occur in a different order than that shown in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or these blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that executes the specified functions or operations, or executes a combination of dedicated hardware and computer instructions.

[0096] Although the above has been described in conjunction with exemplary embodiments, it is understood that the term "exemplary" means merely as an example, not best or optimal. Except as immediately above, nothing described or shown is intended or should be construed to cause the dedication to the public of any component, step, feature, object, benefit, advantage, or equivalent, whether or not it is recited in the claims.

[0097] It will be understood that the terms and expressions used herein have the ordinary meanings as ascribed to such terms and expressions with respect to their corresponding respective fields of study and research, unless otherwise specifically stated herein. Relative terms such as first and second, etc., may be used merely to distinguish one entity or action from another, without necessarily requiring or implying any actual relationship or order between such entities or actions. The terms "comprise", "comprising", or any other variation thereof, are intended to cover a non-exhaustive inclusion, such that a process, method, article, or apparatus that includes a list of elements may include other elements not expressly listed or inherent in such process, method, article, or apparatus, rather than including only those elements. An element beginning with "a" or "an" excludes, without further constraint, the presence of additional identical elements within the process, method, article, or apparatus that includes the element.

[0098] The Abstract of the Disclosure is provided to enable the reader to quickly ascertain the nature of the technical disclosure. The Abstract is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Moreover, in the above Detailed Description, it will be seen that various features are grouped together in various embodiments to streamline the disclosure. This method of disclosure is not to be construed as reflecting an intention that the claimed embodiments have more features than are expressly recited in each claim. Rather, as the following claims reflect, the present subject matter never lies in all features of a single disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim representing separately claimed subject matter in its own right.

Claims

1. A processor; a storage device coupled to the processor; an engine stored in the storage device; A computing device comprising: Execution of the engine by the processor comprises: receiving a raw data set; receiving one or more data quality metric targets corresponding to the received raw data set; determining a schema of the raw data set; identifying an initial set of validation nodes based on the schema of the raw data set; executing said initial set of validator nodes; iteratively expanding and executing a next set of validation nodes based on the schema of the raw data set until a termination criterion is reached; providing a modified data set of the raw data set based on the iterative execution of the initial set and the subsequent set of validation nodes; configuring said computing device to perform operations including device.

2. The device of claim 1 , wherein each validation node includes a data quality check and one or more remediation actions.

3. Executing the validator nodes of the initial set of validator nodes Identifying all possible improvement actions for every data quality check; Transforming the raw data set according to each possible improvement action; calculating a plurality of data quality metrics (DQMs) to evaluate said transformation; The device of claim 1 , comprising:

4. The execution of the validation node is performing logical checks on the raw data set by a validator object to detect one or more anomalies in the raw data set; performing different data transformations on the raw data set by corresponding operators to correct the one or more detected anomalies, and creating a new version of the data for each data transformation; The device of claim 1 , comprising a first stage comprising:

5. The execution of the validator node is generating a data quality metric (DQM) for each new version of the data created by an internal quality evaluator (IQE) module; generating a DQM for the raw data set; The device of claim 4 comprising a second stage comprising:

6. 6. The device of claim 5, wherein each DQM of the second stage includes at least one of: (i) a summary of features in multiple dimensions of the corresponding new version of data created from the raw data set; or (ii) gain or change information of the corresponding new version of data created from the raw data set.

7. The execution of the validator node is performing a comparison of each DQM of each of the new versions of data created to the DQM of the raw data set against the raw data set and assessing an improvement from the raw data set; The device of claim 5 , further comprising a third stage comprising:

8. The execution of the validator node is selecting the operator for the new version of the created data that best meets the data quality metric objectives; The device of claim 7 , further comprising a fourth stage comprising:

9. The device of claim 8 , wherein the operator selected has a largest gap between its corresponding DQM and a DQM of the raw data set that is less than a predefined threshold.

10. Expanding the next set of validating nodes comprises: determining a validator node that best achieves the one or more received quality metric goals; determining the validating nodes based on mining an execution information repository to find all validating nodes that typically occur together; The device of claim 1 , comprising at least one of:

11. A computer-readable storage medium tangibly embodying computer-readable program code having computer-readable instructions that, when executed, cause a computing device to: receiving a raw data set; receiving one or more data quality metric targets corresponding to the received raw data set; determining a schema of the raw data set; identifying an initial set of validation nodes based on the schema of the raw data set; executing said initial set of validator nodes; iteratively expanding and executing a next set of validation nodes based on the schema of the raw data set until a termination criterion is reached; providing a modified data set of the raw data set based on iterative execution of the initial set and the subsequent set of validation nodes; performing a method for improving data quality in a manner that does not waste computational resources, A computer readable storage medium.

12. The computer-readable storage medium of claim 11 , wherein each validation node includes a data quality check and one or more remediation actions.

13. Executing the validator nodes of the initial set of validator nodes Identifying all possible improvement actions for each data quality check; Transforming the raw data set according to each possible improvement action; calculating a plurality of data quality metrics (DQMs) to evaluate said transformation; The computer readable storage medium of claim 11 .

14. The execution of the validation node is performing logical checks on the raw data set by a validator object to detect one or more anomalies in the raw data set; performing different data transformations on the raw data set by corresponding operators to correct the one or more detected anomalies, and creating a new version of the data for each data transformation; a first stage including: generating a data quality metric (DQM) for each new version of the data created by an internal quality evaluator (IQE) module; generating a DQM for the raw data set; a second stage including For each DQM of each of the new versions of data created, a comparison is performed against the raw data set to assess the improvement from the raw data set. a third stage including selecting the operator for the new version of the created data that best meets the data quality metric objectives; a fourth stage including The computer readable storage medium of claim 11 .

15. 15. The computer-readable storage medium of claim 14, wherein the operator selected has a largest gap between its corresponding DQM and a DQM of the raw data set that is less than a predetermined threshold.

16. Expanding the next set of validating nodes comprises: determining a validator node that best achieves one or more of the data quality metric objectives; determining the validating nodes based on mining an execution information repository to find all validating nodes that typically occur together; 12. The computer readable storage medium of claim 11, comprising at least one of:

17. a schema inference module configured to receive an unprocessed data set and determine a schema of the unprocessed data set; a data synthesizer module coupled to the schema estimation module and configured to receive one or more data quality metric targets from a knowledge base corresponding to the received raw data set; A system comprising: The data synthesizer module includes: an initializer module configured to identify an initial set of validation nodes based on the schema of the raw data set; an executor module coupled to the initializer module and configured to execute the initial set of validation nodes; an expander module coupled to the executor module and configured to iteratively expand and execute a next set of validation nodes based on the schema of the raw dataset until a termination criterion is reached; a terminator module coupled to the expander module and configured to iteratively determine the next set of validation nodes to be considered by the expander module and determine when to terminate the iterative determination; an information repository coupled to the executor and terminator modules and configured to provide a modified dataset of the raw dataset based on an iterative execution of the initial set and the subsequent set of validating nodes; Equipped with system.

18. 20. The system of claim 17, wherein each validation node includes a data quality check and one or more remediation actions.

19. The initial set of validating nodes may include: Identify all possible improvement actions for every data quality check, Transforming the raw data set with each possible improvement action; Computing a number of data quality metrics to evaluate the transformation The system of claim 17 configured to:

20. Each validator node: performing logical checks on the raw data set by a validator object to detect one or more anomalies in the raw data set; performing different data transformations on the raw data set by corresponding operators to correct the one or more detected anomalies, and creating a new version of the data for each data transformation; a first stage configured as follows: generating data quality metrics (DQMs) by an internal quality evaluator (IQE) module for each new version of the data created; Generate a DQM for the raw data set a second stage configured as follows: For each DQM of each new version of data created, a comparison is performed against the raw data set. a third stage configured as follows: Selecting the operator for the new version of the created data that best meets the data quality metric objectives. a fourth stage configured as follows: The system of claim 17 , comprising:

21. A method for improving data quality by computer information processing in a manner that does not waste computing resources, comprising: receiving a raw data set; receiving one or more data quality metric targets corresponding to the received raw data set; determining a schema of the raw data set; identifying an initial set of validation nodes based on the schema of the raw data set; executing said initial set of validator nodes; iteratively expanding and executing a next set of validation nodes based on the schema of the raw data set until a termination criterion is reached; providing a modified data set of the raw data set based on iterative execution of the initial set and the subsequent set of validation nodes; 4. A computer-implemented method comprising:

22. 22. The computer-implemented method of claim 21, wherein each validation node includes a data quality check and one or more remediation actions.

23. Executing the validator nodes of the initial set of validator nodes Identifying all possible improvement actions for each data quality check; Transforming the raw data set according to each possible improvement action; calculating a plurality of data quality metrics (DQMs) to evaluate said transformation; 22. The computer-implemented method of claim 21, comprising:

24. The execution of the validation node is performing logical checks on the raw data set by a validator object to detect one or more anomalies in the raw data set; performing different data transformations on the raw data set by corresponding operators to correct the one or more detected anomalies, and creating a new version of the data for each data transformation; a first stage including: generating a data quality metric (DQM) for each new version of the data created by an internal quality evaluator (IQE) module; generating a DQM for the raw data set; a second stage including performing a comparison of each DQM of each new version of data created against the raw data set; a third stage including selecting the operator for the new version of the created data that best meets the data quality metric objectives; a fourth stage including 22. The computer-implemented method of claim 21, comprising:

25. Expanding the next set of validating nodes comprises: determining a validator node that best achieves the one or more received quality metric goals; determining the validating nodes based on mining an execution information repository to find all validating nodes that typically occur together; 22. The computer-implemented method of claim 21, comprising at least one of:

26. A computer program comprising computer readable program instructions causing a computer to carry out a method according to any one of claims 21 to 25.

Citation Information

Patent Citations

  • Data cleansing system and program

    JP2013175096A

  • Data quality analysis and cleansing of source data with respect to a target system

    US20160070725A1

  • Methods and Systems for Automating Clinical Data Mapping and Transformation

    US20190286620A1