Dynamic discovery and correction of data quality problems

Abstract

A computing device, method, and system to improve data quality to save computing resources is disclosed. A computing device receives a raw data set. One or more data quality metric targets corresponding to the received raw data set are received. A pattern of the data set is determined. An initial set of validation nodes is identified based on the pattern of the data set. The initial set of validation nodes is executed. A next set of validation nodes is iteratively expanded and executed based on the pattern of the data set until a termination criterion is reached. A revised data set of the raw data set is provided based on the initial set of validation nodes and the iterative execution of the next set of validation nodes.

Description

Technical Field

[0001] This invention generally relates to computer systems, and more particularly to automatic dynamic data quality assessment and correction. Background Technology

[0002] From the First Industrial Revolution, spearheaded by mechanization powered by water and steam, to the Second Industrial Revolution, which enabled mass production and assembly lines through electricity, the current Fourth Industrial Revolution is transforming what began with the Third Industrial Revolution, which introduced computers and automation. The Fourth Industrial Revolution enhances the Third Industrial Revolution by leveraging intelligent and autonomous systems driven by big data and machine learning. With the increasing amount of available data, the quality of this vast amount of information is a crucial consideration for computers using that data for machine learning.

[0003] In recent years, with the increasing adoption of Industry 4.0, many industrial manufacturing processes are closely monitored in real time by numerous sensors. The quality of the data being used by computing devices has a significant impact on their efficiency, computational accuracy, and the amount of computing resources used. Such specially configured computing devices can be used, for example, to build data-driven artificial intelligence (AI) solutions to predict machine failures, anomaly detection, survival analysis, and more in Industry 4.0. However, current architectures face various challenges when processing large amounts of real-world Internet of Things (IoT) sensor data. For example, the data can be very large (e.g., big data) and involves significant computational resources of the computing device, including processing power, memory, network bandwidth, and time. The data can be noisy, contain missing values, have irregular sampling, and so on. Furthermore, the data processed by computing devices can be heterogeneous and unstructured, collectively referred to here as heterogeneous (e.g., it can involve time series, text, images, audio, different sampling rates, etc.).

[0004] At least one estimate suggests that the data preprocessing phase consumes 80% of the time in the AI ​​model development cycle during the AI ​​development phase. In other words, most of the time involved in analyzing data is spent preparing it into a suitable form from which an AI model can be prepared. Some challenges in automating the data preprocessing phase include the potential uniqueness of each dataset and the resources involved in analyzing its features. Existing methods currently do not provide tools for thorough data cleaning and preprocessing. Consequently, the efficiency, accuracy, and quantity of resources used by computing devices are adversely affected.

[0005] There are many frameworks and defined end-to-end solutions available for AI model development, including machine learning platforms such as sklearn, tensorflow, Tpot, and H2O. However, there is currently no known end-to-end framework and / or library that can effectively cleanse data.

[0006] Therefore, this paper provides a method and system for a computerized approach to dynamically discovering and correcting data quality problems. In one aspect, the teachings of this paper provide automation in the data preprocessing stage. In one embodiment, the iterative process is supervised due to interaction and collaboration from AI practitioners, allowing them to perform quality checks on heterogeneous IoT data. A wide variety of checks are provided through the use of installable function libraries. The data quality assessment and verification discussed in this paper can be highly customized. In one aspect, the system described in this paper is modular, allowing for flexible processes for data quality assessment. For example, there is the flexibility to include various checks and even link them together in a way that suits the data being processed. Summary of the Invention

[0007] According to various embodiments, a computing device, a non-transitory computer-readable storage medium, and a computer-implemented method and system are provided for improving data quality to save computing resources. The process includes receiving a raw dataset; receiving one or more data quality metrics corresponding to the received raw dataset; determining a schema for the dataset; determining an initial set of validation nodes based on the schema; executing the initial set of validation nodes; iteratively expanding and executing the next set of validation nodes based on the schema of the dataset until a termination criterion is met; and providing a corrected dataset of the raw dataset based on the iterative execution of the initial set and the next set of validation nodes.

[0008] In one embodiment, each verification node includes a data quality check action and one or more remedial actions.

[0009] In one embodiment, the execution of verification nodes in the initial set of verification nodes includes: identifying all possible remedial actions for any data quality check, transforming the data using each possible remedial action, and calculating multiple data quality metrics (DQM) to evaluate the transformation.

[0010] In one embodiment, the execution of the verification node includes a first phase, which includes: performing a logical check on the original dataset by a verifier object to detect one or more anomalies in the original dataset; and performing different data transformations on the original dataset by corresponding arithmetic units to generate a new version of the data for each data transformation, thereby correcting one or more detected anomalies.

[0011] In one embodiment, the verification node includes a second phase, which includes: generating a data quality metric (DQM) by an internal quality evaluator (IQE) module for each new version of the data produced, and generating a DQM for the original dataset.

[0012] In one embodiment, each DQM in the second stage includes at least one of the following: (i) a summary of characteristics in multiple dimensions of the corresponding new version of the data generated from the original dataset; or (ii) gain or change information of the corresponding new version of the data generated from the original dataset.

[0013] In one embodiment, the execution of the verification node includes a third phase, which includes performing a comparison with the original dataset for each DQM of each new version of the data produced and the DQM of the original dataset to evaluate the improvement relative to the original dataset.

[0014] In one embodiment, the execution of the verification node includes a fourth phase, which includes: selecting the processor that best satisfies the data quality metric objectives for the generated new version of the data.

[0015] In one embodiment, the selected processor has the highest gap between its corresponding DQM and the original dataset's DQM below a predetermined threshold in order to maintain similarity with the original data.

[0016] In one embodiment, the next set of extended verification nodes includes at least one of the following: determining the verification node that best achieves one or more received quality metric objectives; or determining the verification node based on mining the execution information repository to find all verification nodes that typically appear together.

[0017] According to one embodiment, a system includes a pattern reference module configured to receive a raw dataset and determine a pattern of the raw dataset. A data synthesizer module is coupled to the pattern inference module and configured to receive one or more data quality metrics corresponding to the received raw dataset from a knowledge base. The data synthesizer module includes an initializer module configured to identify an initial set of validation nodes based on the pattern of the dataset. The data synthesizer module also includes an executor module coupled to the initializer module and configured to execute the initial set of validation nodes. The data synthesizer module further includes an expander module coupled to the executor module and configured to iteratively expand and execute the next set of validation nodes based on the pattern of the dataset until a termination criterion is met. The data synthesizer module includes a terminator module coupled to the expander module and configured to iteratively determine the next set of validation nodes to be considered by the expander module and determine when to terminate the iterative determination. An information store is also coupled to the executor module and the terminator module and configured to provide a corrected dataset of the raw dataset based on the iterative execution of the initial and next sets of validation nodes.

[0018] In one embodiment, each verification node includes a data quality check action and one or more remedial actions.

[0019] In one embodiment, the initial set of verification nodes is configured to: identify all possible remedial actions for any data quality check, transform the data using each possible remedial action, and compute multiple data quality metrics to evaluate the transformation.

[0020] In one embodiment, each verification node includes a first phase configured to: perform a logical check on the original dataset by a verifier object to detect one or more anomalies in the original dataset, and perform different data transformations on the original dataset by a corresponding arithmetic unit to generate a new version of the data for each data transformation, thereby correcting one or more detected anomalies.

[0021] In one embodiment, there is a second stage configured to generate a data quality metric (DQM) by an internal quality evaluator (IQE) module for each new version of the data produced, and to generate a DQM for the original dataset.

[0022] In one embodiment, there is a third level configured to perform a comparison with the original dataset for each DQM of each new version of the data generated to evaluate the improvement relative to the original dataset.

[0023] In one embodiment, there is a third stage configured to select the processor that best satisfies the data quality metric objectives of the resulting new version of the data.

[0024] By utilizing the concepts discussed herein, technological improvements are provided in the computational efficiency and accuracy of computing devices processing big data. These and other features will become apparent from the following detailed description of illustrative embodiments of the invention, taken in conjunction with the accompanying drawings. Attached Figure Description

[0025] The accompanying drawings are illustrative embodiments. They do not show all embodiments. Other embodiments may be used alternatively or as an alternative. Details that may be obvious or unnecessary may be omitted to save space or for more efficient illustration. Some embodiments may be implemented with additional components or steps and / or without all components or steps shown. When the same numbers appear in different drawings, they refer to the same or similar components or steps.

[0026] Figure 1 This is a block diagram of an example architecture for a graph-based data quality synthesizer system consistent with the illustrative embodiments.

[0027] Figure 2A high-level block diagram of the verification node is provided, consistent with the illustrative embodiments.

[0028] Figure 3 Provided consistent with the illustrative embodiments Figure 2 The flowchart for Phase 1.

[0029] Figure 4 Provided consistent with the illustrative embodiments Figure 2 A flowchart of the path in stage 2.

[0030] Figure 5 Provided consistent with the illustrative embodiments Figure 2 A flowchart of the path in stage 3.

[0031] Figure 6 It is consistent with the illustrative embodiments. Figure 2 The flowchart for stage 4.

[0032] Figure 7 An illustrative process consistent with the illustrative embodiments is presented, relating to improving data quality to save computing resources.

[0033] Figure 8 A functional block diagram of a computer hardware platform is provided, which can be used to implement a specially configured computing device for a managed dynamic discovery and correction engine.

[0034] Figure 9 A cloud computing environment consistent with the illustrative embodiments is provided.

[0035] Figure 10 It is consistent with the illustrative embodiments. Figure 9 A set of functional abstraction layers for a cloud computing environment. Detailed Implementation

[0036] Overview

[0037] In the following detailed description, numerous specific details are illustrated by way of example to provide a thorough understanding of the relevant teachings. However, it should be understood that the teachings of the invention can be practiced without these details. In other examples, well-known methods, procedures, components, and / or circuits have been described at a relatively high level without detail to avoid unnecessarily obscuring aspects of the teachings.

[0038] This disclosure generally relates to systems and computerized methods for dynamically identifying and correcting data quality problems to improve computational efficiency and accuracy. By utilizing the concepts discussed herein, better computational efficiency and accuracy are provided for computing devices processing large amounts of data. Examples are now shown in the accompanying drawings and discussed below in detail.

[0039] Example Architecture

[0040] Figure 1 This is a block diagram of an example architecture 100 of a graph-based data quality synthesizer system consistent with the illustrative embodiment. There exists an input dataset 102 that can represent raw data stored in a data repository representing large amounts of data. In various embodiments, the input dataset can be homogeneous (e.g., similar in type) or dissimilar, as it originates from various sources and includes different formats. The input dataset is received by a schema inference module 104 configured to infer a schema. As a non-limiting example, the schema may include a marker indicating the data type of each field. For example, column 1 of the table is a numeric column, column 2 of the table is a boolean column, etc. Understanding the schema will help identify checks applicable to that column.

[0041] In one embodiment, pattern inference module 104 can determine what each row and / or column in the input dataset represents. Different methods may be used in various embodiments. In one example, the machine learning model of pattern inference module 104 learns a pattern and automatically provides that pattern, along with the input dataset, to data quality synthesizer 110. In another embodiment, there is a "human in the loop" because the computing device of architecture 100 receives patterns from an administrator of input dataset 102.

[0042] The data quality synthesizer module 110 includes a graph initializer 112, an executor 114, a terminator 116, an expander 118, and an execution information store 120. The graph initializer 112 is configured to initialize a first set of nodes in the graph for performing validation on the data (i.e., the input dataset 102). As used herein, a node is a set of checks to be performed on the data. Each node in the graph represents a data quality check and a corresponding remedial action to transform the input dataset, which will be discussed in more detail later.

[0043] The input to the graph initializer 112 is an input dataset 102 with its annotated pattern from the pattern inference module 104. The output of the graph initializer 112 is a list of validation nodes that serve as the first layer in the data quality graph. A set of checks (e.g., domain-agnostic checks such as null value checks, infinite value checks, column-by-column constant value checks, number of zeros in a column, column-by-column unique value checks, duplicate row checks, duplicate column name checks, duplicate column value checks, low-variance column checks, etc.) are performed, which can be based on pattern information. For example, different types of checks can be performed based on the annotated pattern 104 of the input dataset 102.

[0044] Executor module 114 receives output from graph initializer 112 and performs operations defined by the nodes in the graph. In other words, it performs various checks on the data, as defined by their nodes. In one embodiment, single-node operations are performed relative to multi-node operations based on the size of the dataset. In one example, such as... Figure 2As shown, Phase 1 can be visualized as a graph containing two types of nodes: validators (checkers) and operations (remedial functions). Each validator node can be connected to one or more operator nodes. Graph creation occurs in the graph initializer, but graph execution is performed by the executor module (by traversing from the starting node to each validator and operator defined by the graph, any graph traversal strategy, such as depth-first search, can be used).

[0045] In one embodiment, if a problem is identified (e.g., the output from the actuator module does not provide results within predetermined criteria), a notification is sent to the expander module 118 via the terminator module 116. The expander module 118 is configured to select the set of next nodes to be added to the graph based on the current output from the terminator module 114. The output of the terminator module 116 determines the set of next nodes to consider. Therefore, the expander module 118 receives the output of the terminator. For example, if the data generated by the actuator module 114 is insufficient (i.e., the generated data does not meet the termination criteria of the terminator module), the graph terminator 116 invokes the expander module 118 to provide additional nodes to the graph. For this purpose, the expander module 118 may request the verification library 140, which provides various verification checks through its verification check provider module 142. Verification checks may include general checks 144, AI checks 146, time series checks 148, and language (e.g., domain) checks 150.

[0046] For example, general checks typically examine the structural characteristics and value consistency of data, rather than its content. Examples include the number of rows, columns, similarity between rows and / or columns, missing data, zero values, negative values, etc. General checks are those that are not applicable to a specific data type and can be used in many domains and data modalities. Regarding language checks, they involve checks applicable to specific domains or AI problems with given data.

[0047] A more subtle difference between AI inspections (146) and other methods is that they examine the content of the data and determine whether the content matches the operated field. This allows them to identify any outliers. For example, outliers can be removed or processed using the arithmetic logic unit (ALU) used for AI inspections.

[0048] Regarding time series examination 148, it's about determining the modality of the data, such as seasonality or movement trends in a specific domain. For example, the time series discussed in this article could involve time-ordered data, where each data value in a row is associated with a valid timestamp.

[0049] Language checks 150 (sometimes referred to herein as domain checks) filter out checks that are irrelevant to a specific domain. This can significantly reduce the number of checks performed. Language checks 150 provide a progressive selection of checks tailored to a specific operational domain. One or more of these checks 144 to 150 can be selected based on patterns in the input dataset.

[0050] In one embodiment, each iteration of the execution of executor module 114 is stored in execution information store 120. Execution information store 120 represents a storage of all data quality checks and actions performed on each node in the graph. For example, this store can be used to automatically learn the functions of graph expanders and graph terminators.

[0051] Therefore, the execution information store is dynamically updated after each execution of a set of nodes. The terminator module 116 is configured to determine when to terminate this iterative process of adding a new node to the graph. The iterative process continues until the terminator 116 deems the information provided by the executor module 114 to meet predetermined criteria. For example, the number of successful tests reaches a predetermined threshold.

[0052] Now for reference Figure 2 It provides a high-level block diagram of the verification node 200, consistent with the illustrative embodiment. Figure 2 In the example, verification node 200 comprises four phases: phase 1 (210), phase 2 (230), phase 3 (240), and phase 4 (250). Node 200 represents a self-contained module for detecting and manipulating the input dataset 202 (D). The input to verification node 200 is the raw data from the input dataset 202 (D) with quality issues. The output is the corrected dataset, represented by block 260 (D'). This is discussed in more detail below. Figure 2 The system architecture.

[0053] In phase 1 (210), the validator object 212 receives raw data from the input dataset 202, where logical checks are performed. As used herein, logical checks involve the system checking a given condition, and if that condition is met, it returns a result according to logic. Multiple operators O1, O2, O3, denoted by D1, D2, and D3, are present, which perform different types of data transformations to produce new versions of the data. Although three operators are shown in the example, it is understood that any number of operators can be used based on the desired resolution.

[0054] Now for reference Figure 3 It provides the same as the illustrative embodiments. Figure 2 The block diagram for Phase 1 is shown below. The main logical checks on the raw data 302 are performed by the validator block 310. The input to the validator block 310 is the raw data 302, which may include potential errors. The validator block 310 is configured to detect anomalies in the raw data. If an anomaly is detected in the data, the validator 310 can perform a correction operation on the received raw data 302.

[0055] The validator may select different correction actions, referred to herein as operators (e.g., 320(1)) to 320(3)). For example, validator 310 may be a detector, such as a missing value detector, a duplicate value detector, etc. The corresponding operators 320(1) to 320(3) are configured to provide appropriate correction actions, such as removing rows, removing columns, and / or performing data imputation.

[0056] The output of each of these correction operations 320(1) to 320(3) is a new version of the dataset in which the underlying problems represented by D1 to D3 (330(1) to 330(3)) have been resolved. (Return to Reference) Figure 2 A significant aspect of Data Quality Assurance (DQA) is the linking of validators and potential operators. DQA refers to the entire system described here. Each validator has pointers to a set of callable operations O1toO3, which can be invoked if validation fails. This allows for automatic management of corrections once an anomaly is detected.

[0057] In phase 2 (230), for each of the newly generated versions of data, a data quality metric (DQM) (234(1)) to 234(4)) is generated using the corresponding internal quality evaluator (IQE) (232(1)) to 232(4)). In various embodiments, the IQE may be implemented as a common block or multiple blocks. The parameters may be different for each path. The metric may also be recommended by the user. Each IQE is invoked differently to describe the comparison being performed, for example, IQE 232(1) compares datasets D1 and D, while 232(2) compares datasets D2 and D, and so on. The content of each DQM provides a summary of various characteristics of the multidimensional data. The content may have multiple "quality scorers", such as automated modeling fit, stability detector, group analyzer, similarity detection, etc. The content of each DQM may also include gain or change information from previous versions of data. Note that the raw data from input dataset 202(D) is also processed by its corresponding IQE 232(4) and DQM 234(4) for later comparison. In one embodiment, the amount of data can be reduced if the operator used to remedy missing values ​​removes all rows containing missing values. In this case, V is the missing value check, O is the removal of all rows, and IQE provides a comparison of how much data reduction has occurred.

[0058] Now for reference Figure 4 It provides the same as the illustrative embodiments. Figure 2 A flowchart of the path in Phase 2. The quality metrics for input data 402 and new data are generated by the Internal Quality Evaluator (IQE) 410. IQE 410 is used to find inherent quality metrics for a given dataset 402. (Example...) Figure 4As shown, any dataset D 402 can be passed to IQE 410 to output a data quality metric (DQM) 420. In various embodiments, IQE 410 can independently or in combination with different techniques to evaluate different quality metrics. For example, IQE 410 can use automated modeling fitting to evaluate the data's ability to fit regression and / or classification models. IQE 410 can use a stability detector to detect how statistically stable the data 402 is. Furthermore, it can use a group analyzer to determine how many clusters exist in the data 402. Symmetry detection can be used to determine whether the data distribution around the mean is symmetrical. Further, similarity detection and other techniques can be used.

[0059] The output of IQE 410 is Data Quality Metric (DQM) 420, which in one embodiment provides two main types of information. First, DQM 420 provides a summary of various characteristics of the data across multiple dimensions (e.g., multiple statistics or various types of quality metrics discussed herein). IQE 410 may have multiple quality scorers, such as, but not limited to, automated modeling fits, stability detectors, group analyzers, similarity detectors, etc. Figure 4 As shown. Secondly, the DQM 420 can provide gain or change information from data from previous versions.

[0060] Return to reference Figure 2 In stage 3 (240), each transformed data from its corresponding DQM is aggregated (e.g., compared) with the initial raw data from the input dataset 202 (D) by its corresponding metric aggregator blocks 242 (1) to 242 (2), which have been processed by IQE 232 (4) and by DQM 234 (4) respectively. For example, various data quality metrics (DQMs) are compared to see how much change is observed in the newly generated data from the aggregator to the initial data. The purpose of doing this is to evaluate the effectiveness of the checks and the aggregator. Since the DQMs are arrays, the values ​​are aggregated before they are compared.

[0061] In phase 4, the aggregated scores are passed to decision module 252 to select the appropriate aggregator to use. This decision can be based on various criteria, which may include, but are not limited to, administrator selection, recommendation-based, and / or automatic criteria. In one embodiment, each metric aggregator block 242 provides a measurement of the gap between the changed data provided from its path from phase 2 and the data provided by the output of DQM 234(4). In this way, each metric aggregator can determine which dataset (e.g., D1, D2, or D3) to use. The larger the gap between its path and the output from DQM 234(4), the more significant the change in the initial raw data D, and thus the better. At this point, decision module 252 of phase 4 selects the dataset (e.g., D1, D2, or D3) based on the scores and decision criteria. In one embodiment, the larger the gap between the changed data and the initial dataset as measured by the metric aggregator in each data path, the more desirable the dataset. However, in one embodiment, if the gap is higher than a predetermined threshold, the changed dataset is disqualified because it may be based on an error. Therefore, decision module 252 can select a dataset (D1, D2, or D3) based on the highest gap, but which is below a predetermined threshold. In other words, the most suitable gap is selected. This gap should be sufficient in the positive direction. Secondly, the gap should not be so large that, according to predetermined criteria, the data is dissimilar to the original data. Therefore, array comparison and distance calculation methods are used to compare multiple DQMs (which are arrays of statistical values). These, in turn, provide insight into the gap between the new and original data, since the DQMs are representative of the data they compute. As mentioned above, the gap between the two datasets should be appropriate.

[0062] Now for reference Figure 5 It provides the same as the illustrative embodiments. Figure 2 A flowchart of the path in stage 3. The metrics 520(1) of the new dataset and the metrics 520(2) of the old dataset are compared by the metric aggregator block 530 to determine which operation is more effective in improving the dataset. The larger the gap between the new dataset 520(1) and the old dataset 520(2), the higher the score. The comparison of data metrics between the two datasets can be performed through aggregation. Techniques used in aggregation may include, but are not limited to: averaging, maximizing, weighted averaging, thresholding, custom aggregation, average-of-maximum, maximum-of-average, feature packing, etc. In various embodiments, the result may be an aggregated metric 540 and / or a single similarity / distance score 550 used to characterize improvements in the data.

[0063] Figure 6 It is consistent with the illustrative embodiments. Figure 2The block diagram for stage 4. The decision module 630 is configured based on... Figure 5 The scores and predetermined decision criteria discussed in the context of the selection between datasets 620(1) and 620(2) are used. In various embodiments, the identified criteria may be (i) automatic (e.g., used to make a final decision on which operator to utilize in a node once the aggregate metric for each operation has been computed in stage 3); (ii) administrator-selected (e.g., having an administrator provide their decision on which operator to select); and / or (iii) recommendation-based (e.g., learning over time which operation works best for a given set of data).

[0064] Example process

[0065] Having given the foregoing overview of example architecture 100 and verification node 200, it may now be helpful to consider a more advanced discussion of the example process. To this end, Figure 7 This paper presents an illustrative process related to improving data quality to save computational resources. Process 700 is shown in a logic flowchart as a collection of boxes, representing a sequence of operations that can be implemented using hardware, software, or a combination thereof. In the context of software, these boxes represent computer-executable instructions that, when executed by one or more processors, perform the described operations. Typically, computer-executable instructions can include routines, programs, objects, components, data structures, etc., that perform functions or implement abstract data types. In each process, the order in which operations are described is not intended to be construed as limiting, and any number of described boxes can be combined and / or executed in parallel in any order to implement the process. For discussion purposes, references are made to... Figure 2 The architecture 200 is used to describe the process 700.

[0066] In box 702, the computing device receives the raw dataset 202.

[0067] In box 704, one or more data quality metric objectives corresponding to the received raw dataset are received. As used herein, the term "data quality metric objective" refers to a combination of metrics and configurations that can be provided by the user before the system runs. This includes data quality metrics as well as termination criteria.

[0068] In box 708, determine the schema of the dataset.

[0069] In box 710, the initial set of verification nodes is identified based on the pattern of the dataset.

[0070] In box 712, the initial set of verification nodes is executed.

[0071] In box 714, the pattern of the dataset is iteratively expanded and the next set of validation nodes is executed until the termination criterion is met. The iterative process continues when the termination criterion is determined to be met (i.e., "No" at decision box 716). However, after the termination criterion is determined to be met (i.e., "Yes" at decision box 716), the process proceeds to box 718, where a corrected dataset of the original dataset is provided based on the initial and next set of validation nodes.

[0072] Example computer platform

[0073] As described above, the functions related to automatically matching tasks and participants can be performed using one or more computing devices connected for data communication via wireless or wired communication, such as... Figure 1 As shown, and according to Figure 4 The process is carried out in 400 steps. Figure 8 A functional block diagram of a computer hardware platform 800, specifically configured for implementing a managed dynamic discovery and correction engine 840, is provided. Therefore, the computer hardware platform 800 is capable of communicating with various databases, knowledge bases, verification libraries, etc., as discussed herein. In particular, Figure 8 A network or host computer platform 800 is shown, which can be used to implement a properly configured server.

[0074] The computer platform 800 may include a central processing unit (CPU) 804, a hard disk drive (HDD) 806, random access memory (RAM) and / or read-only memory (ROM) 808, a keyboard 810, a mouse 812, a display 814, and a communication interface 816, which are connected to the system bus 802.

[0075] In one embodiment, the HDD 806 has the capability to include a storage program that can perform various processes, such as a dynamic discovery and correction engine 840, in the manner described herein. The dynamic discovery and correction engine 840 may have various modules configured to perform different functions.

[0076] For example, a pattern inference module 842 may exist, operable to receive the raw dataset and infer its patterns. An initializer module 844 may exist, operable to initialize a first set of nodes in the graph to perform validation on the raw dataset. An executor module 848 may exist, operable to receive output from initializer 844 and perform operations defined by the nodes in the graph provided by initializer 844. As discussed herein, a terminator module 850 may exist, operable to determine the next set of nodes to consider for execution and to provide a termination signal when criteria are met. An expander module 854 may exist, operable to select the next set of nodes to be added to the graph based on the current output from executor module 848 to terminator 850. A validation module 856 may exist, operable to provide various validation checks, including but not limited to general checks, AI checks, time series checks, and language checks, as discussed herein.

[0077] Example cloud platform

[0078] As described above, the functionality related to matching participants to tasks may include the cloud. It should be understood that although this disclosure includes a detailed description of cloud computing, implementations of the teachings set forth herein are not limited to cloud computing environments. Rather, embodiments of this disclosure can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0079] Cloud computing is a microservice delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and microservices) that can be rapidly provisioned and released with minimal management effort or interaction with microservice providers. This cloud model may include at least five features, at least three microservice models, and at least four deployment models.

[0080] The characteristics are as follows:

[0081] On-demand self-service: Cloud consumers can unilaterally and automatically provide computing power, such as server time and network storage, as needed, without requiring manual interaction with the microservice provider.

[0082] Wide Area Network (WAN) Access: Capabilities are available on the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0083] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. Location independence has significance because consumers typically do not control or know the exact location of the resources provided, but can specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0084] Rapid Flexibility: In some cases, the ability to scale outwards and inwards can be provided quickly and flexibly. For consumers, the available capacity often appears unlimited and can be purchased in any quantity at any time.

[0085] Measuring microservices: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the type of microservice (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both the providers and consumers of the microservices being utilized.

[0086] The service model is as follows:

[0087] Software as a microservice (SaaS): The capability offered to consumers is the ability to use the provider's applications running on cloud infrastructure. Applications can be accessed from various client devices through thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application capabilities, with possible exceptions such as limited user-specific application configuration settings.

[0088] As a microservices platform (PaaS), it provides consumers with the ability to deploy applications created or acquired by the consumer onto cloud infrastructure using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and the configuration of possible application hosting environments.

[0089] As the infrastructure for microservices (IaaS): the capabilities provided to consumers are processing, storage, networking, and other basic computing resources that enable consumers to deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but have control over the operating system, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0090] The deployment model is as follows:

[0091] Private cloud: Cloud infrastructure operated solely by an organization. It can be managed by the organization or a third party and can exist inside or outside a building.

[0092] Community cloud: Cloud infrastructure shared by several organizations and supporting a specific community with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can exist on-site or off-site.

[0093] Public cloud: Cloud infrastructure available to the general public or large industrial groups and owned by organizations that sell cloud microservices.

[0094] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a single entity but are bound together by standardized or proprietary technologies that enable data and applications to be ported together (e.g., cloud bursting for load balancing between clouds).

[0095] Cloud computing environments are microservice-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure of a network of interconnected nodes.

[0096] Now for reference Figure 9 The figure depicts an illustrative cloud computing environment 900. As shown, the cloud computing environment 900 includes one or more cloud computing nodes 910 to which local computing devices used by cloud consumers can communicate. These local computing devices include personal digital assistants (PDAs) or cellular phones 954A, desktop computers 954B, laptop computers 954C, and / or automotive computer systems 954N. The nodes 910 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 950 to provide infrastructure, platforms, and / or software as microservices that cloud consumers do not need to maintain on their local computing devices. It should be understood that... Figure 9 The types of computing devices 954A-N shown are for illustrative purposes only, and computing node 910 and cloud computing environment 950 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0097] Now for reference Figure 10 This demonstrates the 950 (cloud computing environment) Figure 9 This provides a set of functional abstractions. It should be understood beforehand that... Figure 10 The components, layers, and functions shown are for illustrative purposes only, and embodiments of this disclosure are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0098] The hardware and software layer 1060 includes hardware and software components. Examples of hardware components include: a host 1061; a server 1062 based on a RISC (Reduced Instruction Set Computer) architecture; a server 1063; a blade server 1064; a storage device 1065; and a network and networking component 1066. In some embodiments, the software components include network application server software 1067 and database software 1068.

[0099] The virtualization layer 1070 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 1071; virtual storage 1072; virtual network 1073, including virtual private network; virtual application and operating system 1074; and virtual client 1075.

[0100] In one example, management layer 1080 may provide the functionality described below. Resource provisioning 1081 provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and pricing 1082 provides cost tracking when utilizing resources in the cloud computing environment, as well as billing or invoicing for consuming these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, and protection for data and other resources. User portal 1083 provides access to the cloud computing environment for consumers and system administrators. Microservice level management 1084 provides cloud resource allocation and management to meet the required microservice level. Microservice level agreement (SLA) planning and fulfillment 1085 provides pre-scheduling and procurement of cloud resources, where future needs are anticipated according to the SLA.

[0101] The workload layer 1090 provides examples of functionalities that can leverage a cloud computing environment. Examples of workloads and functionalities that can be provided from this layer include: mapping and navigation 1091; software development and lifecycle management 1092; virtual classroom education delivery 1093; data analysis and processing 1094; transaction processing 1095; and a dynamic discovery and correction engine 1096, as discussed herein.

[0102] in conclusion

[0103] Various embodiments of the present teachings have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or improvements to existing technologies in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0104] While the content and / or other examples considered to be in the best possible state have been described above, it should be understood that various modifications may be made therein, and the subject matter disclosed herein may be implemented in various forms and examples, and the teachings can be applied to many applications, of which only a few have been described herein. The appended claims are intended to claim protection for any and all applications, modifications, and variations falling within the true scope of this teaching.

[0105] The components, steps, features, purposes, benefits, and advantages discussed herein are merely illustrative. None of them, or the discussion relating to them, is intended to limit the scope of protection. While various advantages have been discussed herein, it will be understood that not all embodiments are necessarily intended to include all advantages. Unless otherwise stated, all measurements, values, ratings, positions, sizes, dimensions, and other specifications set forth in this specification, including those in the appended claims, are approximate and not precise. They are intended to have a reasonable range consistent with the functionality associated with them and with the conventions of the art to which they pertain.

[0106] Many other embodiments have also been considered. These embodiments include those with fewer, additional, and / or different components, steps, features, purposes, benefits, and advantages. These also include embodiments in which components and / or steps are arranged and / or ordered in different ways.

[0107] This document describes aspects of the disclosure with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0108] These computer-readable program instructions may be provided to a processor of a suitably configured computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, and / or other device to function in a certain way, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0109] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0110] The calling flows, flowcharts, and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions indicated in the blocks may occur in a non-consecutive order as indicated in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block illustrated in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0111] While the foregoing has been described in conjunction with exemplary embodiments, it should be understood that the term "exemplary" means only as an example, and not the best or optimal. Apart from what has just been stated above, whether or not it is stated in the claims, nothing stated or described is intended or should be construed as causing any component, step, feature, object, benefit, advantage, or equivalent to be made public.

[0112] It should be understood that, unless otherwise specified herein, the terms and expressions used herein have the general meaning consistent with those in the respective fields of investigation and research to which they pertain. Relational terms such as "first" and "second" may be used merely to distinguish one entity or action from another, without necessarily requiring or implying any actual such relationship or order between these entities or actions. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but may also include other elements not expressly listed or inherent to such a process, method, article, or apparatus. Without further constraints, an element preceded by "a" or "an" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes that element.

[0113] This summary of disclosure is provided to allow the reader to quickly determine the nature of this technical disclosure. It should be understood that it is not intended to interpret or limit the scope or meaning of the claims. Furthermore, as can be seen from the foregoing detailed description, various features have been grouped together in various embodiments to make this disclosure fluent. This method of disclosure should not be construed as reflecting an intention that the claimed embodiments have more features than those expressly recited in each claim. Rather, as reflected in the appended claims, the inventive subject matter lies in fewer than all features of a single disclosed embodiment. Therefore, the following claims are thereby incorporated into the detailed description, wherein each claim is independently claimed as a separate subject matter.

Claims

1. A computing device, comprising: processor; Storage devices coupled to the processor; An engine stored in the storage device, wherein the processor's execution of the engine configures the computing device to perform actions including: Receive the raw dataset; Receive one or more data quality metrics targets corresponding to the received raw dataset; Based on one or more received data quality metrics, determine the pattern of the original dataset; The initial set of verification nodes is identified based on the patterns in the original dataset; Execute the initial set of verification nodes; The pattern is iteratively expanded and the next set of verification nodes is executed based on the original dataset until the termination criterion is met; and Based on the iterative execution of the initial set of verification nodes and the next set of verification nodes, a corrected dataset of the original dataset is provided.

2. The device as claimed in claim 1, wherein, Each verification node includes a data quality check action and one or more remedial actions.

3. The device as claimed in claim 1, wherein, The execution of the verification nodes in the initial set of verification nodes includes: Identify all possible remedial actions for any data quality check; Transform the data using every possible remedial action; and Calculate multiple data quality metrics (DQM) to evaluate the transformation.

4. The device as claimed in claim 1, wherein, The execution of the verification node includes a first phase, which includes: The validator object performs logical checks on the original dataset to detect one or more anomalies in the original dataset; and Different data transformations are performed on the original dataset by corresponding operators to generate new versions of data for each data transformation, thereby correcting one or more detected anomalies.

5. The device as claimed in claim 4, wherein, The execution of the verification node includes a second phase, which includes: For each new version of the data generated, a Data Quality Metric (DQM) is produced by the Internal Quality Evaluator (IQE) module; and Generate a DQM for the original dataset.

6. The device as claimed in claim 5, wherein, Each DQM in the second phase includes at least one of the following: (i) a summary of characteristics in multiple dimensions of the corresponding new version of the data generated from the original dataset; or (ii) gain or change information of the corresponding new version of the data generated from the original dataset.

7. The device as claimed in claim 5, wherein, The execution of the verification node includes a third phase, which includes: For each new version of the data and each DQM of the original dataset, a comparison with the original dataset is performed to evaluate the improvement relative to the original dataset.

8. The device as claimed in claim 7, wherein, The execution of the verification node includes a fourth phase, which includes: Select the processor that best meets the data quality metric objectives for the new version of the data.

9. The device as claimed in claim 8, wherein, The selected processor has a maximum gap below a predetermined threshold between its corresponding DQM and the DQM of the original dataset.

10. The device as claimed in claim 1, wherein, The next set of extended verification nodes includes at least one of the following: Determine the optimal verification node for achieving one or more received quality metric targets; or The verification node is determined by mining the execution information store to find all verification nodes that usually appear together.

11. A computer-readable storage medium tangibly embodying computer-readable program code having computer-readable instructions, which, when executed, cause a computer device to perform a method for improving data quality to save computing resources, the method comprising: Receive the raw dataset; Receive one or more data quality metrics targets corresponding to the received raw dataset; Based on one or more received data quality metrics, determine the pattern of the original dataset; The initial set of verification nodes is identified based on the patterns in the original dataset; Execute the initial set of verification nodes; The pattern based on the original dataset is iteratively expanded and the next set of verification nodes is executed until the termination criterion is met; as well as Based on the iterative execution of the initial set of verification nodes and the next set of verification nodes, a corrected dataset of the original dataset is provided.

12. The computer-readable storage medium of claim 11, wherein, Each verification node includes a data quality check action and one or more remedial actions.

13. The computer-readable storage medium of claim 11, wherein, The execution of the verification nodes in the initial set of verification nodes includes: Identify all possible remedial actions for each data quality check; Transform the data using every possible remedial action; and Calculate multiple data quality metrics (DQM) to evaluate the transformation.

14. The computer-readable storage medium of claim 11, wherein, The execution of the verification node includes: The first phase, which includes: The validator object performs logical checks on the original dataset to detect one or more anomalies in the original dataset; and Different data transformations are performed on the original dataset by corresponding operators to generate new versions of data for each data transformation, thereby correcting one or more detected anomalies; The second phase, which includes: For each new version of the data generated, a Data Quality Metric (DQM) is produced by the Internal Quality Evaluator (IQE) module; and Generate a DQM for the original dataset; The third stage includes: For each DQM of each new version of the data generated, a comparison with the original dataset is performed to evaluate the improvement relative to the original dataset; and The fourth stage includes: Select the processor that best meets the data quality metric objectives for the new version of the data.

15. The computer-readable storage medium of claim 14, wherein, The selected processor has a maximum gap below a predetermined threshold between its corresponding DQM and the DQM of the original dataset.

16. The computer-readable storage medium of claim 11, wherein, The next set of extended verification nodes includes at least one of the following: Determine the verification nodes that best achieve one or more of the data quality metric objectives; or The verification node is determined by mining the execution information store to find all verification nodes that usually appear together.

17. A system comprising: The pattern inference module is configured to receive a raw dataset and determine the pattern of the raw dataset based on one or more data quality metrics corresponding to the received raw dataset. as well as A data synthesizer module, coupled to the pattern inference module, and configured to receive the one or more data quality metric targets from a knowledge base, wherein the data synthesizer module includes: The initializer module is configured to identify an initial set of verification nodes based on patterns in the original dataset; The executor module, which is coupled to the initializer module, is configured to execute the initial set of verification nodes; An expander module, coupled to the executor module, is configured to iteratively expand and execute the next set of verification nodes based on patterns in the original dataset until a termination criterion is met; and A terminator module, coupled to the extender module, is configured to iteratively determine the next set of verification nodes to be considered by the extender module and to determine when to terminate the iterative determination; and An information repository, coupled to the executor module and the terminator module, is configured to provide a corrected dataset of the original dataset based on iterative execution of the initial set and the next set of the verification nodes.

18. The system of claim 17, wherein, Each verification node includes a data quality check action and one or more remedial actions.

19. The system of claim 17, wherein, The initial set of verification nodes is configured as follows: Identify all possible remedial actions for any data quality check; Transform the data using every possible remedial action; and Calculate multiple data quality metrics to evaluate the transformation.

20. The system of claim 17, wherein each verification node comprises: The first stage is configured as follows: The validator object performs logical checks on the original dataset to detect one or more anomalies in the original dataset; as well as Different data transformations are performed on the original dataset by corresponding operators to generate new versions of data for each data transformation, thereby correcting one or more detected anomalies; The second stage is configured as follows: For each new version of the data generated, a Data Quality Metric (DQM) is produced by the Internal Quality Evaluator (IQE) module; and Generate a DQM for the original dataset; The third stage is configured as follows: For each DQM of each new version of the data generated, perform a comparison with the original dataset; as well as The third stage is configured as follows: Select the processor that best meets the data quality metric objectives for the new version of the data.

21. A computer-implemented method for improving data quality to save computing resources, the method comprising: Receive the raw dataset; Receive one or more data quality metrics targets corresponding to the received raw dataset; Based on one or more received data quality metrics, determine the pattern of the original dataset; The initial set of verification nodes to be performed is identified based on the patterns in the original dataset. Execute the initial set of verification nodes; The pattern based on the original dataset is iteratively expanded and the next set of verification nodes is executed until the termination criterion is met; as well as Based on the iterative execution of the initial set of verification nodes and the next set of verification nodes, a corrected dataset of the original dataset is provided.

22. The computer-implemented method as described in claim 21, wherein, Each verification node includes a data quality check action and one or more remedial actions.

23. The computer-implemented method as described in claim 21, wherein, The execution of the verification nodes in the initial set of verification nodes includes: Identify all possible remedial actions for each data quality check; Transform the data using every possible remedial action; and Calculate multiple data quality metrics (DQM) to evaluate the transformation.

24. The computer-implemented method of claim 21, wherein the execution of the verification node comprises: The first phase, which includes: The validator object performs logical checks on the original dataset to detect one or more anomalies in the original dataset; and Different data transformations are performed on the original dataset by corresponding operators to generate new versions of data for each data transformation, thereby correcting one or more detected anomalies; The second phase, which includes: For each new version of the data generated, a Data Quality Metric (DQM) is produced by the Internal Quality Evaluator (IQE) module; and Generate a DQM for the original dataset; The third stage includes: For each DQM of each new version of the data generated, perform a comparison with the original dataset; and The fourth stage includes: Select the processor that best meets the data quality metric objectives for the new version of the data.

25. The computer-implemented method as described in claim 21, wherein, The next set of extended verification nodes includes at least one of the following: Determine the optimal verification node for achieving one or more received quality metric targets; or The verification node is determined by mining the execution information store to find all verification nodes that usually appear together.

Images