Real-time data quality analysis

Abstract

A computer-based method for evaluating data quality involves using a processor to receive multiple updates to data points in a data stream. The processor is also used to compute instances of a data quality metric (DQM) from the data points in the data stream. The DQM instances are configured to differentiate data points in the data stream based on time, with higher weights assigned to DQM instances computed based on newer data points in the data stream. As the processor receives more data points, the DQM instances are continuously updated, while the processor cycles consumed by updating the DQM instances are limited to a threshold.

Description

Background Technology

[0001] This invention generally relates to programmable computers, and more specifically to programmable computers, computer-implemented methods, and computer program products for implementing new data quality metrics and related data quality analysis techniques. According to various aspects of the invention, the new data quality metrics and related data quality analysis techniques are configured to efficiently allocate and utilize the computational resources required to perform incremental data quality analysis on datasets with new or updated data that changes over time.

[0002] In computer processor applications, the phrase "big data" refers to massive datasets that can be computationally analyzed to reveal patterns, trends, and correlations, especially those related to human behavior and interactions. Big data can be utilized by sophisticated computational analytics systems such as machine learning (ML) algorithms, artificial intelligence (AI) algorithms, deep learning algorithms, and Internet of Things (IoT) systems. Because the demand for ML experts has exceeded the supply, user-friendly automated ML / AI computer systems have been developed. Known automated AI / ML systems can automate various AI / ML development tasks, such as model selection, hyperparameter optimization, automated stacking (overall), neural architecture search, pipeline optimization, and feature engineering.

[0003] The performance of automated ML / AI systems largely depends on the extent to which the system's training data is qualitatively cleaned and analyzed to make it suitable for efficient consumption by the ML model. If this is not detected, poor data quality can lead to significant biases in analyses performed by sophisticated AI / ML systems, resulting in inaccurate and misleading results. Because the process of identifying and resolving data quality issues can be labor-intensive, scalable, automated, and interactive data quality tools / systems, known as data quality advisory systems or DQLearn systems, have emerged. Development. For ease of description, the term Data Quality Analysis (DQA) system is used here to refer, generally speaking, to a Data Quality Consultant (or DQLearn) system and other known data quality analysis systems with similar characteristics to DQA / DQLearn systems. The framework of a DQA system performs various data quality analysis tasks, including automatically generating dynamic, executable graphs for performing data validation fine-tuning on a given dataset; establishing a library of validation checks common to many applications; and applying different tools to address persistence and usability issues, making data cleaning a challenging task for data scientists.

[0004] A DQA system performs data quality checks on datasets that continuously flow through a data quality analysis pipeline. Data quality checks involve measuring and / or analyzing different characteristics or properties of the dataset to generate a data quality metric (DQM) that provides feedback to the user regarding data quality. For example, a known data quality check is identifying / measuring NULL (empty) values ​​in a dataset, and the corresponding known DQM is the percentage of empty values ​​in the dataset. In many cases, the data in the analysis includes data that has previously undergone data quality checks, as well as data that has not previously undergone data quality checks (i.e., new / updated data).

[0005] DQA systems continuously update DQM measurements as new / updated data is received, and recalculating these DQMs consumes significant computational resources. To address this issue, known DQA systems can incrementally calculate DQMs. This means that instead of the computationally expensive method of recalculating DQMs on the entire initial dataset each time a dataset is used for data quality assessment, DQMs are calculated "incrementally" by calculating the DQMs for the new / updated data and combining them with DQM information previously calculated for the unchanged portion of the initial dataset.

[0006] Data quality analysis techniques that incrementally compute the DQM for new / updated data offer higher computational resource efficiency than those that do not. However, in known data quality analysis techniques, the DQM itself is static and does not account for the fact that the new / updated data from which the DQM is derived is non-static and frequently changes when viewed over a selected time window. Therefore, known data quality analysis techniques that incrementally determine the DQM for new / updated data still lack efficiency in their allocation and computational resource usage because they cannot well match the constantly changing new / updated data over time. Summary of the Invention

[0007] Embodiments of the present invention relate to a computer-implemented method for evaluating data quality. A non-limiting example of the computer-implemented method includes using a processor to receive multiple updates to data points in a data stream. The processor is also configured to compute instances of a data quality metric (DQM) from the data points in the data stream. The DQM instances are configured to differentiate data points in the data stream based on time, and to assign higher weights to DQM instances computed based on newer data points in the data stream. As the processor receives more data points, the DQM instances are continuously updated, while limiting the processor cycles consumed by updating the DQM instances to a threshold.

[0008] The computer-implemented method described above provides an improvement over known methods of evaluating data quality by offering a DQM that differentiates data points over time and assigns a higher weight to the DQM calculated based on the most recent data points. Therefore, the computer-implemented method efficiently allocates computational resources because it calculates the DQM in a manner that closely matches the constantly changing incoming data points over time.

[0009] The computer-implemented method described above may further include using a processor to generate a data quality assessment using instances of DQM across multiple time ranges; selecting a data quality remediation strategy based at least in part on the data quality assessment; and applying the data quality remediation strategy to the data stream.

[0010] The computer-implemented method described above provides an improvement over known methods for assessing data quality by generating data quality assessments using instances of DQM across multiple time ranges. Therefore, the computer-implemented method efficiently allocates computational resources because it computes DQM in a manner that closely matches the constantly changing incoming data points over time.

[0011] Embodiments of the present invention relate to a computer-implemented method for evaluating data quality. A non-limiting example of the computer-implemented method includes using a processor to receive multiple updates to data points in a data stream. The processor is also configured to compute instances of Data Quality Management (DQM) from the data points in the data stream. The instances of DQM computed over multiple time intervals are analyzed. The processor is further configured to determine outliers in at least one of the multiple time intervals for instances of DQM computed over at least one of the multiple time intervals; and to determine a weighted aggregate value for instances of DQM computed across the multiple time intervals. The lower value of the weighted aggregate value is assigned to instances of DQM in the multiple time intervals that have outliers.

[0012] The computer-implemented method described above provides an improvement over known methods for evaluating data quality by identifying anomalous data points across multiple time intervals and assigning lower weights to those identified as anomalous. Therefore, the computer-implemented method efficiently allocates computational resources because it identifies anomalous behavior among data points in a manner that closely matches the changing nature of the incoming data points over time.

[0013] The computer-implemented method described above may further include calculating a severity level based on outliers, assigning a lower value of a weighted aggregate value to DQM instances in a time interval among a plurality of time intervals with outliers.

[0014] The computer-implemented method described above improves upon known methods for assessing data quality by identifying the severity of anomalous data points across multiple time intervals and assigning lower weights to identified anomalous data points based on their measured severity. Therefore, the computer-implemented method efficiently allocates computational resources because it identifies anomalous behavior among data points in a manner that closely matches the changing nature of the incoming data points over time.

[0015] Embodiments of the present invention also relate to computer systems and computer program products having substantially the same features and functions as the computer-implemented methods described above.

[0016] Embodiments of the present invention relate to a computer program product for evaluating data quality. The computer program product includes a computer-readable storage medium having program instructions embodied therein, which are executable by a processor system to cause the processor system to perform operations. The operations include receiving multiple updates to data points in a data stream, and calculating instances of Data Quality Management (DQM) from the data points in the data stream, wherein the instances of DQM are configured to distinguish data points in the data stream by time, and to assign higher weights to instances of DQM calculated from more recent data points in the data stream. The operations also include continuously updating the instances of DQM as the processor receives more data points, while limiting the processor cycles consumed by updating the instances of DQM to a threshold. The instances of DQM calculated over multiple time intervals are analyzed, and at least one of the multiple time intervals is determined to have outliers for instances of DQM calculated during at least one of the multiple time intervals. A weighted aggregate value of the instances of DQM across the multiple time intervals is calculated, wherein lower values ​​of the weighted aggregate value are assigned to instances of DQM in the multiple time intervals that have outliers.

[0017] The aforementioned computer program product provides an improvement to known methods of assessing data quality by offering a DQM that differentiates data points over time and assigns higher weights to a DQM calculated based on the most recent data points. The aforementioned computer program product also provides an improvement to known methods of assessing data quality by identifying outlier data points across multiple time intervals and assigning lower weights to data points identified as outliers. Therefore, the aforementioned computer program product efficiently allocates computational resources because it calculates the DQM and identifies outlier data points in a manner that closely matches the constantly changing incoming data points over time.

[0018] Embodiments of the present invention also relate to computer-implemented methods and computer systems having substantially the same features and functions as the computer program products described above.

[0019] Additional features and advantages are achieved through the techniques described herein. Other embodiments and aspects are described in detail herein. For a better understanding, refer to the specification and accompanying drawings. Attached Figure Description

[0020] The subject matter considered to be the embodiment is specifically pointed out and clearly claimed in the claims at the end of the specification. The foregoing and other features and advantages of the embodiment will become apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0021] Figure 1 A data quality analysis (DQA) system having a real-time DCA module configured to implement various real-time, non-static data quality analysis features, according to various aspects of the present invention, is described.

[0022] Figure 2 It describes what can be done by Figure 1 The image shows an example of a data quality check performed by a DQA system.

[0023] Figure 3 It describes what can be done by Figure 1 The example shown is a data frame analyzed by the DQA system.

[0024] Figure 4 Examples of data frames with time intervals according to various aspects of the present invention are described, wherein the data frames with time intervals can be generated by... Figure 1 Real-time DQA feature analysis of the DQA system shown;

[0025] Figure 5 It describes what can be made by Figure 1 The workflow implemented by the DQA system shown in the figure;

[0026] Figure 6 Description Figure 5 A block diagram showing additional aspects of the workflow;

[0027] Figure 7 The description illustrates how embodiments of the present invention can be implemented. Figure 1 and 5 A block diagram showing additional details of the real-time DQA module of the DQA system shown;

[0028] Figure 8 The description illustrates how the submodules for calculating the DQM of the modified data, according to various aspects of the present invention, are derived from... Figure 7 The block diagram shown illustrates the implementation of the real-time DQA module.

[0029] Figure 9 The description illustrates how the submodule for calculating the DQM for changing constraints, according to various aspects of the present invention, is derived from... Figure 7The block diagram shown illustrates the implementation of the real-time DQA module.

[0030] Figure 10A The description illustrates how a submodule for calculating the DQM for modifying anomalous data, according to various aspects of the present invention, is... Figure 7 The block diagram shown illustrates the implementation of the real-time DQA module.

[0031] Figure 10B A flowchart is described that illustrates a computer-implemented method according to various aspects of the present invention;

[0032] Figure 11 This description illustrates how a submodule for performing general incremental calculations of DQM, according to various aspects of the present invention, works by... Figure 7 The block diagram shown illustrates the implementation of the real-time DQA module.

[0033] Figure 12 An example of DQM is shown, which can be incrementally computed using submodules for execution. Figure 7 and 11 The DQM is calculated incrementally in the general case shown.

[0034] Figure 13 The description shows that it can be executed by a submodule for use in execution. Figure 7 and 11 The flowchart shown is a computer implementation of a method for incremental computation of the general case of DQM.

[0035] Figure 14 The description shows that it can be executed by a submodule for use in execution. Figure 7 and 11 The flowchart shown is a computer implementation of a method for incremental computation of the general case of DQM.

[0036] Figure 15 It describes what can be made by Figure 7 , 11 The computer-implemented methods and equations implemented by submodules shown in 13 and 14;

[0037] Figure 16 It describes what can be made by Figure 7 , 11 The computer-implemented methods and additional equations implemented by submodules shown in 13 and 14;

[0038] Figure 17 Describes the methods used to achieve Figure 7 , 11 The computer architecture of the computer-implemented methods and submodules shown in 13 and 14;

[0039] Figure 18This describes how a submodule for managing the state of incremental DQM calculations, according to various aspects of the present invention, is composed of... Figure 7 The block diagram shown illustrates the implementation of the real-time DQA module.

[0040] Figure 19 This describes how to use transaction form checks according to various aspects of the present invention. Figure 1 The following is an example of DQA system maintenance status information;

[0041] Figure 20 The use of various aspects of the present invention is described. Figure 1 The DQA system shown is an example of analyzing DQM over time intervals;

[0042] Figure 21 The description illustrates aspects of the invention. Figure 7 A block diagram illustrating an example of how a submodule for efficiently prioritizing DQM computation is implemented in the real-time DQA module of a DQA system.

[0043] Figure 22 The description illustrates how various aspects of the invention can be used. Figure 7 A block diagram showing additional details of the real-time DQA module implementation of the DQA system shown in 19 for the submodules used to efficiently prioritize DQM calculations;

[0044] Figure 23 This describes how a computer-implemented method according to various aspects of the present invention can be executed by a submodule for use. Figure 7 And / or the block diagram of the real-time DQA module of the DQA system shown in 19 to effectively prioritize DQM calculations;

[0045] Figure 24 The computer implementation of the method according to various aspects of the present invention is described in terms of how it can be executed by a submodule for use. Figure 7 And / or the real-time DQA module of the DQA system shown in 19 to efficiently prioritize additional details of DQM computation;

[0046] Figure 25 The illustrations depict how a computer-implemented method according to various aspects of the present invention can be executed by a submodule for use. Figure 7 The real-time DQA module and / or of the DQA system shown Figure 19 The diagram shows an architecture for efficiently prioritizing DQM computation.

[0047] Figure 26 The computer implementation of the method according to various aspects of the present invention is described in terms of how it can be executed by a submodule for use. Figure 7And / or the real-time DQA module of the DQA system shown in 19 to efficiently prioritize additional details of DQM computation;

[0048] Figure 27 A block diagram is described to illustrate a computer system capable of implementing various aspects of the present invention;

[0049] Figure 28 A cloud computing environment according to embodiments of the present invention is described; and

[0050] Figure 29 An abstract model layer according to an embodiment of the present invention is shown.

[0051] In the accompanying drawings and the following detailed description of the disclosed embodiments, the various elements shown in the drawings are provided with three numerical reference numerals where appropriate. The leftmost numeral of each reference numeral corresponds to the figure in which its element is first shown. Detailed Implementation

[0052] For the sake of brevity, conventional techniques associated with carrying out and using various aspects of the present invention may or may not be described in detail herein. In particular, various aspects of the computing systems and specific computer programs used to implement the various technical features described herein are well known. Therefore, for the sake of brevity, many conventional implementation details are only briefly mentioned or omitted entirely herein, without providing well-known system and / or process details.

[0053] Many of the functional units described in this specification are labeled as modules. Embodiments of the invention are applicable to various module implementations. For example, a module can be implemented as a hardware circuit configured to include off-the-shelf semiconductors such as custom VLSI circuitry or gate arrays, logic chips, transistors, or other discrete components. Modules can also be implemented in programmable hardware devices, such as field-programmable gate arrays, programmable array logic, programmable logic devices, etc. Modules can also be implemented in software so that they can be executed by various types of processors. The executable code identifying a module can, for example, include one or more physical or logical blocks of computer instructions, which can be organized, for example, as objects, procedures, or functions. However, the executable code identifying a module does not need to be physically located together, but can include different instructions stored in different locations that, when logically combined, include the module and achieve its intended purpose.

[0054] As previously discussed, configuring a data quality analysis system to incorporate known techniques for incrementally calculating the DQM for new / updated data offers greater computational resource efficiency than data quality analysis systems that do not incrementally calculate the DQM for new / updated data. However, in known techniques for incrementally calculating the DQM, the DQM itself is static and does not take into account the fact that the new / updated data from which the DQM is derived is non-static and frequently changes when viewed over a selected time window. Therefore, known techniques for incrementally determining the DQM for new / updated data remain inefficient in their allocation and use of computational resources because they do not adequately match the constantly changing new / updated data over time.

[0055] Embodiments of the present invention address and overcome the shortcomings of known incremental DQM calculation techniques by providing computer-implemented methods, programmable computing systems, and computer program products configured and arranged to efficiently allocate and utilize the computing resources required to perform data quality analysis techniques that incrementally calculate DQM for new / updated data. More specifically, embodiments of the present invention improve the efficiency of computing resource allocation and utilization by providing DQM and related data quality analysis techniques that well match new / updated data that changes over time, because the DQM and related data quality analysis techniques according to various aspects of the present invention are incremental and non-static. More specifically, the DQM and related data quality analysis techniques according to various aspects of the present invention are non-static because they take into account the fact that the new / updated data from which the DQM and related data quality analysis techniques are derived when viewed over a selected time window is non-static and constantly changing. Therefore, a DQA system with incremental and non-static data quality analysis features according to various aspects of the present invention improves the efficiency of computer resource allocation and utilization compared to a DQA system that only provides incremental data quality analysis features.

[0056] In various aspects of the present invention, a DQA system with incremental and non-static data quality analysis features according to embodiments of the present invention is configured to include a real-time DQA module having a plurality of computer-implemented submodules that perform real-time data quality analysis on changing data by defining a plurality of time-varying incremental DQMs. The plurality of submodules in the real-time DQA module include: a submodule for calculating the DQM for changing data; a submodule for calculating the DQM for time-varying constraints; a submodule for calculating the DQM for anomalous data regions; a submodule for performing general incremental calculations of the DQM; a submodule for managing the state of the incremental DQM calculations; and a submodule for effectively prioritizing the DQM calculations.

[0057] In embodiments of the invention, a submodule for calculating the Data Quality Metric (DQM) for changing data is configured to track data quality changes over time, while also detecting DQM changes across multiple time windows. For example, where the DQM is a measure of missing values ​​in a dataset (e.g., a dataframe in the Python programming language), detected changes in the DQM may include missing values ​​occurring over a predetermined time interval (or time window); recently occurring missing values; and weighting applied to missing values ​​so that missing values ​​from more recent data points are more heavily weighted. As another example, DQM (Data Quality Metric) d1 represents multiple possible data quality checks, including missing data, finding low-variance variables, mean, standard deviation, median, checking constant values, unique values, duplicate values, most frequently occurring values, duplicate values ​​across columns, duplicate rows, etc. In embodiments of the invention, these data quality checks can be applied not only holistically to all new / updated data values ​​but also to certain time intervals or time windows of new / updated data. Therefore, in embodiments of the invention, d1 is generated by data quality checks defined or applied across a specific window of the dataset. In some embodiments of the invention, d1 can be applied over a time interval / window extending from start_time to end_time. In some embodiments of the invention, d1 can be applied to each batch of new / updated data to achieve d1(batch 1), which is calculated for each batch of new / updated data. The values ​​of d1(batch 1) to d1(batch N) can be plotted on the y-axis with time values ​​on the x-axis. In some embodiments of the invention, data points falling into different time intervals / windows can be given different weights for calculating DQM. In some embodiments of the invention, newer data points (e.g., within a predetermined time interval / window defined as the newer time interval / window) can be assigned a higher weight than less recent data points (e.g., within a predetermined time interval / window defined as the less recent time interval / window) to evaluate DQM. In embodiments of the invention, a variety of suitable weighting techniques can be used, including but not limited to exponential weighting (including but not limited to exponentially weighted moving averages). In some embodiments of the invention, each data point can be assigned a different weight based on its time. In some embodiments of the invention, data points are grouped by time intervals / windows, and the same weight can be assigned to sets of data points belonging to the same group. In some embodiments of the invention, older data points can be completely ignored in the DQM calculation. In some embodiments of the invention, various known data quality analysis algorithms can be applied to help identify older data points that will be ignored in the DQM calculation. Therefore, as described above, embodiments of the invention provide a novel non-static DQM that is parameterized by time.The DQA system operating according to various aspects of the present invention is configured and arranged to calculate and visualize a new non-static DQM over any range of data points.

[0058] In embodiments of the invention, a submodule for calculating the DQM for changing constraints is configured to calculate the DQM, which measures the degree to which data points satisfy specific constraints of the DCA system. Because such a “constraint” (or constrained) DQM can change over time, embodiments of the invention calculate the constraint DQM for new / updated data points as they enter the DCA pipeline, rather than calculating a single constraint DQM for the entire dataset. In embodiments of the invention, analyzing the constraint DQM over time enables a more comprehensive data quality analysis of the dataset. Examples of constraints include null values ​​not exceeding a threshold; averages falling within a specific range (e.g., standard deviation, variance, median, etc.); two columns of a data frame having a mathematical relationship or correlation; and so on. Embodiments of the invention recognize and utilize the observation that, for real-time streaming data, the applicability of a given constraint is not a static, fixed property. A constraint that can be applied at one time may or may not be applied to new data being received. Therefore, embodiments of the invention provide analysis of constraints over multiple different time intervals / windows, and the applicability of constraints or sets of constraints according to various aspects of the invention is therefore dynamic and expected to change over time. In some embodiments of the invention, if the constraint DQM changes significantly over time (e.g., changes by a level exceeding a predetermined threshold), information related to the changed constraint DQM can be propagated to users of the DQA system.

[0059] In embodiments of the invention, a submodule for calculating the DQM of an anomalous data region is configured to calculate the DQM, which measures whether data points meet the criteria used to determine whether a data point is “abnormal.” In embodiments of the invention, weights may be assigned to the DQM calculated for data points identified as anomalous. In some embodiments of the invention, the weights may be less than -1 (1) but greater than zero (0) based on the calculated severity level of the anomalousness (e.g., moderately anomalous data may be weighted within approximately 20% of -1). In some embodiments of the invention, anomalous data having a calculated severity level exceeding a threshold may be identified as highly anomalous and assigned a weight of zero (0) (i.e., anomalous data points are ignored when calculating the DQM of an anomalous data region).

[0060] As an example, in an embodiment of the invention, where the DQA system uses the Python programming language, the "correlation" function can be defined as correlation(feature1, feature2, start_time, end_time). This function returns a value between -1 and -1, indicating the correlation level between feature1 and feature2 for a given time interval / window. In an embodiment of the invention, the DQA system is configured to maintain correlation values ​​over several different time intervals. In an embodiment of the invention, time intervals / windows are marked, where the correlation function indicates anomalous behavior within the time interval / window. An example of anomalous behavior of the correlation function is that the correlation (feature1, feature2, start_time, end_time) value changes to a value previously not detected by the DQA system. The time interval / window marking anomalous data points at ti, and according to various aspects of the invention, data points from anomalous time interval / window ti can be omitted in the DQM calculation of ti, or data points from anomalous time interval / window ti can be assigned a lower weight in the DQM calculation of ti. DQM can also notify the user of anomalous time intervals. The user can provide the DQA system with user-selected preferences regarding how anomalous time intervals are handled, such as assigning weights to them for DQM calculation.

[0061] As another example, in embodiments of the invention, the "missing value" function can be defined in the Python programming language as `check_na_columns(df, start_time1, end_time)`, which examines the columns of parameter `df` within a specific time range to look for missing values. According to various aspects of the invention, this function can be used to detect portions of the data that should be assigned higher weights when calculating the DQM. For example, if `check_na_columns` indicates an anomalous proportion of NaN (Not a Number) values, then assigning less weight to the time interval / window may be appropriate when calculating the DQM.

[0062] It should be noted that a higher proportion of NaN values ​​does not necessarily mean that the time interval should be assigned a lower weight. In many cases, DQA systems are configured to search for anomalies in the number of NaN values, which may indicate an unusually high or low number of NaN values. If a particular data interval has a very low proportion of NaN values ​​compared to other intervals, the DQA system can be configured to interpret this as an indication of anomaly, which would mean that this interval should be assigned a lower weight than other intervals with a certain proportion of NaN values ​​that are closer to the average.

[0063] In embodiments of the invention, a submodule for performing incremental calculations of the Data Quality Management (DQM) provides a general method for performing incremental calculations of the DQM. In general, making data quality checks incremental is a challenge. Embodiments of the invention address and overcome this challenge by providing three types of general incremental data quality checks, defined and identified herein as Type I, Type II, and Weighted Type I & II. In embodiments of the invention, Type I data quality checks are data quality checks with a corresponding DQM, which in general can be incrementally performed through decomposition and summation processes. For example, with the DQM defined as q, the data entering the DQA pipeline is organized as data blocks represented as Dw = {D1, D2, ..., Dn} ordered over time, and the quality function is defined as Q(D). General incremental data quality checks according to various aspects of the invention provide a mechanism for decomposing the quality function Q by applying Q to each data block entering the DQA pipeline and storing Q({D1, D2, ..., Dn}) for each data block in the dataset. When data block Dn+1 of dataset D enters the DQA pipeline and contains new / updated data, the Q(D) for the entire dataset is incrementally obtained by applying Q to data block Dn+1 with new / updated data and combining that result with the sum of historical results from applying Q to data blocks {D1, D2, ..., Dn} in dataset D that have not yet been changed. The sum of historical Q results is obtained by... Figure 13 Equation (1) shown indicates that the sum of the new / updated Q results and the historical Q results is determined by... Figure 13 Equation (2) shown in the figure represents this.

[0064] In embodiments of the invention, Type II data quality checking is a data quality check with a corresponding DQM, which, in general, can be incremented using the decomposition / summation process used in Type I. Type I has been modified to take into account the decomposable operations used in associated DQM calculations. For example, when the quality function Q of the DQM calculation is the percentage of null values ​​in dataset D, the percentage calculation Q can be decomposed into two functions P and C, where C counts the total number of data points in dataset D, and where P represents the null values ​​in dataset D. Thus, Q(D) can be calculated as P(D) divided by C(D). According to embodiments of the invention, the data entering the DQA pipeline is organized into data blocks represented as Dw = {D1, D2, ..., Dn}, ordered over time, and the quality function is defined as Q(D). Incremental data quality checking according to various aspects of the present invention provides a mechanism for decomposing the quality function Q by applying P to each data block entering the DQA pipeline; applying C to each data block entering the DQA pipeline; and storing P({D1, D2, ..., Dn}) and C({D1, D2, ..., Dn}) for each data block in the dataset. When data block Dn+1 of dataset D enters the DQA pipeline, and data block Dn+1 has new / updated data, Q(D) for the entire dataset D is incrementally obtained by applying P to data block Dn+1 with new / updated data; applying C to data block Dn+1; and combining this result with the sum of historical results of applying P and C to unchanged data blocks {D1, D2, ..., Dn} in dataset D. Figure 15 Equation (3) shown indicates that the sum of the new / updated P and C results and the historical P and C results is determined by... Figure 15 Equation (4) shown in the figure represents this.

[0065] In embodiments of the invention, weighted Type I and II data quality checks are data quality checks that weight data blocks in an incremental Type I and II calculation scenario. In some embodiments of the invention, different weights can be applied to different data blocks based on any criteria previously described in conjunction with aspects of the invention for applying weights to data checks. In some aspects of the invention, the weight w can be selected based on the time elapsed from the last timestamp of a given data block to the current time. In some aspects of the invention, the weights can be uniformly applied to previous data blocks in the dataset. Examples of how weights can be incorporated into Type I data quality checks according to aspects of the invention are provided below. Figure 16 Equation 5 in the text represents this. An example of how weighting can be incorporated into Type II data quality checks according to various aspects of the invention is provided by... Figure 16 Equation 6 in the text represents this.

[0066] In embodiments of the invention, a submodule for managing the incremental DQM calculation state provides a mechanism for maintaining the state, which is necessary for incremental DQM calculations because incremental calculations require tracking the manner of previous calculations to be used in the incremental calculation. Embodiments of the invention avoid the need for the DQA system to maintain its own historical data quality inspection information database by providing the aforementioned state management submodule, which is configured and arranged to interface with the user system in a manner involving the exchange of state variable information back and forth between the user system or program and the DQA system. In embodiments of the invention, the DQA system can be implemented in a programming language such as Python, and the Python application programming interface (API) of the state management submodule is configured to allow the user program to use the Python API to call various functions related to state management. Various other programming languages ​​(e.g., Java, C++, C, and others) can also be used to implement the DQA system. The state management submodule calculates the DQM and passes the associated state information to the user program. To utilize the state information received from the state management submodule, the user program calls functions of the state management submodule that are configured and arranged to actually analyze the state information. Therefore, the state management submodules according to various aspects of the present invention do not require user programs to have the ability to interpret the format of state variables. User programs only need to use programming language APIs (e.g., Python implementations for DQA systems) to call the state management submodules of the DQA system.

[0067] In embodiments of the invention, submodules for efficiently prioritizing DQM computations are configured and arranged to effectively manage the computational resources of the DQM system used to perform the various non-static and dynamic data quality checks and DQMs described herein. In embodiments of the invention, the computer-implemented method prioritizes the execution of DQM computations based on various factors, including but not limited to the computational overhead required to compute the DQM; the rate of change of a given DQM; whether previously computed DQMs can be used without exceeding a threshold of precision loss; and user-defined constraint priorities. In some embodiments of the invention, machine learning models and / or general simulation algorithms can be used to predict the impact of DQMs on computational resources, and the DQA system can prioritize the execution of DQMs based on these predictions. Example predictions include, but are not limited to, predictions of how much the DQM is expected to change in response to new / updated data; and predictions of the computational overhead of different DQMs performed on a given dataset and dataset size.

[0068] We now turn to a more detailed description of embodiments of the present invention. Figure 1A data quality analysis (DQA) system 100 capable of implementing various aspects of the present invention is described. The DQA system 100 is a programmable computer configured and arranged to evaluate data quality in an interactive, automated, and scalable manner to generate human-readable reports and produce a data pipeline with sufficient quality to be reliably consumed by AI / ML systems. The hardware components of the DQA system 100 can utilize a computer system 2700 (…). Figure 27 It is implemented using the features and functions shown in the diagram. Additionally, the cloud computing system 50 (also shown in the diagram) is also included. Figure 28 (As shown in the diagram) communicates with the DQA system 100 via wired or wireless electronic communication. The cloud computing system 50 can supplement, support, or replace some or all of the functions of the DQA system 100 (in any combination). Additionally, some or all of the functions of the DQA system 100 can be implemented as nodes 10 of the cloud computing system 50 (in...). Figure 28 and 29 (as shown in the image).

[0069] The DQA system 100 can be configured to include a validator 110, a remedy 120, a set of constraints 130, a DQA pipeline 140, and an update module 150, configured and arranged as shown in the figure. The validator 110 is configured to perform various types of data quality checks (predefined or custom) on various types of data. Figure 2 Examples of the types of data quality checks that validator 110 can perform are listed. Validator 110 is configured to perform data quality checks on data from different domains and categories, including general data quality checks applied to general attributes of any dataset, and checks applied to data such as time series data (300...). Figure 3 (as shown) and / or data frame 400 ( Figure 4 Data quality checks are performed on attributes specific to the data format shown.

[0070] Figure 3 The example shown is how time series data 300 is represented using the Python programming language. Typically, time series data structures have dictionary and list attributes. Items are stored sequentially and labeled for easy retrieval.

[0071] Figure 4The data frame 400 shown is an example of how a data frame can be represented in the Python programming language. Typically, a data frame such as data frame 400 is a two-dimensional data structure where data is arranged in rows and columns in a tabular form. The size of a given data frame is flexible; rows and columns can be labeled; and arithmetic operations can be performed on rows and columns. Data frame 400 also depicts time intervals (or windows), which are ranges of consecutive rows representing a subset of the complete dataset. Data frame 400 is an example of how customer information is represented in a data frame, and an example of data quality issues that may exist in a data frame, including, for example, the presence of null values, default entries, and truncated entries. In embodiments of the invention, multiple data frames can be streamed to DQA system 100 for analysis, and various entries in the data frames can change over time. For example, in data frame 400, a new customer row can be added, or the delivery address of an existing customer row can be changed or updated.

[0072] Remediator 120 includes various types of logic that can be invoked and used to correct or remedy data quality problems identified by validator 110. Constraint 130 are custom rules that users can select and apply to customize how the data quality checks performed at validator 110 interpret different statistical values ​​obtained from the data. Pipeline 140 uses validator 110, remediator 120, and constraints 130 to automate the data quality analysis operations performed by validator 110, remediator 120, and constraints 130 on a single dataset.

[0073] Update module 150 is configured to update different Data Quality Metrics (DQMs) after iterations of data quality analysis processing performed by DQA system 100. According to embodiments of the invention, update module 150 includes a real-time data quality analysis (DQA) module 160, configured to implement novel data quality analysis processes according to various aspects of the invention. In these aspects, real-time DQA 160 is configured to perform real-time data quality analysis of changing data by defining multiple non-static and dynamic data quality metrics that vary over time. Specific details on how the real-time DQA module 160 is implemented are detailed in [the following section / document / etc.]. Figure 7 It is shown in the figure and described in more detail hereafter.

[0074] Figure 5 and 6 The document describes workflows illustrating how user 502 interacts with DQA system 100 in two scenarios. The first scenario, labeled Workflow #1, depicts how DQA system 100 performs a one-time data repair when the dataset being analyzed does not include new or updated data. The second scenario, labeled Workflow #2, depicts how DQA system 100 performs a "learning and repair" process when the dataset being analyzed has been updated to include new / updated data. Figure 5In workflow #1, user 502 sets any relevant user input (e.g., data format type, user-selectable constraints, etc.), validator 110 performs a data quality check on the dataset in the analysis, and remediator 120 applies the necessary data repairs to resolve the data quality issues identified by validator 110, generating repaired data 510. Workflow #1 also... Figure 6 The diagram schematically illustrates workflow #1 within three boxes 610, 620, and 150A. Box 610 shows the initial dataset entering block 620, where a data quality assessment is performed (e.g., by validator 110). The data quality assessment performed in box 620 may include the listed example data quality assessments, as well as... Figure 2 The data quality check shown. The operation represented by DQM updating 150 corresponds to the operation performed by... Figure 1 The operation performed by the update module 150 shown is configured to update different data quality metrics after iterations of data quality analysis processing performed by workflows #1 and #2 of the DQA system 100. As previously combined... Figure 1 As mentioned in the description of the DQA system 100 shown, the update module 150 is configured to include a real-time DQA module 160 configured to implement novel data quality analysis processing according to various aspects of the present invention. However, in workflow #1, the real-time DQA module 160 is not invoked because the initial dataset at box 610 does not include new or changed data.

[0075] exist Figure 5In this workflow, workflow #2 describes user 502 providing access to raw data 520, which is branched into pipeline 140 for data quality assessment, while an API (not shown) identifies new data with quality issues 530. Typically, user 502 can specify an initial dataset in the API, and the API is configured to provide information to DQA system 100 identifying portions of the raw data 520 that meet the criteria for being one of the specified datasets (i.e., new data). In this way, the API is configured to evaluate any new data being received to determine whether the new data is an addition or update to a previously received initial dataset or a completely new initial dataset. User 502 also provides pipeline 140 with various user choices or inputs (e.g., knowledge graphs, metadata, configurations, etc.) for use in the overall data quality analysis process to generate repaired data 540, as well as various iterative learning and refinement processes (“learning and repair”) of DQA system 100. When the iterative learning and repair process is redone, the update module 150 is accessed to provide historical data quality metrics that can be utilized during the iterative learning and repair process. Update module 150 also includes the previously described real-time DQA module 160, which is configured to perform real-time data quality analysis of changing data by defining multiple data quality metrics that vary over time. Specific details on how the real-time DQA module 160 is implemented are detailed in [link to article / details]. Figure 7 It is shown in the figure and described in more detail hereafter.

[0076] Workflow #2 is also Figure 6 The diagram schematically illustrates workflow #2 in three boxes 630, 640, and 150A. In box 630, a data quality check is applied to the stream of the initial dataset. In box 640, corresponding new / updated data streams are identified from the initial dataset stream (e.g., via an API). At box 640, the new / updated data streams include several updates received over an extended time period. Block 150A is labeled as Data Quality Metric (DQM) Update 150A and corresponds to Update Module 150 (…). Figure 5 The features and functions are shown in the diagram. In DQM update 150A, DQM is calculated incrementally, meaning that DQM is calculated on the new / updated data, as recalculating the DQM on the entire initial dataset each time for data quality assessment is computationally expensive. For example, if validator 110 needs to apply missing value checking and calculate the missing value DQM on the initial dataset with new / updated data, instead of calculating the missing value DQM for the entire initial dataset, then validator 110 is configured to perform an "incremental update" by calculating the missing value DQM of the new / updated data and combining it with the missing value DQM information previously calculated for the unchanged portion of the initial dataset in update module 150. Similar to... Figure 5The update module 150 shown, DQM update block 150A, includes the previously described real-time DQA module 160, which is configured to perform various aspects of incremental updates in a novel manner. More specifically, the real-time DQA module 160 is configured to perform real-time data quality analysis of changing data by applying multiple DQA methods that define multiple data quality metrics that vary over time.

[0077] Figure 7 Details of how the real-time DQA module 160 is implemented according to an embodiment of the present invention are described. Figure 7 As shown, a data block 702 with new data containing quality issue 530 is provided to an update module 150, which, according to various aspects of the invention, utilizes real-time DQA submodules 720, 730, 740, 750, 760, and 770 to calculate the change DQM 710 over multiple time windows, and to efficiently allocate the computer resources required to calculate the change DQM 710 according to various aspects of the invention. Additional details regarding how the real-time DQA submodules 720, 730, 740, 750, 760, and 770 can be implemented according to embodiments of the invention are provided below. Figure 8-11 It is described in 18, 21-23 and 25 and is subsequently described here.

[0078] Figure 7 It also depicts a cloud computing system 50 that communicates with the update module 150 of the DQA system 100 via wired or wireless electronic communication (also in...). Figure 28 (As shown in the diagram). The cloud computing system 50 can supplement, support, or replace some or all of the functions of the update module 150 of the DQA system 100 (in any combination). Additionally, some or all of the functions of the DQA system 100 can be implemented as node 10 of the cloud computing system 50. Figure 28 and 29 (as shown in the image).

[0079] Figure 8A block diagram illustrating how submodule 720, according to various aspects of the invention, can be configured and arranged to calculate details for changing the DQM of data. When data block 702 and new data 530 with quality issues enter submodule 720, submodule 720 calculates a critical DQM value, but also tracks changes in the critical DQM value over multiple time windows. In this way, changes in the critical DQM value over multiple time windows can be detected. For example, in the case where the critical DQM value includes a percentage of missing value DQM, in the most general implementation, submodule 720 can determine the percentage of missing values ​​over the entire interval. In embodiments of the invention, some of the percentages of missing values ​​in the interval can be determined to be more important than other percentages of missing values ​​in the interval based on one or more criteria. For example, if the most recent percentage of missing values ​​is most important for the relevant application of the dataset, submodule 720 can determine the percentage of missing values ​​that occurred most recently (e.g., within a predetermined recent time window). In some embodiments of the invention, submodule 720 generates a weighted percentage of missing values ​​and assigns weights based on predetermined criteria. For example, in some embodiments of the invention, the predetermined criterion may be that the percentage of missing values ​​from newer data points is given a greater weight than the percentage of missing values ​​from less recent data points. Therefore, the real-time DQM submodule 720 is configured and arranged to calculate the DQM for the changing data itself.

[0080] Figure 9 The illustration depicts how submodule 730, according to various aspects of the invention, can be configured and arranged to calculate for changing constraints (e.g., Figure 1 The diagram shows the details of the DQM for constraint 130. When new data with quality issues 530 arrives, submodule 730 calculates DQM values ​​that assess how well the data points of the new data with quality issues 530 satisfy the constraints. These DQM values ​​can vary over time. Embodiments of the invention calculate the constraint DQM for new data points as they arrive, rather than calculating a single constraint DQM for the entire dataset. If the real-time DQM submodule 730 determines that the constraint DQM has changed significantly over time (e.g., above a predetermined threshold), information related to the changed constraint DQM can be communicated to the user. As a non-limiting example, the constraint DQM could be a value that captures how closely a first feature of a data point is related to a second feature of the data point. Other constraint DQMs can be developed and used based on the specific user application to be evaluated.

[0081] Figure 10AA block diagram depicts details of how the illustrated submodule 740 can be configured and arranged to more effectively utilize the features of the DQA system 100 in identifying anomalous regions in data points to notify the user of the presence of anomalous data. In embodiments of the invention, the update module 150 can identify anomalous data by recognizing patterns in data points and then detecting that the patterns in the data points have changed over time. For example, a data point d1 received between times t2 and t3 can be identified as anomalous because d1 exhibits a different pattern than a data point d2 received earlier between times t1 and t2. The real-time DQA submodule 740 is configured to utilize the anomalous data identification features of the update module 150 by accessing the detected anomalous data points and assigning weights to the DQM calculated for the data points that have been identified as anomalous. In some embodiments of the invention, the real-time DQA submodule 740 can be configured to assign weights less than one (1) but greater than zero (0) to anomalous data points based on the severity level of the calculated anomalousness, using, for example, a numerical scoring system. For example, the DQM computer can assign a weight of approximately 20% of a certain value to data points identified as slightly anomalous by update module 150. In some embodiments of the invention, real-time DQA submodule 740 can be configured to assign a weight of zero (0) to anomalous data points, meaning that anomalous data points are ignored in the DQM calculation. In some embodiments of the invention, the assignment of zero (0) weights to anomalous data can be based on update module 150 identifying highly anomalous data based on a numerical scoring system.

[0082] In an exemplary implementation of submodule 740, the dataset ds1 received between times t3 and t2 is determined by submodule 740 to have a higher or lower proportion of null values ​​than the previously received dataset ds2 between times t1 and t2. A higher or lower proportion of null values ​​can be considered anomalous depending on the pattern on which the determination of the dataset anomaly is based. A pattern where a low proportion of null values ​​is followed by a high proportion of null values ​​can cause the high proportion of null values ​​to be marked as anomalous. A pattern where a high proportion of zero values ​​is followed by a low proportion of null values ​​may cause the low proportion of null values ​​to be marked as anomalous. If a recent low proportion of null values ​​may still be anomalous if it meets another criterion for being considered anomalous, then a pattern of low proportions followed by another low proportion of null values ​​may still cause the recent proportion of null values ​​to be marked as anomalous. Submodule 740 may assign a lower weight (i.e., between zero (0) and one (1)) or even ignore d1 when calculating the DQM 710 associated with d1, because the null values ​​associated with d1 and d2 shift anomalously over time (from t1 to t3).

[0083] Figure 10B The diagram illustrates that it can be constructed from submodule 740 ( Figure 10AThe flowchart illustrates a computer-implemented method 1000. In an embodiment of the invention, method 1000 is implemented by submodule 740 to compute DQM on a changing dataset when anomalies are used. In block 1001, submodule 740 identifies one or more anomalous data regions. For example, according to various aspects of the invention, an anomalous data region may be a data region in which the proportion of null values ​​exceeds the proportion of null values ​​in at least one other data region by a threshold. Alternatively, an anomalous data region may include a data region in which the proportion of null values ​​differs from the proportion of null values ​​in other data by a threshold. A wide variety of other criteria can be used to identify anomalous data regions. For example, an anomalous data region may correspond to a region in which the proportion of data values ​​that are outliers exceeds a threshold. An anomalous data region may also correspond to a region having statistical characteristics that differ from other data regions. For example, an anomalous data region may have a mean, median, modulus, variance, and / or standard deviation that differs from other data by a threshold.

[0084] In box 1002, submodule 740 assigns weights to different data regions identified in box 1001. In some embodiments of the invention, box 1002 may assign lower weights to abnormal regions.

[0085] In block 1003, submodule 740 uses the weights calculated in block 1002 to calculate one or more aggregated data quality metrics.

[0086] The DQA system 100, operating within submodule 740, can process data of different modalities. For example, one modality could be time-series data (e.g., Figure 3 The time series data shown is 300), while another modality can be tabular data (e.g., Figure 4 (Data frame 400 shown). In embodiments of the invention, different non-static and dynamic DQMs can be provided for each of these different modalities.

[0087] In embodiments of the invention, non-static and dynamic DQMs can be customized to the type of analytical task associated with the data in the analysis. For example, classification tasks may have some DQMs associated with them, while regression or clustering tasks may have other DQMs associated with them. More generally, the user (e.g., Figure 5 The user 502 shown can utilize the DQA system 100 to perform specific analytical tasks, which may involve some combination of regression, classification, and clustering. The DQA system 100 according to various aspects of the invention can be used by the user to provide specific non-static and dynamic versions of DQM suitable for such specific tasks.

[0088] Figure 11A block diagram illustrating details of how submodule 750 can be configured to perform data quality checks "incrementally" on an initial dataset using novel and "general" methods that incrementally compute a changed DQM 710 for new / updated data. In general, making data quality checks incremental is a challenge. Embodiments of the invention address and overcome this challenge by providing three types of general incremental data quality checks, defined and identified as Type I, Type II, and Weighted Type I & II. In embodiments of the invention, Type I data quality checks are data quality checks with a corresponding DQM that, in general, can be made incremental by breaking down the data quality checks into blocks and then performing a summation process. In embodiments of the invention, Type II data quality checking is a data quality check performed by decomposing a data quality check function (e.g., Q) into its sub-functions (e.g., P and C); applying Type I decomposition / sum operations to each sub-function (e.g., P and C); and aggregating the information from the applied sub-functions (e.g., P and C) to achieve a data quality check (e.g., Q) that is incrementally processed. In embodiments of the invention, weighted Type I and / or weighted Type II data quality checks are data quality checks performed under Type I or Type II, which can be weighted by selecting different weights for different data blocks. Figure 12 Examples of data quality checks that can be performed in DQA system 100 using Type I and / or Type II general case incremental data quality checks are described. Figure 12 The example data quality check shown can also be weighted according to weighted type I and weighted type II techniques.

[0089] Figure 13 The description illustrates an embodiment of the invention. Figure 7 and 11 The flowchart shown is of method 1300 implemented by submodule 750 of update module 150, which is used to perform type I general incremental data quality checks. In method 1300, the incremental data quality check function is Q, and the data quality check function Q applied to dataset D is Q(D). Method 1300 begins at box 1302 and then moves to box 1304 to organize the next dataset D into time-ordered data blocks {D1, D2, ..., Dn}. In box 1306, the next data block in {D1, D2, ..., Dn} is streamed to DQA pipeline 140. Figure 1 and 5(As shown in the diagram) for analysis as the current data block. In decision box 1308, a query is made regarding whether the current data block includes new / updated data. If the answer to the query at decision box 1308 is no, then method 1300 moves to decision box 1310 to determine whether the data quality function Q has previously been applied to the current data block. If the answer to the query at decision box 1310 is no, then the data quality function Q is applied to the current data block at box 1312, and method 1300 proceeds to decision box 1316. If the answer to the query at decision box 1310 is yes, then method 1300 proceeds to decision box 1316.

[0090] Returning to decision box 1308, if the answer to the query at decision box 1308 is yes, then method 1300 proceeds to box 1314, applies the data quality function Q to the new / updated current data block, and proceeds to decision box 1316. In decision box 1316, a query is made regarding whether the current data block is the last data block in dataset D. If the answer to the query at decision box 1316 is no, then method 1300 returns to box 1306 to process the next data block in dataset D. If the answer to the query at decision box 1316 is yes, then method 1300 advances to box 1318 and computes an incrementing Q(D), which is equal to ∑Q(previously computed data blocks of D) plus Q(new / updated data block). Method 1300 returns from box 1318 to box 1304 to process the next dataset.

[0091] According to various aspects of the invention, block 1318 applies Q incrementally to the entire dataset D by applying Q to data blocks with new / updated data and combining that result with the sum of historical results of applying Q to unchanged data blocks {D1, D2, ..., Dn} in the dataset D. The sum of historical Q results is... Figure 15 Equation (1) shows that the sum of the new / updated Q results and the historical Q results is determined by... Figure 15 Equation (2) shown represents this.

[0092] Figure 14 The description illustrates an embodiment of the invention. Figure 7 and 11 The flowchart shown illustrates a method 1400 implemented by submodule 750 of update module 150, which performs a type II general case incremental data quality check. In method 1400, the incremental data quality check function is Q, and the data quality check function Q applied to dataset D is Q(D). Figure 14In the example depicted, the data quality function Q is the percentage of null values ​​in dataset D; Q equals subfunction P divided by subfunction C; subfunction P is the total number of NULLs in dataset D and subfunction C is the count of all data points in dataset D. Method 1400 begins at box 1402 and then moves to box 1404 to organize the next dataset D into time-ordered data blocks {D1, D2, ..., Dn}. In box 1406, the next data block in {D1, D2, ..., Dn} is streamed into the DQA pipeline 140 (…). Figure 1 and 5 (As shown) is used as the current data block for analysis. In decision box 1408, a query is made regarding whether the current data block includes new / updated data. If the answer to the query at decision box 1408 is no, then method 1400 moves to decision box 1410 to determine whether sub-functions P and C have previously been applied to the current data block. If the answer to the query at decision box 1410 is no, then sub-functions P and C are applied to the current data block at box 1412, and method 1400 proceeds to decision box 1416. If the answer to the query at decision box 1410 is yes, then method 1400 proceeds to decision box 1416.

[0093] Returning to decision box 1408, if the answer to the query at decision box 1408 is yes, then method 1400 proceeds to box 1414, applying subfunctions P and C to the new / updated current data block, and then proceeds to decision box 1416. In decision box 1416, a query is performed regarding whether the current data block is the last data block in dataset D. If the answer to the query at decision box 1416 is "no", then method 1400 returns to box 1416 to process the next data block in dataset D. If the answer to the query at decision box 1416 is "yes", then method 1400 advances to box 1418 and computes an incrementing Q(D), which is equal to ∑P(previously computed data block of D) / (∑C(previously computed data block of D)+C(new / updated data block))+∑P(new / updated data block of D) / (∑C(previously computed data block of D)+C(new / updated data block of D)). From box 1418, method 1400 returns to box 1404 to process the next dataset 1404.

[0094] According to various aspects of the invention, block 1418 incrementally applies Q to the entire dataset D using a type I (method 1300) decomposition / summation process, which has been modified to account for decomposable operations used in the associated DQM calculation. In example method 1400, where the quality function Q of the DQM calculation is the percentage of null values ​​in dataset D, the percentage calculation Q can be decomposed into two functions P and C, where C counts the total number of data points in dataset D, and where P represents the null values ​​in dataset D. Accordingly, Q(D) can be calculated as P(D) divided by C(D). The general incremental data quality check in method 1400 according to various aspects of the invention provides a mechanism for decomposing the quality function Q by applying P to each data block entering the DQA pipeline; applying C to each data block entering the DQA pipeline; and storing P({D1, D2, ..., Dn}) and C({D1, D2, ..., Dn}) for each data block in the dataset. Q(D) for the entire dataset D is obtained incrementally by applying P to data blocks with new / updated data; applying C to said data blocks with new / updated data; and combining this result with the sum of historical results of applying P and C to unchanged data blocks {D1, D2, ..., Dn} in dataset D. The sum of historical P and C results is obtained by... Figure 15 Equation (3) shown indicates that the sum of the new / updated P and C results and the historical P and C results is determined by... Figure 15 Equation (4) shown in the figure represents this. Although method 1400 is described as being for a specific example, where function Q can be represented as subfunction P divided by subfunction C, method 1400 is applicable to a wide variety of functions, as long as the function can be decomposed into subfunctions and then recombined.

[0095] Figure 16 Equations representing weighted Type I & II data quality checks according to various aspects of the invention are described. In some embodiments of the invention, different weights can be applied to different data blocks based on any criteria previously described in conjunction with various aspects of the invention for applying weights to data checks. In some aspects of the invention, the weight w can be selected based on the time elapsed from the last timestamp of a given data block to the current time. In some aspects of the invention, the weights can be uniformly applied to previous data blocks in the dataset. Examples of how weights can be incorporated into Type I data quality checks according to various aspects of the invention are provided below. Figure 16 Equation 5 in the text represents this. An example of how weighting can be incorporated into Type II data quality checks according to various aspects of the invention is provided by... Figure 16 Equation 6 in the text represents this.

[0096] Figure 17 The architecture of submodule 750A is described, illustrating how submodule 750A can be implemented according to an embodiment of the invention. Figure 7 and 11 Details are shown in the diagram. Submodule 750A can be used to utilize... Figure 15 and 16 Equations 1-6 shown are used to achieve this. Figure 13 and 14 Methods 1300 and / or 1400 are shown. For example... Figure 17 As shown, submodule 750A includes a local analyzer 752 configured to perform data quality analysis checks on incoming data blocks and send its results to mini-storage 753 for subsequent use by quality metric generator 755 and optionally by weight generator 754 in calculating incremental data quality metric Q(Dw U Dn+1).

[0097] Figure 18 A block diagram illustrating details of how submodule 760, according to an embodiment of the invention, can be configured to manage the state for incremental DQM calculations. In embodiments of the invention, state management submodule 760 provides a mechanism for maintaining the state, which is necessary for performing incremental DQM calculations because incremental calculations need to track how previous calculations will be used in the incremental calculations. Embodiments of the invention avoid making the DQA system 100 ( Figure 1 and 5 (As shown in the diagram) This necessitates maintaining its own historical data quality inspection information database. Block 762 is configured and arranged to execute functions for calculating the DQM, calculating DQM updates, and passing state variables back and forth between submodule 760 and the user program shown in block 766. Block 764 is configured and arranged to execute functions for analyzing and outputting data quality results, and passing state variables back and forth between submodule 760 and the user program shown in block 766. In an embodiment of the invention, the DQA system 100 is written in the Python programming language, and the Python API of the state management submodule 760 is configured in block 766 to enable the user program to use the Python API to call various functions related to state management. The state management submodule 760 calculates the DQM in block 762 and passes the associated state information to the user program in block 766. To utilize the state information received from the state management submodule 760, in block 766, the user program in block 764 calls functions of the state management submodule 760, which are configured and arranged to actually analyze the data and output data quality variables. Therefore, the use of blocks 762 and 764 in the state management submodule 760A according to various aspects of the present invention does not require the user program at block 766 to have the ability to interpret the format of the state variables. At block 766, the user program only needs to use the Python API to call the state management submodule 760.

[0098] Figure 19 Described Figure 18 The example shown illustrates how the state management module 760 can be configured to maintain state information regarding previous DQM calculations. For instance, block 762 can calculate the DQM for data region d1. State information corresponding to these DQMs is maintained (e.g., the proportion of null values ​​in d1, the average of all or part of d1, the mean, standard deviation, etc.); block 762 calculates the DQM for a new region d2. Although in some embodiments of the invention, this state information can be in the DQA system 100 (… Figure 1 and 5 While state information is maintained in a file system or database (as shown in the diagram), embodiments of the present invention also address situations where maintaining state information using a file system or database is not feasible. Therefore, embodiments of the present invention provide blocks 762, 764 configured and arranged to transfer state information (i.e., state variables) between blocks 762, 764 and a client program (shown in block 766) that accesses the DQA system 100 via its API. This configuration enables the state management submodule 760 to generate state variables and transfer them between the DQA system 100 and the client program (i.e., block 766) via the DQA system 100's API.

[0099] In a specific example where the DQA system 100 is implemented in the Python programming language, a data region is passed to the DQA system 100 using a Python (e.g., Pandas) DataFrame. The DQM, implemented as a Python function or method, can accept the parameter `interval_info_list`, where each element in the list contains information about the data region, such as its relative position within the entire dataset. Other fields corresponding to the DQM of the data region can be included, such as the proportion of missing values, the proportion of infinity values, and the proportion of zero values.

[0100] For example, a Python function (or method) for incrementally checking for null values ​​in a dataset can be implemented as follows. This function updates the null value metric as a new data region is received. The function could have the following signature: `check_na_columns_incremental(df, offset, Interval_info_list)`, where `df` is a data frame containing the data for the data region; `offset` represents the relative position of the data region within the entire dataset; and `interval_info_list` includes the results of analysis from previous data regions. Therefore, the function `check_na_columns_incremental` analyzes `df`, appends the results to `interval_info_list`, and returns the updated value of `interval_info_list`.

[0101] In embodiments of the present invention, the user program (also referred to as the client program) 766 does not need to process the internal structure of the interval_info_list variable. The user program 766 can simply pass the interval_info_list variable to the DQA system 100 via the API and rely on the DQA system 100 to interpret the interval_info_list variable.

[0102] Figure 20 Examples of how a DQA system 100, in some embodiments of the invention, provides a method for analyzing DQM over time based on state information maintained in the interval_info_list parameter are described. A function or method for implementing this is analyze_data_by_interval(interval_info_list), which provides several types of analysis, including analyzing DQM over a single data region and aggregated DQM over multiple data regions. This function / method can detect trends and anomalies across different data regions. It can also plot data quality metrics corresponding to different data regions and across different time intervals. It can also assign different weights to different data regions when calculating the aggregated DQM. In some embodiments of the invention, more recent data can be assigned a higher weight compared to less recent data. In some embodiments of the invention, assigning different weights to different regions when calculating the aggregated DQM can be achieved by using an exponentially weighted moving average.

[0103] Figure 21-26 Details are described regarding how a submodule 770 for efficient prioritization of DQM computation is implemented according to embodiments of the invention. Because achieving the most accurate and up-to-date DQM at all times can be prohibitively costly, submodule 770 is configured and arranged to strike a reasonable trade-off between providing a reasonable data quality estimate and avoiding excessive computational resource usage, particularly for large datasets. Reasonable choices can be made regarding the frequency of recalculating the DQM and the frequency at which the most suitable DQM is selected for recalculation. Factors that typically increase the likelihood of excessive computational resources include the size of the dataset and the number of DQM users that need to be tracked.

[0104] In embodiments of the invention, submodule 770 is configured and arranged to maintain information about the performance of different DQMs as a function of data size and other possible characteristics of the data. Submodule 770 is configured to maintain historical data about DQM performance. When analyzing a new dataset, submodule 770 is configured to maintain historical data (HR) in the historical recorder (e.g., Figure 21The history recorder 771 shown maintains persistent information about performance and other execution characteristics. Submodule 770 can analyze HR to better evaluate the performance of DQM calculations performed by submodule 770. HR maintains information about the execution of DQM. When a DQM function is executed, HR records information such as: the data quality function name and parameters; the size / dimension of the dataset being analyzed; the CPU time consumed by the function execution; the wall clock time consumed by the function execution; I / O and / or network overhead, if significant; and the hardware and software used to execute the function. For tabular data (e.g., ... Figure 4 As shown in data frame 400, HR maintains information such as the number of rows, the number of columns, and information about the data type of the columns (e.g., number, string, category, etc.). HR allows submodule 770 to create performance distributions for all interested DQMs (e.g., ...). Figure 21 The DQM performance distribution shown is 772. For a given dataset and dataset size, submodule 770 can therefore estimate the overhead of different DQMs performed on that dataset. Submodule 770 is further configured to maintain information about how the DQM changes as the data itself changes. Submodule 770 is configured to use this information to predict how much the DQM is expected to change in response to new data. Such a change prediction can be performed using simple calculations (e.g., Figure 21 The simulation algorithm 775 shown) or more complex machine learning models (e.g., Figure 21 The machine learning algorithm and model 774 shown are used for this purpose.

[0105] Figure 21A block diagram illustrating a functional architecture for implementing submodule 770 according to an embodiment of the invention to efficiently prioritize DQM computation, thereby effectively managing the computational resources of DQM system 100 for performing the various non-static and dynamic data quality checks and DQM described herein. The architecture of submodule 770 includes a history recorder 771 and one or more prediction algorithms 773 configured to generate predictions 776. In embodiments of the invention, the history recorder 771 includes DQM performance profiles 772, each of which may include past executions of test workloads; past executions of actual workloads; and other sources of DQM performance data. Prediction algorithms 773 may include machine learning algorithms and models 774 and / or simulation algorithms 775. Machine learning algorithms and models 774 may be trained to generate a model of DQM system 100, including the dataset analyzed by DQM system 100 and the associated computational overhead. Simulation algorithms 775 are configured to model or simulate the performance of DQM system 100, including the computational resources required to perform various data quality checks and analyses. The prediction algorithm 773 is configured to generate predictions 773, which may include, but are not limited to, predicting how much a given DQM will be expected to change in response to new / updated data; predicting the cost of different DQMs performed on a given dataset and dataset size; and / or other DQM cost-efficiency predictions.

[0106] Figure 22 Another block diagram illustrating the functional characteristics of submodule 770 according to an embodiment of the invention is provided. In embodiments of the invention, submodule 770 is configured to prioritize the execution of DQM computation based on various factors, including but not limited to the computational overhead required to compute the DQM; the rate of change of a given DQM; whether a previously computed DQM can be used without exceeding a threshold of precision loss; and constraint priorities set by the user. In embodiments of the invention, submodule 770 is configured to focus its analysis on constraints of selected types, including but not limited to constraints most closely related to important characteristics of the application (user-specified or known from past experience of the application); constraints that change most frequently; and constraints with low computational cost, having low computational and update overhead. In embodiments of the invention, submodule 770 is configured to use these characteristics to rank constraints, such that higher-ranked constraints are evaluated more frequently.

[0107] Figure 23 A block diagram describing the details of the algorithm implemented by submodule 770A according to an embodiment of the present invention is provided. Figure 24 The various aspects of the algorithm implemented by submodule 770A are described. Therefore, Figure 23 The following description of the submodule 770 shown also refers to Figure 24The project described herein. According to various aspects of the invention, submodule 770A includes all the features and functionality of submodule 770 previously described herein, with added functionality of the algorithms described in submodule 770A. In embodiments of the invention, the algorithm of submodule 770A is configured to efficiently prioritize DQM computation and thereby efficiently manage the computational resources of DQM system 100 used to perform the various non-static and dynamic data quality checks and DQMs described herein. The algorithm of submodule 770A begins at block 2301, where DQM system 100 maintains past statistics on computational overhead o and rate of change f for different DQMs. Variable f represents the magnitude of the change in DQM when new data is received. Block 2301 is executed continuously over time. In block 2302, the user (e.g., Figure 5 The user 502 shown can optionally assign importance values ​​i to one or more metrics. The default value of i is one (1) for cases where the user does not provide an importance value. In block 2303, submodule 770A calculates the order r(o, f, i) for each DQM. In embodiments of the invention, various different functions can be used for r. In one example, the function rating = a*o + b*f + c*i is used to assign ratings, where a is a negative constant; b is a positive constant; c is a positive constant; o is the overhead for calculating the DQM; f is the rate of change of the DQM when new data is received; i is the importance of the DQM provided by the user; and if no user value is selected, the default value one (1) is used. The order function r sorts the DQMs in descending order of rating. In block 2304, submodule 770A can dedicate tmax time to updating the DQMs. In block 2305, submodule 770A performs the updates to the DQMs in the order specified by the order function r. In block 2306, after all DQMs have been updated or if tmax has been exceeded, submodule 770A stops updating the DQMs.

[0108] Figure 25 A block diagram depicting the details of an algorithm implemented by submodule 770B according to an embodiment of the present invention is provided, and Figure 26 Depicting by Figure 25 The algorithm implemented by submodule 770B shown herein, therefore, for Figure 25 The following description of the submodule 770B shown also refers to Figure 26The items shown are described herein. According to various aspects of the invention, submodule 770B includes all the features and functions of submodule 770 previously described herein, with added functionality of the algorithms described in submodule B. In embodiments of the invention, the algorithm of submodule 770B is configured to efficiently prioritize DQM computation, thereby efficiently managing the computational resources of the DQM system 100 used to perform the various non-static and dynamic data quality checks and DQM described herein. According to various aspects of the invention, the algorithm of submodule 770B avoids the problem that if the DQA system 100 always runs out of time to compute DQM (i.e., always exceeds tmax), lower-ranked DQMs may never be updated.

[0109] The algorithm of submodule 770B begins in block 2501, where submodule 770B maintains past statistics on the computational overhead o and rate of change f for different DQMs. The variable f represents the magnitude of the DQM change when new data is received. Block 2501 is executed continuously over time. In block 2502, the user (e.g., Figure 5 The user 502 shown can optionally assign importance values ​​i to one or more metrics. The default i value is one (1) for cases where the user does not provide an importance value. Submodule 770B calculates a rating r1(o, f, i) for each DQM in block 2503. In embodiments of the invention, r1 can use various different functions. In one example, the function r1 = a*o + b*f + c*i is used to assign ratings, where a is a negative constant; b is a positive constant; c is a positive constant; o is the overhead used to calculate the DQM; f is the rate of change of the DQM when new data is received; i is the importance of the DQM provided by the user; and if no user selection is provided, the default value one (1) is used. The value of r1 indicates the relative frequency at which submodule 770 should update the DQM. All r1 values ​​should be positive. To ensure that all DQMs are calculated at least some time, the ratio between the highest and lowest values ​​is chosen to not exceed a threshold (i.e., it should not be too high), even if tmax is always exceeded. In block 2504, submodule 770B may dedicate the tmax time to updating the DQM. In block 2505, submodule 770B performs the update of the DQM. Each DQM is updated at a frequency proportional to its r1 value. In block 2506, after all DQMs have been updated or if tmax has been exceeded, submodule 770B stops performing the update of the DQM.

[0110] Figure 27A high-level block diagram of a computer system 2700 is depicted, which can be used to implement one or more computer processing operations according to various aspects of the present invention. Although an exemplary computer system 2700 is shown, the computer system 2700 includes a communication path 2726 connecting the computer system 2700 to an additional system (not shown), and may include one or more wide area networks (WANs) and / or local area networks (LANs), such as the Internet, intranets, and / or wireless communication networks. The computer system 2700 and the additional system communicate via the communication path 2726, for example, to transfer data between them.

[0111] Computer system 2700 includes one or more processors, such as processor 2702. Processor 2702 is connected to communication infrastructure 2704 (e.g., a communication bus, jumper, or network). Computer system 2700 may include display interface 2706, which forwards graphics, text, and other data from communication infrastructure 2704 (or from a frame buffer, not shown) for display on display unit 2708. Computer system 2700 also includes main memory 2710, preferably random access memory (RAM), and may also include secondary memory 2712. Secondary memory 2712 may include, for example, hard disk drive 2714 and / or removable storage drive 2716, which represents, for example, a floppy disk drive, magnetic tape drive, or optical disk drive. Removable storage drive 2716 reads from and / or writes to removable storage unit 2718 in a manner known to those skilled in the art. Removable storage unit 2718 represents, for example, a floppy disk, compact disk, magnetic tape or optical disk, flash memory drive, solid-state memory, etc., read from and written to by removable storage drive 2716. As will be understood, the removable storage unit 2718 includes a computer-readable medium in which computer software and / or data are stored.

[0112] In alternative embodiments, auxiliary memory 2712 may include other similar means for allowing computer programs or other instructions to be loaded into the computer system. Such means may include, for example, removable storage unit 2720 and interface 2722. Examples of such means may include packages and package interfaces (e.g., found in video game devices), removable memory chips (e.g., EPROM or PROM) and associated sockets, and other removable storage units 2720 and interfaces 2722 that allow software and data to be transferred from removable storage unit 2720 to computer system 2700.

[0113] Computer system 2700 may also include a communication interface 2724. Communication interface 2724 allows software and data to be transferred between the computer system and external devices. Examples of communication interface 2724 may include a modem, a network interface (e.g., an Ethernet card), a communication port, or a PCM-CIA slot and card. The software and data transmitted via communication interface 2724 are in the form of signals, which may be, for example, electronic, electromagnetic, optical, or other signals that can be received by communication interface 2724. These signals are provided to communication interface 2724 via a communication path (i.e., channel) 2726. Communication path 2726 carries signals and may be implemented using wires or cables, optical fibers, telephone lines, cellular telephone links, RF links, and / or other communication channels.

[0114] In this specification, the terms "computer program medium," "computer-usable medium," "computer program product," and "computer-readable medium" are used to generally refer to media such as main memory 2710 and secondary memory 2712, removable storage drive 2716, and hard disks installed in hard disk drive 2714. The computer program (also referred to as computer control logic) is stored in main memory 2710 and / or secondary memory 2712. The computer program may also be received via communication interface 2724. When run, such a computer program enables the computer system to perform the features of the invention as discussed herein. In particular, when run, the computer program enables processor 2702 to perform the features of the computer system. Therefore, such a computer program represents the controller of the computer system.

[0115] The technical effects and benefits of the disclosed DQA system for performing real-time data quality analysis include, but are not limited to, the following. Embodiments of the present invention provide a DQA system that performs accurate data analysis checks in several problem domains, particularly in applications where new data is continuously streamed in. The DQA system generates data quality metrics that are continuously updated as new data is received. The DQA system according to various aspects of the present invention also provides novel methods for performing data quality assessments while data is continuously streamed in.

[0116] An additional technical benefit of the DQA system according to various aspects of the present invention lies in its definition of multiple metrics that change over time. While the disclosed DQA system provides incremental calculations for multiple different data quality metrics, it significantly goes beyond past work in defining new metrics that improve upon existing metrics for static datasets. The disclosed DQA system is configured to define new metrics when considering inflowing new data, applicable to continuously changing data rather than simply using existing metrics. The disclosed DQA system allows data quality checks to be defined across specific windows of the dataset.

[0117] An additional technical benefit of the DQA system according to various aspects of the present invention is that different weights can be assigned to different time periods for calculating data quality metrics. Typically, more recent data points can be assigned higher weights to evaluate data quality metrics compared to less recent data points. Each sample or data point can be assigned a different weight based on its time. Samples can be grouped by time intervals, and the same weight can be assigned to sample sets belonging to the same group. In some cases, older values ​​can be completely ignored. Different algorithms can be applied to determine which earlier values ​​should be ignored.

[0118] An additional technical benefit of DQA systems is that they can provide time-parameterized data quality metrics. These metrics can be calculated and visualized across any range of data points.

[0119] An additional technical benefit of the DQA system according to various aspects of the invention is that it is configured to provide a complete analysis of a dataset, including constraints applicable to the dataset. The DQA system according to various aspects of the invention utilizes the observation that, for real-time streaming data, the applicability of constraints is not a static, fixed property. A constraint may be applicable at a particular time but not applicable to newly received data. Therefore, the disclosed DQA system is configured to provide analysis of constraints across multiple time scales. Thus, the disclosed DQA system treats the applicability of constraints or sets of constraints as dynamic and anticipates changes over time.

[0120] An additional technical benefit of the DQA system according to various aspects of the present invention is that it maintains relevant values ​​at several different time intervals and can mark the time intervals in which anomalous behavior is detected. An example of anomalous behavior is that the value of a relevant data point changes to a previously unseen value. The disclosed DQA system is configured to disclose the time intervals of anomalous behavior to the user. In calculating the overall quality assessment, the anomalous time intervals can be omitted or assigned a lower weight.

[0121] An additional technical benefit of the DQA system according to various aspects of the present invention is that it can handle data of different modalities (e.g., time series data and / or tabular data). The disclosed DQA system is configured to provide different data quality metrics for each type of modality it is configured to process.

[0122] An additional technical benefit of the DQA system according to various aspects of the present invention is that it tailors data quality metrics to the type of analytical task associated with the data. For example, classification tasks have certain data quality metrics associated with them, while regression or clustering tasks have other data quality metrics associated with them. More generally, users can perform specific analytical tasks, which may involve some combination of regression, classification, and clustering. The disclosed DQA system can provide specific data quality metrics suitable for such specific tasks.

[0123] An additional technical benefit of the DQA system according to various aspects of the present invention is that it is configured to maintain state information about previously calculated data quality metrics. For example, when the system calculates a data quality metric for a new area, state information corresponding to the calculated data quality metric can be maintained. This state information can be maintained in a file system or a database. In cases where maintaining state information using a file system or database is not feasible, the disclosed DQA system is configured to transfer state information between the accessing client program and the disclosed DQA system via an API. Thus, the disclosed DQA system generates state variables, and once the state variables are created, they are transferred between the disclosed DQA system and the client program via the API.

[0124] An additional technical benefit of the DQA system according to various aspects of the present invention lies in its ability to effectively manage the trade-off between efficiency and accuracy of data quality metrics. Achieving the most accurate and up-to-date data quality metrics at all times can be prohibitively expensive. Therefore, the disclosed DQA system makes an appropriate trade-off in providing reasonable data quality estimates without using excessive computational resources. The disclosed DQA system is configured to make electronically intelligent choices regarding both the frequency of recalculating data quality metrics and the selection of the most appropriate data quality metric for recalculation.

[0125] An additional technical benefit of the DQA system according to various aspects of the present invention is that it maintains information about the performance of different data quality metrics as a function of data size and other possible data characteristics. The disclosed DQA system maintains historical data about the performance of novel data quality metrics generated by the DQA system. When analyzing a new dataset, the disclosed DQA system maintains persistent information about performance and other execution characteristics in a history recorder (HR). The HR is analyzed to better understand the performance of our data quality metrics. The HR maintains information about the execution of the data quality metrics. Therefore, for a given dataset and dataset size, the disclosed DQA system can estimate the overhead of different data quality metrics performed on that dataset.

[0126] An additional technical benefit of the DQA system according to various aspects of the invention is its maintenance of information about how data quality metrics change as the data itself changes. The disclosed DQA system uses this information to predict how much the data quality metrics are expected to change in response to new data. Such predictions of change can be made using simple computations (e.g., using simulation algorithms) or more complex machine learning models.

[0127] An additional technical benefit of the DQA system according to various aspects of the present invention is that it limits the invocation of data quality metrics with high overhead and allows data quality metrics with lower overhead to be executed more frequently.

[0128] An additional technical benefit of the DQA system according to various aspects of the present invention lies in its focus on the data change rate and the data quality metric itself. If the change rate is high, the data quality metric needs to be recalculated more frequently.

[0129] An additional technical benefit of the DQA system according to various aspects of the present invention is that, as more data is received, it can use simple computation and predictive models to estimate how much the data quality metric is expected to change. The disclosed DQA system is configured to recalculate the data metric expected to change the most.

[0130] An additional technical benefit of the DQA system according to various aspects of the invention is that, when it recalculates the novel data quality metrics described herein, it generates updated information about how much the data quality metrics change in response to changes in the data. This information can be used to update predictive models about how the data quality metrics change with changes in the input data. Thus, when the disclosed DQA system is executed, it becomes more intelligent over time in predicting the behavior of data quality metrics and more accurate over time in calculating performance metrics (with limited computational resources).

[0131] An additional technical benefit of the DQA system according to various aspects of the present invention is that users have the ability to assign importance scores to data quality metrics. A higher importance score indicates that the most recent score of the data quality metric is more important.

[0132] It should be understood beforehand that although this disclosure includes a detailed description of cloud computing, the implementation of the teachings described herein is not limited to a cloud computing environment. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0133] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with service providers. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0134] The characteristics are as follows:

[0135] On-demand self-service: Cloud consumers can unilaterally and automatically provide computing power, such as server time and network storage, as needed, without requiring manual interaction with the service provider.

[0136] Wide Area Network (WAN) Access: Capabilities are available on the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0137] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. Location independence has significance because consumers typically do not control or know the exact location of the resources provided, but can specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0138] Rapid Flexibility: In some cases, the ability to scale outwards and inwards quickly and flexibly can be provided. For consumers, the available capacity often appears unlimited and can be purchased in any quantity at any time.

[0139] Measurement services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency for both service providers and consumers.

[0140] The service model is as follows:

[0141] Software as a Service (SaaS): The capability offered to consumers is the ability to use the provider's applications running on cloud infrastructure. Applications can be accessed from various client devices through thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application capabilities, with possible exceptions such as limited user-specific application configuration settings.

[0142] Platform as a Service (PaaS): This provides consumers with the ability to deploy consumer-created or acquired applications onto cloud infrastructure using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and the configuration of any application hosting environments.

[0143] Infrastructure as a Service (IaaS): This provides consumers with the capability to deliver processing, storage, networking, and other basic computing resources that enable them to deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they do have control over the operating system, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0144] The deployment model is as follows:

[0145] Private cloud: Cloud infrastructure operated solely by an organization. It can be managed by the organization or a third party and can exist inside or outside a building.

[0146] Community cloud: Cloud infrastructure shared by several organizations and supporting a specific community with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can exist on-site or off-site.

[0147] Public cloud: Cloud infrastructure available to the general public or large industrial groups and owned by organizations that sell cloud services.

[0148] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a single entity but are bound together by standardized or proprietary technologies that enable data and applications to be ported together (e.g., cloud bursting for load balancing between clouds).

[0149] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure of a network of interconnected nodes.

[0150] Now for reference Figure 27The illustration depicts a cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 that can communicate with local computing devices used by cloud consumers, such as personal digital assistants (PDAs) or cellular phones 54A, desktop computers 54B, laptop computers 54C, and / or automotive computer systems 54N. The nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 50 to provide infrastructure, platform, and / or software as a service, without requiring cloud consumers to maintain resources on their local computing devices. It should be understood that... Figure 27 The types of computing devices 54A-N shown are for illustrative purposes only, and computing node 10 and cloud computing environment 50 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0151] Now for reference Figure 28 This demonstrates a cloud computing environment of 50 ( Figure 27 This provides a set of functional abstractions. It should be understood beforehand that... Figure 28 The components, layers, and functions shown are for illustrative purposes only, and embodiments of the invention are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0152] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: a host 61; a server 62 based on a RISC (Reduced Instruction Set Computer) architecture; a server 63; a blade server 64; a storage device 65; and a network and network components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0153] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 71; virtual storage 72; virtual network 73, including virtual private network; virtual application and operating system 74; and virtual client 75.

[0154] In one example, management layer 80 can provide the following functionalities: Resource Provisioning 81 provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and Pricing 82 provides cost tracking when utilizing resources in the cloud computing environment, as well as billing or invoicing for consuming these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, and protection for data and other resources. User Portal 83 provides access to the cloud computing environment for consumers and system administrators. Service Level Management 84 provides cloud resource allocation and management to ensure that required service levels are met. Service Level Agreement (SLA) Planning and Fulfillment 85 provides pre-scheduling and procurement of cloud resources, where future needs are anticipated according to the SLA.

[0155] Workload layer 90 provides examples of functionalities that can be leveraged in a cloud computing environment. Examples of workloads and functionalities that can be provided from this layer include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analytics and processing 94; transaction processing 95; and performing effective real-time data quality analysis 96.

[0156] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the terms “comprising” and / or “including” as used in this specification designate the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0157] The following definitions and abbreviations are used to interpret the claims and specification. As used herein, the terms “comprising,” “including,” “having,” “containing,” or any other variations thereof are intended to cover a non-exclusive inclusion. For example, a composition, mixture, process, method, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such compositions, mixtures, processes, methods, articles, or apparatus.

[0158] Additionally, the term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs. The terms "at least one" and "one or more" are understood to include any integer greater than or equal to one, i.e., one, two, three, four, etc. The term "multiple" should be understood to include any integer greater than or equal to two, i.e., two, three, four, five, etc. The term "connection" can include both indirect "connection" and direct "connection."

[0159] The terms “about,” “substantially,” “approximately,” and variations thereof are intended to include a degree of error associated with a measurement of a specific quantity based on the equipment available at the time of filing this application. For example, “about” may include a range of ±8%, 5%, or 2% of a given value.

[0160] This invention can be a system, method, and / or computer program product at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform aspects of the invention.

[0161] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or recessed structures with instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0162] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the respective computing / processing device.

[0163] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages ​​such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to perform aspects of this invention, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions to personalize the electronic circuits by utilizing the status information of the computer-readable program instructions.

[0164] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0165] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other devices to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0166] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0167] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions indicated in the blocks may occur in a non-consecutive order as shown in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0168] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or improvements to existing technologies on the market, or to enable others skilled in the art to understand the embodiments described herein.

Claims

1. A computer-implemented method for evaluating data quality, the computer-implemented method comprising: The processor receives multiple updates to data points in the data stream. An example of using the processor to calculate the data quality metric DQM from the data points in the data stream; The instances of DQM are configured to differentiate the data points in the data stream based on time, and to assign higher weights to instances of DQM calculated based on more recent data points in the data stream. as well as As the processor receives more data points, the DQM instance is continuously updated, while the processor cycles consumed by updating the DQM instance are limited to a threshold.

2. The computer-implemented method of claim 1, further comprising generating a data quality assessment using instances of the DQM spanning multiple time ranges.

3. The computer-implemented method of claim 2, further comprising using the processor to select a data quality remediation strategy based at least in part on the data quality assessment.

4. The computer-implemented method of claim 3, further comprising using the processor to apply the data quality remediation strategy to the data stream.

5. The computer-implemented method of claim 1, wherein the DQM is selected from the group consisting of: The number of lost data points in the data stream; The number of replica data points in the data stream; The number of null values ​​in the data points of the data stream; and The percentage of null values ​​in the data points of the data stream.

6. A computer-implemented method for evaluating data quality, the computer-implemented method comprising: The processor receives multiple updates to data points in the data stream. An example of using the processor to calculate the data quality metric DQM from the data points in the data stream; Analyze instances of DQM calculated over multiple time intervals; Determine at least one of the plurality of time intervals that has an outlier value for an instance of the DQM calculated for at least one of the plurality of time intervals; as well as Calculate the weighted aggregate value of the instances of the DQM across the multiple time intervals; Instances of DQM in the time intervals with outliers are assigned the lower value of the weighted aggregate value.

7. The computer-implemented method of claim 6, wherein determining outliers of at least one of a plurality of time intervals having an instance of DQM calculated for at least one period of the plurality of time intervals includes analyzing patterns of data points.

8. The computer-implemented method of claim 7, wherein analyzing the pattern of the data points includes comparing the pattern of the data points with an expected pattern of the data points.

9. The computer-implemented method of claim 6, wherein, based on the severity level of the calculated outlier, the lower value of the weighted aggregate value is assigned to instances of the DQM in the plurality of time intervals having the outlier.

10. The computer-implemented method of claim 6, wherein the DQM is selected from the group consisting of: The number of lost data points in the data stream; The number of replica data points in the data stream; The number of null values ​​in the data points of the data stream; and The percentage of null values ​​in the data points of the data stream.

11. A computer system for evaluating data quality, the computer system including a memory communicatively coupled to a processor, wherein the processor is configured to perform operations including: Receive multiple updates to data points in the data stream; An instance of calculating the data quality metric DQM from the data points in the data stream; The instances of DQM are configured to differentiate the data points in the data stream based on time, and to assign higher weights to instances of DQM calculated based on more recent data points in the data stream. as well as As the processor receives more data points, the DQM instance is continuously updated, while the processor cycles consumed by updating the DQM instance are limited to a threshold.

12. The computer system of claim 11, wherein the operation further includes generating a data quality assessment using instances of the DQM across multiple time ranges.

13. The computer system of claim 12, wherein the operation further includes selecting a data quality remediation strategy based at least in part on the data quality assessment.

14. The computer system of claim 13, wherein the operation further includes applying the data quality remediation strategy to the data stream.

15. The computer system of claim 11, wherein the DQM is selected from the group consisting of: The number of lost data points in the data stream; The number of replica data points in the data stream; The number of null values ​​in the data points of the data stream; and The percentage of null values ​​in the data points of the data stream.

16. A computer system for evaluating data quality, the computer system including a memory communicatively coupled to a processor, wherein the processor is configured to perform operations including: Receive multiple updates to data points in the data stream; An instance of calculating the data quality metric DQM from the data points in the data stream; Analyze instances of DQM calculated over multiple time intervals; Determine at least one of the plurality of time intervals that has an outlier value for an instance of the DQM calculated for at least one of the plurality of time intervals; as well as Calculate the weighted aggregate value of the instances of the DQM across the multiple time intervals; Instances of DQM in the time intervals with outliers are assigned the lower value of the weighted aggregate value.

17. The computer system of claim 16, wherein determining that at least one of the plurality of time intervals has an outlier for an instance of the DQM calculated for at least one of the plurality of time intervals includes analyzing the pattern of the data points.

18. The computer system of claim 17, wherein analyzing the pattern of the data points includes comparing the pattern of the data points with a desired pattern of the data points.

19. The computer system of claim 16, wherein, based on the severity level of the calculated outlier, the lower value of the weighted aggregate value is assigned to instances of the DQM in the plurality of time intervals having the outlier.

20. The computer system of claim 16, wherein the DQM is selected from the group consisting of: The number of lost data points in the data stream; The number of replica data points in the data stream; The number of null values ​​in the data points of the data stream; and The percentage of null values ​​in the data points of the data stream.

21. A computer program product for evaluating data quality, the computer program product comprising a computer-readable storage medium having program instructions embodied therein, the program instructions being executable by a processor system to cause the processor system to perform operations, the operations including: Receive multiple updates to data points in the data stream; An instance of calculating the data quality metric DQM from the data points in the data stream; The instances of DQM are configured to differentiate the data points in the data stream based on time, and to assign higher weights to instances of DQM calculated based on more recent data points in the data stream. as well as As the processor receives more data points, the DQM instance is continuously updated, while the processor cycles consumed by updating the DQM instance are limited to a threshold.

22. The computer program product of claim 21, wherein the operation further comprises: Analyze instances of DQM calculated over multiple time intervals; Determine at least one of the plurality of time intervals that has an outlier value for an instance of the DQM calculated for at least one of the plurality of time intervals; as well as Calculate the weighted aggregate value of the instances of the DQM across the multiple time intervals; DQM instances with outliers in multiple time intervals are assigned the lower of the weighted aggregate value.

23. The computer program product of claim 22, wherein the operation further comprises: Generate data quality assessments using DQM instances spanning multiple time ranges; The data quality remediation strategy is selected based at least in part on the aforementioned data quality assessment; as well as The data quality remediation strategy is applied to the data stream.

24. The computer program product of claim 22, wherein determining that at least one of the plurality of time intervals has an outlier for the instance of the DQM calculated for at least one of the plurality of time intervals includes analyzing the pattern of the data points.

25. The computer program product of claim 24, wherein analyzing the pattern of the data points includes comparing the pattern of the data points with an expected pattern of the data points.

Images