Systems and methods for heterogeneous data analysis

Abstract

Various techniques for analyzing complex datasets and identifying potential problems in them are disclosed. A validation schema may be provided to the data analysis engine that defines the types of data analysis to be performed on a given dataset. One or more validators associated with different types of data analysis may be selected based at least partially on the validation schema. In various examples, each validator includes one or more executable validator modules configured to analyze the data contained in the dataset, at least partially based on the data modality and / or data type (e.g., time series, continuous, categorical, multidimensional, etc.). A validator results report may be generated according to the output of the validator modules. In some examples, an aggregation of validator results may be determined and included in the validator results report.

Description

Technical Field

[0001] Background

[0001] This application claims the benefit and priority of U.S. Provisional Patent Application No. 63 / 444,707, filed on February 10, 2023, and titled "SYSTEMS AND METHODS FOR HETEROGENEOUS DATA ANALYSIS," and U.S. Provisional Patent Application No. 63 / 523,684, filed on June 28, 2023, and titled "SYSTEMS AND METHODS FOR HETEROGENEOUS DATA ANALYSIS," the contents of which are hereby incorporated by reference in their entirety.

Background Art

[0002]

[0002] In an ideal machine learning setting where data is homogeneous and unimodal, users face various issues at the data preprocessing stage that can degrade the performance of the trained model without their notice. Issues related to data redundancy, distribution shift, and anomalous data points do not prevent model learning but ultimately degrade performance at test time. Discovering these "potential problems" requires extensive manual effort and insight, and even then, it may be impossible to discover such problems if the scope of data-related issues is large. In real-world environments, the situation is even more difficult because data is often obtained from diverse sources, contains noise and outliers, and exhibits significant shifts due to exogenous noise. In such cases, it becomes completely impossible to discover data-related problems manually.

Brief Description of the Drawings

[0003]

[0003] Many aspects of the present disclosure can be better understood by referring to the following drawings. The components of the drawings are not necessarily to scale, and instead, emphasis is placed on clearly showing the principles of the present disclosure. Further, in the drawings, like reference numerals indicate corresponding parts throughout the several views.

[0004] [Figure 1]

[0004] This is a drawing illustrating a data analysis framework according to various embodiments of the present disclosure. [Figure 2]

[0005] These are diagrams of network environments according to various embodiments of the present disclosure. [Figure 3]

[0006] This diagram illustrates how, according to various embodiments of this disclosure, it is possible to leverage the richness of multidimensional data validation methods with a single representation function for each data modality. [Figure 4]

[0007] This flowchart shows examples of functionality implemented as part of an application running in a computing environment within the network environment shown in Figure 2, according to various embodiments of this disclosure. [Figure 5] This flowchart shows examples of functionality implemented as part of an application running in a computing environment within the network environment shown in Figure 2, according to various embodiments of this disclosure. [Figure 6] This flowchart shows examples of functionality implemented as part of an application running in a computing environment within the network environment shown in Figure 2, according to various embodiments of this disclosure. [Modes for carrying out the invention]

[0005]

[0008] Various methods for analyzing complex datasets and identifying potential problems in them are disclosed. According to various embodiments, a validation schema may be provided to the data analysis system that defines the types of data analysis to be performed on a given dataset. One or more validators associated with different types of data analysis may be selected based at least partially on the validation schema. In various examples, each validator includes one or more executable validator modules configured to analyze the data contained in the dataset, at least partially based on the data modality and / or data type (e.g., time series, continuous, categorical, multidimensional, etc.). A validator results report may be generated according to the output of the validator modules. In some examples, an aggregation of validator results may be determined and included in the validator results report.

[0006]

[0009] As data scales both horizontally and vertically, manual data cleaning and inspection strategies, once simply inefficient, prove unfeasible. This is especially true for tasks requiring the examination of multiple data columns simultaneously, such as confounding variable analysis and multimodal anomaly detection. Furthermore, multiple environments exist that rely on live-streaming predictions or AutoML-based configurations, based on mixed data domains and modalities. In such configurations, the timeline for "data validation" becomes even shorter, and the data becomes even more cluttered. These issues are significant challenges for the machine learning community. Machine learning models trained on erroneous data or used on out-of-distribution data can lead to undetected failures, resulting in incorrect predictions being accepted by end-users. Investigating these problems is time-consuming, sometimes taking months to pinpoint the root cause.

[0007]

[0010] Known validation tools for identifying specific data-related problems are highly effective at discovering one type of problem, but they are not easily scalable. Some of these validation tools can (1) identify anomalies and outliers in multimodal datasets, time-series data, and image data; (2) identify dataset shifts that may degrade performance during inference; and (3) check for "linting errors" such as improperly preprocessed data or misencoding errors. Furthermore, purpose-specific validation systems work well, but they are not modular and lack generalizability for new use cases. One known system performs online validation of home energy systems and detects false results. Another known system includes an unsupervised system for inferring validation rules on online sensor data and validating them in real time.

[0008]

[0011] Some known validation tools build frameworks to support arbitrary checks within the context of specific pipelines or toolsets, but they are not designed for highly heterogeneous data. One example of a validation tool is a rule-based validation framework for pandas dataframes. Another example of a validation tool is introducing TensorFlow Data Validation as a component of a larger machine learning pipeline (TFX) and performing data validation from within the TFX framework. Another example of a validation tool is testing explicit and declarative assumptions about the data. Another example of a validation tool is using the R package to perform statistical tests on the data.

[0009]

[0012] While all the aforementioned tools require an explicit declaration of validation, some tools attempt to infer what should be validated based on the data itself. One example of a validation tool infers data validation patterns from a data lake and uses those patterns for validation. Another validation tool uses a genetic algorithm to identify validation rules.

[0010]

[0013] There are also research areas concerning the overall challenges, trends, and best practices related to data validation. For example, one practice involves a five-role "line of defense," including human-in-the-loop validators who review and approve the work of data scientists and data owners to mitigate the risks of machine learning models. Other challenges in data validation include the importance of human oversight in data validation, as well as the limitations of data validation in big data environments and opportunities for improvement.

[0011]

[0014] To overcome the shortcomings of known data validation techniques, this disclosure relates to a modular and extensible data validation framework aimed at identifying data-related issues in real-world data. The design principles of the data analysis system in this disclosure are selected to make the data analysis system an effective and extensible tool for discovering problems that may be encountered in real-world datasets that are potentially noisy, heterogeneous, sparse, multimodal, miscoded, and improperly preprocessed.

[0012]

[0015] Now, looking at Figure 1, an example of the framework of the data analysis system 100 of this disclosure is shown with various examples. As shown in Figure 1, a validation schema 103 that identifies desired validators 106 (e.g., 106a, 106b, 106c, 106d, 106e) for analyzing a dataset 109, and data objects 112 that define the dataset 102 and its divisions (e.g., training dataset, validation dataset, test dataset, inference dataset, etc.) may be provided to the data analysis engine 115. In various examples, each validator 106 includes one or more executable validator modules 118 (e.g., 118a, 118b, 118c, 118d, 118e, 118f, 118g, 118h, 118i, 118j, 118k, 118l, 118m, 118n, 118o, 118p) configured to analyze the data contained in the dataset 109, at least partially based on the data modality and / or data type being analyzed (e.g., time series, continuous, categorical, multidimensional, etc.). In various examples, each executed validator module 118 may return validator results 121 associated with the corresponding analysis.

[0013]

[0016] In various examples, the data analysis engine 115 automatically identifies the appropriate validators 106 defined by the validation schema 103 and determines the relevant validator module 118 to be executed on the dataset 109, at least partially based on the data types contained in the dataset 109 (which may be defined by the data objects 112). In various examples, the validator results report may be generated according to the validator results 121 output from each validator module 118. In some examples, the validator results 121 correspond to a ranking of scores associated with the detected problems. In other examples, the validator results 121 may include probabilities associated with the problems in the data. In some examples, aggregated validator results 124 corresponding to the aggregation of the validator results 121 associated with each validator 106 may be calculated and included in the validator results report provided to the user.

[0014]

[0017] In the example in Figure 1, the dashed lines represent unused validator modules 118 and validators 106. For example, validator module 118 may be excluded because dataset 109 does not contain columns of the appropriate data type, or because it has been manually excluded from the validation schema 103. In various examples, validator 106 may be excluded by the data analysis engine 115 when it is excluded from the validation schema 103. In the example in Figure 1, dataset 109 contains only multidimensional data. Thus, only the multidimensional validator modules 118 (e.g., 118c, 118d, 118e, 118g, 118h, 118i, 118j, 118n, 118o, 118p) of the selected validators 106 (e.g., 106b, 106d, 106e) are selected by the data analysis engine 115 to analyze the data contained in dataset 109.

[0015]

[0018] In various examples, dataset 109 may include a torch dataset describing the data type of each column. This torch dataset is described using data objects 112 that map each column to its respective data type. In order to utilize some of the validators 106 of the data analysis system 100, dataset 109 may need to be preprocessed and split so that the data analysis engine 115 can detect errors from the preprocessing stage (e.g., miscoding errors) or the splitting stage (e.g., splits that are not independent and identically distributed (IID)). In some examples, if the user does not want to include these checks, an unsplit raw dataset 109 may be provided. In some examples, if the user wants to override the default run configuration, the user can provide a validation schema 103 that specifies the validations to include and their options. From there, the data analysis engine 115 automatically identifies the appropriate validator modules 118 from the validators 106 selected for each data type, applies them, and returns the validator results 121 to the user.

[0016]

[0019] Various examples show that the abstraction of data modalities present in the validator / validator module architecture of the data analysis system 100 makes it possible to support highly heterogeneous data with minimal specifications. Since the validator 106 automatically identifies the appropriate validator module 118 for a given data type, additional validator modules 118 may be written and incorporated into the given validator 106 to support custom data types and extend fixed validation types to new data types. This is particularly effective when the input dataset 109 contains a large number of modalities, as the data analysis system 100 can automatically apply the appropriate validator module 118 for each new data modality.

[0017]

[0020] Attempting to perform numerous validations for a large number of custom data modalities can result in an impractical number of additional validator modules 118 that need to be written. For example, consider a scenario where you want to perform anomaly detection validation, OOD validation during inference, and conditional independence testing for three new data modalities not yet supported by the data analysis framework. A simple approach would require the user to write nine new methods, one for each data modality-validation combination. Therefore, it may be simpler to maintain a set of representation functions (e.g., word2vec or ResNet encoder) that map each data modality to a multidimensional continuous vector containing a compressed representation of the data modality, and then use the existing set of multidimensional validator methods to perform all checks in the data analysis system. In the above example, the user would only need to include three representation functions to perform all nine validation combinations. Alternatively, validator modules 118 could be created to map high-dimensional feature spaces, such as image or audio data, to low-dimensional descriptive representations, such as histograms or band-limited spectrographs. This method allows users to benefit from the existing set of multidimensional continuous validator modules 118 implemented in the data analysis system 110 without necessarily having to train a model to learn representations of new data types.

[0018]

[0021] This may be a quicker way to include new data types in the complete validation suite, but certain data types may require specific validator modules 118 to map directly from high-dimensional data to validator results 121. In such cases, the data analysis system 100 allows the user to write additional validator modules 118 for that data type.

[0019]

[0022] According to various examples, when creating a new validator method or module, a new class can be added that extends the base validator 106. In various examples, the new class is added by the user interacting with the data analysis system. To create a validator module 118, the following information is required: (1) which part of the data object is needed (e.g., for testing, partitioning, inference, etc.); (2) the data types to be supported (e.g., time series, continuous, categorical, multidimensional, etc.); (3) the settings for each module call (e.g., description of normality testing in a commercially available library, etc.); and (4) predefined data calls from validator 106 using options that may be included in the validation schema 103. In various examples, a user interacting with a client device 206 communicating with the data analysis engine 115 can create a new validator module 118 by providing the necessary information. Figure 5 provides a supplementary discussion regarding the addition of a validator module 118.

[0020]

[0023] One design challenge of the data analysis system 100 in this disclosure is to maximize value for both non-expert and expert users by enabling any degree of customization and complexity in the validation strategy without sacrificing usability. An ideal data validation pipeline is not only arbitrarily customizable but also functions as a ready-to-use solution that can be integrated inline into machine learning pipelines with minimal configuration. In light of this design challenge and overall goal, the data analysis system 100 in this disclosure adheres to the following design principles: (1) to abstract design details to the greatest extent possible; (2) to provide reasonable and overrideable default settings to deliver ready-to-use functionality; and (3) to facilitate method extensibility.

[0021]

[0024] To use the analysis system of the present disclosure, it is sufficient for a non-expert user to understand what they are trying to evaluate with their data. Therefore, there is no need to identify or understand the underlying methods for examining these evaluations. For example, consider a user who wants to perform anomaly detection. The user can switch the on / off of this function with one line of code without considering the data types within their data or knowing anything about the anomaly detection method. To achieve this, the data analysis system of the present disclosure divides the verification function into a highly implementation-dependent modular component (referred to as the validator module 118) and an easily switchable component (referred to as the I-validator 106).

[0022]

[0025] A validator method or module includes executable code that, when executed, performs a specific type of test on data problems related to a specific data type. These validator methods mainly receive a data set, perform an evaluation on it, and are composed of the code necessary to generate positive or negative results. Some examples of validator methods include the following: (1) the Mann-Whitney U test for examining the distribution shift between training / test splits in tabular data, (2) the kernel conditional independence (KCI) test for verifying causal hypotheses for vector-valued data, (3) an isolation forest trained / evaluated with an image histogram for identifying anomalies in image data, and / or other types of techniques or modules.

[0023]

[0026] Each switchable component is referred to as a validator, and these are a collection of data methods that do not depend on data types and are sequentially applied to the data set. Each validator targets a single problem that can occur in the data, and the single problem includes, for example, shifts between different data splits, detection of outliers and anomalies, violation of parametric assumptions of the data, violation of the expected causal (casual) structure / conditional independence in the data, and / or other types of problems.<​

[0027] According to various embodiments, the data analysis system 100 of the present disclosure is operable immediately with minimal setup on the data sets used by the user in the pipeline. In various examples, the data analysis system 100 further provides an option to override these default settings to accommodate the needs of expert users. In various examples, each validator 106 can include attributes to be included in the default settings, at least partially based on whether the validator 106 should be included in an immediately available validation setup associated with the validator system.

[0025]

[0028] According to various examples, the validator 106 can include specifications of methods or validator modules 118 to be used in default settings for inspecting a given problem. However, if not all validator modules 118 included in the validator 106 are desirable for a given situation, the user can describe a custom validator 106 with a set of their own validator modules 118. When the validator 106 is applied to a data set, the backend of the data analysis system 100 applies, in default settings, all validator modules 118 described for a particular data type to all columns of that data type in the data set 109. However, this behavior can also be overridden by including a regular expression in each validator module 118, and the set of columns to which the validator module 118 is applied can be restricted. This is useful when dealing with large data sets 109 with regular column names. As a result of all these default options, the system is operable without options and can also be configured to operate exactly as the user desires.

[0026]

[0029] Various examples demonstrate that the ease of module extensibility allows users to easily integrate new validations (e.g., validator modules 118) into the data analysis system 100 framework without being limited to an existing set of validator modules 118, and without spending more time than it would take to write a standalone validator module 118. Furthermore, modularity and extensibility improve the scale and comprehensiveness of readily available validator modules 118 that can be incorporated into the validation schema 103 with a single line of code. For example, a user interacting with the data analysis engine 115 can create additional validator modules 118, which can then be associated with their corresponding validators 106 and stored in the data store 215 for future use by the user and / or other users of the data analysis engine 115. Thus, users have an incentive to contribute to building an ecosystem of validator modules 118 and validators 106 implemented in the data analysis system 110.

[0027]

[0030] In various examples, the ease of extensibility of this disclosure is most evident in the module design of the validator modules 118 applied to each dataset 109. These validator modules 118 contain functional static classes scoped to apply a single method to a small number of data types. The validator modules 118 do not explicitly include suggestions for solutions to data-related problems. Forcing the inclusion of potential solutions to such data-related problems would significantly impair the ease of extensibility and potentially limit the scope to which users can write and share their own methods. Furthermore, when data-related problems are discovered, there are usually multiple options for users to improve model performance by appropriately modifying the experimental setup, and since choosing the optimal option is a delicate judgment, it is best to leave that judgment to the user.

[0028]

[0031] In various cases, the validator module 118 returns interpretable and actionable validator results 121. For example, a validator module 118 that includes statistical components can obtain p-values ​​under the null hypothesis that there are no problems with the data. A validator module 118 that includes scoring components, such as an anomaly detection method, can return internal anomaly scores and rankings for each sample in the dataset, allowing users to quickly identify the most suspicious examples and take appropriate action. An out-of-distribution (OOD) validator module 118 applied during inference can indicate where an OOD score stands in the context of the training set.

[0029]

[0032] The following outlines the system and its components of the present invention, followed by a discussion of their operation. The following description provides illustrative examples of the operation of various components of this disclosure, but the use of these examples does not preclude other embodiments that are consistent with the principles disclosed therein.

[0030]

[0033] Referring to Figure 2, various embodiments of the network environment 200 are shown. The network environment 200 may include a computing environment 203 and client devices 206, which can communicate data with each other via a network 209. The network 209 may include a wide area network (WAN), a local area network (LAN), a personal area network (PAN), or a combination thereof. These networks may include wired or wireless components, or a combination thereof. Wired networks may include Ethernet networks, cable networks, fiber optic networks, and telephone networks such as dial-up, digital subscriber line (DSL), and Integrated Services Digital Network (ISDN). Wireless networks may include cellular networks, satellite networks, Institute of Electrical and Electronics Engineers (IEEE) 802.11 wireless networks (i.e., Wi-Fi®), Bluetooth® networks, microwave transmission networks, and other networks that rely on wireless broadcasting. The network 209 may also include combinations of two or more networks 209. Examples of network 209 may include the internet, intranets, extranets, virtual private networks (VPNs), and similar networks.

[0031]

[0034] The computing environment 203 may include one or more computing devices, including a processor, memory, and / or network interfaces. For example, such computing devices may be configured to perform calculations on behalf of other computing devices or applications. As another example, such computing devices may host and / or provide content to other computing devices upon request.

[0032]

[0035] Furthermore, the computing environment 203 may employ multiple computing devices that can be located in one or more server banks or computer banks or other configurations. Such computing devices may be located in a single location or distributed across many geographically different locations. For example, the computing environment 203 may include a set of multiple computing devices that can include hosted computing resources, grid computing resources, edge computing resources, or any other distributed computing configuration. In some cases, the computing environment 203 may accommodate elastic computing resources, where the allocated capacity for processing, networking, storage, or other computing-related resources may change over time.

[0033]

[0036] The computing environment 203 can run various applications or other functions. Components running on the computing environment 203 include the data analysis engine 115 and other applications, services, processes, systems, engines, or functions not described in detail herein.

[0034]

[0037] The data analysis engine 115 can analyze the data contained in the provided dataset 109 to detect any potential problems present in the dataset 109. In various examples, the data analysis engine 115 can obtain a data object 112 containing the dataset 109, a mapping defining the types of data contained in the dataset 109, any partitions associated with the dataset 109 (e.g., inference data, test data, training data, etc.), and / or other information. Furthermore, the data analysis engine 115 can obtain a validation schema 103 that can be used to define the types of validators 106 used to evaluate the data contained in the dataset 109.

[0035]

[0038] In various examples, the validation schema 103 may include a list of validators 106 used to analyze the data. In various examples, the validators 106 may include parametric assumption validators, conditional independence validators, distribution shift validators, anomaly validators, out-of-distribution (OOD) inference validators, and / or other types of validators. In some examples, the validators 106 may be predefined. In other examples, the validators 106 may be user-defined and may be generated at least in part based on user interaction with the data analysis engine 115.

[0036]

[0039] In some examples, the validation schema 103 may include options associated with the use of the validator 106 on the data contained in the dataset 109. For example, the validation schema 103 may further define columns of the dataset 109 associated with the data to be analyzed, or it may include columns of the dataset 109 that are not to be evaluated, thereby providing filter characteristics associated with the test. The validation schema 103 may also include other information that the data analysis engine 115 can use to select and execute the validator module 118.

[0037]

[0040] In various examples, each validator 106 may be associated with one or more validator modules 118 configured to analyze data contained in a dataset 109 associated with one or more different data types. For example, the data contained in the dataset 109 may include time-series data, continuous data, categorical data, multidimensional data, and / or other types of data. A validator module 118 associated with a given validator 106 may be configured to analyze one or more different data types. For example, a first validator module 118 associated with a first validator 106 may be configured to evaluate continuous and multidimensional data, but not time-series data or categorical data.

[0038]

[0041] According to various examples, the data analysis engine 115 can select a validator 106 for data analysis of the dataset 109, at least partially based on the validation schema 103. Once an appropriate validator 106 is selected, the data analysis engine 115 can select a validator module 118 associated with the validator 106, at least partially based on the type of data to be evaluated. The type of data to be evaluated can be determined by a mapping contained in the dataset 109, which identifies the type of data corresponding to a given column in the dataset 109.

[0039]

[0042] Once validators 106 and validator modules 118 are selected for data analysis, the data analysis engine 115 can run each validator module 118 on the data contained in the dataset 109 to determine the validator results 121 shown in Figure 1, which are included in the output of each validator module 118. In various examples, the validator results may include probability values ​​and / or score values ​​based on the test type of a given validator module 118. In the example of scored values, the scored data can be ranked, and the ranking can be included in the validation result output.

[0040]

[0043] In some examples, the data analysis engine 115 includes a validator results report 212 that contains validator results 121 associated with each validator module 118 that was executed. In some examples, the validator results report 212 includes validator results 121 associated with each validator module 118 that was executed for the corresponding data evaluated. In other examples, the validator results report 212 includes an aggregated validator results 124, as shown in Figure 1, which corresponds to an aggregated validator results 121 of the validator modules 118 for a given validator 106. In this example, the aggregated validator results 124 can provide a more accurate summary of the data results and highlight data areas that may contain issues that need to be addressed. In various examples, the data analysis engine 115 can display the validator results report 212 on a display device of a client device 206 or other computing device.

[0041]

[0044] Furthermore, various data is stored in a data store 215 accessible from the computing environment 203. The data store 215 may represent multiple data stores 215, including relational or non-relational databases (such as object-oriented databases, hierarchical databases, hash tables, or similar key-value data stores), as well as other data storage applications or data structures. Additionally, these databases, data storage applications, and / or data structures can be used in combination to provide a single logical data store. The data stored in the data store 215 is associated with the operation of various applications or functional entities described below. This data may include validators 106, validation schemas 103, transformation libraries 210, datasets 109, validator result reports 212, and potentially other data.

[0042]

[0045] The validator 106 includes components that address a single problem that may occur in the data, such as shifts between different data partitions, detection of outliers and anomalies, violations of parametric assumptions in the data, violations of expected causal structure / conditional independence in the data, and / or other types of problems. According to various examples, the validator 106 may include a parametric assumption validator, a conditional independence validator, a distribution shift validator, an anomaly detection validator, an out-of-distribution (OOD) inference validator, and / or other types of validators. Each validator 106 includes a collection of data type-independent data modules 118 that are applied sequentially to the dataset 109.

[0043]

[0046] A validator module 118 contains executable code that, when executed, performs a specific type of test on a data issue relating to a particular data type. In particular, validator modules 118 are configured to analyze the data contained in dataset 109, at least partially based on the data modality and / or data type being analyzed (e.g., time series, continuous, categorical, multidimensional, etc.). These validator modules 118 consist mainly of the code necessary to receive dataset 109, perform evaluations on it, and generate positive or negative results. Some examples of validator modules 118 include: (1) a Mann-Whitney U test to examine distribution shifts between training / test splits in tabular data; (2) a kernel conditional independence (KCI) test to verify causal assumptions for vector-valued data; (3) an isolation forest trained / evaluated on image histograms for identifying anomalies in image data; and / or other types of methods or modules. In various examples, each executed validator module 118 may return a validator result 121 associated with the corresponding analysis associated with the validator module 118.

[0044]

[0047] In various examples, a validator module 118 designed to examine distributional shifts between data splits can provide p-values ​​under the null hypothesis that data from two data splits (e.g., training, validation, test, inference) are drawn from the same distribution. Similarly, in the case of a pointwise OOD test, the data analysis system 100 can accept the training / validation set and the data points on which the model is used, and output (1) a set of underlying OOD scores from the model output in the context of the training set and (2) a ranking of the samples by their likelihood of being anomalous. In some examples, if the inference dataset is significantly smaller than the training dataset, the validator module 118 can provide percentile values ​​indicating where the anomalous score of a data point would be placed as a percentile in the context of the larger dataset.

[0045]

[0048] In various examples, validator 106 and the corresponding validator module 118 can perform anomaly detection. In some examples, the validator module 118 of the anomaly detection validator 106 can include an unsupervised anomaly detection method. This is because it is not expected that the user will provide labeled anomaly samples, and labeled anomalies are rare in nature. As an example, various types of anomalies in tabular data can be identified using the out-of-the-box unsupervised anomaly detection method of the open-source anomaly detection toolkit pyOD. Furthermore, for high-dimensional data such as images or audio, a set of validator modules 118 for multidimensional data can be utilized using a low-dimensional representation with one of the strategies described above. In various examples, anomaly detection is supported for aligned time-series data, multidimensional data, and continuous data.

[0046]

[0049] In various examples, validator 106 and the corresponding validator module 118 can assess data imbalances. When dealing with categorical and continuous data, certain categorical and continuous variables may exhibit an unusual amount of imbalance. While there are several ways to manage data imbalances during model development, data imbalances are easily overlooked, and model performance can be unnoticed. According to various examples, one or more validator modules 118 of the data analysis system 100 of this disclosure can implicitly check for class imbalances and continuity imbalances and warn the user of such problems before training the model.

[0047]

[0050] In various examples, the validator 106 and the corresponding validator module 118 can evaluate the data in relation to parametric and statistical assumptions. One exemplary use case of the data analysis system 100 is in a heterogeneous machine learning setting, but the assumption validation phase can be particularly useful for statistical models that have multiple strong assumptions about the underlying data (such as parametric assumptions or other forms of conformance to distributions). Other assumptions related to the data, such as linearity and equality of variances, can also be quickly and easily checked by the data analysis system in various examples.

[0048]

[0051] In various examples, validator 106 and the corresponding validator module 118 can evaluate data with respect to causal / conditional independence assumptions. Causal independence assumptions and conditional independence assumptions are particularly useful in machine learning settings where some inductive knowledge exists about the causal structure of the data or the data generation process. This type of assumption is useful in tabular settings where the causal structure is already known.

[0049]

[0052] In various scenarios, a user can create validators 106 and / or corresponding validator modules 118 that can be sent from a client device 206 to a data analysis engine 115. For example, a user can interact with a user interface 224 associated with the data analysis engine 115 and use code to define the functionality of validators 106 and / or associated validator modules 118. Thus, validators 106 and / or associated validator modules 118 can be sent from the client device 206 to the data analysis engine 115.

[0050]

[0053] The validation schema 103 may include data defining validators 106 (e.g., 106a, 106b, 106c, 106d, 106e) for analyzing the dataset 109, and data objects 112 defining the dataset 102 and its partitions (e.g., training dataset, validation dataset, test dataset, inference dataset, etc.) may be provided to the data analysis engine 115. In various examples, the validation schema 103 may include a list defining at least a subset of multiple validators 106 that can be used to analyze the dataset 109. In some examples, the validation schema 103 may include options associated with the use of validators 106 with respect to the data contained in the dataset 109. For example, the validation schema 103 may further define columns of the dataset 109 associated with the data being analyzed, or it may include columns of the dataset 109 that are not being evaluated, thereby providing filter properties associated with the test. In some examples, the validation schema 109 may include information defining one or more data-specific transformations that may be necessary to transform all columns of one data type contained in the dataset 109 into another data type. For example, the validation schema 109 can identify one or more representation functions (e.g., word2vec or ResNet encoder) that map each data modality to a multidimensional continuous vector containing a compressed representation, and then perform all checks of the data analysis system using an existing set of multidimensional validator methods.

[0051]

[0054] The validation schema 103 may further include other information that the data analysis engine 115 can use to select and execute the validator module 118. In various examples, the validation schema 103 is user-defined. In other examples, the validation schema 103 can be predefined. Thus, in some examples, the validation schema 103 may be selected from among several predefined validation schemas 103, at least partially based on the desired type of evaluation for a given dataset 109.

[0052]

[0055] In some examples, the validation schema 103 may be empty or show options, and therefore may not define any of the multiple validators 106 that can be used to analyze the dataset 109. In this example, the data analysis engine 115 can select validators 106 that can be automatically applied based at least partially on the dataset 109 being analyzed.

[0053]

[0056] The transformation library 210 may include predefined transformation functions that enable the effective transformation of data from unsupported data types to supported data types. In some examples, the transformation function can map each data modality to a multidimensional continuous vector containing a compressed representation of the data modality, and then perform all checks of the data analysis system using an existing set of multidimensional validator methods. In other examples, the transformation function can map the data contained in dataset 109 from a high-dimensional feature space (e.g., image, audio data, etc.) to a low-dimensional descriptive representation (e.g., histogram, bandwidth-limited spectrograph, etc.).

[0054]

[0057] The dataset 109 may include data representing information associated with one or more electronically storable objects (e.g., measurements, statistics, etc.). In various examples, the dataset 109 may include time-series data, continuous data, categorical data, multidimensional data, and / or other types of data. In various examples, the dataset 109 may include a torch dataset describing the type of data in each of its columns. In some examples, the dataset 109 may need to be preprocessed and split so that the data analysis engine 115 can detect errors from the preprocessing stage (e.g., miscoding errors) or the splitting stage (e.g., splits that are not independent and identically distributed (IID)). In some examples, the dataset 109 may be associated with data objects 112 that define the dataset 109 and its splits (e.g., training dataset, validation dataset, test dataset, inference dataset, etc.) and map each column to its type of data.

[0055]

[0058] The validator results report 212 includes validator results 121 associated with each validator module 118 that was executed. In some examples, the validator results report 212 includes validator results 121 associated with each validator module 118 executed on the corresponding data evaluated. In other examples, the validator results report 212 includes aggregated validator results 124, which corresponds to an aggregated version of the validator results 121 of the validator modules 118 for a given validator 106. In this example, the aggregated validator results 124 can provide a more accurate summary of the data results and highlight data areas that may contain issues that need to be addressed. In various examples, the data analysis engine 115 can display the validator results report 212 on a display device of a client device 206 or another computing device.

[0056]

[0059] The computing environment 203 may further include a data cache 219. The data cache 219 can store transformed data output from a given transformation function. According to various examples, the data contained in the dataset 109 can be transformed into one or more data types using transformation functions contained in the transformation library 210 and / or transformation functions contained in and defined in the validation schema 103. Once the data is transformed, the transformed data can be stored in the data cache 219. Therefore, when different validator modules 118 are executed on the data, the transformed data can be accessed from the data cache 219 without having to re-execute the transformation process each time the validator module 118 is applied and / or when different data partitions are analyzed.

[0057]

[0060] Client device 206 is representative of multiple client devices that can be connected to network 209. Client device 206 may include processor-based systems such as computer systems. Such computer systems can be embodied in the form of personal computers (e.g., desktop computers, laptop computers, or similar devices), mobile computing devices (e.g., personal digital assistants, mobile phones, smartphones, webpads, tablet computer systems, music players, portable game consoles, e-book readers, and similar devices), media playback devices (e.g., media streaming devices, Blu-ray® players, digital video disc (DVD) players, set-top boxes, and similar devices), video game consoles, or other devices with similar functionality. Client device 206 may include one or more displays 218, such as liquid crystal displays (LCDs), gas plasma-based flat panel displays, organic light-emitting diode (OLED) displays, electrophoretic ink ("E-ink") displays, projectors, or other types of display devices. In some cases, the display 218 can be a component of the client device 206, or it can be connected to the client device 206 via a wired or wireless connection.

[0058]

[0061] The client device 206 may be configured to run various applications, such as the client application 221 or other applications. The client application 221 may run on the client device 206 and access network content provided by the computing environment 203 or other servers, thereby rendering the user interface 224 on the display 218. For this purpose, the client application 221 may include a browser, a dedicated application, or other executable files, and the user interface 224 may include a network page, an application screen, or other user mechanisms for obtaining user input. The client device 206 may also be configured to run applications other than the client application 221, such as an email application, a social networking application, a word processor, a spreadsheet, or other applications.

[0059]

[0062] Figure 2 illustrates a data analysis engine 115 running as a backend system within a computing environment 203. Note that in some examples, the data analysis engine 115 includes extension components that can be integrated into a standalone system or a client device 206. Therefore, some or all of the functions and components of the computing environment may be included in or run by the client device 206.

[0060]

[0063] Next, a general description of the operation of various components of the network environment 200 is provided with respect to an exemplary case study. To demonstrate the practical value of the data analysis system 100, the data analysis system 100 of this disclosure is applied to various different medical datasets 109 in the case study. This case study is intended to represent the conditions of a highly multimodal environment in which a user queries data from a number of different datasets 109 across different modalities. In medical research, it is often necessary to examine multiple different types of data modalities (e.g., images, genetic data, demographic data, sequencing data, etc.) from multiple different sources. The datasets 109 in this case study were selected to mimic this example.

[0061]

[0064] This case study examined several different datasets 109. These datasets 109 included multiple breast cancer datasets, cardiac datasets, lymphangiography datasets, thyroid datasets, and molecular datasets containing molecular attributes. The included datasets 109 contained both categorical and continuous data, both univariate and multivariate data, and were related to several different disease types and the analyses collected. During evaluation, binary categorical data were included in multidimensional data vectors. Non-binary categorical data were not included in this evaluation.

[0062]

[0065] For each dataset 109, 20 samples were extracted and held out for use at "inference time". These samples were intended to represent the online data that the model would encounter at inference time after training and deployment. The remainder of the datasets (the "all except inference" set) was used to reconstruct the training, validation, and test datasets by performing a 60 / 20 / 20 random split. All of these datasets (the inference dataset, the "all except inference" dataset, the training dataset, the validation dataset, the test dataset, and the entire dataset) were contained in the data object 112 that was input to the data analysis engine 115.

[0063]

[0066] Three validators 106 are selected to be executed by the data analysis engine 115, at least partially based on the validation schema 103, dataset 109, and data object 112 received by the data analysis engine 115. For example, the validation schema 103 defining the types of validators 106 to run on dataset 109 may be received by the data analysis engine 115 from a client device 206, from a data store 215 containing one or more predefined validation schemas 103, and / or from other entities. In various examples, the data analysis engine 115 can identify validator modules 118 associated with validators 106, at least partially based on the types of data contained in dataset 109, which may be defined by dataset 109 and / or the corresponding data object 112.

[0064]

[0067] These validator modules 118 examined for anomalous samples in an unsupervised manner; distribution shifts between training, test, and validation datasets; and potential out-of-distribution issues during inference. Multiple ADBench validator modules 118 were used to detect anomalous samples, including principal component analysis (PCA) outlier detection, isolation forest, and cluster-based local outlier factor (CBLOF) outlier detection algorithms. Multiple statistical validator modules 118 were performed to detect dataset shifts, including the two-sample Kolmogorov-Smirnov test, Mann-Whitney U test, and Kruskal-Wallis test. The p-values ​​for distribution shifts in multivariate data were adjusted using Bonferroni correction. Results from the validator modules 118 associated with the anomaly detection validator 106 are shown in Table 1. The anomaly detection that performed best varied depending on the type of dataset. This suggests the importance of using a variety of different validator modules 118. TIFF2026509100000002.tif59170

[0065]

[0068] To detect out-of-distribution samples during inference, the aforementioned anomaly detection validator module 118 was used by training it on all samples except 20 holdout samples, applying the validator module 118 to the 20 samples in the inference dataset, and evaluating them with the provided anomaly labels. In the exemplary case study, no statistically significant results indicating a shift between the training, validation, and test data were detected. The OOD validator results 121 during inference from these tests are shown in Table 2. TIFF2026509100000003.tif59170

[0066]

[0069] To investigate the relationships between each validator module 118 for anomaly detection, Spearman correlations between rank vectors obtained from each validator module 118 were observed. The results of tank mean and Borda count rank aggregation are also included. The five ranked validator modules 118 were found to exhibit a mean correlation r = 0.853. This suggests that the rankings among the validator modules 118 each contain their own unique information sources.

[0067]

[0070] Following various examples, each validator module 118 (e.g., isolation forest, PCA, cblof) assigned a score to each sample in each dataset 109 for the anomalous content within the test cases. These scores were converted into a list of “anomalousness ranks” for each sample in dataset 109. The data analysis engine 115 then aggregated these ranks by (a) averaging and (b) using a Borda count ranking method. While some datasets 109 and validator modules 118 (e.g., PCA / iforest for the cardiac dataset) showed some redundancy between the validator modules 118, in most cases a considerable amount of inconsistency was observed between the anomaly detection validator modules 118, indicating the need for multiple detection methods. In the test cases, there was no substantial difference between rank aggregation by Borda count and rank averaging.

[0068]

[0071] Turning to Figure 3, a diagram illustrates how the richness of the multidimensional data validator module 118 can be leveraged with a single representation function for each data modality 303. The data modality 303 can include discrete data, sequence data, grid data, graph / point cloud data, and / or other types of data modalities. In various examples of this disclosure, the data analysis engine 115 can apply dimensionality reduction and representation learning techniques to map to a wide range of vector representations of the data modality 303. According to various examples, Figure 3 shows how different types of validators 103 can be used to analyze various forms of data. For example, data with a discrete-based data modality 303 can correspond to categorical data. The data analysis engine 115 can then determine whether the data is univariate or multivariate and select the appropriate validator 106 and validator module 118 to use to evaluate the data. Similarly, data representing a point cloud-based data modality 303 can be transformed using machine learning transformations included in the data library 210. These conversion processes can convert the data into a vector representation, which is then further analyzed by validator 106 and the corresponding validator module 118, at least partially based on the type of data represented by the vector.

[0069]

[0072] In various examples, data transformed by transformation processes within the transformation library 210 and / or by transformation processes provided in the validation schema 103 can be stored in the data cache 219. In this example, data can be transformed and analyzed by validator 106 and the corresponding validator module 118, but the transformed data can be retrieved from the data cache 109 for subsequent analysis by other validators 106 and / or validator modules 118. Therefore, data analysis can be performed on different subsets or partitions of data without recalculating the transformation processes.

[0070]

[0073] Referring to Figure 4, a flowchart 400 is shown that provides an example of some operation of the data analysis engine 115. The flowchart in Figure 4 is just one example of many different types of functional configurations that can be used to perform the operation of the illustrated part of the data analysis engine 115. Alternatively, the flowchart in Figure 4 can be considered as illustrating an example of elements of the method implemented within the network environment 200.

[0071]

[0074] In particular, Figure 4 relates to facilitating customizable and modular analysis of a dataset 109 using a validator 106 defined in a validation schema 103, according to the framework of the data analysis system 100. To overcome the shortcomings of known data validation techniques, this disclosure relates to a modular and extensible data validation framework aimed at identifying data-related issues in real-world data. The use of the validator 106 and the corresponding validator module 118 of the data analysis system 100 provides an effective and extensible tool for discovering problems that may be encountered in real-world datasets, which may be potentially noisy, heterogeneous, sparse, multimodal, misencoded, and improperly preprocessed validator modules 118.

[0072]

[0075] In various examples, the use of validator 106, validator module 118, and the ability to define what to test using validation schema 103 maximizes value for both non-expert and expert users by enabling any degree of customization and complexity in the validation strategy without sacrificing usability. In various examples, the validation framework not only functions as a customizable solution but also as a ready-to-use solution that can be integrated inline into machine learning pipelines for high-dimensional data with minimal configuration. Thus, the data analysis system 100 shown in Figure 4 adheres to the following design principles: (1) maximizing abstraction of design details, (2) providing reasonable and overrideable default settings to deliver ready-to-use functionality, and (3) facilitating method extensibility.

[0073]

[0076] To use the analysis system 100 of this disclosure, non-expert users only need to know what aspects of their data they intend to evaluate. Therefore, they do not need to identify or understand the underlying methods for inspecting these evaluations. For example, consider a user who wants to perform anomaly detection. The user can switch this function on and off with a single line of code, without considering the types of data in their data or knowing anything about anomaly detection methods. In various examples, this can be included when defining the validation schema 103. To achieve this, the data analysis system of this disclosure divides the validation function into highly implementation-dependent modular components (referred to as validator modules 118) and easily switchable components (referred to as validators 106).

[0074]

[0077] Starting from box 403, the data analysis engine 115 receives the dataset 109 and the validation schema 103. In various examples, the data analysis engine 115 receives the dataset 109 and / or the validation schema 103 from the client device 206. For example, the data analysis engine 115 may receive the dataset 109 and the validation schema 103 from the client device 206 in response to one or more user interactions with the user interface 224 associated with the data analysis engine 115, which is displayed on the client device 206. For example, a user can define the type of validator 106 to use by turning on / off the functionality associated with the desired validator 106 with a single line of code or through user interaction with the user interface 224, without considering the types of data in the data or knowing anything about the associated validator module 118. In some examples, the dataset 109 and / or the validation schema 103 are received along with a request to analyze the data in the dataset 109. In various examples, the request may include the dataset 109 and / or the validation schema 103. In another example, the request could include the location of a datastore to access dataset 109 and / or validation schema 103.

[0075]

[0078] In various examples, dataset 109 may include time-series data, continuous data, categorical data, multidimensional data, and / or other types of data. In various examples, dataset 109 may include a torch dataset describing the data type of each of its columns. In some examples, dataset 109 may need to be preprocessed and split so that the data analysis engine 115 can detect errors from the preprocessing stage (e.g., miscoding errors) or the splitting stage (e.g., splits that are not independent and identically distributed (IID)). In some examples, the data analysis engine 115 receives a data object 112 from a client device 206. In various examples, dataset 109 may be associated with a data object 112. The data object 112 defines dataset 109 and one or more of the dataset and its splits (e.g., training dataset, validation dataset, test dataset, inference dataset, etc.). In various examples, the data object 112 may include a mapping of each column of dataset 109 to its respective data type (e.g., time-series, continuous, categorical, multidimensional, etc.).

[0076]

[0079] In various examples, the validation schema 103 includes data defining at least a subset of validators 106 used to analyze the dataset 109, and a data object 112 defining the dataset 109. In some examples, the data object 112 can define partitions of the dataset 109. In various examples, the validation schema 103 can include a list of validators 106 used to analyze the dataset 109. In some examples, the validation schema 103 can include options associated with the use of validators 106 with respect to the data contained in the dataset 109. For example, the validation schema 103 can further define columns of the dataset 109 associated with the data being analyzed, or it can include columns of the dataset 109 that are not being evaluated, thereby providing filtering properties associated with the test. In some examples, regular expressions can be used to facilitate column selection.

[0077]

[0080] The validation schema 103 may further include other information that the data analysis engine 115 can use to select and execute the validator module 118. In various examples, the validation schema 103 is user-defined. For example, the validation schema 103 may be user-defined in response to a user interaction with the user interface 224 displayed on the client device 206, which includes the selection of validator components associated with the validator 106. In various examples, the validator 106 defined in the validation schema 103 includes multiple validator modules 118. In various examples, the multiple validator modules 118 associated with the validator 106 defined in the validation schema 118 may be unknown to the user. In various examples, the ability for the user to define the validation schema 103 without knowledge of the associated validator modules 118 allows for any degree of customization and complexity in the validation strategy without sacrificing usability. In other examples, the validation schema 103 may be predefined. Therefore, in some examples, a validation schema 103 may be selected from among several predefined validation schemas 103 based at least partially on the desired type of evaluation for a given dataset 109.

[0078]

[0081] In some examples, the validation schema 103 may be empty or show default options, and therefore may not define any of the multiple validators 106 that can be used to analyze the dataset 109. In this example, the data analysis engine 115 can select validators 106 that can be automatically applied based at least partially on the dataset 109 being analyzed.

[0079]

[0082] In box 406, the data analysis engine 115 can select one or more validators 106 to analyze the data in the dataset 109, based at least partially on the validation schema 103. In various examples, the validation schema 103 defines at least a subset of multiple validators 106 for analyzing the data contained in the dataset 109. The selected validator 106 is one of the multiple validators and is included in the subset of the multiple validators 106. In various examples, the validator 106 includes one of the following: a parametric assumption validator, a conditional independence validator, a distribution shift validator, an anomaly detection validator, an out-of-distribution (OOD) inference validator, and / or other types of validators. In some examples, the validation schema 103 can include multiple validators 106 used to analyze the provided dataset 109. Thus, the data analysis engine 115 can select multiple validators 106 based at least partially on the list of validators 106 included in the validation schema 103.

[0080]

[0083] In box 409, the data analysis engine 115 selects a validator module 118 associated with the selected validator 106. In some examples, the validator module 118 is selected based at least partially on the data type of the data in the dataset 109 to be analyzed, the data partitioning associated with the dataset 109, the inclusion criteria defined in the validation schema 103, the column names of the data to be analyzed, and / or other factors. In particular, the validator module 118 is configured to analyze the data contained in the dataset 109 based at least partially on the data modality, the type of data to be analyzed, the inclusion criteria defined in the validation schema 103, and / or other factors. The data types of the data to be analyzed include time series data, continuous data, categorical data, multidimensional data, and / or other types of data. These validator modules 118 mainly consist of the code necessary to receive the dataset 109, perform evaluations on it, and generate validator results 121. In some examples, the validation schema 103 defines the type of data to be analyzed. In some examples, a data object 112 received with the dataset 109 defines the types of data and / or data partitions contained in the dataset 109. Each validator 106 contains a collection of data type-independent data modules 118 that are applied sequentially to the dataset 109. In some examples, a validator module 118 is one of several validator modules 118 associated with the validator 106. In various examples, each validator module 118 associated with the validator 106 is configured to analyze at least one of several data types (e.g., time series, continuous, categorical, multidimensional, etc.). Thus, the data analysis engine 115 can select several validator modules 118 based at least partially on the data types of the data in the dataset 109 being analyzed and / or the data partitions present in the dataset 109.

[0081]

[0084] In box 412, the data analysis engine 115 determines whether the data needs to be transformed. For example, if the data has been previously transformed using a transformation library 210 and / or a transformation function in the validation schema, the transformed data may be stored in the data cache 109 for subsequent use. Thus, if the data has been previously transformed and the transformed data is stored in the data cache 109, the data analysis engine 115 determines that the data does not need to be transformed and proceeds to box 418, where the data analysis engine 115 retrieves the transformed data from the data cache 219. However, if the data analysis engine 115 determines that the data needs to be transformed, the data analysis engine 115 proceeds to box 415.

[0082]

[0085] In box 415, the data analysis engine 115 applies transformation functions to transform data to be applied to a given validator module 118. For example, the data analysis engine 115 can apply one or more data-specific transformation functions to transform all columns of a certain data type contained in a dataset 109 into another data type supported by a given validator module 118. In some examples, the transformation functions can be included in the validation schema 109. In other examples, the transformation functions can be stored in the data store 215 within the transformation library 210. For example, the transformation functions may be contained in the transformation library 210, which may include predefined transformation processes that enable the effective transformation of data from unsupported data types to supported data types. In some examples, the transformation functions can map each data modality to a multidimensional continuous vector containing a compressed representation of the data modality, and then perform all checks of the data analysis system using an existing set of multidimensional validator methods. In other examples, the transformation function can map the data contained in dataset 109 from a high-dimensional feature space (e.g., image, audio data, etc.) to a low-dimensional descriptive representation (e.g., histogram, bandwidth-limited spectrograph, etc.). In various examples, the transformed data can be stored in the data cache 219 for subsequent use by other validators 106 and / or other validator modules 118.

[0083]

[0086] In box 421, the data analysis engine 115 executes the validator module 118 to determine the validation results associated with the corresponding data in the dataset being analyzed by applying the corresponding data to the functions of the validator module 118. The validator module 118 contains executable code that, when executed, performs a specific type of test on a data issue relating to a particular data type. Some examples of validator module 119 include: (1) the Mann-Whitney U test to examine distribution shifts between training / test splits in tabular data; (2) the kernel conditional independence (KCI) test to verify the independence assumption for vector-valued data; (3) an isolation forest trained / evaluated on image histograms to identify anomalies in image data; and / or other types of methods or modules.

[0084]

[0087] In box 424, the data analysis engine 115 determines the results associated with the validator module 118. In various examples, each validator module 118 executed may return validator results 121 associated with the corresponding analysis of the dataset 109 performed by the validator module 118. In some examples, the results include probability values. In other examples, the results include scores, and the tested data may be ranked according to these scores.

[0085]

[0088] In box 427, the data analysis engine 115 generates a validator results report 212 containing validator results 121. The validator results report 212 includes validator results 121 associated with each of the validator modules 118 that were executed. In some examples, the validator results report 212 includes validator results 121 associated with the validator modules 118 that were executed on the corresponding data evaluated. In other examples, the validator results report 212 includes aggregated validator results 124, which corresponds to an aggregate of the validator results 121 of multiple validator modules 118 executed on a given validator 106. In this example, the aggregated validator results 124 can provide a more accurate summary of the data results and highlight data areas that may contain issues that need to be addressed.

[0086]

[0089] In box 430, the data analysis engine 115 sends the validator result report 212 to the client device 206 so that it is displayed on the display 218 of the client device 206. In some examples, the data analysis engine 115 sends the validation result report 212 to the client device 206, which can then display the validation result report 212 on its display 218. For example, the data analysis engine 115 may generate a user interface 224 containing the validator result report 212, or generate user interface code to generate a user interface 224 containing the validator result report 212, and send the user interface 142 or the user interface code to the client device 206. This part of the process then proceeds to completion.

[0087]

[0090] Referring next to Figure 5, a flowchart 500 is shown that provides an example of some operation of the data analysis engine 115. The flowchart in Figure 5 is just one example of many different types of functional configurations that can be used to perform the operation of the illustrated part of the data analysis engine 115. Alternatively, the flowchart in Figure 5 can be considered as illustrating an example of elements of the method implemented within the network environment 200.

[0088]

[0091] In particular, Figure 5 relates to the ability for users to create new validator modules 118 for use within the data analysis system framework according to various embodiments. The ease of module extensibility allows users to easily integrate new validations (e.g., validator modules 118) into the data analysis system 100 framework without being limited to the existing set of validator modules 118, and without spending more time than it would take to write a standalone validator module 118. Furthermore, modularity and extensibility improve the scale and comprehensiveness of the ready-to-use validator modules 118 that can be incorporated into the validation schema 103 with a single line of code.

[0089]

[0092] The ease of extensibility of this disclosure is most evident in the module design of the validator module 118 applied to each dataset 109. The validator module 118 contains functional static classes scoped to apply a single method to a small number of data types. The validator module 118 does not explicitly include suggestions for solutions to data-related problems. Forcing the inclusion of potential solutions to such data-related problems would significantly impair the ease of extensibility and potentially limit the scope to which users can write and share their own methods. Furthermore, when data-related problems are discovered, there are usually multiple options for users to improve model performance by appropriately modifying the experimental setup, and since choosing the optimal option is a delicate judgment, it is best to leave that judgment to the user.

[0090]

[0093] Starting with box 503, the data analysis engine 115 receives a request to create a validator module 118. For example, the data analysis engine 115 may receive a request to create a validator module 118 from the client device 206 in response to user interaction with the user interface 224 associated with the data analysis engine 115, which is displayed on the client device 206. In this example, the user can determine that a certain type of data analysis is not supported or is not included in the validator module 118 associated with a validator 106 that is available to the data analysis engine 115. However, if the desired type of data analysis is not included in the validator module 118 associated with a given validator 106, the user can request the creation of a new validator module 118 by creating a request through interaction with the user interface 224.

[0091]

[0094] In box 506, the data analysis engine 115 selects a validator 106 associated with a validator module request. For example, the request may identify a validator 106. In various examples, validator 106 may include a parametric assumption validator, a conditional independence validator, a distribution shift validator, anomaly detection validator, an out-of-distribution (OOD) inference validator, and / or other types of validators 106.

[0092]

[0095] In box 509, the data analysis engine 115 determines which parts of the data object 112 to analyze. The data object 112 defines the dataset 109 and its divisions (e.g., training dataset, validation dataset, test dataset, inference dataset, etc.), mapping each column to its data type. In this example, the user can define whether the created validator module 118 will use the entire dataset 109, the training dataset, validation dataset, test dataset, inference dataset, and / or other parts of the dataset defined by the data object 112. In various examples, the data analysis engine 115 determines which parts of the data object 112 will be analyzed by the created validator module 118, depending on one or more user interactions of the user with the user interface 224 associated with the data analysis engine 115. In some examples, this information is included in the request. In other examples, this information is provided to the user via the user interface 224 and is included depending on the displayed prompts.

[0093]

[0096] In box 512, the data analysis engine 115 determines the data types supported by the new validator module 118. For example, the new validator module 118 may support time-series data, continuous data, categorical data, multidimensional data, and / or other types of data. In various examples, the data analysis engine 115 determines the data types supported by the created validator module 118 in response to one or more user interactions by the user with the user interface 224 associated with the data analysis engine 115. For example, the user may define the supported data types when creating a new validator module 118. In some examples, this information is included in the request. In other examples, this information is provided to the user via the user interface 224 and is included in response to displayed prompts.

[0094]

[0097] In box 515, the data analysis engine 115 determines data call information for the validator module 118. In various examples, the data call information may include configuration of method calls contained within the validator module. In some examples, this information includes calls to commercially available libraries that can be used and / or executed to obtain the necessary information. Furthermore, this information may include configuration data that defines data calls from the validator 106 using options that may be included in the validation schema 103. In other examples, this information may include executable code provided by a user requesting an additional validator module 118. In various examples, the data analysis engine 115 determines data call information for the validator module 118 in response to one or more user interactions of the user with the user interface 224 associated with the data analysis engine 115. In some examples, this information is included in the request. In other examples, this information is provided to the user via the user interface 224 and is included in response to prompts displayed.

[0095]

[0098] In box 518, the data analysis engine 115 creates the validator module 118. In various examples, the validator module 118 is created using information obtained and / or determined from boxes 506-515. This information includes, for example, (1) which parts of the data object are needed (e.g., for testing, splitting, inference, etc.); (2) the data types to support (e.g., time series, continuous, categorical, multidimensional, etc.); (3) the settings for each module call (e.g., the description of normality testing in a commercial library, etc.); and (4) predefined data calls from validator 106 using options that may be included in the validation schema 103.

[0096]

[0099] In box 521, the data analysis engine 115 stores the validator module 118 in the data store 215, associating it with the appropriate validator 106. This part of the process then proceeds to completion.

[0097]

[0100] Referring to Figure 6, a flowchart 600 is shown that provides an example of some operation of the data analysis engine 115. The flowchart in Figure 6 is just one example of many different types of functional configurations that can be used to perform the operation of the illustrated part of the data analysis engine 115. Alternatively, the flowchart in Figure 6 can be considered as illustrating an example of elements of the method implemented within the network environment 200.

[0098]

[0101] In particular, Figure 6 relates to generating aggregated validator results 124 based at least in part on the validator results 121 of the executed validator module 118. According to various embodiments, the user only needs to know what they are trying to evaluate in their data and does not need to identify or understand the underlying methods for checking these evaluations. Therefore, if multiple modules 118 are executed for a given validator 106, the user only needs to know the aggregated results associated with the executed module 118 and does not need to know the results for each executed module 118.

[0099]

[0102] Starting from box 603, the data analysis engine 115 selects validators 106 from the validation schema 103. In various examples, the validation schema 103 defines at least a subset of validators 106 for analyzing the data contained in the dataset 109. The selected validator 106 is one of multiple validators and is included in a subset of multiple validators 106. In various examples, validator 106 includes one of the following: parametric assumption validators, conditional independence validators, distribution shift validators, anomaly detection validators, out-of-distribution (OOD) inference validators, and / or other types of validators. In some examples, the validation schema 103 can include multiple validators 106 used for analyzing the provided dataset 109. Thus, the data analysis engine 115 can select multiple validators 106 based at least partially on the list of validators 106 included in the validation schema 103.

[0100]

[0103] In box 606, the data analysis engine 115 selects a validator module 118 associated with the selected validator 106. In some examples, the validator module 118 is selected based at least partially on the data type of the data in the dataset 109, the data partitioning associated with the dataset 109, the inclusion criteria defined in the validation schema 103, the column names of the data to be analyzed, and / or other factors. In particular, the validator module 118 is configured to analyze the data contained in the dataset 109 based at least partially on the data modality, the type of data to be analyzed (e.g., time series, continuous, categorical, multidimensional, etc.), the inclusion criteria defined in the validation schema 103, and / or other factors. These validator modules 118 mainly consist of the code necessary to receive the dataset 109, perform evaluations on it, and generate validator results 121. In some examples, the validation schema 103 can define the data to be analyzed. For example, the validation schema 103 can define the column names of the data to be analyzed, the data types of the data to be analyzed, the data partitions to be analyzed, and other types of inclusion criteria that define which data in the dataset 109 is to be analyzed. In some examples, the data object 112 received with the dataset 109 defines the types of data contained in the dataset 109. Each validator 106 contains a collection of data type-independent data modules 118 that are applied sequentially to the dataset 109.

[0101]

[0104] In box 609, the data analysis engine 115 executes the validator module 118 to determine the validation results associated with the corresponding data in the dataset being analyzed by applying the corresponding data to the functions of the validator module 118. The validator module 118 contains executable code that, when executed, performs a specific type of test on a data issue relating to a particular data type. Some examples of validator module 119 include: (1) the Mann-Whitney U test to examine distribution shifts between training / test splits in tabular data; (2) the kernel conditional independence (KCI) test to test causal assumptions for vector-valued data; (3) an isolation forest trained / evaluated on image histograms to identify anomalies in image data; and / or other types of methods or modules.

[0102]

[0105] In box 612, the data analysis engine 115 determines the results associated with the validator module 118. In various examples, each validator module 118 executed may return validator results 121 associated with the corresponding analysis of the dataset 109 performed by the validator module 118. In some examples, the results include probability values. In other examples, the results include scores, and the tested data may be ranked according to these scores.

[0103]

[0106] In box 615, the data analysis engine 115 determines whether there are any additional validator modules 118 associated with validator 106 that are available for execution. For example, validator 106 may be associated with multiple validator modules 118 that support specific types of data. In the example in Figure 1, validator modules 118a and 118b under validator 106a supported continuous data. Thus, if the data type to be evaluated includes continuous data and validator module 118a has already been executed, the data analysis engine 115 can determine that validator module 118b also needs to be executed. If additional validator modules 118 need to be executed, the data analysis engine 115 returns to box 606. For example, a second validator module can be selected from the multiple validator modules associated with validator 106, at least partially based on the data type of the data in the dataset being analyzed. The second validator module can be executed to determine a second validation result associated with the data in the dataset being analyzed. Otherwise, the data analysis engine 115 proceeds to box 618.

[0104]

[0107] In box 618, the data analysis engine 115 aggregates the validator results 121 associated with each validator module 118 to generate an aggregated result 124. The aggregated validator result 124 corresponds to the aggregated validator results 121 of multiple validator modules 118 executed on a given validator 106. For example, if a first validator module and a second validator module 118 are executed, the data analysis engine 115 can generate an aggregated validator result 124 based at least partially on the first validator result 121 associated with the first validator module 118 and the second validator result 121 associated with the second validator module 118. In this example, the aggregated validator result 124 can provide a more accurate summary of the data results and highlight data areas that may contain issues that need to be addressed. In some examples, the validator results 121 of different validator modules 118 can be converted into an "anomaly rank" for each sample in the dataset 109. Next, the data analysis engine 115 can generate an aggregated result 124 by aggregating those ranks. For example, the data analysis engine 115 can generate it by (a) averaging them and (b) using a ranking method based on the Borda count. However, it should be noted that the generation of the aggregated result 124 is not limited to the use of averaging and the ranking method based on the Borda count, and any type of rank aggregation method can be used.

[0105]

[0108] In box 621, the data analysis engine 115 generates a validator results report 212 containing the aggregated validator results 124 and sends the validator results report 212 to the client device 206 so that it is displayed on the client device 206's display 218. In some examples, the data analysis engine 115 can send the validation results report 212 to the client device 206, which can then display the validation results report 212 on its display 218. For example, the data analysis engine 115 may generate a user interface 224 containing the validator results report 212, or it may generate user interface code to generate the user interface 224 containing the validator results report 212, and can send the user interface 142 or the user interface code to the client device 206. This part of the process then proceeds to completion.

[0106]

[0109] The aforementioned software components are stored in the memory of their respective computing devices and are executable by the processor of those computing devices. In this sense, the term “executable” means a program file in a format that can ultimately be executed by the processor. Examples of executable programs include compiled programs that can be loaded into the random access portion of memory and converted into machine code in a format executable by the processor, source code that can be loaded into the random access portion of memory, such as object code, and expressed in a suitable format executable by the processor, or source code that can generate instructions for the random access portion of memory that can be interpreted by another executable program and executed by the processor. Executable programs can be stored in any part or component of memory, including random access memory (RAM), read-only memory (ROM), hard drives, solid-state drives, Universal Serial Bus (USB) flash drives, memory cards, optical discs such as compact discs (CDs) or digital multipurpose discs (DVDs), floppy disks, magnetic tapes, or other memory components.

[0107]

[0110] Memory includes both volatile and non-volatile memory, as well as data storage components. Volatile components are those that do not retain data values ​​when power is lost. Non-volatile components retain data even when power is lost. Therefore, memory may include random access memory (RAM), read-only memory (ROM), hard disk drives, solid-state drives, USB flash drives, memory cards accessed via memory card readers, floppy disks accessed via corresponding floppy disk drives, optical disks accessed via optical disk drives, magnetic tapes accessed via appropriate tape drives, or other memory components, or any two or more combinations of these memory components. Furthermore, RAM may include devices such as static random access memory (SRAM), dynamic random access memory (DRAM), or magnetic random access memory (MRAM). ROM may include programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or other similar memory devices.

[0108]

[0111] The applications and systems described herein can be implemented as software or code executed on general-purpose hardware as described above, but alternatively, they can also be implemented as dedicated hardware, or as a combination of software / general-purpose hardware and dedicated hardware. When implemented as dedicated hardware, each application and system can be implemented as a circuit or state machine employing one or a combination of several technologies. These technologies may include, but are not limited to, discrete logic circuits having logic gates to perform various logic functions when one or more data signals are applied, application-specific integrated circuits (ASICs) with appropriate logic gates, field-programmable gate arrays (FPGAs), or other components. Such technologies are generally well known to those skilled in the art and are therefore not described in detail herein.

[0109]

[0112] The flowchart illustrates the functions and operations when implementing some of the various embodiments of this disclosure. When implemented in software, each block may represent a module, segment, or portion of code containing program instructions for performing a specified logical function. Program instructions may be implemented in the form of source code containing human-readable statements written in a programming language, or in the form of machine code containing numerical instructions recognizable by an appropriate execution system, such as a processor in a computer system. Machine code can be converted from source code through various processes. For example, machine code can be generated from source code using a compiler before running the corresponding application. As another example, machine code may be generated from source code in parallel with execution by an interpreter. Other methods may also be available. When implemented in hardware, each block may represent a circuit or multiple interconnected circuits for implementing a specified logical function or function.

[0110]

[0113] While flowcharts illustrate a specific execution order, it is understood that the execution order may differ from that shown. For example, the execution order of two or more blocks may be different from the order shown and may be swapped. Also, two or more blocks shown consecutively may be executed simultaneously or partially simultaneously. Furthermore, in some embodiments, one or more blocks shown in a flowchart may be skipped or omitted. In addition, any number of counters, state variables, warning semaphores, or messages may be added to the logical flows described herein for various purposes, such as improving usability, accounting, performance measurement, or troubleshooting. All such modifications are understood to be within the scope of this disclosure.

[0111]

[0114] Furthermore, any logic or application described herein, including software or code, may be embodied by an instruction execution system, such as a processor in a computer system or other system, or in any non-temporary computer-readable medium for use in connection with an instruction execution system. In this sense, logic may include statements, including instructions and declarations, that can be obtained from a computer-readable medium and executed by an instruction execution system. In the context of this disclosure, “computer-readable medium” can be any medium that can store, preserve, or hold the logic or application described herein, used by or in connection with an instruction execution system. In addition, a collection of computer-readable media distributed across multiple computing devices (e.g., a storage area network, a distributed file system, a clustered file system, or a database) can also be considered collectively as non-temporary computer-readable media.

[0112]

[0115] Computer-readable media can include any one of various physical media, such as magnetic media, optical media, and semiconductor media. More specific examples of suitable computer-readable media include, but are not limited to, magnetic tapes, magnetic floppy disks, magnetic hard drives, memory cards, solid-state drives, USB flash drives, or optical discs. Furthermore, computer-readable media can be random-access memory (RAM), including static random-access memory (SRAM) and dynamic random-access memory (DRAM) or magnetic random-access memory (MRAM). In addition, computer-readable media may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or other types of memory devices.

[0113]

[0116] Furthermore, the logic or applications described herein can be implemented and structured in various ways. For example, one or more of the described applications can be implemented as modules or components of a single application. Furthermore, one or more of the described applications may run on shared or separate computing devices, or a combination thereof. For example, multiple applications described herein may run on the same computing device, or on multiple computing devices within the same computing environment 203.

[0114]

[0117] In addition to the foregoing, the various embodiments of this disclosure include, but are not limited to, those set forth in the following clauses.

[0115]

[0118] Clause 1 A system for facilitating customizable and modular data analysis, comprising a computing device including a processor and memory, and machine-readable instructions stored in memory, which, when executed by the processor, cause the computing device to: receive a dataset and a validation schema from a client device; select validators for analyzing data in the dataset, at least partially based on the validation schema; select validator modules associated with the validators, at least partially based on the data types of data in the dataset to be analyzed; execute the validator modules to determine validation results associated with the data in the dataset to be analyzed; generate a validator results report including the validation results; and send the validator results report to the client device.

[0116]

[0119] Clause 2: A validator is one of several validators, and the validation schema defines at least a subset of the several validators used to analyze the dataset, and the validator is included in the subset of the several validators as described in Clause 1.

[0117]

[0120] Clause 3: The validation schema is user-defined in response to user interaction with a user interface displayed on a client device, and the user interaction includes the selection of validator components associated with the validator, as described in Clause 1 or 2.

[0118]

[0121] Clause 4 The system described in Clause 3, wherein the validator includes multiple validator modules, and each validator module of the multiple validator modules is configured to parse one or more data types from a set of different data types, and the validator module is one of the multiple validator modules, and the multiple validator modules associated with the validator are unknown to the user defining the validation schema.

[0119]

[0122] Clause 5 A system described in any one of Clauses 1 to 4, wherein the validator includes one of the following: a parametric assumption validator, a conditional independence validator, a distribution shift validator, an anomaly detection validator, or an out-of-distribution (OOD) inference validator.

[0120]

[0123] Clause 6: A system described in any one of Clauses 1 to 5, wherein the data type includes one of time-series data, continuous data, categorical data, or multidimensional data.

[0121]

[0124] Clause 7 The system described in any one of Clauses 1 to 6, wherein the validator module is one of several validator modules associated with a validator, and each validator module is configured to analyze at least one of several data types.

[0122]

[0125] Clause 8 The system according to any one of Clauses 1 to 7, wherein the validator module is a first validator module among a plurality of validator modules associated with the validator, the validation result includes the first validation result, and when a machine-readable instruction is executed, it causes a computing device to further: select a second validator module from the plurality of validator modules associated with the validator, at least partially based on the data type of the data in the dataset being analyzed; execute the second validator module to determine a second validation result associated with the data in the dataset being analyzed; and generate an aggregated result, at least partially based on the first and second validation results.

[0123]

[0126] Clause 9 A system described in any one of Clauses 1 to 8, in which, when a machine-readable instruction is executed, the computing device further causes the computing device to receive a data object associated with the dataset, the data object defining the dataset and one or more partitions of the dataset, one or more partitions comprising at least one of the training dataset, validation dataset, test dataset, or inference dataset.

[0124]

[0127] The system described in any one of Clauses 1 to 9, wherein a validator is one of a plurality of validators, and when a machine-readable instruction is executed, it causes a computing device to receive from a client device a request to create a new validator module, the request comprising identification information of a particular validator among the plurality of validators, one or more data types supported by the new validator module, and data call information for the new validator module; creates the validator module based at least in part on the request; and stores the new validator module in association with the particular validator.

[0125]

[0128] Clause 11 A method for facilitating customizable and modular data analysis, comprising: receiving a dataset and validation schema from a client device by at least one computing device; selecting validators for analyzing data in the dataset, at least partially based on the validation schema, by at least one computing device; selecting validator modules associated with the validators, at least partially based on the data types of data in the dataset to be analyzed, by at least one computing device; running the validator modules by at least one computing device to determine validation results associated with the data in the dataset to be analyzed; generating a validator results report containing the validation results, by at least one computing device; and sending the validator results report to a client device by at least one computing device.

[0126]

[0129] Clause 12 The method according to Clause 11, wherein the validator is one of several validators, the validation schema defines at least a subset of the several validators to be used to parse the dataset, and the validator is included in the subset of the several validators.

[0127]

[0130] Clause 13 The method according to Clause 11 or 12, wherein the validation schema is user-defined in response to user interaction with a user interface displayed on a client device, and the user interaction includes the selection of validator components associated with the validator.

[0128]

[0131] The method of Clause 14, wherein the validator comprises multiple validator modules, and each validator module of the multiple validator modules is configured to parse one or more data types from a plurality of different data types, and the validator module is one of the multiple validator modules, and the multiple validator modules associated with the validator are unknown to the user defining the validation schema.

[0129]

[0132] Clause 15 The method described in any one of Clauses 11 to 14, wherein the validator includes one of the following: a parametric assumption validator, a conditional independence validator, a distribution shift validator, an anomaly detection validator, or an out-of-distribution (OOD) inference validator.

[0130]

[0133] Clause 16 The method described in any one of Clauses 11 to 15, wherein the data type includes one of time-series data, continuous data, categorical data, or multidimensional data.

[0131]

[0134] The method described in any one of the clauses 11 to 16, wherein the validator module is one of several validator modules associated with a validator, and each validator module is configured to analyze at least one of several data types.

[0132]

[0135] Clause 18 A method according to any one of Clauses 11 to 17, wherein the validator module is a first validator module among a plurality of validator modules associated with a validator, and the validation result includes a first validation result, further comprising: selecting a second validator module from the plurality of validator modules associated with the validator, at least in part on the data type of the data in the dataset to be analyzed; running the second validator module to determine a second validation result associated with the data in the dataset to be analyzed; and generating an aggregated result, at least in part on the first and second validation results, wherein the validator result report includes the aggregated result.

[0133]

[0136] A method according to any one of the clauses 11 to 18, further comprising receiving a data object associated with a dataset, wherein the data object defines a dataset and one or more splits of the dataset, one or more splits comprising at least one of a training dataset, a validation dataset, a test dataset, or an inference dataset.

[0134]

[0137] Clause 20 A method of any one of Clauses 11 to 19, wherein the validator is one of a plurality of validators, the method further comprising: receiving a request from a client device to create a new validator module, the request including identification information of a particular validator among the plurality of validators, one or more data types to be supported by the new validator module, and data call information for the new validator module; creating the validator module at least in part based on the request; and storing the new validator module in association with the particular validator.

[0135]

[0138] Clause 21 A non-temporary computer-readable medium containing machine-readable instructions for facilitating customizable and modular data analysis, wherein, when the machine-readable instructions are executed by the processor of a computing device, the computing device causes at least: to receive a dataset and a validation schema from a client device; to select validators for analyzing data in the dataset based at least partially on the validation schema; to select validator modules associated with the validators based at least partially on the data types of data in the dataset to be analyzed; to execute the validator modules to determine validation results associated with the data in the dataset to be analyzed; to generate a validator results report containing the validation results; and to transmit the validator results report to the client device.

[0136]

[0139] Clause 22 The validator is one of several validators, the validation schema defines at least a subset of the several validators to be used to analyze the dataset, and the validator is included in the subset of the several validators in the non-temporary computer-readable media as described in Clause 21.

[0137]

[0140] Clause 23 The validation schema is user-defined in response to user interaction with a user interface displayed on a client device, and the user interaction includes the selection of validator components associated with the validator, as described in Clause 21 or 22, in a non-transient computer-readable medium.

[0138]

[0141] Clause 24: A non-temporary computer-readable medium as described in Clause 23, in which the validator includes multiple validator modules, and each validator module of the multiple validator modules is configured to parse one or more data types from a set of different data types, and the validator module is one of the multiple validator modules, and the multiple validator modules associated with the validator are unknown to the user defining the validation schema.

[0139]

[0142] Clause 25 Non-transient computer-readable media as described in any one of Clauses 21 to 24, in which the validator includes one of the following: a parametric assumption validator, a conditional independence validator, a distribution shift validator, an anomaly detection validator, or an out-of-distribution (OOD) inference validator.

[0140]

[0143] Clause 26 Non-temporary computer-readable media as described in any one of Clauses 21 to 15, wherein the data type includes one of time-series data, continuous data, categorical data, or multidimensional data.

[0141]

[0144] Clause 27 A non-temporary computer-readable medium as described in any one of Clauses 21 to 26, wherein the validator module is one of several validator modules associated with a validator, and each validator module is configured to analyze at least one of several data types.

[0142]

[0145] Clause 28 A non-temporary computer-readable medium as described in any one of Clauses 21 to 27, wherein the validator module is a first validator module among a plurality of validator modules associated with the validator, the validation result includes the first validation result, and when a machine-readable instruction is executed, it causes a computing device to at least: select a second validator module from the plurality of validator modules associated with the validator based at least in part on the data type of the data in the dataset being analyzed; execute the second validator module to determine a second validation result associated with the data in the dataset being analyzed; and generate an aggregated result based at least in part on the first and second validation results.

[0143]

[0146] Clause 29 A non-transient computer-readable medium as described in any one of Clauses 21 to 28, in which, when a machine-readable instruction is executed, the computing device further causes the computing device to receive a data object associated with the dataset, the data object defining the dataset and one or more splits of the dataset, one or more splits comprising at least one of the training dataset, validation dataset, test dataset, or inference dataset.

[0144]

[0147] Clause 30 A non-temporary computer-readable medium as described in any one of Clauses 21 to 29, wherein the validator is one of a plurality of validators, and when a machine-readable instruction is executed, it further causes a computing device to receive from a client device a request to create a new validator module, the request comprising identification information of a particular validator among the plurality of validators, one or more data types supported by the new validator module, and data call information for the new validator module; and causes the computing device to create the validator module at least in part based on the request; and causes the new validator module to be stored in association with the particular validator.

[0145]

[0148] The system described in any one of the clauses 1 to 10, wherein, when a machine-readable instruction is executed, it causes a computing device to convert at least a portion of the data in a dataset into a different data format, at least partially based on one or more conversion functions.

[0146]

[0149] The system described in Clause 31, wherein, when a machine-readable instruction is executed, it further causes a computing device to store at least the converted data in a data cache.

[0147]

[0150] Clause 33: The system described in Clause 31 or 32, wherein one or more conversion functions are included in the conversion library.

[0148]

[0151] Clause 34 The method described in any one of Clauses 11 to 20, further comprising converting at least a portion of the data in a dataset into a different data format, at least in part, based on one or more conversion functions.

[0149]

[0152] Clause 35 The method of Clause 34, wherein, when a machine-readable instruction is executed, it further causes a computing device to store at least the converted data in a data cache.

[0150]

[0153] Clause 36 The method according to Clause 34 or 35, wherein one or more conversion functions are included in the conversion library.

[0151]

[0154] Clause 37 A non-transient computer-readable medium as described in any one of Clauses 22 to 30, wherein, when a machine-readable instruction is executed, it causes a computing device to convert at least a portion of the data in a dataset into a different data format, at least partially based on one or more conversion functions.

[0152]

[0155] Clause 38 A non-temporary computer-readable medium as described in Clause 37, which, when a machine-readable instruction is executed, further causes a computing device to store at least the converted data in a data cache.

[0153]

[0156] Clause 39 A non-temporary computer-readable medium as described in Clause 37 or 38, which contains one or more conversion functions in a conversion library.

[0154] Disjunctive expressions such as "at least one of X, Y, or Z" are generally understood in a general context to indicate that an item, term, etc., may be one of X, Y, or Z, or any combination thereof (e.g., X;Y;Z;X or Y;X or Z;Y or Z;X, Y, or Z, etc.), unless otherwise specified. Therefore, such disjunctive expressions generally do not mean, and should not be interpreted, that "at least one X," "at least one Y," or "at least one Z" must each exist in a particular embodiment.

[0155]

[0158] It should be emphasized that the embodiments described above in this disclosure are merely possible examples presented to clearly illustrate the principles of this disclosure. Many variations and modifications can be made to the embodiments described above without substantially departing from the spirit and principles of this disclosure. All such modifications and variations are intended to be within the scope of this disclosure and protected by the following claims.

Claims

1. A system that facilitates customizable and modular data analysis, A computing device including a processor and memory; A machine-readable instruction stored in the memory, which, when executed by the processor, outputs to the computing device at least: Receive the dataset and validation schema from the client device; Based at least partially on the validation schema, a validator is selected to analyze the data in the dataset; The validator module associated with the validator is selected based at least partially on the data type of the data in the dataset to be analyzed; The validator module is executed to determine the validation results associated with the data in the dataset to be analyzed; Generate a validator results report including the aforementioned verification results; and, The validator result report is sent to the client device. Machine-readable instructions A system that includes this.

2. The system according to claim 1, wherein the validator is one of a plurality of validators, the validation schema defines at least a subset of the plurality of validators to be used to analyze the dataset, and the validator is included in the subset of the plurality of validators.

3. The system according to claim 1 or 2, wherein the validation schema is user-defined in response to user interaction with a user interface displayed on a client device, and the user interaction includes the selection of validator components associated with the validator.

4. The system according to claim 3, wherein the validator includes a plurality of validator modules, each validator module of the plurality of validator modules is configured to analyze one or more data types from a plurality of different data types, the validator module is one of the plurality of validator modules, and the plurality of validator modules associated with the validator are unknown to the user who defines the validation schema.

5. The system according to any one of claims 1 to 4, wherein the validator includes one of a parametric assumption validator, a conditional independence validator, a distribution shift validator, an anomaly detection validator, or an out-of-distribution (OOD) inference validator.

6. The system according to any one of claims 1 to 5, wherein the data type includes one of time-series data, continuous data, categorical data, or multidimensional data.

7. The system according to any one of claims 1 to 6, wherein the validator module is one of a plurality of validator modules associated with the validator, and each validator module is configured to analyze at least one of a plurality of data types.

8. A system according to any one of claims 1 to 7, wherein the validator module is a first validator module among a plurality of validator modules associated with the validator, the verification result includes a first verification result, and when the machine-readable instruction is executed, the computing device further provides at least: Based at least partially on the data type of the data in the dataset to be analyzed, a second validator module is selected from the plurality of validator modules associated with the validator; The second validator module is executed to determine a second validation result associated with the data in the dataset to be analyzed; and, Based at least partially on the first and second verification results, an aggregated result is generated. The validator results report includes the aggregated results in the system.

9. The system according to any one of claims 1 to 8, wherein when the machine-readable instruction is executed, the computing device further causes the computing device to receive a data object associated with the dataset, the data object defining the dataset and one or more partitions of the dataset, the one or more partitions comprising at least one of a training dataset, a validation dataset, a test dataset, or an inference dataset.

10. A system according to any one of claims 1 to 9, wherein the validator is one of a plurality of validators, and when the machine-readable instruction is executed, the computing device further provides The client device receives a request to create a new validator module, the request including identification information for a specific validator among the plurality of validators, one or more data types supported by the new validator module, and data call information for the new validator module; Create the validator module based at least partially on the aforementioned request; and, A system for storing the aforementioned new validator module in association with the aforementioned specific validator.

11. When the machine-readable instruction is executed, the computing device will also receive at least: The system according to any one of claims 1 to 10, which causes at least a portion of the data in the dataset to be converted to a different data format based at least in part on one or more conversion functions.

12. The system according to claim 11, wherein when the machine-readable instruction is executed, the computing device further causes at least the converted data to be stored in a data cache.

13. The system according to claim 11, wherein the one or more conversion functions are included in the conversion library.

14. A method for facilitating customizable and modular data analysis, At least one computing device receives the dataset and validation schema from the client device; Selecting validators for analyzing data in the dataset, at least partially based on a validation schema, by at least one computing device; Selecting a validator module associated with the validator by the at least one computing device, at least partially based on the data type of the data in the dataset to be analyzed; The validator module is executed by the at least one computing device to determine the validation results associated with the data in the dataset being analyzed; A validator results report including the verification results is generated by the at least one computing device; and The validator result report is transmitted to the client device by the at least one computing device. Methods that include...

15. The method according to claim 14, wherein the validator is one of a plurality of validators, the validation schema defines at least a subset of the plurality of validators to be used to analyze the dataset, and the validator is included in the subset of the plurality of validators.

16. The method according to claim 14 or 15, wherein the validation schema is user-defined in response to user interaction with a user interface displayed on a client device, and the user interaction includes the selection of validator components associated with the validator.

17. The method according to claim 16, wherein the validator includes a plurality of validator modules, each validator module of the plurality of validator modules is configured to analyze one or more data types from a plurality of different data types, the validator module is one of the plurality of validator modules, and the plurality of validator modules associated with the validator are unknown to the user who defines the validation schema.

18. The method according to any one of claims 14 to 17, wherein the validator includes one of a parametric assumption validator, a conditional independence validator, a distribution shift validator, an anomaly detection validator, or an out-of-distribution (OOD) inference validator.

19. The method according to any one of claims 14 to 18, wherein the data type includes one of time-series data, continuous data, categorical data, or multidimensional data.

20. The method according to any one of claims 14 to 19, wherein the validator module is one of a plurality of validator modules associated with the validator, and each validator module is configured to analyze at least one of a plurality of data types.

21. The method according to any one of claims 14 to 20, wherein the validator module is a first validator module among a plurality of validator modules associated with the validator, and the verification result includes a first verification result, further, Selecting a second validator module from the plurality of validator modules associated with the validator, at least partially based on the data type of the data in the dataset to be analyzed; Executing the second validator module to determine a second validation result associated with the data in the dataset being analyzed; and This includes generating an aggregated result based at least partially on the first verification result and the second verification result, A method wherein the validator results report includes the aggregated results.

22. A method according to any one of claims 14 to 21, further comprising receiving a data object associated with the dataset, wherein the data object defines the dataset and one or more splits of the dataset, the one or more splits comprising at least one of a training dataset, a validation dataset, a test dataset, or an inference dataset.

23. The method according to any one of claims 14 to 22, wherein the validator is one of a plurality of validators, further, Receiving a request from the client device to create a new validator module, the request including identification information for a specific validator among the plurality of validators, one or more data types supported by the new validator module, and data call information for the new validator module; Creating the validator module based at least in part on the aforementioned request; and A method comprising storing the new validator module in association with the specific validator.

24. The method according to any one of claims 14 to 23, further comprising converting at least a portion of the data in the dataset into a different data format based at least in part on one or more conversion functions.

25. The method according to claim 24, wherein when a machine-readable instruction is executed, the computing device further causes at least the converted data to be stored in a data cache.

26. The method according to claim 24, wherein the one or more conversion functions are included in the conversion library.

27. A non-temporary computer-readable medium containing machine-readable instructions for facilitating customizable and modular data analysis, wherein when the machine-readable instructions are executed by the processor of a computing device, the computing device receives at least: Receive the dataset and validation schema from the client device; Based at least partially on the validation schema, select validators to analyze the data in the dataset; The validator module associated with the validator is selected based at least partially on the data type of the data in the dataset to be analyzed; The validator module is executed to determine the validation results associated with the data in the dataset to be analyzed; Generate a validator results report including the aforementioned verification results; and, The validator result report is sent to the client device. A non-temporary computer-readable medium.

28. The non-temporary computer-readable medium according to claim 27, wherein the validator is one of a plurality of validators, the validation schema defines at least a subset of the plurality of validators to be used to analyze the dataset, and the validator is included in the subset of the plurality of validators.

29. The non-temporary computer-readable medium according to claim 27 or 28, wherein the validation schema is user-defined in response to user interaction with a user interface displayed on a client device, and the user interaction includes the selection of validator components associated with the validator.

30. The non-temporary computer-readable medium according to claim 29, wherein the validator comprises a plurality of validator modules, each validator module of the plurality of validator modules is configured to analyze one or more data types from a plurality of different data types, the validator module is one of the plurality of validator modules, and the plurality of validator modules associated with the validator are unknown to the user defining the validation schema.

31. The non-temporary computer-readable medium according to any one of claims 27 to 30, wherein the validator includes one of a parametric assumption validator, a conditional independence validator, a distribution shift validator, an anomaly detection validator, or an out-of-distribution (OOD) inference validator.

32. A non-temporary computer-readable medium according to any one of claims 27 to 31, wherein the data type includes one of time-series data, continuous data, categorical data, or multidimensional data.

33. The non-temporary computer-readable medium according to any one of claims 27 to 32, wherein the validator module is one of a plurality of validator modules associated with the validator, and each validator module is configured to analyze at least one of a plurality of data types.

34. A non-temporary computer-readable medium according to any one of claims 27 to 33, wherein the validator module is a first validator module among a plurality of validator modules associated with the validator, the validation result includes a first validation result, and when the machine-readable instruction is executed, the computing device further provides at least: Based at least partially on the data type of the data in the dataset to be analyzed, a second validator module is selected from the plurality of validator modules associated with the validator; The second validator module is executed to determine a second validation result associated with the data in the dataset to be analyzed; and, Based at least partially on the first and second verification results, an aggregated result is generated. The validator results report is a non-temporary, computer-readable medium containing the aggregated results.

35. The non-temporary computer-readable medium according to any one of claims 27 to 34, wherein when the machine-readable instruction is executed, the computing device further causes the computing device to receive a data object associated with the dataset, the data object defines the dataset and one or more partitions of the dataset, the one or more partitions comprising at least one of a training dataset, a validation dataset, a test dataset, or an inference dataset.

36. A non-temporary computer-readable medium according to any one of claims 27 to 35, wherein the validator is one of a plurality of validators, and when the machine-readable instruction is executed, the computing device further... The client device receives a request to create a new validator module, the request including identification information for a specific validator among the plurality of validators, one or more data types supported by the new validator module, and data call information for the new validator module; Create the validator module based at least partially on the aforementioned request; and, The new validator module is stored in association with the specific validator. A non-temporary computer-readable medium.

37. When the machine-readable instruction is executed, the computing device further receives at least: Convert at least a portion of the data in the dataset into a different data format based at least partially on one or more conversion functions. A non-temporary computer-readable medium according to any one of claims 27 to 36.

38. The non-temporary computer-readable medium according to claim 37, wherein when the machine-readable instruction is executed, the computing device further causes at least the converted data to be stored in a data cache.

39. The non-temporary computer-readable medium according to claim 37, wherein the one or more conversion functions are included in the conversion library.

Images