System and method for predicting correct or missing data and data anomalies

Abstract

The present disclosure relates to systems and methods for predicting and correcting data anomalies. In an exemplary aspect, data is received by a system. The system can analyze the data by profiling the data for certain profiling statistics (e.g., minimum, maximum, average, cardinality, etc.). At least one machine learning algorithm (e.g., a random forest algorithm) can be applied to the profiled data to confirm potential relationships between certain data columns in the data. Once certain relationships are confirmed, related data can be extracted to form a project set. A second machine learning algorithm (e.g., a frequent pattern growth algorithm) can be applied to the project set to confirm certain frequencies of related values ​​in the project set. Low frequency values ​​can represent anomalies in the data set. If an anomaly is detected, the system can be configured to provide intelligent repair actions, such as replacing certain values ​​and / or filling missing values.

Description

Technical Field

[0001] The present disclosure relates to systems and methods for predicting correct or missing data, data anomalies, and overall data quality using associative machine learning algorithms such as random forests and frequent pattern growing. Background Art

[0002] Entities maintain a large amount of data that may be related (or associated) to certain data classifications. Compared with column-level verification, such entities may have difficulty understanding the attribute relationships within the data set, and it is difficult to verify record-level information. For example, a zip code is usually related to a street address and a state. An entity may have a valid zip code and a valid state code, but the zip code and the state code together do not form a valid record (for example, because the zip code is not located in the state). In these data sets, there may be data anomalies such as the one just described. At present, entities must manually confirm these data anomalies by determining whether certain data points are actually abnormal. In order to make this determination, entities usually have to confirm certain relationships between data points (for example, zip codes are related to street addresses and have a specific digital format consisting of nine digits). Confirming these relationships in the data set is cumbersome and time-consuming. The current process of trying to confirm the relationship also produces errors because the manual confirmation of data relationships often ignores or simply loses the relationship between certain data points. In this way, the current method of confirming data anomalies is inconsistent and inefficient, in part because confirming data anomalies usually requires evaluating the relationship between data points (or lack of such a relationship). If a particular data received looks unusual compared to previously received data, this may indicate an anomaly, but it may also be a false positive. Similarly, a particular data received that is not frequently received from a particular entity may also indicate an anomaly, or alternatively, it may not be an anomaly but simply a less frequently received data point. Untangling large datasets to identify data anomalies is necessary to predict the data quality of the dataset, and currently, accurately predicting the correct or missing data for a given record and the overall data quality of a dataset is difficult and unreliable because today's methods for identifying data anomalies are error-prone and inefficient.

[0003] As such, there is an increased need for systems and methods that can address the challenges of identifying data anomalies, identifying data relationships, and accurately and efficiently predicting the correct data set.

[0004] It is with respect to these and other general considerations that the various aspects disclosed herein have been made.In addition, although relatively specific problems may be discussed, it should be understood that the examples should not be limited to solving specific problems identified in the background of this disclosure or elsewhere. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1An example of a distributed system for predicting correct or missing data and data quality as described herein is shown.

[0006] Figure 2 An exemplary input processor for predicting correct or missing data and data quality as described herein is shown.

[0007] Figure 3A and 3B An exemplary method for predicting correct or missing data and data quality of a data set as described herein is shown.

[0008] Figures 4A-4C An exemplary architecture for implementing systems and methods for predicting correct or missing data and data quality is shown.

[0009] Figure 5 An example of a suitable operating environment is shown in which one or more of the present embodiments may be implemented. DETAILED DESCRIPTION

[0010] Aspects of the present disclosure are described more fully below with reference to the accompanying drawings, which form a part of this document and illustrate specific exemplary aspects. However, different aspects of the present disclosure can be implemented in many different forms and should not be interpreted as being limited to the aspects described herein; rather, these aspects are provided so that the present disclosure will be thorough and complete and the scope of these aspects will be fully conveyed to those skilled in the art. Various aspects can be practiced as methods, systems, or devices. Therefore, various aspects can take the form of hardware implementations, fully software implementations, or implementations combining software and hardware aspects. Therefore, the following detailed description should not be taken as limiting.

[0011] Figure 1 An example of a distributed system for predicting correct or missing data and data quality as described herein is shown. The proposed exemplary system 100 is a combination of interdependent components that interact to form an integrated whole for consolidating and enriching data on a data market. The components of the system can be hardware components or software implemented on and / or executed by the hardware components of the system. For example, the system 100 includes client devices 102, 104, and 106, local databases 110, 112, and 114, a network 108, and server devices 116, 118, and / or 120.

[0012] The client devices 102, 104, and 106 may be configured to receive and transmit data. For example, the client devices 102, 104, and 106 may include client-specific data. The client devices may also be configured to receive data that can be ingested and analyzed at the client devices 102, 104, and / or 106 from various data sources (e.g., data lakes, databases, flat files, data streams, etc.) via one or more networks 108. The received data may be stored in local databases 110, 112, and 114. The received data may be analyzed by applying at least one machine learning (ML) algorithm to the data. The received data may be transmitted to one or more servers 116, 118, and / or 120 via one or more networks 108 and / or satellites 122. One or more servers 116, 118, and / or 120 may be third-party servers and / or servers owned by the client. Depending on, for example, the size of the data, the complexity of the analysis to be performed on the data, etc., the data may be transmitted to one or more servers 116, 118, and / or 120 for further processing, analysis, and / or storage. In other examples, the received data may be stored in a server (in addition to or in lieu of a local client device and a local database) and may be analyzed to determine correct or missing data and the data quality of the data set. The received data along with a data quality predictor (e.g., a score, a table, etc.) may be transmitted from the client server to a third party server via one or more networks 108 and / or satellites 122.

[0013] In various aspects, client devices (e.g., client devices 102, 104, and 106) may have access to one or more data sets or data sources and / or databases that include customer-specific data. In other aspects, client devices 102, 104, and 106 may be equipped to receive broadband and / or satellite signals that carry customer data. The data may include raw data sets, data sets that have been analyzed (e.g., have been subjected to the application of at least one ML algorithm), and / or a mixture of both. The signals and information that client devices 102, 104, and 106 may receive may be transmitted from satellite 122. Satellite 122 may also be configured to communicate with one or more networks 108, in addition to being able to communicate directly with client devices 102, 104, and 106. In some examples, client devices may be mobile phones, laptops, tablets, smart home devices, landlines, and wearable devices (e.g., smart watches), among other devices.

[0014] To further illustrate the network topology, client devices 102, 104, and / or 106 (and their corresponding local databases 110, 112, and 114) can receive data from a client. On the client devices 102, 104, and / or 106, a parser and / or rule engine can be installed and applied to the received data. Comparison and analysis of the received data can occur locally on the client devices 102, 104, and / or 106. In an alternative embodiment, the received data can be compared and analyzed at one or more servers 116, 118, and / or 120. Once compared and analyzed, the received data can be manipulated to confirm which data columns have the strongest relationship. In some examples, a predictive relationship / association indicator can be generated. The predictive indicator can be generated at one or more client devices 102, 104, and / or 106, one or more servers 116, 118, and / or 120, or a combination of both. The predictive indicator output can be generated locally or externally and shared via one or more networks 108.

[0015] Once the association of the data set is confirmed, a transaction item set can be generated. The item set can be generated at one or more client devices 102, 104, and / or 106, or generated externally at one or more servers 116, 118, and / or 120. The transaction item set can then be analyzed using a second ML algorithm (e.g., a frequent pattern growth (FPG) algorithm) to confirm the frequency of the confirmed data associations. A data quality indicator can be generated that indicates which data associations may be less frequent and present in the data set. Less frequent data associations can indicate data anomalies. The output can be generated locally at one or more client devices 102, 104, and / or 106, and / or one or more servers 116, 118, and / or 120. Regardless of where the DQ output is generated, the DQ output can be shared with a variety of devices via one or more networks 108.

[0016] Figure 2An exemplary input processor for predicting correct or missing data and data quality as described herein is shown. Input processor 200 can be embedded in a client device (e.g., client device 102, 104, and / or 106), a remote network server device (e.g., device 116, 118, and / or 120), and other devices capable of implementing systems and methods for predicting correct or missing data and data quality. The input processing system includes one or more data processors and is capable of executing algorithms, software routines, and / or instructions based on processed data provided by at least one data source (e.g., a data lake, a database, a flat file, a data stream, etc.). The input processing system can be a factory-installed system or an additional unit of a specific device. In addition, the input processing system can be a general-purpose computer or a specialized special-purpose computer. There is no limitation on the location of the input processing system relative to a client or a remote network server device, etc.

[0017] according to Figure 2 In the embodiment shown in , the disclosed system may include a memory 205, one or more processors 210, a communication module 215, a data collection module 220, an ML algorithm module 225, and a data or DQ recommendation module 230. Other embodiments of the present technology may include some, all, or none of these modules and components, as well as other modules, applications, data, and / or components. However, some embodiments may combine two or more of these modules and components into a single module and / or associate a portion of the functionality of one or more of these modules with different modules.

[0018] The memory 205 may store instructions for running one or more applications or modules on one or more processors 210. For example, the memory 205 may be used in one or more embodiments to hold all or part of the instructions required to perform the functions of the data collection module 220, the ML algorithm module 225, and / or the data / DQ suggestion module 230, and the communication module 215. In general, the memory 205 may include any device, mechanism, or populated data structure for storing information. According to some embodiments of the present disclosure, the memory 205 may include but is not limited to any type of volatile memory, non-volatile memory, and dynamic memory. For example, the memory 205 may be a random access memory, a memory storage device, an optical storage device, a magnetic medium, a floppy disk, a tape, a hard disk drive, a SIMM, a SDRAM, a RDRAM, a DDR, a RAM, a SODIMM, an EPROM, an EEPROM, an optical disk, a DVD, and / or the like. According to some embodiments, the memory 205 may include one or more disk drives, a flash drive, one or more databases, one or more tables, one or more files, a local cache memory, a processor cache memory, a relational database, a flat database, and / or the like. Furthermore, those of ordinary skill in the art will appreciate that many additional devices and techniques for storing information may be used as memory 205 .

[0019] The communication module 215 is associated with sending / receiving information (e.g., data collected by the data collection module 220, data analyzed / processed according to the ML algorithm module 225, and / or predicted correct or missing data, and data quality information from the DQ recommendation module 230), commands received via a client device or server device, other client devices, remote network servers, etc. These communications can use any suitable technology type, such as Bluetooth, WiFi, WiMax, cellular (e.g., 5G), single-hop communication, multi-hop communication, dedicated short-range communication (DSRC), or proprietary communication protocols. In some embodiments, the communication module 215 sends information (e.g., data) collected by the data collection module 220 (e.g., batches, large batches, and / or continuous feeds), processed by the ML algorithm module 225 (e.g., the data set may be profiled and analyzed using a random forest algorithm to predict associations, and the frequency of associations may be further analyzed using a frequent pattern growing algorithm), and generated as output by the DQ recommendation module 230 (e.g.,), and / or sent to the client devices 102, 104, and / or 106, and the memory 205 to be stored for future use. In some examples, the communication module may be built on the HTTP protocol through one or more secure REST servers using RESTful services.

[0020] The data collection module 220 is configured to receive and analyze data. For example, the data collection module 220 can receive data related to a data set. The data collection module 220 can then analyze the data, including capturing statistics, including but not limited to one or more minimum values, one or more maximum values, cardinality, confidence frequencies, and categorical value encodings, among other statistics. In some examples, cardinality can refer to the number of different values ​​captured in the data set.

[0021] The data collection module 220 may also be configured to identify certain data values ​​that are inherently low confidence, low support, and / or high frequency. Such data values ​​may be excluded by the data collection module 220 before being sent to the ML algorithm module 225.

[0022] One or more ML algorithm modules 225 are configured to receive a data set and apply at least one machine learning (ML) algorithm to the data set. For example, after the data set is received and analyzed by the data collection module 220, the data set can be transmitted to the ML algorithm module 225. The ML algorithm module 225 can be configured to analyze the data set, and in particular, to analyze the columns of the data set. Each column label can be compared with other columns to confirm the relationship between the columns in the data set.

[0023] In order to confirm the relationship between the columns, the ML algorithm module 225 can apply the random forest (RF) algorithm to the data set. The features and attributes required to start / build the ML algorithm can be obtained during the profiling or other analysis steps. The data set can be profiled first, and the descriptive statistics obtained from the profiling step can be used as input to the RF algorithm. The random forest algorithm can build multiple decision trees and merge them together to obtain more accurate and stable predictions about the relationship between the data sets. Other algorithms that can be applied to the data set include, but are not limited to, classifier models, XGBoost (and other tree-based integrated machine learning algorithms using gradient boosting), and other ML models from such as MLlib (Apache Spark's machine learning library).

[0024] After applying the RF algorithm (or other ML algorithm), an output may be generated that may represent the strongest relationships between the data columns. For example, a relationship representing the relationship between the column label "zip code" and the column label "street address" may be represented because zip codes are often associated with street addresses. The label "SSN" may be associated with the column labels "first name" and "last name" because social security numbers are often associated with an individual's identity. These data columns may be basketed together based on feature importance, explanatory coefficients, and other model accuracy / relationship determinations.

[0025] Once the relevant data columns are identified, the headers of the data can be concatenated for each row. The ML algorithm module 225 can be configured to create a new transaction data set called an "item set" based on the concatenation of the column headers of each data row. The ML algorithm module 225 can then apply a second ML algorithm (such as a frequent pattern growing algorithm) to further predict the frequency of the relationship. In other words, the application of the RF algorithm can indicate whether certain data items are related, and the additional application of the frequent pattern growing algorithm can indicate how frequently these relationships occur in the data set.

[0026] The ML algorithm module 225 can be configured to calculate a probability matrix based on the frequency of data relationships. The frequency of the relationship basket can indicate whether there are certain data anomalies in the data set. Less frequent relationships can indicate a higher probability of data anomalies, while more frequent relationships can indicate a lower probability of data anomalies.

[0027] After the data set is processed by the ML algorithm module 225, the DQ suggestion module 230 can be configured to receive processing information from the ML algorithm module 225 indicating the frequency of certain data relationships (i.e., certain data columns are related to other data columns). In some instances, the DQ suggestion module 230 can output intelligent suggestions that suggest certain actions to be taken on the data to fix any confirmed data anomalies. In other examples, a table indicating the associations between certain data records and the frequency of these data relationships can be displayed (e.g., the system can highlight potential relationships in an output table showing two records with the same SSN, name, and gender but with two different email addresses). Less frequent data relationships can be highlighted to indicate outlier data points or suspicious data that may be erroneous.

[0028] Figure 3A and 3BAn exemplary method for predicting data quality of a data set as described herein is shown. Method 300 is divided into two separate sub-methods - sub-method 300A and sub-method 300B. Sub-method 300A is a relationship finder sub-method that confirms the relationship between the data sets (if any). Sub-method 300A begins with receiving data step 302. The data can be received in the form of a data lake, a database, a flat file, a data stream, or other ingestible data format. The data can then be analyzed at step 304 to ensure that the data is in a format that can be read by the systems and methods described herein (for example, step 304 may include a user interface button that can be pressed to start profiling). The data can be profiled at step 306, where certain data statistics such as minimum, maximum, average, cardinality indicators, categorical columns, etc. can be captured. After the data values ​​are profiled, a brute force comparison can then be applied to the data set at step 308. For example, exemplary attribute A can be compared to all other attributes in the data set, and exemplary attribute B can be compared to all other attributes in the data set. The brute force comparison output can display a table indicating which column headers can be related to the data values. For example, the following table may be displayed for an exemplary comparison of column 1 ("col1") with other columns in a data set:

[0029]

[0030]

[0031] At step 310, a first machine learning algorithm may be applied to the data set. The first ML algorithm may be a random forest algorithm, which may be applied to understand which data columns have potential relationships with each other. At step 312, those column relationships may be confirmed based on the output of the RF algorithm. For example, a zip code data column may be associated with a street address. A bank account number data column may be associated with a bank name data column. An SSN data column may be associated with a first name data column and a last name data column. An IP address data column may be associated with a machine identifier data column.

[0032] Once the relationships between the data columns are confirmed, at step 314, the resulting output of these relationships is added to the basket based on the automatic selection of data column relationships and / or the user-selected relationships of the data columns. The data column relationship basket is then analyzed at step 316, where the data column headers are concatenated to create a transaction item set at step 318. The transaction item set at step 318 is then provided as input to a second ML algorithm, such as a frequent pattern growth model, at step 320. At step 322, the FPG model can calculate a probability matrix of item set combinations to determine specific patterns in the item set. The patterns confirmed at step 322 can represent the frequency of relationships (or lack of relationships) between certain data columns.

[0033] Based on the frequency of data relationships confirmed by the FPG model and the information derived from the RF model, a data prediction can be generated at step 324, which can predict correct or missing data and the data quality of the data set. For example, less frequent relationships in the data can indicate that certain data values ​​are abnormal or erroneous. More frequent relationships in the data can indicate that the data values ​​include fewer anomalies and errors.

[0034] In one specific example, a bank may be analyzing loan data. The data may be limited to a specific geographic area. Certain relationships in the data may be confirmed by the RF algorithm at step 310, such as the relationship between certain bank customer account numbers and SSN numbers. In addition, the frequency of these relationships may be confirmed by applying the FPG model at step 320. However, although the DQ prediction system described herein can confirm a certain relationship, the relationship may be suspicious, for example, because the claimed account number may not comply with the standard format guidelines for account numbers (e.g., an account number with 6 digits instead of 8 digits). In this way, the system can mark the data value as an anomaly because the data value has a low confidence score. Another example of anomaly detection can be based on transaction volume in a specific geographic area. Although there may be relationships between data columns, the frequency of a specific transaction volume being associated with a specific type of bank transaction (e.g., deposits, withdrawals, loans, etc.) may be low. Since the frequency of a specific relationship in the data set is low, the system can also mark this transaction as a potential anomaly.

[0035] In yet another example, a data set may include columns with zip codes and state identifier codes. One data entry may include zip code "21087" with a state identifier code of "MD" (Maryland), while another data entry may include the same zip code "21087" but with a state code of "NY" (New York). The system described herein may identify the data anomaly, apply at least one machine learning algorithm to the data set, and recommend at least one corrective action (i.e., identify which data entry is correct and which data entry is incorrect). Although the zip code itself is in the zip code column and is formatted correctly, one of the data records is incorrect (21087 is a zip code associated with Maryland).

[0036] In an additional example, a data set may include an asset symbol, such as a publicly traded stock or cryptocurrency. An asset may be traded on multiple exchanges with the same symbol. However, if a particular asset appears on an exchange where the asset has never been traded before, the system can confirm the anomaly and recommend one or more corrective actions. From a data quality perspective, the asset symbol may be correct and formatted correctly, but the underlying quality of the data record (asset symbol plus exchange) may be incorrect because the exchange does not actually support trading of the asset.

[0037] Figures 4A-4C An exemplary architecture for implementing systems and methods for predicting correct or missing data and data quality is shown. Environment 400 includes data sources, which may include, but are not limited to, data lake 402, database 404, flat file 406, and data stream 408. Once the data is received by the system, data extraction service 410 can be applied to the raw input data. Extraction service 410 can reformat the data into a format suitable for processing. The data can be fed into a profiler 412 continuously / in batches / in large quantities. Profiler 412 can be configured to implement a rule engine 414, which can include statistics for analyzing data. Such statistics can include maximum, minimum, average, and cardinality of data values. Certain data columns can be classified and confidence values ​​can be initially assigned to certain data. The frequency of certain data types can also be analyzed by profiler 412.

[0038] The profiled data may be compared and analyzed at 416. In some examples, the profiled data may be further analyzed by applying certain rules to prepare a data set for application of an ML algorithm. For example, certain missing values ​​in the data set may be filled with dummy values ​​(e.g., n / a for categorical values, 0 for numerical values, or replacing the mean). In another example, the profiled data may be decomposed to provide more relevant information to the ML algorithm. For example, if sales data varies depending on the day of the week, the day of the week (e.g., Friday) may be separated from the actual date (April 15, 2021). Certain min-max normalization may also be applied to the data set, such as applying confidence intervals or filtering based on maximum and / or minimum extreme values. For example, before applying a certain ML algorithm, in some instances, the top 2.5% and bottom 2.5% of the values ​​may be discarded.

[0039] Various artificial intelligence / machine learning tools 418 may be applied to the classifier module 420. The classifier module 420 may classify the profiled data into certain data categories, such as strings, integers, and other discrete objects.

[0040] The features of the data sets may be compared at 422, which may indicate which ML algorithm to apply to the data sets. Once the ML algorithm is applied to the data sets, at 426, the output may represent predicted relationships and / or associations between the data sets.

[0041] Once the relationship of the data set is confirmed, the problem confirmation process can be started. First, a data set can be selected at 428. The selected data set can be a subset of the initial data set input into the system. The selected data set can be a data set of related data columns and / or possible anomalies confirmed. The selected data set can be transformed into a transaction item set 430, in which the column headers with data values ​​can be linked together. Once the item set is created, additional AI / ML toolsets can be applied to the item set, such as ML algorithms like frequent pattern growth, which can confirm the frequency of relationships in the data set (that is, how often certain data columns appear in relationships with other data columns). Based on the output of the FPG model, a prediction model can be established at 434, in which data anomalies can be confirmed. In addition, the user of the system can be presented with a suggested repair action for cleaning data. For example, both data anomalies and / or potential errors can be presented to the user. The most likely results and / or predictions of a specific data set can also be presented to the user. Specifically, the user can accept a specific prediction value in order to clean the data set (for example, the system can predict the correct value of a specific data entry that may be wrong, and the user can accept the proposed change).

[0042] Figure 5 An example of a suitable operating environment in which one or more embodiments of the present embodiment can be implemented is shown. This is only an example of a suitable operating environment and is not intended to limit the scope of use or functionality. Other well-known computing systems, environments and / or configurations that may be suitable for use include, but are not limited to, personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, programmable consumer electronics (e.g., smartphones), network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0043] In its most basic configuration, the operating environment 500 typically includes at least one processing unit 502 and memory 504. Depending on the exact configuration and type of computing device, the memory 504 (which stores information related to detected devices, association information, personal gateway settings, and instructions for executing the methods disclosed herein, etc.) can be volatile (e.g., RAM), non-volatile (e.g., ROM, flash memory, etc.), or some combination of the two. Figure 5506. In addition, the environment 500 may also include storage devices (removable 508 and / or non-removable 510), including but not limited to disks or optical disks or tapes. Similarly, the environment 500 may also have one or more input devices 514, such as a keyboard, mouse, pen, voice input, etc., and / or one or more output devices 516, such as a display, speaker, printer, etc. The environment may also include one or more communication connections 512, such as LAN, WAN, point-to-point, etc.

[0044] The operating environment 500 typically includes at least some form of computer-readable media. Computer-readable media can be any available media that can be accessed by the processing unit 502 or other devices that include the operating environment. By way of example, and not limitation, computer-readable media can include computer storage media and communication media. Computer storage media include volatile and non-volatile removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, cassettes, magnetic tape, disk storage, or other magnetic storage devices, or any other tangible media that can be used to store the desired information. Computer storage media do not include communication media.

[0045] Communication media embodies non-transitory computer-readable instructions, data structures, program modules, or other data. Computer-readable instructions may be transmitted in a modulated data signal such as a carrier wave or other transport mechanism, and include any information delivery media. The term "modulated data signal" refers to a signal that has one or more of its characteristics set or changed in a manner that encodes the information in the signal. By way of example, and not limitation, communication media include wired media, such as a wired network or a direct wired connection, and wireless media, such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.

[0046] Operating environment 500 can be a single computer that operates in a networked environment using a logical connection to one or more remote computers. The remote computer can be a personal computer, a server, a router, a network PC, a peer device, or other public network node, and typically includes many or all of the above elements and other elements that are not mentioned in this way. The logical connection can include any method supported by an available communication medium. Such a networked environment is often located in an office, an enterprise-wide computer network, an intranet, and the Internet.

[0047] For example, the above text describes various aspects of the present disclosure with reference to the block diagrams and / or operational diagrams of the methods, systems, and computer program products according to various aspects of the present disclosure. The functions / actions marked in the blocks may not occur in the order shown in any flow chart. For example, two blocks shown in succession may in fact be substantially executed concurrently, or the blocks may sometimes be executed in reverse order, depending on the functions / actions involved.

[0048] The description and illustration of one or more aspects provided in this application are not intended to limit or restrict the scope of the disclosure claimed for protection in any way. The multiple aspects, examples and details provided in this application are considered to be sufficient to convey the possession thereof, and enable others to make and use the best mode of the disclosure claimed for protection. The disclosure claimed for protection should not be interpreted as being limited to any aspect, example or detail provided in this application. Whether shown and described in combination or individually, various features (structures and methods) are intended to be selectively included or omitted to produce an embodiment with a specific feature set. After providing the description and illustration of this application, those skilled in the art can envision changes, modifications and alternative aspects that fall within the spirit of the broader aspects of the general inventive concept embodied in this application, without departing from the broader scope of the disclosure claimed for protection.

[0049] In summary, it should be understood that specific embodiments of the present invention have been described herein for illustrative purposes, but that various modifications may be made without departing from the scope of the present invention. Therefore, the present invention is not limited except by the appended claims.

Claims

1. A method for predicting and correcting data anomalies in a data set, comprising: receiving a first data set; analyzing the first data set; applying a first machine learning algorithm to the first data set to identify at least one relationship between a first data column and a second data column in the first data set; generating a second data set based on the confirmation of the at least one relationship between the first data column and the second data column in the first data set, wherein the second data set is a subset of the first data set; concatenate a plurality of column headers in the second data set; generating an itemset based on the concatenation of the plurality of column headers in the second data set; applying a second machine learning algorithm to the set of items to identify at least one frequency value associated with the at least one relationship between the first data column and the second data column in the first data set, wherein the second machine learning algorithm is a frequent pattern growing algorithm; Based on the at least one frequency value, identifying at least one outlier in the set of items, wherein the at least one outlier is associated with a low frequency value; and At least one suggestion for repairing the at least one outlier is generated, and the at least one outlier is repaired by replacing the at least one outlier with a replacement value associated with a high frequency value.

2. The method according to claim 1, wherein: Identifying at least one outlier in the itemset includes identifying at least one missing value in the itemset.

3. The method according to claim 2, wherein: The at least one suggestion for fixing the at least one outlier includes at least one suggested value for filling the at least one missing value in the itemset.

4. The method according to claim 3, wherein: The at least one suggested value is at least one of: zero and an average value.

5. The method according to claim 1, wherein: The first machine learning algorithm is a random forest algorithm.

6. The method according to claim 1, wherein: The at least one outlier in the itemset is a relational anomaly in at least one data record.

7. The method according to claim 6, wherein: The at least one suggestion is a suggestion for correcting the relationship anomaly in the at least one data record.

8. The method according to claim 7, wherein: The suggestion for correcting the relationship anomaly includes at least one replacement value.

9. The method according to claim 1, further comprising: receiving at least one user input accepting the at least one suggestion; as well as Based on the at least one user input, the at least one suggestion is applied to the set of items.

10. The method according to claim 1, wherein: The first data set and the second data set are at least one of: a data file, a data lake, a flat file, and a data stream.

11. A system for predicting and correcting data anomalies in a data set, comprising: a memory configured to store non-transitory computer-readable instructions; as well as a processor communicatively coupled to the memory, wherein the processor, when executing the non-transitory computer-readable instructions, is configured to: receiving a first data set; analyzing the first data set; applying a first machine learning algorithm to the first data set to identify at least one relationship between a first data column and a second data column in the first data set; generating a second data set based on the confirmation of the at least one relationship between the first data column and the second data column in the first data set, wherein the second data set is a subset of the first data set; concatenate a plurality of column headers in the second data set; generating an itemset based on the concatenation of the plurality of column headers in the second data set; applying a second machine learning algorithm to the set of items to identify at least one frequency value associated with the at least one relationship between the first data column and the second data column in the first data set, wherein the second machine learning algorithm is a frequent pattern growing algorithm; Based on the at least one frequency value, identifying at least one outlier value in at least one data record in the itemset, wherein the at least one outlier value is associated with a low frequency value; and At least one suggestion for repairing the at least one outlier is generated, and the at least one outlier is repaired by replacing the at least one outlier with a replacement value associated with a high frequency value.

12. The system according to claim 11, wherein: The at least one outlier in the at least one data record is an anomaly in the relationship between at least one data value in the first data column and at least one data value in the second data column.

13. The system according to claim 12, wherein: The relationship anomaly is related to the geographical location.

14. The system according to claim 12, wherein: The relationship anomaly is associated with an asset symbol, wherein the asset symbol is at least one of: a stock symbol and a cryptocurrency symbol.

15. The system according to claim 11, wherein: The first machine learning algorithm is a random forest algorithm.

16. The system of claim 11, wherein: The at least one abnormal value is at least one of: a missing value, an incorrect value, and a format error.

17. The system of claim 11, further configured to generate at least one output table displaying the at least one outlier in the at least one data record in the itemset and a corrected version of the at least one data record without the at least one outlier.

18. A computer-readable storage medium storing non-transitory computer-executable instructions, which when executed, cause a computing system to perform a method for predicting and correcting data anomalies, the method comprising: receiving a first data set; analyzing the first data set; applying a first machine learning algorithm to the first data set to identify at least one relationship between a first data column and a second data column in the first data set; generating a second data set based on the confirmation of the at least one relationship between the first data column and the second data column in the first data set, wherein the second data set is a subset of the first data set; concatenate a plurality of column headers in the second data set; generating an itemset based on the concatenation of the plurality of column headers in the second data set; applying a second machine learning algorithm to the set of items to identify at least one frequency value associated with the at least one relationship between the first data column and the second data column in the first data set, wherein the second machine learning algorithm is a frequent pattern growing algorithm; Based on the at least one frequency value, identifying at least one outlier in the set of items, wherein the at least one outlier is associated with a low frequency value; and At least one suggestion for repairing the at least one outlier is generated, and the at least one outlier is repaired by replacing the at least one outlier with a replacement value associated with a high frequency value.

Images