Method, system, computer-readable medium, and program product for discovering semantic meaning of a data field from profile data of the data field

Abstract

A data processing system for discovering semantic meaning of a field included in one or more data sets, the data processing system being configured to identify a field included in the one or more data sets, wherein the field has an identifier. For the field, the system analyzes data values ​​of the field to generate a data profile, accesses a plurality of label proposal tests, and generates a set of label proposals by applying the plurality of label proposal tests to the data profile. The system determines similarities between the label proposals and selects a classification. The system identifies one of the label proposals as identifying the semantic meaning. The system stores the identifier of the field together with the identified one of the label proposals that identifies the semantic meaning.

Description

[0001] Priority declaration

[0002] This application claims priority to U.S. patent application serial number 62 / 855,233 filed on May 31, 2019 and U.S. patent application serial number 16 / 794,361 filed on February 19, 2020, each of which is hereby incorporated by reference in its entirety. Technical Field

[0003] The present disclosure relates to classification of data. More specifically, the present disclosure relates to classifying data fields by analyzing data profiles and metadata of the data fields. Background Art

[0004] A data set may include data content stored in a data field. A data field may indicate the type of data content stored in the data field. Summary of the invention

[0005] The following systems and processes described in the present disclosure may include one or more of the following advantages. In many systems, existing data sets may be very large and undocumented. In some cases, if the data set is labeled, the labels may be inconsistent or technical in nature, making it difficult to determine business information from the labels. Due to legacy systems and changes in system versions over time, it is difficult to determine what each part of the data set is in actual context. Therefore, there is a need to develop annotations for such data sets.

[0006] These systems and processes are configured to quickly and in some cases automatically provide labels for data sets. The systems and processes described in this document are configured to perform a series of statistical checks on the metadata and data content of a data set in order to discover, classify and label the data content of the data set. Discovering data content includes determining that data content (or data fields) exist in a data set. For example, the systems and processes described in this disclosure provide for marking of data fields (e.g., columns in a table or database) that are not labeled or labeled in a meaningless manner. These systems and processes are configured to discover that specific data fields exist in a data set and mark specific data fields. In order to perform this labeling, the systems and processes described in this document are configured to map descriptive definitions to fields of a data set, such as by assigning business terms or business domains to fields of a data set. The labels generated for the data fields provide information about the data content stored in the data fields. This information can indicate what type of data content (e.g., phone number, name, date, etc.) is included in the data field.

[0007] This information can be used by computing systems for a variety of applications. For example, applications that can use the generated dataset labels can include data quality enforcement, personal data anonymization, data masking, personally identifiable information (PII) reporting, test data management, dataset annotation, etc. For example, annotated data can include PII and may need to be masked. In another example, the systems and processes described in this document can provide a mechanism for labeling third-party data from different sources. This may be useful for systems where system administrators are responsible for knowing and understanding what data is in the datasets stored on the system, such as for regulatory reasons.

[0008] In addition, the systems and processes described herein enable the construction of an infrastructure that allows data standards to be applied across many different data items from different data sources. The infrastructure can be easily modified (e.g., by adding definitions to classification tests through a user interface) so that additional tests can be added as new data requirements are identified.

[0009] In one aspect, a system is configured to identify a field included in one or more data sets, wherein the field has an identifier; and for the field: one or more data values ​​of the field are analyzed by a data processing system to generate a data profile; a plurality of label proposal tests are accessed; a set of label proposals is generated based on at least applying the plurality of label proposal tests to the data profile; similarities between label proposals in the set of label proposals are determined; a classification is selected based at least on the similarities between the label proposals in the set; a graphical user interface is presented based on the classification to request input when identifying a label proposal that identifies a semantic meaning or to determine that no input is required; one of the label proposals is identified as identifying the semantic meaning; and the identifier of the field is stored in a data storage area together with the identified one of the label proposals that identifies the semantic meaning. The function of the preamble is to explain the general technical environment of the present invention.

[0010] In some embodiments, analyzing the one or more data values ​​of the field includes determining the format of the data values ​​of the field. Analyzing the data values ​​of the field includes determining statistics representing the data values ​​included in the field. The statistics include at least one of the minimum length of the data values ​​of the field, the maximum length of the data values ​​of the field, the most common data value of the field, the least common data value of the field, the maximum data value of the field, and the minimum data value of the field. Applying the multiple label proposal tests includes: determining that the field includes a primary key of one of the one or more data sets; and selecting a label proposal test related to the primary key from the multiple label proposal tests. Applying the multiple label proposal tests includes: performing a metadata comparison of the data values ​​of the field with terms in a glossary. Applying the multiple label proposal tests includes: determining a pattern represented by the data values ​​stored in the field according to the data profile; determining a specific label mapped to the pattern; and marking the field with the specific label. Applying the multiple label proposal tests includes: retrieving a list of values ​​representing a data set; comparing a data value of the field to the list of values; in response to the comparison, determining that a threshold number of data values ​​match the values ​​in the list; and in response to the determination, marking the field with a particular label that specifies the data set. Applying the multiple label proposal tests includes: generating at least two labels for the field; and determining whether the at least two labels are mutually exclusive or mutually inclusive.

[0011] In some embodiments, the system is configured to perform actions further comprising: in response to applying the plurality of label proposal tests, determining a relationship between the field and another field of the one or more data sets. The relationship comprises one of: indicating that a first data value of the field determines a second data value stored in another field, indicating that the first data value is related to the second data value, or indicating that the first data value is the same as the second data value. The plurality of label proposal tests are each associated with at least one weight value, the method further comprising: updating a weight value associated with at least one label proposal test; and reapplying the label proposal test to the data profile using the updated weight value.

[0012] In some embodiments, the actions further include training the plurality of label proposal tests using a machine learning process. The actions include retrieving one or more data quality rules assigned to the label proposal that specifies the semantic meaning from a data quality rule environment; and assigning one of the one or more data quality rules to the field. Comparing the label proposals generated by the label proposal tests includes: applying a score value to each label proposal; for each label in the label proposals, combining the score values ​​associated with the label; and sorting the labels according to the score values ​​associated with each label.

[0013] In some embodiments, the actions further include: receiving a validation of the label proposals from the plurality of label proposal tests; and in response to receiving the validation, weighting the plurality of label proposal tests with the label proposals. The data store includes a data dictionary. In some embodiments, the actions further include: outputting the label proposals to a data quality rules environment. In some embodiments, the actions further include: reducing another number of errors in processing data for the field using data quality rules from the data quality environment based on the identified one of the label proposals relative to a number of errors in processing data for the field without using the identified one of the label proposals.

[0014] In one aspect, a system includes a data storage device storing instructions; and at least one processor configured to execute the instructions stored in the data storage device to perform operations including: identifying a field included in one or more data sets, wherein the field has an identifier; and for the field: parsing, by a data processing system, one or more data values ​​of the field to generate a data profile; accessing multiple label proposal tests; generating a set of label proposals based on applying at least the multiple label proposal tests to the data profile; determining similarities between label proposals in the set of label proposals; selecting a classification based at least on the similarities between the label proposals in the set; presenting a graphical user interface requesting input when identifying a label proposal that identifies a semantic meaning or determining that no input is required based on the classification; identifying one of the label proposals as identifying the semantic meaning; and storing the identifier of the field in a data storage area together with the identified one of the label proposals that identifies the semantic meaning.

[0015] In some embodiments, analyzing the one or more data values ​​of the field includes determining the format of the data values ​​of the field. Analyzing the data values ​​of the field includes determining statistics representing the data values ​​included in the field. The statistics include at least one of the minimum length of the data values ​​of the field, the maximum length of the data values ​​of the field, the most common data value of the field, the least common data value of the field, the maximum data value of the field, and the minimum data value of the field. Applying the multiple label proposal tests includes: determining that the field includes a primary key of one of the one or more data sets; and selecting a label proposal test related to the primary key from the multiple label proposal tests. Applying the multiple label proposal tests includes: performing a metadata comparison of the data values ​​of the field with terms in a glossary. Applying the multiple label proposal tests includes: determining a pattern represented by the data values ​​stored in the field according to the data profile; determining a specific label mapped to the pattern; and marking the field with the specific label. Applying the multiple label proposal tests includes: retrieving a list of values ​​representing a data set; comparing a data value of the field to the list of values; in response to the comparison, determining that a threshold number of data values ​​match the values ​​in the list; and in response to the determination, marking the field with a particular label that specifies the data set. Applying the multiple label proposal tests includes: generating at least two labels for the field; and determining whether the at least two labels are mutually exclusive or mutually inclusive.

[0016] In some embodiments, the system is configured to perform actions further comprising: in response to applying the plurality of label proposal tests, determining a relationship between the field and another field of the one or more data sets. The relationship comprises one of: indicating that a first data value of the field determines a second data value stored in another field, indicating that the first data value is related to the second data value, or indicating that the first data value is the same as the second data value. The plurality of label proposal tests are each associated with at least one weight value, the method further comprising: updating a weight value associated with at least one label proposal test; and reapplying the label proposal test to the data profile using the updated weight value.

[0017] In some embodiments, the actions further include training the plurality of label proposal tests using a machine learning process. The actions include retrieving one or more data quality rules assigned to the label proposal that specifies the semantic meaning from a data quality rule environment; and assigning one of the one or more data quality rules to the field. Comparing the label proposals generated by the label proposal tests includes: applying a score value to each label proposal; for each label in the label proposals, combining the score values ​​associated with the label; and sorting the labels according to the score values ​​associated with each label.

[0018] In some embodiments, the actions further include: receiving a validation of the label proposals from the plurality of label proposal tests; and in response to receiving the validation, weighting the plurality of label proposal tests with the label proposals. The data store includes a data dictionary. In some embodiments, the actions further include: outputting the label proposals to a data quality rules environment. In some embodiments, the actions further include: reducing another number of errors in processing data for the field using data quality rules from the data quality environment based on the identified one of the label proposals relative to a number of errors in processing data for the field without using the identified one of the label proposals.

[0019] In one aspect, a system includes: a device for identifying a field included in one or more data sets, wherein the field has an identifier; for the field: a device for parsing one or more data values ​​of the field by a data processing system to generate a data profile; a device for accessing multiple label proposal tests; a device for generating a set of label proposals based on applying at least the multiple label proposal tests to the data profile; a device for determining similarities between label proposals in the set of label proposals; a device for selecting a classification based at least on the similarities between the label proposals in the set; a device for presenting a graphical user interface that requests input when identifying a label proposal that identifies a semantic meaning or determines that no input is required based on the classification; a device for identifying one of the label proposals as identifying the semantic meaning; and a device for storing the identifier of the field in a data storage area together with the identified one of the label proposals that identifies the semantic meaning.

[0020] In some embodiments, analyzing the one or more data values ​​of the field includes determining the format of the data values ​​of the field. Analyzing the data values ​​of the field includes determining statistics representing the data values ​​included in the field. The statistics include at least one of the minimum length of the data values ​​of the field, the maximum length of the data values ​​of the field, the most common data value of the field, the least common data value of the field, the maximum data value of the field, and the minimum data value of the field. Applying the multiple label proposal tests includes: determining that the field includes a primary key of one of the one or more data sets; and selecting a label proposal test related to the primary key from the multiple label proposal tests. Applying the multiple label proposal tests includes: performing a metadata comparison of the data values ​​of the field with terms in a glossary. Applying the multiple label proposal tests includes: determining a pattern represented by the data values ​​stored in the field according to the data profile; determining a specific label mapped to the pattern; and marking the field with the specific label. Applying the multiple label proposal tests includes: retrieving a list of values ​​representing a data set; comparing a data value of the field to the list of values; in response to the comparison, determining that a threshold number of data values ​​match the values ​​in the list; and in response to the determination, marking the field with a particular label that specifies the data set. Applying the multiple label proposal tests includes: generating at least two labels for the field; and determining whether the at least two labels are mutually exclusive or mutually inclusive.

[0021] In some embodiments, the system is configured to perform actions further comprising: in response to applying the plurality of label proposal tests, determining a relationship between the field and another field of the one or more data sets. The relationship comprises one of: indicating that a first data value of the field determines a second data value stored in another field, indicating that the first data value is related to the second data value, or indicating that the first data value is the same as the second data value. The plurality of label proposal tests are each associated with at least one weight value, the method further comprising: updating a weight value associated with at least one label proposal test; and reapplying the label proposal test to the data profile using the updated weight value.

[0022] In some embodiments, the actions further include training the plurality of label proposal tests using a machine learning process. The actions include retrieving one or more data quality rules assigned to the label proposal that specifies the semantic meaning from a data quality rule environment; and assigning one of the one or more data quality rules to the field. Comparing the label proposals generated by the label proposal tests includes: applying a score value to each label proposal; for each label in the label proposals, combining the score values ​​associated with the label; and sorting the labels according to the score values ​​associated with each label.

[0023] In some embodiments, the actions further include: receiving a validation of the label proposals from the plurality of label proposal tests; and in response to receiving the validation, weighting the plurality of label proposal tests with the label proposals. The data store includes a data dictionary. In some embodiments, the actions further include: outputting the label proposals to a data quality rules environment. In some embodiments, the actions further include: reducing another number of errors in processing data for the field using data quality rules from the data quality environment based on the identified one of the label proposals relative to a number of errors in processing data for the field without using the identified one of the label proposals.

[0024] Other features and advantages of the system will become apparent from the following description and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1A to Figure 1B is a block diagram of a system for discovering, classifying, and tagging data fields by analyzing data profiles generated from data of the data fields.

[0026] FIG. 2A to FIG. 2E is a block diagram illustrating how execution system 202 parses, categorizes, and tags data fields to identify the semantic meaning of the data fields.

[0027] Figures 3A to 3G is a block diagram of the process used to profile, classify, and tag data fields to identify their semantic meaning.

[0028] Figure 4 is a flow chart illustrating operations performed by a logic engine of an execution system to discover, classify, and tag data fields by analyzing data profiles generated from data of the data fields.

[0029] Figure 5 is a block diagram of the process used to profile, classify, and tag data fields to identify their semantic meaning.

[0030] Fig. 6A is a depiction of a user interface showing the results of a data profiling process.

[0031] Figure 6B is a depiction of a user interface showing the results of a classification process.

[0032] Figure 6C is a depiction of a user interface showing recommendations for labels for discovered data fields.

[0033] Fig.6D is a depiction of an example user interface showing the results of marking a data field.

[0034] Figure 7 is a flow chart illustrating an example process for discovering, classifying, and tagging data fields by analyzing data profiles generated from data of the data fields. DETAILED DESCRIPTION

[0035] refer to Figure 1A to Figure 1B , shows a networked system 100 for discovering, classifying, and labeling data fields by analyzing data profiles generated from data of the data fields. The networked system 100 includes an execution system 102, one or more data sources 112, a reference database 116, and a data dictionary database 114. The execution system 102 is configured to receive source data from data sources 112a, 112b (collectively referred to as data sources 112) in the networked system 100. The execution system 102 is configured to analyze the source data received from the data source 112 to generate a data profile representing the source data of the data source 112. The execution system 102 is configured to analyze the profile data to discover, classify (e.g., test) portions of the source data, and associate portions of the source data with labels representing the semantic meaning of those portions of the source data. A portion of the source data may include a field in the source data. In other words, the execution system 102 uses the profile data to determine the semantic meaning of one or more fields of the source data using multiple classification tests. Classifying the source data includes associating probabilities with portions of the source data. The probability specifies the likelihood that a portion of the source data corresponds to a label included in the data dictionary database 114 .

[0036] Typically, the execution system 102 includes a profile data module 104, a classification module 105, a test module 106, a result validation module 108, and a load data module 110, which together are configured to discover, classify, and label data fields of data received from a data source 112. A general description of the labeling process (also referred to as a semantic discovery process) of the execution system 102 is now described. The profile data module 104 receives source data 112 and generates profile data. As described subsequently, the profile data includes metadata representing statistical properties of the source data and fields of the source data. The profile data may also include data specifying relationships (e.g., statistical correlations, dependencies, etc.) between or among fields of the source data.

[0037] The classification module 105 classifies each field as having a data type (eg, date field, number field, string, etc.) The classification module checks the tag index of the reference database 116 to determine if the field (or other fields with the same field name) has been tagged with a semantic tag.

[0038] Typically, each field in a data set is associated with an identifier, such as a field name. An identifier may also include a field address that specifies the location of the field in the data set. For example, an identifier may specify in which table the field is and which column of the table represents the field. In some embodiments, the identifier is part of a tag index that associates the identifier for each field with the semantic tags for those fields. The identifier may be the name of the field, a key value, or some other value that identifies the field.

[0039] The test module 106 includes a plurality of classifiers, each of which is configured to operate on field data values, field names, and profile data for each field. The classifiers (also referred to as tests) of the test module 106 each use a different approach to propose labels to be associated with a field, which is described in more detail below. The test uses a label dictionary from a data dictionary database 114 as a label library from which one or more labels can be proposed to identify the semantic meaning of the tested field. Due to the use of different approaches, the tests can propose different labels from each other, or the proposed labels from all tests can be the same. The proposed labels (and their scores, if applicable) for each test from the test module 106 are sent to the result verification module 108.

[0040] The result verification module 108 classifies the proposed tags for the field based on the similarity between the proposed tags tested. If the proposed tags are all the same, the similarity is high and a match is found. The result verification module 108 can also classify the proposed tag groups for each field from the test module 106 as recommended tags (where one of the proposed tags is a recommended tag). The result verification module 108 may find that there is little similarity between the proposals and recommend that the user further explore the results. The result verification module 108 can also classify the proposed tags from the test module 106 as ignored tags, as described later. The execution system 102 can receive verification data (e.g., received from the user through a user interface) that verifies that the proposed tags correctly identify the semantic meaning of the field. After verification, the proposed tags will be assigned to the field, and the tag index will be updated to show that the field is associated with the assigned tag. The load data module 110 sends the tag index to the reference database 116 for access by future iterations of the labeling process (also known as the semantic discovery process), or for access by downstream applications. The functions of modules 104, 105, 106, 108 and 110 will then be described with respect to Figures 2A to 3G Describe in more detail.

[0041] The data dictionary database 114 includes a list of labels that are candidate labels for identifying the semantic meaning of a field of source data. Each label of the data dictionary database 114 is associated with one or more attributes. The attribute describes the semantic meaning of the label and may include labels, associations, or other metadata that provide context for the semantic meaning of each label to the execution system 102. For example, the attributes may include statistical metadata describing the value of a given field, labels for a set of specific values ​​of the field (e.g., a list of city names, zip codes, etc.), specified data formats (e.g., date formats), relationships between or among fields of a data set, and the like. During the testing (classification) process, the execution system 102 is configured to determine the extent to which the attributes of the label are closely consistent with the attributes of the field of the source data. The classification test is configured to analyze the profile data of the field and the content of the field itself to determine which labels of the data dictionary database 114 identify the semantic meaning of the field. Before the execution system 102 performs the semantic discovery process, the data dictionary database 114 may be populated with a list of labels and label attributes.

[0042] like Figure 1A As shown in , the data dictionary database 114 is extensible because new labels can be added to the data dictionary database 114 as data with new semantic meanings are processed by the execution system. In some embodiments, a set of seed labels can be automatically provided to initialize the system, and if the existing labels of the data dictionary database 114 do not identify the semantic meaning of the field, additional labels can be added as needed. For example, if the execution system 102 fails to find a label in the data dictionary database 114 with attributes that satisfy the classification test (e.g., the score for each test is below a threshold), the execution system 102 can prompt the user to define a new label for the data dictionary database 114 with attributes corresponding to the unlabeled field. In subsequent tests, the execution system 102 is configured to propose newly defined labels to label subsequent fields with similar attributes.

[0043] The execution system 102 is configured to automatically or in response to user input, label the source data portion with one or more tags that identify the semantic meaning of the source data portion. The discovery, classification, and labeling of the source data can be iterated to improve the classification of the source data and improve the accuracy of the labeled source data. The labeling of the source data includes metadata associated with the source data as a label index, as described subsequently. Once the discovery, classification, and labeling of the source data are completed, the label index is loaded into the reference database 116 for use by one or more downstream applications.

[0044] like Figure 1BAs shown, once the execution system 102 has performed the analysis, classification, and tagging of the source data, the load data module 110 can send the data to each of the data dictionary database 114 and the reference database 116. If additional tags are defined during or as a result of the semantic discovery process, the newly defined tags are added to the data dictionary database 114 for use by the execution system 102 (e.g., the test module 106) in future tagging. In addition or in the alternative, the load data module 110 sends tag index data to the reference database 116. For the source data, the tag index describes which tag is associated with each field of the source data that has been tagged by the execution system 102. For example, for each tag, the tag index can include a list of each field associated with the tag. Alternatively, for each field of the source data, the tag index can describe which tags are associated with the field. The tag index is stored in the reference database 116. In a subsequent tagging process of the execution system 102, the tag index can be checked (e.g., by the classification module 105) to determine whether a particular field has been tagged. This can allow the execution system 102 to bypass the testing process and simply label the field with a known label, thereby reducing the amount of data processed to label the field. In some implementations, data indicating that a label has been applied to the field (or similar fields) can be used during the testing process to increase the accuracy of the tested classifier, as described subsequently.

[0045] The tag index can be provided to or retrieved by one or more downstream applications for various purposes. The tag index provides a quick reference for downstream applications to determine the meaning of data values ​​of a data set without the downstream applications having to analyze the data set. For example, an application only needs to refer to the tag index to determine the semantic meaning of a field. The tag index can indicate whether a particular field includes personally identifiable information (PII) and how to mask the data of the field. The tag index can indicate what the schema of the data set is (e.g., whether the field is a primary key or a foreign key). The tag index can indicate what the semantic relationship between fields is by specifying the attributes of the tags (e.g., whether they are related to each other, whether there is a dependency, etc.). Because the semantic meaning is included in the attributes of each tag in the tag, the application only needs to refer to the tag index instead of scanning the data set itself to retrieve the semantic information. Reducing the amount of data to be processed by downstream applications reduces the processing resources of the application because only the tag index is needed to determine the semantic meaning of each field in the data set. In an example, the tag index can indicate which fields in the data set include PII. Without scanning every field of the data set, the application can mask only those fields with PII as needed. Therefore, the application can access the data storage area storing the data set less frequently and can transmit less data (reducing bandwidth usage).

[0046] In some embodiments, a data dictionary may also be provided to the application. The relatively small data footprint of the data dictionary and tag index is very useful when the data set is large. The size of the tag index and data dictionary each grows slowly relative to the size of the data set itself. The tag index and data dictionary provide all the context that the application needs to access only the relevant portion of the data set, rather than requiring the entire data set to be retrieved to provide sufficient context for the application. The application can simply refer to the data set's tags and the attributes of the tags to determine the context of the fields in the data set.

[0047] The source data of the data source 112 may include several different types of data. In one example, the source data of (multiple) data sources 112 include a table with data fields. The execution system 102 is configured to discover, classify and mark the data fields of the table. For example, the execution system 102 analyzes the data content of each discovery field of the source data and determines what the data content of the data field represents. The execution system 102 classifies each data field by associating the data field with a known tag (e.g., by assigning a probability value to it). The marked data of the table can be output to a data storage device 114, which other applications and systems can access to operate on the marked data. The process of discovering, classifying and marking the data fields of the data table is described in detail later. The data source 112 may include any type of computing system. For example, the data source 112 may include a mainframe, a database, unstructured data provided from a third party, a data lake, a personal computer, a large-scale network, etc.

[0048] In the present disclosure, the process for discovering, classifying and marking data fields of data tables is used as an example to illustrate the functions of execution system 102 and networked system 100. However, although the data field is an example of something that execution system 102 is configured to discover, classify and detect, execution system 102 can operate on other types of data. For example, execution system 102 receives application data from an application. Execution system 102 is configured to discover, classify and mark different data of the application. The data may include different files stored and accessed for operating the application. Execution system 102 can be used to discover PII stored by the application, discover malware, changes to application files, etc. In another example, execution system 102 can analyze files stored in a file system (e.g., on a personal computer). Execution system 102 can scan the file system to identify a specific file subset defined by a user. For example, a user may want to remove work files from a home computer before reformatting the home computer. Execution system 102 can be configured to scan the file system of the home computer and mark all work files. The enforcement system 102 may be configured to tag data for data subject rights, such as right to be forgotten, data erasure, subject access request, data correction request, data suspension, data portability, and consent. Many other applications are possible.

[0049] In order to discover, classify and mark parts of the source data (such as data fields), the execution system 102 is configured to access a reference database 116 of one or more files 118. The files 118 provide the execution system 102 with a context for performing the discovery, classification and marking of the source data. In some embodiments, the reference database 116 may store a lookup table that stores the relationship between the values ​​found in the entries of the data fields. For example, the data storage device may include a lookup table that matches the code to a glossary, and the execution system 102 may refer to the glossary during the discovery, classification and marking of the source data. The files 118 of the reference database 116 may include weight values ​​for classification. For example, the weight values ​​may indicate to the execution system 102 the probability that two terms (e.g., business terms) of the tagged source data 112 are related to each other. These values ​​may be generated during the initial iteration of the discovery, classification and marking of the source data 118, and updated by the user or automatically during subsequent iterations.

[0050] The files 118 may be defined before the execution system 102 discovers, classifies, and labels source data, during the discovery, classification, and labeling, or after the discovery, classification, and labeling in an iterative process. A development environment (not shown) of the networked system 100 may provide a means by which a user may write to the reference database 116 or update the files 118 of the data storage device. For example, the development environment may include a user interface that provides feedback to a user of the execution system 102. For example, the user interface of the development environment may display a report that shows how the execution system 102 performed, such as what data fields were labeled and what the probability of each classification by the execution system 102 was. Examples of feedback provided to the user and user interfaces are described in detail later.

[0051] Typically, the execution system 102 includes one or more processors configured to execute the logic of the profile data module 104, the classification module 105, the test module 106, the result verification module 108, and the load data module 110. The operations of each of the profile data module 104, the classification module 105, the test module 106, the result verification module 108, and the load data module 110 can be performed by batch processing or in real time. In addition, the execution system 102 can perform the operations of each of the modules 104, 105, 106, 108, 110 substantially simultaneously or during different time periods. For example, in some embodiments, the profile data module 104 generates profile data representing a profile of the source data from the data source 112 at a first time. Later, once all data from the data source 112 for a given time period has been profiled by the profile data module 104, the classification module 105, the testing module 106, the result validation module 108, and the load data module 110 can analyze the profile data to discover, classify, and mark data fields of the source data, and load the data into the data storage device 116 for one or more downstream applications.

[0052] FIG. 2A to FIG. 2E is a block diagram illustrating how the execution system 202 parses, categorizes, and tags data fields to identify the semantic meaning of the data fields. Figure 2A, a process 200a for profiling a data source by a profile data module 104 is shown. The profile data module 104 is configured to receive (254) source data (e.g., a table, a file, etc.) and generate a data profile of the source data. The profile data module 104 discovers (256) fields of the source data (e.g., one or more data sets). The profile data module 104 may discover the fields by identifying table rows in the source data, finding field names, references to the fields, or using any similar process. The profile data module 104 determines (258) statistical properties of the data fields and generates (260) profile data including those statistical properties. The profile data identifies patterns in the source data. More specifically, the profile data includes statistics about the values ​​of the data fields of the tables of the source data. For example, the profile data may include information specifying whether the data values ​​of the data fields include numeric data, character strings, etc. For example, the statistics about the data values ​​may include the maximum value, minimum value, standard deviation, average value, etc. of the values ​​included in each of the data fields (if the data is numeric). In some implementations, statistics about the data may include how many digits or characters are in each entry of the data value. For example, the data profile may indicate that each data value of a data field includes seven (or ten) digits, which may provide a contextual clue indicating that the data field includes a telephone number. For example, the data profile may indicate that each entry of a data field includes a value from a small set of values, which may be used to trigger a comparison of the lookup table by the test module 106.

[0053] The data acquisition of the profile data module 104 includes analyzing the field names of the fields, the locations of the fields in the table (or files in the file system), and the schema of the analyzed data. In other words, data acquisition occurs at the field level, the data set level, and the schema level.

[0054] For the field level, the profile data module 104 analyzes the value of the field and the entries of the field to generate profile data. The profile data module 104 can determine whether the value of the field or its entry is empty, blank, valid for a specific data type, etc. The profile data may include statistics on the empty percentage, blank percentage, and percentage of each field value. The profile data module 104 may also generate data indicating changes in these percentages relative to baseline percentages (which may be specified by the user through the development environment or automatically generated). In another example, the profile data may include an indication of whether the data of the entry is valid for an implicit data type. For example, if a data field is known to be a string field, but date data is found, it can be inferred that the data is invalid for the entry. In another example, the profile data may include an indication that the data of the entry is valid for a specified format (e.g., two decimal places are required, but no explicit type is specified). In some embodiments, after generating the profile data, the test module 106 performs some of the functions in this function.

[0055] For the data set level, the profile data module 104 can provide statistics that associate a portion of the source data with the data set of the entire source data. For example, the profile data module 104 includes an indication of natural key uniqueness and key cardinality. The profile data module 104 indicates whether there are duplications on the key values ​​of the source data. The profile data module 104 indicates whether there are duplications on approximate natural key matches. The profile data module 104 indicates the count of records with one or more of these features. In some embodiments, for example, after the field is determined to be a key value, this information is determined by the result verification module 108.

[0056] For the pattern level, the profile data module 104 is configured to determine statistical information about data fields of the source data that change over time. For example, the profile data module 104 detects whether there are added or removed columns in a new version of the source data. The profile data module 104 detects delimited fields. In some embodiments, the profile data module 104 performs a lineage count to determine whether any records have been removed from a previous version of the source data. Other pattern evolutions may be detected. In some embodiments, this function is performed by the result validation module 108 after a data field is found in the profile data.

[0057] In some embodiments, the profile data can be improved if contextual data is available in the reference database 116. For example, if a user specifies a format for one or more data fields of the source data, the profile data can generate additional statistical information about these data fields. This can be part of an iterative process. For example, once a data field is found in a first iteration (but if classification fails for that iteration), the user can view the data content of the data field and provide additional information to the execution system 102 for analysis by the profile data module 104 (and other data modules). For example, if a user specifies that all data fields should include only numeric data, the profile data module 104 can quickly determine which data is invalid and provide statistical measures of that information in the data profile.

[0058] The profile data module 104 generates profile data from the source data by scanning the entire data set of the source data before generating the profile data. The profile data module 104 does not need to copy the entire data set from the source data, but this can be performed to generate the profile data. Because the data set may be large (e.g., many gigabytes in size or even many terabytes in size), it may be impractical to copy the data in bulk to a local system for analysis. Typically, the profile data module 104 scans the source data during a period of low activity at the data source.

[0059] Go to Figure 2B, a process 200b for classifying data fields using profile data by the classification module 105 is shown. The classification module 105 is configured to receive (262) profile data and receive (264) source data including fields. For each field of the source data, the classification module 105 is configured to look up (266) (e.g., from the reference database 116) a tag index of existing tags for the discovered field of the source data. These tags can be from a previous iteration of the tagging process, or the tag index (e.g., an initial tag index) can be manually generated, imported, or otherwise obtained. However, the tag index need not exist before performing the tagging process.

[0060] For a field, the classification module determines (268) whether the field has been associated with a tag in the tag index. If the field has not been tagged, or if there is no tag index, the classification module 105 determines that no tag is found for the field. If necessary, the classification module 105 generates a new tag index to populate the semantic tag. The classification module 105 performs (270) classification of the field data type. The classification can be based on the profile data of the field, the field name, and the value of the field. For example, the classification module 105 can determine that the field is a "date" field. In some embodiments, the classification module 105 can determine that the field is a numeric field, a string field, or other such data type. When the classification module 105 determines the data type of the field, the semantic meaning of the field (and therefore the semantic tag) is determined by the testing module 106, as described subsequently. For example, the classification module 105 can determine that the field is a date field, and the testing module 106 determines that the date of the date field is the customer's "date of birth". In another example, the testing module 106 determines that the numeric field is a "user ID" field. Many other such examples are possible. The classification module 105 generates 272 classification data as classification output 276 to be sent to the testing module 106 to find semantic meaning. The classification data is labeled with a data type determined by the classification module 105.

[0061] If a label is found, the classification module generates (274) label data that can be passed to the testing module 106 and the result verification module 108. The label data informs the testing module 106 and the result verification module 108 that the field has been labeled. This can be used to weight or suggest a label for the classifier applied to the field. However, the field can be reclassified by the classification module 105 and retested by the testing module 106 to confirm that the label is accurate and potentially update the label attributes of the label in the data dictionary database 114. For example, if the testing module 106 finds that the existing label matches poorly, a new label can be suggested. If the user selects an existing label (e.g., as presented by the result verification module 108), the label data can be used as a sign to the execution system 102 that the label attributes of the label in the data dictionary database 114 do not represent the data value marked by the label, and these attributes can be updated.

[0062] In some implementations, the classification module 105 may be updated through multiple iterations using machine learning methods. For example, if a field is found to have been flagged, the classifier may determine that further classification may be bypassed. In another example, the score applied to the field may be updated based on additional data received from the data source 112 or from user input. The classification module 105 may determine that the testing module 108 should perform a different test(s) than in a previous iteration. For example, if the fingerprint analysis was inconclusive in a previous iteration, the classifier may determine that the test should be bypassed (or replaced with another test) in a subsequent iteration.

[0063] Go to Figure 2C , a process 200c for classifying a field as being associated with a semantic tag by the testing module 106 is shown. The testing module 106 is configured to classify the source data of the data set using statistics in the profile data and using additional contextual information (such as a lookup table 442) provided in the reference database 116. The classification output 276 of the classification module 105 is used to provide context for the data type for each field and provide an existing label (if any) for the field. The testing module 106 is configured to receive candidate labels 440 from the data dictionary database 114. The candidate labels are a library of existing labels, each label is associated with one or more attributes that identify the semantic meaning of the data field (e.g., for downstream applications or users). As previously explained, the attributes associated with each label in the data dictionary database 114 provide the semantic meaning of the label. The testing module 106 determines which of these candidate labels is most closely associated with the attributes of the data field determined by the data profile, the field name, and the data value of the field.

[0064] The test module 106 includes multiple tests (or classifiers), each test being performed by a different test module to associate one or more labels with each tested field. For example, after the profile data module 104 discovers a field of a data set, the test module determines the extent to which the attributes of the field closely correspond to the attributes of each candidate label in the candidate labels. Each test uses different data and methods to propose one or more labels. Because different tests use different classification methods, the proposed labels for each test are not necessarily the same. The proposed labels are confirmed in the result confirmation module 108, as described subsequently. Using different tests to identify the semantic meaning of a field makes the determination of the semantic meaning more robust than using any single test because it does not rely on a single factor (e.g., the field name, or including a specific value or a set of values ​​in the field, etc.) as an indication of the semantic meaning of the field.

[0065] Testing of the data in each data field may include determining one or more of the following: the overall level of the data in the data set (how often a value appears in the data field), the type of data found (e.g., dates or numbers are saved as strings), the domain of the data field, the discovery of key fields, the determination of whether the field is a single word or a description field, etc. For example, the classification of the data may include date and time analysis. Accordingly, the testing module 106 receives the profile data from the profile data module 104 and performs a series of statistically based functions to identify, classify, and test the field details against a set of known label types. The rules for the testing vary depending on the data type, which may be identified by the classification module 105 in the classification output 276, or in some cases included in the profile data generated by the profile data module 104.

[0066] The test module 106 is configured to perform a plurality of different classification tests on the entries in the field names and data fields to determine how to label the data fields. The test module 106 receives the classification output 276, the candidate labels 440, and any reference data 442 from the reference database 116 and provides these data to the test. The test includes pattern analysis 410, business term analysis 414, fingerprint analysis 418, and keyword search 422. As previously explained, although the table with field names and field data is described as an illustrative example, the tests 410, 414, 418, and 422 of the test module 106 can be performed on other data types. Examples of classification tests performed on data fields and data entries of source data can include fuzzy matching plans, column data plans, business term matching plans, keyword matching, fingerprint recognition plans (e.g., contextual data lookup), pattern matching, and confirmation.

[0067] The fuzzy matching logic of the test module 106 includes logic for fuzzy matching field names from a term dictionary. Typically, fuzzy matching is configured to look for a match between a field name and a term in a dictionary when an exact match cannot be found. The system looks for less accurate dictionary terms. For example, the test module 106 sets the threshold for fuzzy matching to a percentage value less than 100, and the dictionary database (e.g., database 116) will then return any matches in its memory that correspond to (e.g., are greater than) that percentage. In some embodiments, a probability score is assigned to each match. The probability score can be presented to a user.

[0068] The business term analysis 414 of the test module 106 includes logic for matching data field names with business terms known in the glossary. Often, business terms can be placed in different contexts or business term groups. The test module 106 performs a check to find the occurrence of a particular word or term in another word or term. This can include a reference to a particular specification. For example, the test module 106 receives a specification indicating different abbreviations of street names, such as "st", "ln", "ave", "pl", "ct", etc. The test module 106 performs a check to determine whether any of those abbreviations are included in the data field. If one or more of the data elements in the specification are included in the data field, the test module 106 determines that the data field includes a street name. This piece of information can be used alone, such as determining that the data field includes a street name and should therefore be labeled as such. This piece of information can also indicate that the data field includes other information such as an address. The test module 106 uses the determination that a street name is present in the column in addition to other data to determine whether the data field includes an address, only a street name, or some other data. In another example, the phrase "date of birth" can be identified and associated with such a label. Other matching strategies may include matching using fuzzy matching, synonyms, etc. Typically, the test module 106 associates a probability score with each match. A user may generate a specification that configures the logic, such as through a development environment.

[0069] The pattern matching analysis 410 of the test module 106 uses the data content of the field (in addition to or in place of the field name). The pattern matching type used for pattern matching can be determined by the test module 106 based on the results of the classification data 276. For example, the classification data 276 can identify the data type of the field, such as whether the data is numeric. In this example, the profile data also indicates that each entry in the data field is 13 to 18 characters long. This can indicate to the test module 106 that the data field may be a credit card number data field. To confirm this, the test module 106 can perform one or more pattern tests on the data of the suspicious data field. For example, the first 4 to 6 digits of each entry can be checked against the issuer code table. The last digit can include a check digit defined by the Luhn test. If a threshold percentage of the entries of the data field meet each of these patterns, the test module 106 can conclude that the field holds a credit card number and associate the field name with an appropriate label and probability. For pattern matching logic, both the data itself for a given field and the pattern of the data in that field (eg, identified in profile data) can be used to discern which pattern tests to run and what labels to apply to a given data field.

[0070] The testing module 106 determines (408) whether to perform a pattern analysis test 410 on the source data 112. The determination can be a result of the classification data 276. The pattern matching analysis 410 uses the profile data to determine whether the source data 112 conforms to a predetermined pattern indicating a candidate field label. For example, if the data for a field has a particular length and composition, the pattern matching analysis 410 identifies the corresponding candidate label. The pattern score applied to the label can be a function of how closely the identified pattern matches the predetermined pattern, how unique the pattern is, or any number of factors. The weight can be adjusted based on the uniqueness of the pattern. For example, unusual patterns may correspond to higher weight values. If the values ​​of the field are found to closely match the pattern (on all or some of the values), the score may be higher than if only a few values ​​match the pattern.

[0071] The test module 106 may include a keyword search test 422. Keyword testing (similar to business term matching testing) includes data-based testing that involves searching for specific keywords within a data field. For example, to find an address, the test module 106 searches for common address words, such as "street," "road," "boulevard," etc. This test may be extended by the user, who may add new keyword files to the specification of the reference database 116. Keyword testing may be used to find words within a phrase or portion of a word, such as an address and a company name, where there is a limited set of common words that may uniquely identify a data field.

[0072] The test module 106 may determine (420) whether to perform keyword analysis 422 on the field names. In this example, if some of the fields are still not associated with a tag value, the test module 106 will perform a keyword matching test. Checking whether the field name (and possibly the field value) includes one or more keywords from the table can help the test module 106 associate a specific tag with the field. The test module 106 performs a keyword search 422 and generates a keyword search score.

[0073] The test module 106 may include a fingerprint recognition test 418 for analyzing the data values ​​of the source data. The logic of fingerprint recognition generally includes a data search for the entire data field. The fingerprint recognition logic includes a test based on the data value. The logic of fingerprinting the data field includes comparing a known list (e.g., from the reference database 116) with the data of the data field to determine whether the data of the data field is related to the data of the list. For example, the data from the data field can be compared with a list of names, a list of state names, a list of city names, etc. The fingerprint (known data list) is usually representative, not comprehensive. In other words, the fingerprint does not need to include every example of the value that is part of the fingerprint set. Generally, the fingerprint can include a selected example value that roughly represents the most common value that should appear in the data entry of the data field. For example, for a name table, the fingerprint does not need all possible names in the United States, but it is sufficient to select a set of the most popular names. For example, the top 100 names usually give enough data to show the correlation between the data field and the data fingerprint. The data fingerprint can be generated from the master data. For example, the system may include 50 to 100 most populous cities in the United States to check whether the data field corresponds to the city name. Users can add new domains to the specification to increase fingerprinting testing capabilities for specific systems.

[0074] The test module 106 determines (416) whether to check the fingerprint table for the field. Fingerprinting can be effective when the list of possible values ​​for the field is long, but some of the values ​​are expected to be more common than others. For example, city names, street names, and even first and last names are good candidates for fingerprinting analysis. The test module 106 performs fingerprint analysis 418 and generates a fingerprint score.

[0075] The test module 106 determines (412) whether to run business term analysis. Business term analysis may be useful when there are many unique terms in the source data 112 that may correspond to business terms used to label fields. If business analysis is performed, the test module 106 performs 414 business term analysis by comparing the field names to the business terms to find matches and generate labels and their associated probabilities. Similar to the pattern matching analysis 410, the results of the business term analysis 414 are associated with weights and scores.

[0076] The test module 106 can be configured to compare results across tests to improve the results using validation logic. For example, validation logic can be used to verify business term matching using another classification test. For example, a test that specifies that a data field includes a maiden name value should also pass a classification test for the surname field.

[0077] The testing module 106 is configured to execute machine learning logic in which the classification of a previous data set (e.g., from a particular source) or a previous iteration of the same data set is remembered and has an impact on which tests are selected for subsequent iterations and how the probability values ​​for those subsequent iterations are determined. The machine learning logic is trained on the data set and can apply weights developed using the training data to classify new data of the data set.

[0078] Each of the tests 410, 414, 418, and 422 can output one or more proposed tags for the data field being analyzed. The tests 410, 414, 418, and 422 do not need to be consistent. Each proposed tag can be associated with a score (not shown) and a weight value (not shown). The result verification module 108 can use the score and weight of each tag to propose a specific tag in the proposed tags as the semantic meaning of the identification field, and classify the tags (or classify the tags) into categories. The categories (described later) indicate the degree of consistency between the tests, and therefore the confidence of the proposed tags is suggested as the semantic meaning of the data of the identification field.

[0079] To perform the tests, the test module 106 receives data 276, 442, and 440 and determines whether to perform each test. For example, if the type of data received is not formatted for a particular test, the test cannot be performed. Any combination of tests can be performed. The test module determines (408) whether to perform pattern analysis 410, determines (412) whether to perform business term analysis 414, determines (416) whether to perform fingerprint analysis 418, and determines (420) whether to perform keyword search 422. Each test 410, 414, 418, and 422 generates one or more proposed tags and outputs the proposed tags. The test results including all proposed tags are concatenated (424) and sent to the result verification module 108 as the test results 306.

[0080] In some embodiments, if a result has not been found from an earlier test, a subsequent test may be performed. In some embodiments, the test module 106 prepares the reference data by retrieving data from the reference database 116 based on the source data 112 being analyzed. For example, the test module 106 may retrieve data from the reference database 116 based on the location of the source data 112, the table name in the source data, user input, etc. The test module 106 may determine which lookup tables to access and check for each field of the source data. The selection of the lookup may be performed based on the profile data received from the profile data module 104.

[0081] In some implementations, the test module 106 determines whether to run a pattern analysis. Pattern analysis can be used to analyze source data 112 over time and as a whole. For example, if a field is missing, added, deleted, etc., the information can be used to flag other data fields.

[0082] In some embodiments, once the tests have each been completed, the testing module 106 combines the associated scores to generate a final score, which is shown along with the associated proposed label to which the score applies. In some embodiments, the upstream score affects the downstream score, so that the scores are indistinguishable from each other except that they represent a score that is updated as each analysis occurs. In some embodiments, each proposed label and its score are reported separately to the result verification module 108, which then determines how to categorize the test results 306.

[0083] In some embodiments, the order of the tests may be such that the more processing intensive tests are placed last. If other tests fail, the more time intensive tests may be a last resort. Ordering the tests in this manner may reduce the processing time used to mark the source data 112 on the execution system 102.

[0084] Go to Figure 2D , the result validation module 108 is configured to execute validation logic. Once the test module 106 has run the classification test on the source data and determined the probability values ​​for the labels of the data fields of the source data, the result validation module 108 performs a check to determine whether the classification result indicates a high confidence or whether further testing should be performed.

[0085] The result confirmation module 108 receives the test results 306 of different tests performed by the test module 106 on the source data, and determines whether the results confirm or conflict with each other. The test results 306 of the test module 106 are classified into several classification categories by the result confirmation module 108. These categories include matching categories 290, recommendation categories 291, research categories 292 and ignored categories 293. Each of these categories indicates the similarity between the label proposals in the test results 306. For example, in all the same label proposals, the test results have a high level of similarity. If each label proposal is different, the test results have a low level of similarity. Similarity can be more than just a voting mechanism for each test in the test. Each label proposal is associated with a weighted score value. If a label proposal for a test does not match other label proposals, but is associated with a relatively high score value and a larger weight compared to other proposed labels for other tests, then although three of the four tests are consistent, the similarity can be identified as low.

[0086] Depending on the identified category, the result validation module 108 may automatically validate the semantic meaning of the tag identification field, or prompt the user to manually validate the tag. Validation may be accomplished through the client device on a user interface, as described subsequently.

[0087] The match category 290 indicates the highest level of confidence for the tag (e.g., 100 matches). In some embodiments, the matching tags are automatically included in the metadata associated with the source data. However, the result verification module 108 can still be configured to present the results to the user for manual verification. Typically, the match classification indicates that all performed tests proposed the same tag. In some embodiments, if the tags do not all match, but when any objection tags are below a score threshold indicating overall consistency between the tests, then a match category can be selected.

[0088] The recommended category 291 generally indicates that at least one tag has a high-quality association with the data field. However, the recommended tags are generally below the threshold level set for the highest confidence, and further verification is preferred. In some embodiments, the recommended category indicates several high-quality tags associated with the data field. In some embodiments, the result validation module 108 sorts and lists the recommended tags, each with a probability score, which can help the user select the best tag(s) for the data field.

[0089] Study category 292 indicates values ​​or data fields that have some interesting statistical properties that do not pass any particular test. For example, a data field may look as if it should have meaning, but no test has a proposed label, or the tests have or the proposed labels have scores below a given threshold. For example, a data field may include profile attributes that indicate that the data field is an important domain or field in the source data, but a label is not recommended (or recommended above a threshold probability). Typically, such a result indicates that additional rules should be added to the test (e.g., the test should be expanded or changed in some way) and executed again.

[0090] Ignore category 293 indicates that the data field is designated as uninteresting, or that the field has not triggered a test and has no profile attributes showing that the data field is important. Typically, the Ignore category is shown when the received data has errors or no discernible pattern. Although the Ignore category may indicate that a new tag should be defined, the Ignore category usually indicates that the field includes miscellaneous data with no particular structure.

[0091] Typically, to categorize (e.g., classify) the test results 306 into categories, the result validation module 108 performs process 285. The result validation module 108 compares (286) the test results from the fields. For each field, the result validation module assigns (288) the test results 306 to a category based on the level of similarity.

[0092] In addition to generating a classification file specifying a classification category, the result validation module 108 is also configured to generate other files. Other files include a profile annotation file. A profile annotation file is a file that describes the results of the classification analysis process. Other files include a discovery result summary that describes the combined output of the business term matching process and the fingerprint recognition process.

[0093] The result verification module 108 is configured to determine whether the data value is an outlier based on the total number and range of each data field. An outlier includes a data value that does not conform to the identified relationship or format of the data field. The result verification module 108 determines the outlier based on the clustering and prediction of the classified relationship of the test module 106.

[0094] The process described above can be iterated to improve the accuracy of the classification and enable the user to update the classification test to obtain better results. As previously explained, in some embodiments, machine learning logic can be used to train (multiple) classifiers during each iteration to facilitate the process. Generally, once the profile data module 104 generates the profile data, the process of the test module 106 and the result verification module 108 will be iterated; unless the source data itself is updated, there is no need to repeatedly generate new profile data.

[0095] Once the profile data module 104, the classification module 105, the test module 106, and the result validation module 108 have generated tags for each of the data fields, the load data module 110 may load metadata including the tag index into the reference database 116. Figure 2E , the load data module 110 performs a process 299 for updating the label index and loading data into the reference database 116. The load data module 110 receives (295) a field name and receives (296) a manually or automatically validated proposed label for the field. The load data module 110 concatenates (297) the field name and the proposed label. The load data module updates the label index by associating the label with the location of the field in the data set. The load data module concatenates (298) the label of the data set being analyzed into the label index, and the execution system 202 and downstream applications can reference the label indexed data set.

[0096] Typically, the reference database 116 can be accessed by one or more downstream computing systems for various applications. For example, the generated dataset labels can be used for data quality enforcement, personal data anonymization, data masking, (PII) reporting, test data management, dataset annotation, etc.

[0097] The load data module 110 is configured to package the metadata and source data into a software package that can be used by one or more other computing systems. For example, once the profile data is generated, the operations of the classification module 105, the test module 106, and the result verification module 108 can each be configured to run multiple instances of their processes in parallel on different partitions of the source data. For example, if the source data includes multiple tables, the source data can be partitioned by table. The test module 106 and the result verification module 108 can run instances of their logical processes on multiple tables simultaneously to increase the throughput of the process of the execution system 102. Once the source data is marked, the load data module 110 can be configured to merge the partitions back together and store the marked source data in the reference database 116.

[0098] Figures 3A to 3G It is a block diagram of the process used to parse, classify, and tag data fields to identify their semantic meaning. Figure 3AIn the example of FIG. 1 , block diagram 300a shows that source data 302 is partitioned so that each table 302a, 302b is processed in parallel by the execution system 102. The source data 302 includes fields and values ​​of entries in the fields. As previously discussed, although tables and fields are shown as examples of source data, source data may include other types of data. The profile data module 104 receives the source data 302 and stores index data 302c in the reference database 116. The index data 302c may be an index to a table of the source data 302. The index data 302c is then modified to include a label for each of the fields in the source data 302 and thus becomes a label index. Figure 3A As shown, for each field, index data 302 c associates the field name with the field position in source data 302 .

[0099] Source data 302 includes two tables, table 302a and table 302b. The first table 302a includes dates, city names, and alphanumeric codes. The second table 302b includes names, phone numbers, and dates. While it may be obvious to a user that a city name is indeed a city name or that a phone number is indeed a phone number, it may be difficult for a user to determine the purpose of an alphanumeric code or to determine what a date represents. The execution system 102 is configured to address this problem by classifying each of the fields of the first table 302a and the second table 302b as previously described.

[0100] Typically, the profile data module 104 first analyzes all tables of the source data 302 to determine statistical characteristics about the tables. Once the profile data 304a, 304b is generated for each of the tables 302a, 302b, respectively, the execution system 102 divides the first table 302a into first partitions 304a and the second table 302b into second partitions 304b.

[0101] As previously discussed, the profile data 304 includes statistical information about the field names and the entries for each of the fields. For example, for table 302a, the profile data 304a identifies four fields: Date 1, Date 2, Place, and ID. The profile data module 104 has determined that the data values ​​(also referred to as entries) for Date 1 include the same format: ## / ## / ####. In fact, the profile data module 104 can determine that almost all values ​​have this format and discard outliers. In addition to determining the format of the value of Date 1, the profile data module 104 also determines that the values ​​of the first two digits of Date 1 fall within the range of 1 to 12. This data can indicate to the test module 106 that the value of the entry for Date 1 is a date.

[0102] Similarly, the profile data module 104 performs format analysis and statistical analysis on each of the fields Date 2, Place, and ID. The entry for Date 2 also has a ## / ## / #### format. The profile data module 104 also determines that the value of the last four digits of the Date 2 portion is always greater than the corresponding value of the last four digits of the entry for Date 1. This indicates that if both Date 1 and Date 2 include dates, then for each entry in table 302a, the date value of Date 2 is always later in time than the corresponding date value of Date 1. The profile data module 104 determines that Place is a string in which a two-character code always follows the first word. The profile data module 104 determines that ID always includes an alphanumeric string of length 7, which can provide the test module 106 with a clue that the value of ID is a code, a user ID, etc.

[0103] The profile data module 104 also analyzes table 302b to generate profile data 302b. Typically, before partitioning occurs, tables 302a and 302b are analyzed together. The profile data module 104 determines that the last name (Last) of table 302b includes a string, and the value of "Smith" is the most common entry (x2). The profile data module 104 determines that the first name (First) of table 302b includes a string, and the value of "John" is the most common entry (x3). The profile data module 104 determines that M / F includes a single character that is one of the following two values: "M" or "F". The profile data module 104 determines that the format of the number includes (###)###-####, and the number (or the first three numbers) in the brackets is from a finite set. The profile data module 104 determines that the comment (Comment) includes a string, and the first two digits are between 1 and 12.

[0104] Once generated by the profile data module 104, the profile data 304 is sent to the test module 106. The test module 106 is configured to operate on the tables in the partitions 306a and 306b in parallel. The profile data module 104 sends the profile data 304 to the classification module 105.

[0105] Go to Figure 3B, block diagram 300b shows how the classification module 105 processes the profile data 304 and the source data 302 to classify the data type of each field. The classification module 105 receives (401) the profile data 304 from the profile data module 104 and the label index 220 from the reference database 116. The classification module 105 selects (405) a field from the source data. In this example, the field date1 is selected for classification. The classification is performed on the source data 302 field by field. The profile data 314 of the selected field is classified based on whether there is an existing label for the field. Here, the status of the field date1 is shown as <NOTYET LABELED>. The classification module 105 classifies the field date1 to determine that the field is a date field and indicates any label for the field (in this case, no label is found). This information is the classification output 286 sent to the testing module 106.

[0106] Go to Figure 3C , Figure 300c shows how the test module 106 proposes (multiple) labels for the field date1. The figure 300c shows a test of a single field of a data set at a first time T1. The test is performed field by field, but typically all fields are tested for the data set. The test module 106 generates the classification output 306 by using clues provided by the classification output 286, the profile data 304, and the data stored in the reference database 116 including the lookup table 442. For example, as previously described, the classification module 105 determines that the first field is a "date" field based on the format of date1 of table 302a. The additional information that the first part is a number between 1 and 12 (and the system uses the US date symbol) further indicates that date1 is a date field. The test module 106 assigns a probability that date1 is a date field 95. However, without further context, the test module 106 cannot determine what kind of date the date field represents. The date field of date1 can include a date of registration for service, a date of birth, a renewal date, etc. The test module can determine that the Date 1 field is a "Date of Birth" field because the test uses the contextual information of the reference database 116 and the tag table 440. The tag table 440 includes a list of defined tags for the data set being analyzed. As previously described, the tag table 440 can be modified as needed to add additional tags. Figure 3C As shown, each tag of tag table 440 serves as one or more attributes that describe what characteristics the data the tag is supposed to describe. For example, a valid phone number has the first three digits of a set of possible numbers from a table of area codes in a reference database. For example, the date in the Date of Expiration field will always be later than the date in the Date of Birth field (if present). These attributes, which indicate complex relationships between field values ​​and other characteristics of the data, indicate that the data has a semantic meaning identified by the associated tag.

[0107] To test the field date 1, the test module 106 performs tests 410, 414, 418, and 422 in any combination as previously described. Each test proposes one or more tags. For example, the pattern matching analysis 410 proposes a birth date tag proposal 318 with a score of 0.80 and a weight of 1.5. For example, the business term analysis 414 proposes an expiration date tag proposal 320 with a score of 0.75 and a weight of 1.1. For example, the fingerprint analysis 418 proposes a birth date tag proposal 322 with a score of 0.60 and a weight of 1.75. For example, the keyword search analysis 422 proposes a birth date tag proposal 324 with a score of 0 and a weight of 0.5. The weights can be predetermined based on feedback applied by the user to the execution system 102, or can be automatically adjusted by a machine learning process. The score indicates the strength of the proposal by the test. For example, a score of 0 can be given for the keyword search test 422 because the available data is not properly formatted for such analysis. Therefore, the proposal of this test is discarded in this case. For example, a score of 1 indicates a high confidence that the proposed tag is the correct test. For example, if all attributes of a particular tag in tag table 440 are identified in source data 302 and profile data 304 for field Date 1, then the test may determine that the particular tag accurately identifies the semantic meaning of the field.

[0108] The testing module 106 connects 424 the tag proposals 318, 320, 322, and 324 into the test results 326. The test results 326 include the proposed tags and their associated scores and weights. The result validation module 108 uses the scores and weights to determine how similar the tag proposals are to each other in order to classify the proposals into categories.

[0109] Go to Figure 3D , Figure 300d shows a second time T2, after which each of the fields in the source data has been tested. Test results 326 for Date 1 are shown, and these test results are connected with other test results for other fields. For example, the test module 106 determines that Date 2 is an expiration date with a score of 95, similar to determining that Date 1 is a date of birth. The test module 106 has the additional context that for each entry, the date is always after the date value of Date 1. Additional information can be accessed to further define the date. In this example, the test proposes "Expiration Date" as a potential field name. The test module 106 assigns a score of 53 to Date 2, which is labeled "Date of Birth", based on fingerprint analysis.

[0110] The test module 106 performs a lookup based on the values ​​in the place entries. As previously described, the test module 106 can use a lookup table, keyword matching, fuzzy matching logic, etc. to determine that the value of the string is a city name and that the two-letter string in each entry represents a state abbreviation. In some implementations, the test module 106 is configured to parse the two parts of each value of the place entry to determine that the entry value represents both a city name and a state name.

[0111] The test module 106 references a specific lookup table provided by the system administrator for the source data 302. Because the system obtains data from the Department of Motor Vehicles, one of the tables included by the user is a driver's license ID table. The test module 106 can determine that the format of the ID value matches the format of the driver's license ID in the lookup table. In this case, the fingerprint recognition algorithm is applied. Although some matches are found between the ID and the lookup table values, there are only a few matches in this example. The test module 106 assigns a corresponding score 62 to the tag. Subsequent iterations in this example can provide additional value to the fingerprint recognition algorithm to obtain a stronger match. However, in other examples, the fingerprint recognition algorithm can provide a high confidence match on the first try.

[0112] In parallel with analyzing table 302a in partition 306a, test module 106 is configured to analyze profile data and content of table 302b in partition 306b. In this example, test module 106 applies one or more classifiers to the last name of table 302b to determine that the field represents a last name. For example, the classification algorithm previously described can be applied to the data of table 302b. Because the most common value ("Smith") only appears twice, test module 106 assigns a "last name" label with a lower score of 35 to the field. However, for the first name, based on a similar analysis of the lookup table, because the first name "John" appears more frequently (and is identified as a first name value), the first name score is 85.

[0113] The test module 106 performs a check against the single character code of the M / F value of table 302b. The gender assignment table is found and the M / F value matches those specified in the table. M / F is assigned a "gender" label with a score of 98.

[0114] For number, the test module 106 determines that the format of each entry (or most of the entries) corresponds to a telephone number. As a second check, the first three digits are compared to the area code table. Because the first three digits of each value of number match the values ​​in the area code table (not shown), the test module 106 assigns the number a "telephone number" label with a score of 100. For annotation, the test module 106 determines that no test found a possible label. The test results 306 can be saved in partitions 306a, 306b based on the table being analyzed.

[0115] Go to Figure 3E , Figure 300e shows how the result validation module 108 processes the test result 326 for field date1. Figure 3E The time T1 for processing a particular field Date 1 is shown, as each field is classified on a field-by-field basis. The result validation module 108 receives the test result 326 for the field Date 1.

[0116] The result confirmation module 308 compares the label proposals of the test result 326 with each other at module 286. The data count of the birth label proposal is 3, while the date count of the expiration label proposal is 1. In some embodiments, a simple voting mechanism can be used to determine which label(s) should be shown to the user as the final label proposed to identify the semantic meaning of a field such as date1. In some embodiments, scores and weights can be used to determine which label to propose and what category is associated with the proposed label. For example, the result confirmation module 108 can sort the proposed labels by their weighted scores and select the highest ranked proposed label. In some embodiments, all proposed labels are shown to the user for verification (if more than one label is proposed).

[0117] The result validation module 108 assigns the proposed tags to categories based on similarities between the tags proposed by module 288. The categories may be selected based on weight scores, counts of each proposed tag, etc. The tag proposal(s) for the field may be assigned to one of categories 290, 291, 292, or 293 using a function of counts, scores, and weights. Here, because the proposed tags were not consistent in the test, and because the objection test was associated with a sufficiently high weighted score value, the recommended category 291 was selected (rather than the matching category 290). The result validation module will likely prompt the user to verify the categorized proposal 289.

[0118] The report generation module 380 can generate a report 381 showing how each field in the field is categorized and what label is proposed. For example, a histogram is shown including counts for each category and which fields and proposed labels are in each category. Typically, there are fewer fields that need to be studied, and more fields that have recommended or matching proposals.

[0119] Go to Figure 3F, block diagram 300e shows time T2 of the result validation module after all fields have associated categorized label proposals. The test results and assigned categories 221 are sent to a test result validation module 399. The validation module 399 may cause a user interface to be presented to a user, such as on a validation data source 111 (e.g., a client device). Validation data 309 is received via source 111, and validated labels are assigned to fields as assigned labels 308. Each field is associated with an assigned label that identifies the semantic meaning of the data for that field.

[0120] Go to Figure 3G , Figure 330f shows how the load data module 110 stores the assigned tags 308 of the fields of the source data 302 and how the tag index is updated / generated. The load data module 110 receives index data 302c from the reference database 116. The index 302c is used to generate the location of each field assigned to the tag for each tag in the data dictionary database 114. The tag index 310 shows this association. When downstream applications manipulate the source data 302, the tag index 310 can be referenced to reduce errors in data processing and quickly find the semantic meaning of a given field without further analyzing the data. This makes it possible for downstream applications to filter personal identification data, for example, without actually accessing the data itself.

[0121] Figure 4 4 is a flowchart illustrating operations performed by the logic engine of the execution system for discovering, classifying and marking data fields by analyzing data profiles generated by the data of the data fields. As previously described, once the results validation module 108 performs the analysis, the analysis can be repeated if the results are inconclusive.

[0122] Figure 5 A block diagram of an execution system 102 for discovering, classifying, and tagging data fields by analyzing a data profile generated by the data of the data field is shown. The execution system 102 is shown in an example environment 500 in which a particular field 208 is tagged from source data 202 including fields 202a, 202b, and 202c. In this example, the extensibility of the system is shown because a tag has been associated with the field 208.

[0123] exist Figure 5 In the example of , source data 202 includes a field 208 named "Field_XYZ", which is a technical field name. Execution system 102 is configured to receive source data 202 and tag field 208 with a business term indicating what kind of value is stored in the field. Field 208 includes data entries, such as data entry 210 including the string "Austin TX".

[0124] The profile data module 104 is configured to receive source data 202 and generate profile data from the source data. Although this example is limited to specific data entries, the source data 202 generally includes many entries. Generally, the entries can be scanned to obtain statistical information. As previously described, once the profile data is generated for the field (or generally, the table), the test module 106 performs one or more classification tests. The classification 106 can refer to the reference database 116 to determine whether the field name "Field_XYZ" already has a label. In this example, a direct lookup result is found in the label index 154. It is known (e.g., by a specification generated by a user or by some other means) that the field name "Field_XYZ" represents the field name of "city name". Because the relationship is clearly defined in the index 154, the test module 106 can score the relationship with a high confidence, or pass the result to the result confirmation module 108. The test module 106 can use the knowledge of what "city name" may be to further determine what value is included in the value 210.

[0125] The tags identified by the testing module 106 are sent to the result validation module 108. The result validation module 108 classifies the tag into the "match" category 218, determines whether the tag should be automatically applied, and requests validation if necessary. The result validation module 108 validates the different classification tests as previously described to output a ranked list of recommendations for labeling the field 208. Once the tag is validated 220, the assigned tag is sent to update the index table 152 if necessary. In some embodiments, the attributes of the tag city name can be updated based on the attributes of the field 208.

[0126] Go to Fig. 6A , shows a depiction of a user interface 500 that shows the results of a data profiling process, such as performed by the profile data module 104. The user interface 500 includes a header 501 and rows 502a-e and columns 504, 506, 508, and 510. Each of the rows 502a-e represents a field of the source data 112. In some embodiments, different rows represent the same field (such as row 502a and row 502b), but each row shows a different detected relationship between the field and the other fields. For example, row 502a shows that the field "Date 1" is related to the field "Date 2", while row 502b shows that the field "Last Name" determines the field "ID".

[0127] Column 504 shows each of the fields of the source data 112. Column 506 shows the type of relationship detected between the field and a related field of the source data 112, as shown in column 508. Column 510 includes a confidence score (e.g., probability) that the fields of columns 504 and 508 are related to each other, as indicated by column 506.

[0128] The testing module 106 can use the relationships detected in the source data to determine the label value for each of the fields. For example, determining that a ZIP code field is related to a state field can provide clues about what each field means in practice. This is useful if the names of the fields are technical or otherwise do not indicate the type of data included in the field.

[0129] Go to Figure 6B , shows a depiction of a user interface 520 including the results of the classification process of the test module 106. Window 524 shows a list of discovery fields 526a-e. In this example, the fields from Fig. 6A 504 and 508 of the columns. Box 526a shows which fields have been analyzed by the testing module 106. In this example, the field "Date 1" has been analyzed. In some implementations, each field can be selected to show details about how the testing module 106 analyzed the finding field and how the score was generated. For example, the bold outline of button 526a shows that the button has been selected.

[0130] Window 528 shows the analysis details of the selected discovery field of window 524. Here, window 528 shows that four classifier tests are applied: pattern matching analysis, business term analysis, fingerprint analysis and keyword search. Each test is associated with a corresponding weighted score. The final adjusted score is also shown in box 536.

[0131] Window 534 shows a list 530 of candidate tags and their adjusted scores. Here, the "Date of Birth" tag is associated with a score of 95. The second tag "Expiration Date" is associated with a score of 33. Although the candidate tags are shown sorted according to their scores, the candidate tags may be shown in any order. The score of 95 matches the adjusted score shown in window 528. The analysis performed and the score shown in window 528 are associated with candidate tag 532a "Date of Birth".

[0132] Window 534 shows additional details about the classification process of the test module 106. The number of proposed tags can be shown. In this example, the number is 2. In another example, multiple iterations for analyzing one or more of the discovery fields can be shown.

[0133] Figure 6C 5 is a depiction of a user interface 540 showing an example of the results of the result validation module 108. The user interface 540 includes a proposed label pane 542 in which labels that may represent data fields are shown. A window 552 shows which fields were found in the analysis received by the result validation module 108. The found field 554 has a name of "place" that is included in Fig. 6A and as regards Figure 6B The proposed tag 556 having the value of "city name" is recommended to the user as a tag for the field "place".

[0134] The recommended item information pane 544 includes data describing the discovery fields of the recommended tags. For example, the current tag(s) 558 assigned to the field can be shown. In this example, the field has not yet been labeled (perhaps user input is pending). The values ​​of the entries in the data fields can be shown in the pane 574. Links between fields can be shown in the menu 572. In this example, values ​​including "Albany", "Boston", "Cleveland", "Detroit", "Evanston", and "Fargo" are shown.

[0135] The profile summary pane 550 shows a summary of the contents of the field. For example, the summary pane 550 may include statistics about the field, such as total values, missing values, multiple different / unique values, the minimum and maximum length of an entry, the characters included in an entry, the (multiple) data types of the entry, etc. In this example, the field to be labeled "type" has 28 values, with 28 unique values. All values ​​are between 3 and 11 characters in length and are of string type. In addition to the data of the entry, the test module 106 can also use these clues to determine that the label "type" should be recommended.

[0136] Common values ​​pane 548 shows which values ​​are most common in a field and their relative proportions in the values ​​of the field. In this example, all different values ​​of the field occur a similar number of times.

[0137] Pattern pane 546 shows common patterns for each of the entries for the field. For example, if a date is present, the pattern might include ## / ## / ####, as described above. Here, no particular pattern dominates the field, although values ​​including the "New_" pattern are the most common. Although these specific profile data are shown in panes 550, 548, 546, any data that can help a user approve a proposed candidate name for a discovery field may be shown.

[0138] Go to Fig.6D, a depiction of user interface 560 shows recommendations for labels for discovered data fields. The user interface is presented in a tabular format, where the values ​​of the table are interactive. Column 562 shows the field names received in the source data. Column 564 shows which labels have been assigned to the fields. As previously described, each field can be associated with more than one label. Column 566 shows proposed labels for the fields of column 562. Column 568 identifies which test(s) the testing module 106 ran on the field, how strong the results were, and what category of recommendation was made for the label. Column 570 shows the user's pending decision regarding the fields of column 562. For example, the user may be prompted to accept / reject the proposed label, rerun the test with updated parameters, etc.

[0139] In this example, in the last row, the discovery field "Date 1" is shown, which has been FIG. 6A to FIG. 6C The discovery, analysis, classification, and labeling are performed as described. The field "Date 1" is assigned the label "Date of Birth". As shown in column 568, the category of the label is the "Recommended" category because the score associated with the candidate label is 95, as shown in FIG. Figure 6B The analysis performed includes pattern matching, among other analyses. The decision to assign a label to the field "Date 1" is open, as the label has already been assigned and no further action is required by the user.

[0140] Figure 7 A flow chart of a process 600 for labeling a discovery field with a candidate label is shown. The process 600 includes identifying (602) a field included in one or more data sets, wherein the field has an identifier. The process 600 includes parsing (604) one or more data values ​​of the field by a data processing system to generate a data profile. The process 600 includes accessing (606) a plurality of label proposal tests. The process 600 includes generating (608) a set of label proposals based on applying at least the plurality of label proposal tests to the data profile. The process 600 includes determining (610) similarities between label proposals in the set of label proposals. The process 600 includes selecting (612) a classification based at least on similarities between label proposals in the set. The process 600 includes determining (614) whether user input is required. The process 600 includes presenting (616) a graphical user interface requesting input if user input is required. The process 600 includes identifying (618) one of the label proposals as identifying a semantic meaning. The process includes storing (620) an identifier of the field in a data store along with an identified one of the tag proposals that identifies a semantic meaning.

[0141] The above techniques can be implemented using software to be executed on a computer. For example, the software forms a process in one or more computer programs that is executed on one or more programming or programmable computer systems (which can be various architectures such as distributed, client / server or grid), each of which includes at least one processor, at least one data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device or port, and at least one output device or port. The software can form one or more modules of a larger program (e.g., providing other services related to the design and configuration of diagrams and flow charts). The nodes, links, and elements of the diagram can be implemented as data structures stored in a computer-readable medium or other organized data that conforms to a data model stored in a data repository.

[0142] In some embodiments, the data processing system described in this specification includes one or more aspects. In one aspect, a method implemented by a data processing system for discovering the semantic meaning of data included in a field of one or more data sets includes identifying a field included in one or more data sets, wherein the field is associated with an identifier. The method includes, for the field, parsing one or more data values ​​of the field by the data processing system to generate a data profile. The method includes, for the field, accessing multiple label proposal tests based on applying at least multiple label proposal tests to the data profile. The method includes, for the field, generating a set of label proposals. The method includes, for the field, determining the similarity between label proposals in the set of label proposals. The method includes, for the field, selecting a classification based at least on the similarity between the label proposals in the set. The method includes, for the field, presenting a graphical user interface for requesting input when identifying a label proposal that identifies a semantic meaning or determining that no input is required based on the classification. The method includes, for the field, identifying one of the label proposals as identifying the semantic meaning. The method includes, for the field, storing an identifier of the field in a data storage area together with an identified one of the label proposals that identifies the semantic meaning. The method includes receiving a request to process one or more data sets including a specific identifier. The method includes accessing the specific identifier and a specific tag associated with the specific identifier from a data storage area in response to the request. The method includes determining one or more data processing rules for processing at least a portion of the one or more data sets based on the specific tag. The method includes processing at least a portion of the one or more data sets according to the selected one or more data processing rules.

[0143] Typically, the stored identifier of the field together with the identified one of the label proposals that identifies the semantic meaning is an index such as a label index. When the data processing system is processing a field of a data set, the data processing system does not need to access the value of the field to determine how to process the value. Instead, the data processing system can simply access the associated label in the label index of the field, and then access a set of specific rules for processing data (e.g., a set of data quality rules, data masking rules, etc.). The set of specific rules is associated with the label in the label index. Therefore, the data processing system can process the data set more accurately because the data processing system has selected the appropriate processing rules to process the value of the tagged field. This reduces errors in processing the data values ​​of the fields of various applications such as data masking, pattern analysis of data sets, application of data quality rules, selection of test data, etc. For example, labels can be associated with attributes (e.g., in a label table as previously described) to indicate to the data processing system what the structure of the data set is (e.g., whether the field corresponds to a key value field). Then, the data processing system can process the data set according to the attributes associated with the label of the tagged field. In this example, these attributes may indicate that the field includes a key value. Thus, in one aspect, the data processing system uses the stored identifier to access a specific data processing rule or operation to process the field marked by the proposed tag.

[0144] In one aspect, profiling one or more data values ​​of a field includes determining a format of the data values ​​of the field. In one aspect, profiling the data values ​​of a field includes determining a statistical value representative of the data values ​​included in the field.

[0145] In one aspect, the statistical value includes at least one of a minimum length of data values ​​of the field, a maximum length of data values ​​of the field, a most common data value of the field, a least common data value of the field, a maximum data value of the field, and a minimum data value of the field.

[0146] In one aspect, applying the plurality of tag proposal tests includes determining that a field comprises a primary key of one of the one or more data sets, and selecting a tag proposal test of the plurality of tag proposal tests that is associated with the primary key.

[0147] In one aspect, applying the plurality of tag proposal tests includes performing a metadata comparison of data values ​​of fields with terms in a glossary.

[0148] In one aspect, applying the plurality of tag proposal tests includes determining, from a data profile, a pattern represented by data values ​​stored in a field, determining a particular tag that maps to the pattern, and marking the field with the particular tag.

[0149] In one aspect, applying the multiple label proposal tests includes retrieving a list of values ​​representing a data set, comparing a data value of a field to the list of values, determining in response to the comparison that a threshold number of data values ​​match the values ​​of the list, and marking the field with a particular label for the specified data set in response to the determination.

[0150] In one aspect, applying the plurality of label proposal tests includes generating at least two labels for the field, and determining whether the at least two labels are mutually exclusive or mutually inclusive.

[0151] In one aspect, the method includes determining a relationship between a field and another field of one or more data sets in response to applying the plurality of label proposal tests. In one aspect, the relationship includes one of indicating that a first data value of the field determines a second data value stored in another field, indicating that the first data value is related to the second data value, or indicating that the first data value is the same as the second data value.

[0152] In one aspect, the plurality of label proposal tests are each associated with at least one weight value. The method includes updating the weight value associated with at least one label proposal test, and reapplying the label proposal test to the data profile using the updated weight value. In one aspect, the method includes training the plurality of label proposal tests using a machine learning process.

[0153] In one aspect, the method includes retrieving one or more data quality rules assigned to a tag proposal specifying a semantic meaning from a data quality rule environment, and assigning a data quality rule of the one or more data quality rules to a field.

[0154] In one aspect, comparing the label proposals generated by the label proposal tests includes applying a score value to each label proposal, for each label in the label proposals, combining the score values ​​associated with the label, and ranking the labels according to the score values ​​associated with each label.

[0155] In one aspect, the method includes receiving a validation of the tag proposals from the plurality of tag proposal tests, and weighting the plurality of tag proposal tests with the tag proposals in response to receiving the validation.

[0156] In one aspect, the data store includes a data dictionary.

[0157] In one aspect, the method includes outputting a label proposal to a data quality rules environment. In one aspect, the method includes reducing another number of errors in processing data for the field using data quality rules from the data quality environment based on an identified one of the label proposals relative to a number of errors in processing data for the field without using the identified one of the label proposals.

[0158] In one aspect, a data processing system for discovering the semantic meaning of a field included in one or more data sets includes a data storage area storing instructions and at least one processor configured to execute the instructions stored by the data storage device. The operations include identifying a field included in one or more data sets, wherein the field is associated with an identifier. The operations include, for the field, parsing one or more data values ​​of the field by the data processing system to generate a data profile. The operations include, for the field, accessing multiple label proposal tests based on applying at least multiple label proposal tests to the data profile. The operations include, for the field, generating a set of label proposals. The operations include, for the field, determining the similarity between label proposals in the set of label proposals. The operations include, for the field, selecting a classification based on at least the similarity between the label proposals in the set. The operations include, for the field, presenting a graphical user interface that requests input when identifying a label proposal that identifies a semantic meaning or determining that no input is required based on the classification. The operations include, for the field, identifying one of the label proposals as identifying a semantic meaning. The operations include, for the field, storing an identifier of the field in the data storage area together with an identified one of the label proposals that identifies the semantic meaning. The operations include receiving a request to process one or more data sets including a specific identifier. The operations include accessing the specific identifier and a specific tag associated with the specific identifier from a data storage area in response to the request. The operations include determining one or more data processing rules for processing at least a portion of the one or more data sets based on the specific tag. The operations include processing at least a portion of the one or more data sets according to the selected one or more data processing rules.

[0159] Typically, the stored identifier of the field together with the identified one of the label proposals that identifies the semantic meaning is an index such as a label index. When the data processing system is processing a field of a data set, the data processing system does not need to access the value of the field to determine how to process the value. Instead, the data processing system can simply access the associated label in the label index of the field, and then access a set of specific rules for processing data (e.g., a set of data quality rules, data masking rules, etc.). The set of specific rules is associated with the label in the label index. Therefore, the data processing system can process the data set more accurately because the data processing system has selected the appropriate processing rules to process the value of the tagged field. This reduces errors in processing the data values ​​of the fields of various applications such as data masking, pattern analysis of data sets, application of data quality rules, selection of test data, etc. For example, labels can be associated with attributes (e.g., in a label table as previously described) to indicate to the data processing system what the structure of the data set is (e.g., whether the field corresponds to a key value field). Then, the data processing system can process the data set according to the attributes associated with the label of the tagged field. In this example, these attributes may indicate that the field includes a key value. Thus, in one aspect, the data processing system uses the stored identifier to access a specific data processing rule or operation to process the field marked by the proposed tag.

[0160] In one aspect, profiling one or more data values ​​of a field includes determining a format of the data values ​​of the field. In one aspect, profiling the data values ​​of a field includes determining a statistical value representative of the data values ​​included in the field.

[0161] In one aspect, the statistical value includes at least one of a minimum length of data values ​​of the field, a maximum length of data values ​​of the field, a most common data value of the field, a least common data value of the field, a maximum data value of the field, and a minimum data value of the field.

[0162] In one aspect, applying the plurality of tag proposal tests includes determining that a field comprises a primary key of one of the one or more data sets, and selecting a tag proposal test of the plurality of tag proposal tests that is associated with the primary key.

[0163] In one aspect, applying the plurality of tag proposal tests includes performing a metadata comparison of data values ​​of fields with terms in a glossary.

[0164] In one aspect, applying the plurality of tag proposal tests includes determining, from a data profile, a pattern represented by data values ​​stored in a field, determining a particular tag that maps to the pattern, and marking the field with the particular tag.

[0165] In one aspect, applying the multiple label proposal tests includes retrieving a list of values ​​representing a data set, comparing a data value of a field to the list of values, determining in response to the comparison that a threshold number of data values ​​match the values ​​of the list, and marking the field with a particular label for the specified data set in response to the determination.

[0166] In one aspect, applying the plurality of label proposal tests includes generating at least two labels for the field, and determining whether the at least two labels are mutually exclusive or mutually inclusive.

[0167] In one aspect, the operations include determining a relationship between the field and another field of the one or more data sets in response to applying the plurality of tag proposal tests. In one aspect, the relationship includes one of indicating that a first data value of the field determines a second data value stored in another field, indicating that the first data value is related to the second data value, or indicating that the first data value is the same as the second data value.

[0168] In one aspect, the plurality of label proposal tests are each associated with at least one weight value. The operations include updating the weight value associated with at least one label proposal test, and reapplying the label proposal test to the data profile using the updated weight value. In one aspect, the operations include training the plurality of label proposal tests using a machine learning process.

[0169] In one aspect, the operations include retrieving one or more data quality rules assigned to a tag proposal specifying a semantic meaning from a data quality rule environment, and assigning a data quality rule of the one or more data quality rules to a field.

[0170] In one aspect, comparing the label proposals generated by the label proposal tests includes applying a score value to each label proposal, for each label in the label proposals, combining the score values ​​associated with the label, and ranking the labels according to the score values ​​associated with each label.

[0171] In one aspect, the operations include receiving validation of the tag proposals from the plurality of tag proposal tests, and weighting the plurality of tag proposal tests with the tag proposals in response to receiving the validation.

[0172] In one aspect, the data store includes a data dictionary.

[0173] In one aspect, the operations include outputting a label proposal to a data quality rules environment. In one aspect, the operations include reducing another number of errors in processing data for the field using data quality rules from the data quality environment based on the identified one of the label proposals relative to a number of errors in processing data for the field without using the identified one of the label proposals.

[0174] On the one hand, one or more non-transitory computer-readable media storing instructions for discovering the semantic meaning of a field included in one or more data sets, the instructions being executable by one or more processors configured to perform operations. The operations include identifying a field included in one or more data sets, wherein the field is associated with an identifier. The operations include, for the field, parsing one or more data values ​​of the field by a data processing system to generate a data profile. The operations include, for the field, accessing multiple label proposal tests based on at least applying multiple label proposal tests to the data profile. The operations include, for the field, generating a set of label proposals. The operations include, for the field, determining similarities between label proposals in the set of label proposals. The operations include, for the field, selecting a classification based at least on the similarities between label proposals in the set. The operations include, for the field, presenting a graphical user interface for requesting input when identifying a label proposal that identifies a semantic meaning or determining that no input is required based on the classification. The operations include, for the field, identifying one of the label proposals as identifying a semantic meaning. The operations include, for the field, storing in a data store an identifier of the field along with an identified one of the tag proposals that identifies the semantic meaning.

[0175] Typically, the stored identifier of the field together with the identified one of the label proposals that identifies the semantic meaning is an index such as a label index. When the data processing system is processing a field of a data set, the data processing system does not need to access the value of the field to determine how to process the value. Instead, the data processing system can simply access the associated label in the label index of the field, and then access a set of specific rules for processing data (e.g., a set of data quality rules, data masking rules, etc.). The set of specific rules is associated with the label in the label index. Therefore, the data processing system can process the data set more accurately because the data processing system has selected the appropriate processing rules to process the value of the tagged field. This reduces errors in processing the data values ​​of the fields of various applications such as data masking, pattern analysis of data sets, application of data quality rules, selection of test data, etc. For example, labels can be associated with attributes (e.g., in a label table as previously described) to indicate to the data processing system what the structure of the data set is (e.g., whether the field corresponds to a key value field). Then, the data processing system can process the data set according to the attributes associated with the label of the tagged field. In this example, these attributes may indicate that the field includes a key value. Thus, in one aspect, the data processing system uses the stored identifier to access a specific data processing rule or operation to process the field marked by the proposed tag.

[0176] In one aspect, profiling one or more data values ​​of a field includes determining a format of the data values ​​of the field. In one aspect, profiling the data values ​​of a field includes determining a statistical value representative of the data values ​​included in the field.

[0177] In one aspect, the statistical value includes at least one of a minimum length of data values ​​of the field, a maximum length of data values ​​of the field, a most common data value of the field, a least common data value of the field, a maximum data value of the field, and a minimum data value of the field.

[0178] In one aspect, applying the plurality of tag proposal tests includes determining that a field comprises a primary key of one of the one or more data sets, and selecting a tag proposal test of the plurality of tag proposal tests that is associated with the primary key.

[0179] In one aspect, applying the plurality of tag proposal tests includes performing a metadata comparison of data values ​​of fields with terms in a glossary.

[0180] In one aspect, applying the plurality of tag proposal tests includes determining, from a data profile, a pattern represented by data values ​​stored in a field, determining a particular tag that maps to the pattern, and marking the field with the particular tag.

[0181] In one aspect, applying the multiple label proposal tests includes retrieving a list of values ​​representing a data set, comparing a data value of a field to the list of values, determining in response to the comparison that a threshold number of data values ​​match the values ​​of the list, and marking the field with a particular label for the specified data set in response to the determination.

[0182] In one aspect, applying the plurality of label proposal tests includes generating at least two labels for the field, and determining whether the at least two labels are mutually exclusive or mutually inclusive.

[0183] In one aspect, the operations include determining a relationship between the field and another field of the one or more data sets in response to applying the plurality of tag proposal tests. In one aspect, the relationship includes one of indicating that a first data value of the field determines a second data value stored in another field, indicating that the first data value is related to the second data value, or indicating that the first data value is the same as the second data value.

[0184] In one aspect, the plurality of label proposal tests are each associated with at least one weight value. The operations include updating the weight value associated with at least one label proposal test, and reapplying the label proposal test to the data profile using the updated weight value. In one aspect, the operations include training the plurality of label proposal tests using a machine learning process.

[0185] In one aspect, the operations include retrieving one or more data quality rules assigned to a tag proposal specifying a semantic meaning from a data quality rule environment, and assigning a data quality rule of the one or more data quality rules to a field.

[0186] In one aspect, comparing the label proposals generated by the label proposal tests includes applying a score value to each label proposal, for each label in the label proposals, combining the score values ​​associated with the label, and ranking the labels according to the score values ​​associated with each label.

[0187] In one aspect, the operations include receiving validation of the tag proposals from the plurality of tag proposal tests, and weighting the plurality of tag proposal tests with the tag proposals in response to receiving the validation.

[0188] In one aspect, the data store includes a data dictionary.

[0189] In one aspect, the operations include outputting a label proposal to a data quality rules environment. In one aspect, the operations include reducing another number of errors in processing data for the field using data quality rules from the data quality environment based on the identified one of the label proposals relative to a number of errors in processing data for the field without using the identified one of the label proposals.

[0190] The techniques described herein may be implemented in digital electronic circuit systems, or in computer hardware, firmware, software, or a combination thereof. The apparatus may be implemented in a computer program product tangibly embodied or stored in a machine-readable storage device (e.g., a non-transitory machine-readable storage device, a machine-readable hardware storage device, etc.) for execution by a programmable processor; and the method actions may be performed by a programmable processor executing an instruction program to perform functions by operating on input data and generating outputs. The embodiments described herein, as well as the claims and other embodiments of the techniques described herein, may be advantageously implemented in one or more computer programs that may be executed on a programmable system comprising: at least one programmable processor coupled to receive data and instructions from a data storage system, at least one input device, and at least one output device, and to transmit data and instructions to a data storage system, at least one input device, and at least one output device. Each computer program may be implemented in a high-level procedural or object-oriented programming language, or in an assembly language or machine language if desired; and in any case, the language may be a compiled language or an interpreted language.

[0191] Processors suitable for executing computer programs include, for example, both general-purpose microprocessors and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. Essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or be operably coupled to receive data from or transmit data to the one or more mass storage devices, or both receive and transmit data. Computer-readable media for embodying computer program instructions and data include all forms of non-volatile memory, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into a dedicated logic circuit system. Any of the above may be supplemented by or incorporated into an ASIC (Application Specific Integrated Circuit).

[0192] To provide interaction with a user, embodiments may be implemented on a computer having a display device (e.g., an LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound input, voice input, or tactile input.

[0193] Embodiments may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with implementations of the embodiments), or any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs) (e.g., the Internet).

[0194] The system and method or part thereof can use a collection of servers on the Internet that utilizes the Hypertext Transfer Protocol (HTTP) - the "World Wide Web" (Web or WWW). HTTP is a known application protocol that provides users with access to resources, which can be information in different formats, such as text, graphics, images, sounds, videos, Hypertext Markup Language (HTML), and programs. After the user specifies a link, the client computer sends a TCP / IP request to the Web server and receives information, which can be another Web page formatted according to HTML. The user can also access other pages on the same or other servers by following instructions on the screen, entering specific data, or clicking on selected icons. It should also be noted that any type of selection device known to those skilled in the art, such as check boxes, drop-down boxes, etc., can be used for embodiments that use web pages to allow users to select options for given components. The server runs on a variety of platforms, including UNIX machines, but other platforms such as Windows 2000 / 2003, Windows NT, Sun, Linux, and Macintosh can also be used. Computer users can view information available on a server or network on the Web by using browsing software (such as Firefox, Netscape Navigator, Microsoft Internet Explorer, or Mosaic browser). The computing system can include a client and a server. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0195] Other embodiments are within the scope and spirit of the specification and claims. For example, due to the nature of software, the functions described above may be implemented using software, hardware, firmware, hardwiring, or any combination of these. Features that implement the functions may also be physically located in various locations, including being distributed so that parts of the functions are implemented at different physical locations. The use of the term "one" herein and throughout the application is not used in a limiting manner, and therefore is not meant to exclude multiple meanings or "one or more" meanings of the term "one". In addition, with respect to the priority claimed for the provisional patent application, it should be understood that the provisional patent application is not restrictive, but includes examples of how the technology described herein may be implemented.

[0196] Various embodiments have been described. The invention may also (alternatively) be described by way of the aspects numbered 1 to 76 below.

[0197] aspect

[0198] 1. A method implemented by a data processing system for discovering semantic meaning of data included in a field of one or more data sets, the method comprising:

[0199] identifying a field included in one or more data sets, wherein the field is associated with an identifier; and

[0200] For this field:

[0201] parsing, by a data processing system, one or more data values ​​of the field to generate a data profile;

[0202] Access multiple tabs to propose tests;

[0203] generating a set of tag proposals based on applying at least the plurality of tag proposal tests to the data profile;

[0204] determining similarities between label proposals in the set of label proposals;

[0205] A classification is selected based at least on the similarity between label proposals in the group;

[0206] presenting a graphical user interface that requests input when identifying a tag proposal that identifies a semantic meaning based on the classification or determining that no input is required;

[0207] identifying one of the label proposals as identifying the semantic meaning; and

[0208] An identifier for the field is stored in a data store along with an identified one of the label proposals that identifies the semantic meaning.

[0209] 2. The method of aspect 1, wherein parsing one or more data values ​​of the field comprises determining a format of the data value of the field.

[0210] 3. A method as described in aspect 1, wherein analyzing the data values ​​of the field includes determining a statistical value representative of the data values ​​included in the field.

[0211] 4. A method as described in aspect 3, wherein the statistical value includes at least one of the minimum length of the data value of the field, the maximum length of the data value of the field, the most common data value of the field, the least common data value of the field, the maximum data value of the field and the minimum data value of the field.

[0212] 5. The method according to aspect 1, wherein applying the plurality of tag proposal tests comprises:

[0213] determining that the field comprises a primary key of a data set in the one or more data sets; and

[0214] Select a tag proposal test associated with the primary key from among the multiple tag proposal tests.

[0215] 6. The method of aspect 1, wherein applying the plurality of tag proposal tests comprises:

[0216] Performs a comparison of the data value of this field with the metadata of the terms in the glossary.

[0217] 7. The method of clause 1, wherein applying the plurality of tag proposal tests comprises:

[0218] determining, based on the data profile, a pattern represented by the data value stored in the field;

[0219] determining the specific tags that are mapped to the pattern; and

[0220] Label the field with that specific label.

[0221] 8. The method of clause 1, wherein applying the plurality of tag proposal tests comprises:

[0222] Retrieve a list of values ​​representing a collection of data;

[0223] Compare the data value of the field with the list of values;

[0224] In response to the comparing, determining that a threshold number of data values ​​match the values ​​in the list; and

[0225] Responsive to the determination, the field is labeled with a particular tag that specifies the data set.

[0226] 9. The method of clause 1, wherein applying the plurality of tags to propose a test comprises:

[0227] Generate at least two labels for the field; and

[0228] It is determined whether the at least two tags are mutually exclusive or mutually inclusive.

[0229] 10. The method according to aspect 1, further comprising:

[0230] Responsive to applying the plurality of label proposal tests, a relationship between the field and another field of the one or more data sets is determined.

[0231] 11. A method as described in aspect 10, wherein the relationship includes one of the following: indicating that the first data value of the field determines the second data value stored in another field, indicating that the first data value is related to the second data value, or indicating that the first data value is the same as the second data value.

[0232] 12. The method of aspect 1, wherein each of the plurality of tag proposal tests is associated with at least one weight value, the method further comprising:

[0233] updating a weight value associated with at least one tag proposal test; and

[0234] The tag proposal test is reapplied to the data profile using the updated weight values.

[0235] 13. The method of aspect 1 further comprises training the plurality of label proposal tests using a machine learning process.

[0236] 14. The method according to aspect 1, further comprising:

[0237] retrieving from a data quality rule environment one or more data quality rules assigned to the tag proposal specifying the semantic meaning; and

[0238] A data quality rule of the one or more data quality rules is assigned to the field.

[0239] 15. The method of clause 1, wherein comparing the tag proposals generated by the tag proposal tests comprises:

[0240] Apply a score value to each label proposal;

[0241] For each of the label proposals, combining the score values ​​associated with the label; and

[0242] The tags are sorted according to the score value associated with each tag.

[0243] 16. The method according to aspect 1, further comprising:

[0244] receiving validation of the tag proposals from the plurality of tag proposal tests; and

[0245] In response to receiving the validation, the plurality of tag proposal tests are weighted with the tag proposals.

[0246] 17. The method of aspect 1, wherein the data storage area comprises a data dictionary.

[0247] 18. The method according to aspect 1, further comprising:

[0248] Output these label proposals to the data quality rules environment.

[0249] 19. The method according to aspect 18, further comprising:

[0250] Relative to the number of errors in processing data for the field without using the identified one of the label proposals, another number of errors in processing data for the field using data quality rules from the data quality environment is reduced based on the identified one of the label proposals.

[0251] 20. A data processing system for discovering semantic meaning of fields included in one or more data sets, the system comprising:

[0252] A data storage device storing instructions; and

[0253] At least one processor, the at least one processor being configured to execute instructions stored in the data storage device to perform operations including:

[0254] identifying a field included in one or more data sets, wherein the field has an identifier; and

[0255] For this field:

[0256] parsing, by a data processing system, one or more data values ​​of the field to generate a data profile;

[0257] Access multiple tabs to propose tests;

[0258] generating a set of tag proposals based on applying at least the plurality of tag proposal tests to the data profile;

[0259] determining similarities between label proposals in the set of label proposals;

[0260] A classification is selected based at least on the similarity between label proposals in the group;

[0261] presenting a graphical user interface that requests input when identifying a tag proposal that identifies a semantic meaning based on the classification or determining that no input is required;

[0262] identifying one of the label proposals as identifying the semantic meaning; and

[0263] An identifier for the field is stored in a data store along with an identified one of the label proposals that identifies the semantic meaning.

[0264] 21. The system of clause 20, wherein parsing the one or more data values ​​of the field comprises determining a format of the data values ​​of the field.

[0265] 22. The system of clause 20, wherein profiling the data values ​​of the field comprises determining a statistical value representative of the data values ​​included in the field.

[0266] 23. A system as described in aspect 22, wherein the statistical value includes at least one of a minimum length of data values ​​of the field, a maximum length of data values ​​of the field, a most common data value of the field, a least common data value of the field, a maximum data value of the field, and a minimum data value of the field.

[0267] 24. The system of clause 20, wherein applying the plurality of tag proposal tests comprises:

[0268] determining that the field comprises a primary key of a data set in the one or more data sets; and

[0269] Select a tag proposal test associated with the primary key from among the multiple tag proposal tests.

[0270] 25. The system of clause 20, wherein applying the plurality of tag proposal tests comprises:

[0271] Performs a comparison of the data value of this field with the metadata of the terms in the glossary.

[0272] 26. The system of clause 20, wherein applying the plurality of tag proposal tests comprises:

[0273] determining, based on the data profile, a pattern represented by the data value stored in the field;

[0274] determining the specific tags that are mapped to the pattern; and

[0275] Label the field with that specific label.

[0276] 27. The system of clause 20, wherein applying the plurality of tag proposal tests comprises:

[0277] Retrieve a list of values ​​representing a collection of data;

[0278] Compare the data value of the field with the list of values;

[0279] In response to the comparing, determining that a threshold number of data values ​​match the values ​​in the list; and

[0280] Responsive to the determination, the field is labeled with a particular tag that specifies the data set.

[0281] 28. The system of clause 20, wherein applying the plurality of tag proposal tests comprises:

[0282] Generate at least two labels for the field; and

[0283] It is determined whether the at least two tags are mutually exclusive or mutually inclusive.

[0284] 29. The system of aspect 20, wherein the operations further comprise:

[0285] Responsive to applying the plurality of label proposal tests, a relationship between the field and another field of the one or more data sets is determined.

[0286] 30. The system of aspect 29, wherein the relationship comprises one of: indicating that a first data value of the field determines a second data value stored in another field, indicating that the first data value is related to the second data value, or indicating that the first data value is the same as the second data value.

[0287] 31. The system of clause 20, wherein the plurality of tag proposal tests are each associated with at least one weight value, the operations further comprising:

[0288] updating a weight value associated with at least one tag proposal test; and

[0289] The tag proposal test is reapplied to the data profile using the updated weight values.

[0290] 32. The system of clause 20, the operations further comprising training the plurality of label proposal tests using a machine learning process.

[0291] 33. The system of aspect 20, wherein the operations further comprise:

[0292] retrieving from a data quality rule environment one or more data quality rules assigned to the tag proposal specifying the semantic meaning; and

[0293] A data quality rule of the one or more data quality rules is assigned to the field.

[0294] 34. The system of clause 20, wherein comparing the tag proposals generated by the tag proposal tests comprises:

[0295] Apply a score value to each label proposal;

[0296] For each of the label proposals, combining the score values ​​associated with the label; and

[0297] The tags are sorted according to the score value associated with each tag.

[0298] 35. The system of aspect 20, wherein the operations further comprise:

[0299] receiving validation of the tag proposals from the plurality of tag proposal tests; and

[0300] In response to receiving the validation, the plurality of tag proposal tests are weighted with the tag proposals.

[0301] 36. The system of clause 20, wherein the data storage area comprises a data dictionary.

[0302] 37. The system of aspect 20, wherein the operations further comprise:

[0303] Output these label proposals to the data quality rules environment.

[0304] 38. The system of aspect 37, the operations further comprising:

[0305] Relative to the number of errors in processing data for the field without using the identified one of the label proposals, another number of errors in processing data for the field using data quality rules from the data quality environment is reduced based on the identified one of the label proposals.

[0306] 39. One or more non-transitory computer readable media storing instructions for discovering semantic meaning of fields included in one or more data sets, the instructions executable by one or more processors configured to perform operations comprising:

[0307] identifying a field included in one or more data sets, wherein the field has an identifier; and

[0308] For this field:

[0309] parsing, by a data processing system, one or more data values ​​of the field to generate a data profile;

[0310] Access multiple tabs to propose tests;

[0311] generating a set of tag proposals based on applying at least the plurality of tag proposal tests to the data profile;

[0312] determining similarities between label proposals in the set of label proposals;

[0313] A classification is selected based at least on the similarity between label proposals in the group;

[0314] presenting a graphical user interface that requests input when identifying a tag proposal that identifies a semantic meaning based on the classification or determining that no input is required;

[0315] identifying one of the label proposals as identifying the semantic meaning; and

[0316] An identifier for the field is stored in a data store along with an identified one of the label proposals that identifies the semantic meaning.

[0317] 40. The one or more non-transitory computer-readable media of clause 39, wherein parsing the one or more data values ​​of the field comprises determining a format of the data values ​​of the field.

[0318] 41. The one or more non-transitory computer-readable media of clause 39, wherein profiling the data values ​​of the field comprises determining a statistical value representative of the data values ​​included in the field.

[0319] 42. One or more non-transitory computer-readable media as described in aspect 41, wherein the statistical value includes at least one of a minimum length of a data value of the field, a maximum length of a data value of the field, a most common data value of the field, a least common data value of the field, a maximum data value of the field, and a minimum data value of the field.

[0320] 43. The one or more non-transitory computer-readable media of clause 39, wherein applying the plurality of tag proposal tests comprises:

[0321] determining that the field comprises a primary key of a data set in the one or more data sets; and

[0322] Select a tag proposal test associated with the primary key from among the multiple tag proposal tests.

[0323] 44. The one or more non-transitory computer-readable media of clause 39, wherein applying the plurality of tag proposal tests comprises:

[0324] Performs a comparison of the data value of this field with the metadata of the terms in the glossary.

[0325] 45. The one or more non-transitory computer-readable media of clause 39, wherein applying the plurality of tag proposal tests comprises:

[0326] determining, based on the data profile, a pattern represented by the data value stored in the field;

[0327] determining the specific tags that are mapped to the pattern; and

[0328] Label the field with that specific label.

[0329] 46. ​​The one or more non-transitory computer-readable media of clause 39, wherein applying the plurality of tag proposal tests comprises:

[0330] Retrieve a list of values ​​representing a collection of data;

[0331] Compare the data value of the field with the list of values;

[0332] In response to the comparing, determining that a threshold number of data values ​​match the values ​​in the list; and

[0333] Responsive to the determination, the field is labeled with a particular tag that specifies the data set.

[0334] 47. The one or more non-transitory computer-readable media of clause 39, wherein applying the plurality of tag proposal tests comprises:

[0335] Generate at least two labels for the field; and

[0336] It is determined whether the at least two tags are mutually exclusive or mutually inclusive.

[0337] 48. The one or more non-transitory computer-readable media of clause 39, the operations further comprising:

[0338] Responsive to applying the plurality of label proposal tests, a relationship between the field and another field of the one or more data sets is determined.

[0339] 49. One or more non-transitory computer-readable media as described in aspect 48, wherein the relationship includes one of: indicating that a first data value of the field determines a second data value stored in another field, indicating that the first data value is related to the second data value, or indicating that the first data value is the same as the second data value.

[0340] 50. The one or more non-transitory computer-readable media of clause 39, wherein the plurality of tag proposal tests are each associated with at least one weight value, the operations further comprising:

[0341] updating a weight value associated with at least one tag proposal test; and

[0342] The tag proposal test is reapplied to the data profile using the updated weight values.

[0343] 51. The one or more non-transitory computer-readable media of clause 39, the operations further comprising training the plurality of label proposal tests using a machine learning process.

[0344] 52. The one or more non-transitory computer-readable media of clause 39, the operations further comprising:

[0345] retrieving from a data quality rule environment one or more data quality rules assigned to the tag proposal specifying the semantic meaning; and

[0346] A data quality rule of the one or more data quality rules is assigned to the field.

[0347] 53. The one or more non-transitory computer-readable media of clause 39, wherein comparing the tag proposals generated from the tag proposal test comprises:

[0348] Apply a score value to each label proposal;

[0349] For each of the label proposals, combining the score values ​​associated with the label; and

[0350] The tags are sorted according to the score value associated with each tag.

[0351] 54. The one or more non-transitory computer-readable media of clause 39, the operations further comprising:

[0352] receiving validation of the tag proposals from the plurality of tag proposal tests; and

[0353] In response to receiving the validation, the plurality of tag proposal tests are weighted with the tag proposals.

[0354] 55. The one or more non-transitory computer-readable media of clause 39, wherein the data storage area comprises a data dictionary.

[0355] 56. The one or more non-transitory computer-readable media of clause 39, the operations further comprising:

[0356] Output these label proposals to the data quality rules environment.

[0357] 57. The one or more non-transitory computer-readable media of clause 56, the operations further comprising:

[0358] Relative to the number of errors in processing data for the field without using the identified one of the label proposals, another number of errors in processing data for the field using data quality rules from the data quality environment is reduced based on the identified one of the label proposals.

[0359] 58. A data processing system for discovering semantic meaning of fields included in one or more data sets, the system comprising:

[0360] means for identifying a field included in one or more data sets, wherein the field has an identifier;

[0361] For this field:

[0362] means for parsing, by a data processing system, one or more data values ​​of the field to generate a data profile;

[0363] means for accessing a plurality of label proposal tests;

[0364] means for generating a set of tag proposals based on at least applying the plurality of tag proposal tests to the data profile;

[0365] means for determining similarities between tag proposals in the set of tag proposals;

[0366] means for selecting a classification based at least on similarities between tag proposals in the set;

[0367] means for presenting a graphical user interface requesting input when identifying a tag proposal identifying a semantic meaning, or determining that no input is required, based on the classification;

[0368] means for identifying one of the tag proposals as identifying the semantic meaning; and

[0369] Means for storing in a data store an identifier for the field together with an identified one of the tag proposals that identifies the semantic meaning.

[0370] 59. The system of clause 58, wherein the means for parsing the one or more data values ​​of the field comprises:

[0371] Means for determining the format of the data value for this field.

[0372] 60. The system of clause 58, wherein the means for parsing the data value of the field comprises:

[0373] Means for determining a statistical value representative of the data values ​​included in the field.

[0374] 61. A system as described in aspect 60, wherein the statistical value includes at least one of a minimum length of data values ​​of the field, a maximum length of data values ​​of the field, a most common data value of the field, a least common data value of the field, a maximum data value of the field, and a minimum data value of the field.

[0375] 62. The system of clause 58, wherein the means for applying the plurality of tag proposal tests comprises:

[0376] means for determining that the field comprises a primary key of one of the one or more data sets; and

[0377] Means for selecting a tag proposal test from the plurality of tag proposal tests that is associated with the primary key.

[0378] 63. The system of clause 58, wherein applying the plurality of tag proposal tests comprises:

[0379] Means for performing a comparison of the data value of the field with metadata of terms in the glossary.

[0380] 64. The system of clause 58, wherein applying the plurality of tag proposal tests comprises:

[0381] means for determining, from the data profile, a pattern represented by the data value stored in the field; and

[0382] means for determining a specific label mapped to the pattern; and

[0383] Means for marking the field with the particular label.

[0384] 65. The system of clause 58, wherein the means for applying the plurality of tag proposal tests comprises:

[0385] means for retrieving a list of values ​​representing a set of data;

[0386] means for comparing the data value of the field with the list of values;

[0387] means for determining, in response to the comparing, that a threshold number of the data values ​​match the values ​​of the list; and

[0388] Means for marking the field with a particular tag specifying the data set in response to the determination.

[0389] 66. The system of clause 58, wherein applying the plurality of tag proposal tests comprises:

[0390] means for generating at least two tags for the field; and

[0391] Means for determining whether the at least two tags are mutually exclusive or mutually inclusive.

[0392] 67. The system of aspect 58, further comprising:

[0393] Means for determining a relationship between the field and another field of the one or more data sets in response to applying the plurality of tag proposal tests.

[0394] 68. A system as described in aspect 67, wherein the relationship includes one of the following: indicating that the first data value of the field determines the second data value stored in another field, indicating that the first data value is related to the second data value, or indicating that the first data value is the same as the second data value.

[0395] 69. The system of clause 58, wherein the plurality of tag proposal tests are each associated with at least one weight value, the system further comprising:

[0396] means for updating a weight value associated with at least one tag proposal test; and

[0397] Means for reapplying the tag proposal test to the data profile using the updated weight value.

[0398] 70. The system of aspect 58, further comprising:

[0399] Means for training the plurality of label proposal tests using a machine learning process.

[0400] 71. The system of aspect 58, further comprising:

[0401] means for retrieving from a data quality rule environment one or more data quality rules assigned to a tag proposal specifying the semantic meaning; and

[0402] Means for assigning a data quality rule of the one or more data quality rules to the field.

[0403] 72. The system of clause 58, wherein the means for comparing the label proposals generated by the label proposal tests comprises:

[0404] means for applying a score value to each label proposal;

[0405] means for combining, for each of the tag proposals, a score value associated with the tag; and

[0406] Means for sorting the tags according to the score value associated with each tag.

[0407] 73. The system of aspect 58, further comprising:

[0408] means for receiving validation of the tag proposals from the plurality of tag proposal tests; and

[0409] Means for weighting the plurality of tag proposal tests with the tag proposal in response to receiving the verification.

[0410] 74. The system of clause 58, wherein the data storage area comprises a data dictionary.

[0411] 75. The system of aspect 58, further comprising:

[0412] Means for outputting the label proposal to a data quality rules environment.

[0413] 76. The system of aspect 75, further comprising:

[0414] Means for reducing another number of errors in processing data for the field using data quality rules from the data quality environment based on the identified one of the label proposals relative to the number of errors in processing data for the field without using the identified one of the label proposals.

[0415] Several embodiments of the present invention have been described above. However, it will be appreciated by those skilled in the art that various modifications may be made therefrom without departing from the present invention as defined by the scope of the following claims including their equivalents.

Claims

1. A method implemented by a data processing system for discovering semantic meaning of data included in a field of one or more data sets, the method comprising: identifying a field included in one or more data sets, wherein the field is associated with an identifier; as well as For this field: parsing, by a data processing system, one or more data values ​​of the field to generate a data profile; Access multiple tabs to propose tests; generating a set of label proposals based on applying at least the plurality of label proposal tests to the data profile, wherein the label proposal tests are configured to generate label proposals including labels proposed to provide semantic meaning for one or more data values ​​of the field; determining similarities between label proposals in the set of label proposals; A classification is selected based at least on the similarity between label proposals in the group; presenting a graphical user interface that requests input when identifying a tag proposal that identifies a semantic meaning based on the classification or determining that no input is required; identifying one of the label proposals as identifying the semantic meaning; and storing an identifier of the field in a data store along with an identified one of the label proposals that identifies the semantic meaning; receiving a request to process the one or more data sets including a specific identifier; In response to the request, accessing from the data store the specific identifier and the specific tag associated with the specific identifier; determining, based on the particular tag, one or more data processing rules for processing at least a portion of the one or more data sets; and At least the portion of the one or more data sets is processed according to the selected one or more data processing rules.

2. The method of claim 1, wherein: Parsing the one or more data values ​​of the field includes determining a format of the data values ​​of the field.

3. The method of claim 1, wherein: Profiling the data values ​​of the field includes determining statistics representative of the data values ​​included in the field.

4. The method of claim 3, wherein: The statistical value includes at least one of a minimum length of data values ​​of the field, a maximum length of data values ​​of the field, a most common data value of the field, a least common data value of the field, a maximum data value of the field, and a minimum data value of the field.

5. The method of claim 1, wherein: Applying this multiple label proposal test includes: determining that the field comprises a primary key of a data set in the one or more data sets; and Select a tag proposal test associated with the primary key from among the multiple tag proposal tests.

6. The method of claim 1, wherein: Applying this multiple label proposal test includes: Performs a comparison of the data value of this field with the metadata of the terms in the glossary.

7. The method of claim 1, wherein: Applying this multiple label proposal test includes: determining a pattern of data values ​​stored in the field according to the data profile; determining the specific tags that are mapped to the pattern; and Label the field with that specific label.

8. The method of claim 1, wherein: Applying this multiple label proposal test includes: Retrieve a list of values ​​representing a collection of data; Compare the data value of the field with the list of values; In response to the comparing, determining that a threshold number of data values ​​match the values ​​in the list; and Responsive to the determination, the field is labeled with a particular tag that specifies the data set.

9. The method of claim 1, wherein: Applying this multiple label proposal test includes: Generate at least two labels for the field; and It is determined whether the at least two tags are mutually exclusive or mutually inclusive.

10. The method of claim 1, further comprising: Responsive to applying the plurality of label proposal tests, a relationship between the field and another field of the one or more data sets is determined.

11. The method of claim 10, wherein: The relationship includes one of indicating that a first data value of the field determines a second data value stored in another field, indicating that the first data value is related to the second data value, or indicating that the first data value is the same as the second data value.

12. The method of claim 1, wherein: The plurality of tag proposal tests are each associated with at least one weight value, the method further comprising: updating a weight value associated with at least one tag proposal test; and The tag proposal test is reapplied to the data profile using the updated weight values.

13. The method of claim 1, further comprising training the plurality of label proposal tests using a machine learning process.

14. The method of claim 1, further comprising: retrieving from a data quality rule environment one or more data quality rules assigned to the tag proposal specifying the semantic meaning; as well as A data quality rule of the one or more data quality rules is assigned to the field.

15. The method of claim 1, wherein: Comparisons of label proposals generated by these label proposal tests include: Apply a score value to each label proposal; For each of the label proposals, combining the score values ​​associated with the label; and The tags are sorted according to the score value associated with each tag.

16. The method of claim 1, further comprising: receiving validation of the label proposals from the plurality of label proposal tests; as well as In response to receiving the validation, the plurality of tag proposal tests are weighted with the tag proposals.

17. The method of claim 1, wherein: The data store includes a data dictionary.

18. The method of claim 1, further comprising: Output these label proposals to the data quality rules environment.

19. The method of claim 18, further comprising: Relative to the number of errors in processing the data of the field without using the identified one of the label proposals, another number of errors in processing the data of the field using data quality rules from the data quality rule environment is reduced based on the identified one of the label proposals.

20. A data processing system for discovering semantic meaning of fields included in one or more data sets, the system comprising: A data storage device storing instructions; as well as At least one processor is configured to execute instructions stored in the data storage device to perform the steps of any one of the methods of claims 1 to 19.

21. One or more non-transitory computer-readable media storing instructions that, when executed by a processor, implement the steps of the method of any one of claims 1 to 19.

22. A computer program product comprising computer instructions, which when executed by a processor implement the steps of the method according to any one of claims 1 to 19.

Images