Techniques for discovering and updating the semantic meaning of data fields

Abstract

1. A technique for discovering the semantic meaning of data in fields included in one or more datasets, comprising: identifying a first field having a previously assigned label that indicates the semantic meaning of the first field; identifying a set of one or more candidate labels for possible assignment to the first field in place of the previously assigned label; evaluating whether the first candidate label should be assigned to the first field using the previously identified label score and the first candidate label score; wherein, when the first candidate label score is greater than the previously identified label score by at least a first threshold amount, the evaluating includes presenting the first candidate label to a user by generating an interface in which the user can provide input indicating whether the first candidate label should be assigned to the first field in place of the previously identified label.

Description

[Technical Field]

[0001] The present disclosure relates to computerized discovery of the semantic meaning of data. More particularly, the present disclosure relates to techniques for updating the semantic meaning of data. [Background technology]

[0002] An organization (e.g., a multinational corporation, e-commerce company, or other business) may store its data in a dataset that includes multiple fields. The data stored in each of the fields may represent a particular type of information. For example, a dataset may store a user's address information, including a zip code field, a street name field, a house number field, a city field, a state field, and a country field. In another example, a dataset may store information about bank accounts in an account number field, an account holder field, and a balance field. Summary of the Invention [Problem to be solved by the invention]

[0003] Systems and processes such as those described in this disclosure may include one or more of the following advantages: In many systems, existing datasets can be very large and undocumented. In some cases, when datasets are labeled, the labels may be inconsistent or technical in nature, making it difficult to ascertain business information from the labels. Due to legacy systems and system versioning over many years, it can be difficult to identify which parts of a dataset are of practical relevance. Annotations for such datasets need to be developed. [Means for solving the problem]

[0004] These systems and processes are configured to provide labels for datasets quickly and, in some cases, automatically. The systems and processes described herein are configured to discover, classify, and label data content of a dataset by performing a series of statistical checks on the metadata and data content within the dataset. Discovering data content includes identifying data content (or data fields) within the dataset. For example, the systems and processes described herein provide for labeling data fields (e.g., columns in a table or database) that are unlabeled or labeled in a meaningless manner. These systems and processes are configured to discover the presence of specific data fields within a dataset and label the specific data fields. To perform this labeling, the systems and processes described herein are configured to map descriptive definitions to fields of the dataset, such as by assigning business terms or domains to the dataset fields. The labels generated for data fields provide information about the data content stored in that data field. The information can indicate what type of data content is contained in the data field (e.g., phone number, first name, date, etc.).

[0005] The information can be used by a computing system for a variety of purposes. For example, applications that can use the generated labels for a dataset may include data quality enhancement, personal data anonymization, data masking, personally identifiable information (PII) reporting, test data management, dataset annotation, etc. For example, comment data may contain PII and may require masking. In other examples, the systems and processes described herein can provide a mechanism for labeling third-party data from different sources. This can be beneficial for systems where system administrators need to know and understand what data is in the datasets stored in the system, for example, for regulatory purposes.

[0006] Additionally, the systems and processes described herein enable the creation of an infrastructure that allows data standards to be applied across many different data items from different data sources, which can be easily modified (e.g., by adding definitions to classification tests through a user interface) and additional tests can be added as new data requirements are identified.

[0007] In some embodiments, a method is provided for discovering the semantic meaning of data in fields included in one or more datasets, the method being performed by a data processing system, the method including: using the data processing system to identify a first field having previously assigned labels indicative of the semantic meaning of the first field, the previously assigned labels having corresponding previously identified label scores; identifying a set of one or more candidate labels for possible assignment to the first field in place of the previously assigned labels and a corresponding set of candidate label scores, the set of candidate labels including a first candidate label corresponding to a first candidate label score in the set of candidate label scores; and .... and evaluating whether a first candidate label should be assigned to the first field using the label score, wherein the evaluating includes presenting the first candidate label to a user by generating an interface in which the user can provide input indicating whether the first candidate label should be assigned to the first field instead of the previously identified label when the first candidate label score is greater than the previously identified label score by at least a first threshold amount, and bypassing presentation of the first candidate label to the user when the first candidate label score is not greater than the previously identified label score by at least a threshold amount.

[0008] In some embodiments, bypassing the presentation of the first candidate label to the user includes bypassing the generation of an interface in which the user can provide input indicating whether the first candidate label should be assigned to the first field instead of a previously identified label.

[0009] In some embodiments, the method includes identifying a second field that is different from the first field and that does not have an assigned label that indicates a semantic meaning of the second field, wherein one or more previously identified candidate labels were previously identified for possible assignment to the second field along with a corresponding set of previous candidate label scores, and none of the previously identified candidate labels have been assigned to the second field; and identifying a second set of one or more candidate labels for possible assignment to the second field and a corresponding second set of candidate label scores, wherein the second set of candidate labels is a second candidate label in the second set of candidate label scores. and evaluating whether the second candidate label should be assigned to the second field using the previous candidate label score and the second candidate label score, wherein the evaluating includes presenting the second candidate label to a user by generating an interface in which the user can provide input indicating whether the second candidate label should be assigned to the second field when the second candidate label score is greater than each of the previous candidate label scores by at least a second threshold amount, and bypassing presentation of the second candidate label to the user when the second candidate label score is not greater than each of the previous candidate label scores by at least a threshold amount.

[0010] In some embodiments, the method further includes identifying a third field that does not have an assigned label that indicates a semantic meaning of the third field, wherein the candidate label has not been previously identified for possible assignment to the third field; identifying a third set of one or more candidate labels for possible assignment to the third field and a corresponding third set of candidate label scores, wherein the third set of candidate labels includes a third candidate label that corresponds to a third candidate label score in the third set of candidate label scores; and evaluating whether the third candidate label should be assigned to the third field, wherein the evaluating includes presenting the third candidate label to a user by generating an interface in which the user can provide input indicating whether the third candidate label should be assigned to the third field.

[0011] In some embodiments, identifying one or more sets of candidate labels and corresponding sets of candidate label scores for possible assignment to the first field includes applying a plurality of label proposal tests to the profile of the first field to obtain a plurality of label proposals and corresponding label proposal scores, and using the plurality of label proposals and corresponding label proposal scores to identify one or more sets of candidate labels and corresponding sets of candidate label scores. In some embodiments, the plurality of label proposal tests are associated with a plurality of weight values, and identifying the set of candidate label scores includes using the plurality of label proposal scores and the plurality of weight values ​​to identify the set of candidate label scores.

[0012] In some embodiments, the method further includes, in response to receiving user input indicating assignment of the first candidate label to the first field, storing, in the data store, an identifier of the first field in association with the first candidate label to indicate a semantic meaning of the first field.

[0013] In some embodiments, the method further includes, using a data processing system, after evaluating whether the first candidate label should be assigned to the first field, identifying a label to be assigned to the first field, associating one or more attributes with the first field based on the label assigned to the first field, the one or more attributes indicating that data in the first field includes personally identifiable information (PII), and processing at least some data from the first field based on the one or more attributes. In some embodiments, processing at least some data from the first field based on the one or more attributes includes masking PII in the data from the first field.

[0014] In some embodiments, the method further includes using a data processing system to evaluate whether the first candidate label should be assigned to the first field, and then running at least one software application that applies data quality, anonymizes PII, and / or data masks personally identifiable information (PII) using the label assigned to the first field. In some embodiments, the label assigned to the first field indicates that the first field contains PII.

[0015] In some embodiments, the method further includes, after using the data processing system to evaluate whether a first candidate label should be assigned to the first field, receiving a request to process data managed by the data processing system, and, in response to the request, accessing the label assigned to the first field, identifying one or more data processing rules for processing the data based on the label assigned to the first field, and processing the data in accordance with the identified one or more data processing rules. In some embodiments, the data includes data stored in the first field. In some embodiments, identifying one or more data processing rules for processing the data based on the label assigned to the first field includes identifying, in a label index data structure, the label assigned to the first field; and identifying from the label index data structure, one or more data processing rules associated with the label identified in the label index data structure, wherein the one or more data processing rules are associated with the label in the label index data structure. In some embodiments, the label assigned to the first field indicates that the data stored in the first field includes PII, and the one or more data processing rules include one or more data masking rules for masking the PII. In some embodiments, processing the data in accordance with the identified one or more data processing rules includes masking the PII contained in the first field without scanning the data in the first field to identify whether it includes PII. In some embodiments, the label assigned to the first field indicates that the data stored in the first field includes PII, and the one or more data processing rules include one or more data anonymization rules for anonymizing the PII in the first field.In some embodiments, processing the data in accordance with the identified one or more data processing rules includes processing the data in accordance with the identified one or more data processing rules without parsing the data stored in the first field.

[0016] In some embodiments, a data processing system for discovering semantic meaning of data in fields included in one or more datasets is provided, the data processing system including at least one processor and at least one non-transitory computer-readable storage medium storing instructions that, when executed by the at least one processor, cause the at least one processor to perform a method, the method including: identifying a first field having a previously assigned label indicative of a semantic meaning of the first field, the previously assigned label having a corresponding previously determined label score; and identifying a set of one or more candidate labels for possible assignment to the first field in place of the previously assigned label and a corresponding set of candidate label scores, the set of candidate labels being the first candidate label score in the set of candidate label scores. and evaluating whether the first candidate label should be assigned to the first field using the previously identified label score and the first candidate label score, wherein the evaluating includes presenting the first candidate label to a user by generating an interface in which the user can provide input indicating whether the first candidate label should be assigned to the first field instead of the previously identified label when the first candidate label score is greater than the previously identified label score by at least a first threshold amount, and bypassing presentation of the first candidate label to the user when the first candidate label score is not greater than the previously identified label score by at least a threshold amount.

[0017] In some embodiments, at least one non-transitory computer-readable storage medium is provided that stores instructions that, when executed by at least one processor of a data processing system, cause the at least one processor to perform a method for discovering meaning of data in fields included in one or more datasets, the method including: identifying a first field having a previously assigned label that indicates a semantic meaning of the first field, the previously assigned label having a corresponding previously determined label score; identifying a set of one or more candidate labels for possible assignment to the first field in place of the previously assigned label and a corresponding set of candidate label scores, the set of candidate labels including a first candidate label that corresponds to a first candidate label score in the set of candidate label scores; and determining a value for the previously determined label score and the first candidate label score. and evaluating whether a first candidate label should be assigned to the first field using a label score generator, wherein the evaluating includes presenting the first candidate label to a user by generating an interface in which the user can provide input indicating whether the first candidate label should be assigned to the first field instead of the previously identified label when the first candidate label score is greater than the previously identified label score by at least a first threshold amount, and bypassing presentation of the first candidate label to the user when the first candidate label score is not greater than the previously identified label score by at least a threshold amount.

[0018] In one aspect, the system is configured to: identify a field included in one or more datasets, the field having an identifier; for the field, by a data processing system, profile one or more data values ​​of the field to generate a data profile; access a plurality of label proposal tests; generate a set of label proposals based on applying at least the plurality of label proposal tests to the data profile; identify similarities between the label proposals in the set of label proposals; select a classification based on at least the similarities between the label proposals in the set; render a graphical user interface requiring input in identifying a label proposal that identifies semantic meaning or identify that no input is required based on the classification; identify one of the label proposals as identifying the semantic meaning; and store, in a data store, an identifier for the field together with the identified one of the label proposals that identifies its semantic meaning. The functionality of the preceding sentence is intended to illustrate a general technical environment of the present invention.

[0019] In some implementations, profiling one or more data values ​​of a field includes identifying a format of the data values ​​of the field. Profiling the data values ​​of a field includes identifying statistics representative of the data values ​​contained in the field. The statistics include at least one of a minimum length of data values ​​of the field, a maximum length of data values ​​of the field, a most common data value of the field, a least common data value of the field, a maximum data value of the field, and a minimum data value of the field. Applying a plurality of label suggestion tests includes identifying that the field includes a primary key for a dataset of one or more datasets and selecting a label suggestion test of the plurality of label suggestion tests associated with the primary key. Applying a plurality of label suggestion tests includes performing a metadata comparison of the data values ​​of the field with terms in a glossary. Applying a plurality of label suggestion tests includes identifying, from the data profile, a pattern represented by stored data values ​​of the field, identifying a specific label that maps to the pattern, and labeling the field with the specific label. Applying the multiple label suggestion test includes retrieving a list of values ​​representing the dataset, comparing the data values ​​of the field to the list of values, identifying a threshold number of data values ​​that match the values ​​in the list in response to the comparing, and labeling the field with a particular label that designates the dataset in response to the identifying. Applying the multiple label suggestion test includes generating at least two labels for the field and identifying whether the at least two labels are mutually exclusive or inclusive.

[0020] In some implementations, the system is configured to perform an operation, in response to applying the plurality of label suggestion tests, further including identifying a relationship between the field and another field of the one or more datasets. The relationship includes one of an indication that a first data value of the field determines a second data value stored in the other field, an indication that the first data value correlates with the second data value, or an indication that the first data value is identical to the second data value. The plurality of label suggestion tests are each associated with at least one weight value, and the method further includes updating the weight value associated with the at least one label suggestion test and reapplying the label suggestion test to the data profile using the updated weight value.

[0021] In some implementations, the operations further include training a plurality of label proposal tests using a machine learning process. The operations include retrieving one or more data quality rules from a data quality rule environment to assign to the proposed label that specifies a semantic meaning, and assigning a data quality rule of the one or more data quality rules to the field. Comparing the proposed labels generated from the label proposal tests includes applying a score value to each proposed label, and for each label of the proposed labels, combining the score values ​​associated with that label, and ranking the labels according to the score value associated with each label.

[0022] In some implementations, the operations further include receiving validation of the proposed label from the plurality of label proposal tests and weighting the plurality of label proposal tests with the proposed label in response to receiving the validation. The data store includes a data dictionary. In some implementations, the operations further include outputting the proposed label to a data quality rule environment. In some implementations, the operations further include reducing, based on the identified one of the proposed labels, a number of errors associated with processing data for the field using data quality rules from the data quality environment relative to another number of errors associated with processing data for the field without using the identified one of the proposed labels.

[0023] In one aspect, a system includes a data storage storing instructions; and at least one processor configured to execute the instructions stored by the data storage to perform operations including identifying a field included in one or more datasets, the field having an identifier; for the field, profiling, with a data processing system, one or more data values ​​of the field to generate a data profile; accessing a plurality of label proposal tests; generating a set of label proposals based on applying at least the plurality of label proposal tests to the data profile; identifying similarities between the label proposals in the set of label proposals; selecting a classification based on at least the similarities between the label proposals in the set; rendering a graphical user interface requiring input in identifying label proposals that identify semantic meaning or identifying that no input is required based on the classification; identifying one of the label proposals as identifying semantic meaning; and storing the identifier of the field in a data store together with the identified one of the label proposals that identify semantic meaning.

[0024] In some implementations, profiling one or more data of a field includes identifying a format of data values ​​of the field. Profiling data values ​​of a field includes identifying statistics representative of data values ​​contained in the field. The statistics include at least one of a minimum length of data values ​​of the field, a maximum length of data values ​​of the field, a most common data value of the field, a least common data value of the field, a maximum data value of the field, and a minimum data value of the field. Applying a plurality of label suggestion tests includes identifying that the field includes a primary key for a dataset of one or more datasets and selecting a label suggestion test of the plurality of label suggestion tests associated with the primary key. Applying a plurality of label suggestion tests includes performing a metadata comparison of data values ​​of the field with terms in a glossary. Applying a plurality of label suggestion tests includes identifying a pattern from the data profile representative of stored data values ​​of the field, identifying a specific label that maps to the pattern, and labeling the field with the specific label. Applying the multiple label suggestion test includes retrieving a list of values ​​representing the dataset, comparing the data values ​​of the field to the list of values, identifying a threshold number of data values ​​that match the values ​​in the list in response to the comparing, and labeling the field with a particular label that designates the dataset in response to the identifying. Applying the multiple label suggestion test includes generating at least two labels for the field and identifying whether the at least two labels are mutually exclusive or inclusive.

[0025] In some implementations, the system is configured to perform an operation in response to applying the plurality of label suggestion tests, further including identifying a relationship between the field and another field of the one or more datasets. The relationship includes one of an indication that a first data value of the field determines a second data value stored in the other field, an indication that the first data value correlates with the second data value, or an indication that the first data value is identical to the second data value. The plurality of label suggestion tests are each associated with at least one weight value, and the method further includes updating the weight value associated with the at least one label suggestion test and reapplying the label suggestion test to the data profile using the updated weight value.

[0026] In some implementations, the operations further include training a plurality of label proposal tests using a machine learning process. The operations include retrieving one or more data quality rules from a data quality rule environment to assign to the label proposals that specify semantic meanings, and assigning a data quality rule of the one or more data quality rules to the field. Comparing the label proposals generated from the label proposal tests includes applying a score value to each label proposal, combining, for each label of the label proposals, the score values ​​associated with that label, and ranking the labels according to the score value associated with each label.

[0027] In some implementations, the operations further include receiving validation of the proposed label from the plurality of label proposal tests and, in response to receiving the validation, weighting the plurality of label proposal tests with the proposed label. The data store includes a data dictionary. In some implementations, the operations further include outputting the proposed labels to a data quality rule environment. In some implementations, the operations further include reducing, based on the identified one of the proposed labels, a number of errors associated with processing data for the field using data quality rules from the data quality environment relative to another number of errors associated with processing data for the field without using the identified one of the proposed labels.

[0028] In one aspect, the system includes means for identifying fields included in one or more datasets, the field having an identifier; means for profiling, by a data processing system, one or more data values ​​of the field for the field to generate a data profile; means for accessing a plurality of label proposal tests; means for generating a set of label proposals based on applying at least the plurality of label proposal tests to the data profile; means for identifying similarities between the label proposals in the set of label proposals; means for selecting a classification based on at least the similarities between the label proposals in the set; means for rendering a graphical user interface requiring input in identifying label proposals that identify semantic meaning or for identifying that no input is required based on the classification; means for identifying one of the label proposals as identifying semantic meaning; and means for storing the identifier of the field together with the identified one of the label proposals that identify semantic meaning in a data store.

[0029] In some implementations, profiling one or more data values ​​of a field includes identifying a format of the data values ​​of the field. Profiling the data values ​​of the field includes identifying statistics representative of the data values ​​contained in the field. The statistics include at least one of a minimum length of data values ​​of the field, a maximum length of data values ​​of the field, a most common data value of the field, a least common data value of the field, a maximum data value of the field, and a minimum data value of the field. Applying a multiple label suggestion test includes identifying that the field includes a primary key for a dataset of one or more datasets and selecting a label suggestion test of the multiple label suggestion tests associated with the primary key. Applying a multiple label suggestion test includes performing a metadata comparison of the data values ​​of the field with terms in a glossary. Applying a multiple label suggestion test includes identifying, from the data profile, a pattern represented by stored data values ​​of the field, identifying a specific label that maps to the pattern, and labeling the field with the specific label. Applying the multiple label suggestion test includes retrieving a list of values ​​representing the dataset, comparing the data values ​​of the field to the list of values, determining in response to the comparing that a threshold number of the data matches a value in the list, and labeling the field with a particular label that designates the dataset in response to the determining. Applying the multiple label suggestion test includes generating at least two labels for the field and determining whether the at least two labels are mutually exclusive or inclusive.

[0030] In some implementations, the system is configured to perform an operation, in response to applying the plurality of label suggestion tests, further including identifying a relationship between the field and another field of the one or more datasets. The relationship includes one of an indication that a first data value of the field determines a second data value stored in the other field, an indication that the first data value correlates with the second data value, or an indication that the first data value is identical to the second data value. The plurality of label suggestion tests are each associated with at least one weight value, and the method further includes updating the weight value associated with the at least one label suggestion test and reapplying the label suggestion test to the data profile using the updated weight value.

[0031] In some implementations, the operations further include training a plurality of label proposal tests using a machine learning process. The operations include retrieving one or more data quality rules from a data quality rule environment to assign to the proposed label that specifies a semantic meaning, and assigning a data quality rule of the one or more data quality rules to the field. Comparing the proposed labels generated from the label proposal tests includes applying a score value to each proposed label, and for each label of the proposed labels, combining the score values ​​associated with that label, and ranking the labels according to the score value associated with each label.

[0032] In some implementations, the operations further include receiving validation of the proposed label from the plurality of label proposal tests and weighting the plurality of label proposal tests with the proposed label in response to receiving the validation. The data store includes a data dictionary. In some implementations, the operations further include outputting the proposed label to a data quality rule environment. In some implementations, the operations further include reducing, based on the identified one of the proposed labels, a number of errors associated with processing data for the field using data quality rules from the data quality environment relative to another number of errors associated with processing data for the field without using the identified one of the proposed labels.

[0033] Other features and advantages of the system will become apparent from the following description and from the claims. [Brief explanation of the drawings]

[0034] [Figure 1A-1B] 1 is a block diagram of a system for discovering, classifying, and labeling data fields by analyzing data profiles generated from data in the data fields. [Figures 2A-2E] FIG. 2 is a block diagram illustrating how the execution system 202 profiles, classifies, and labels data fields to identify their semantic meaning. [Figure 3A-3G] FIG. 1 is a block diagram of a process for profiling, classifying, and labeling data fields to identify their semantic meaning. [Figure 4] FIG. 1 is a flow diagram illustrating operations performed by a logic engine of an execution system to discover, classify, and label data fields by analyzing data profiles generated from the data in the data fields. [Figure 5] FIG. 1 is a block diagram of a process for profiling, classifying, and labeling data fields to identify their semantic meaning. [Figure 6A] 1 is a depiction of a user interface showing the results of a data profiling process. [Figure 6B] 1 is a depiction of a user interface showing the results of the classification process. [Figure 6C] 10 is a depiction of a user interface showing recommendations for labels for discovered data fields. [Figure 6D] 10 is a depiction of a user interface showing an example of the results of labeling data fields. [Figure 6E] FIG. 10 is a block diagram illustrating a semantic label update module of the result validation module 108. [Figure 6F] 6F is a depiction of labeling previously labeled fields performed by the semantic label update module of FIG. 6E. [Figure 6G] 6F is a depiction of the reassignment of labels to previously labeled fields performed by the semantic label update module of FIG. 6E. [Figure 6H] 1 is a depiction of a user interface in which a user can provide input indicating assignment of candidate labels to fields. [Figure 6I] 10 is a depiction of a user interface in which the assignment of a candidate label to a field can be accepted or rejected. [Figure 7] FIG. 1 is a flow diagram illustrating an exemplary process for discovering, classifying, and labeling data fields by analyzing data profiles generated from data in the data fields. [Figure 8] FIG. 10 is a flow diagram illustrating an example process for updating the semantic labels of a field. [Figure 9] 1 illustrates an example of a suitable computing system environment on which the techniques described herein may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0035] Described herein are techniques for assigning labels to fields of one or more datasets that indicate the semantic meaning of the fields. The techniques perform various labeling processes to identify labels for the fields. For example, the labeling process for the fields may include profiling data from the fields, performing tests to identify candidate labels, and generating a graphical user interface (GUI) to obtain user input for the candidate labels.

[0036] Labeling of fields in a dataset needs to be updated periodically to accommodate various dynamic factors, such as changes in the data stored in the field, new available labels that may be assigned to the field, and / or updates to the labeling process. The techniques described herein intelligently identify whether the labeling process for a field should be repeated to reduce the amount of computation required to update the labels. The techniques may eliminate some or all of the processes involved in relabeling a previously labeled field. For example, the techniques may eliminate profiling data values ​​in the field and performing tests to identify candidate labels for the previously labeled field. As another example, the techniques may bypass the generation of a GUI to obtain input regarding candidate labels for the previously labeled field.

[0037] 1A-1B, a networked system 100 for discovering, classifying, and labeling data profiles generated from data fields is shown. The networked system 100 includes an execution system 102, one or more data sources 112, a reference database 116, and a data dictionary database 114. The execution system 102 is configured to receive data from data sources 112a, 112b (collectively referred to as data sources 112) within the networked system 100. The execution system 102 is configured to profile the source data received from the data sources 112 to generate a data profile representing the source data of the data sources 112. The execution system 102 is configured to analyze the profile data to discover and classify (e.g., test) portions of the source data and associate them with labels representing the semantic meaning of these portions of the source data. Portions of the source data may include fields within the source data. In other words, the execution system 102 uses the profile data to identify the semantic meaning of one or more fields of the source data using multiple classification tests. Classifying source data involves associating a probability with that portion of the source data. The probability indicates the likelihood that that portion of the source data corresponds to a label contained in the data dictionary database 114. Generally, the execution system 102 includes a profile data module 104, a classification module 105, a test module 106, a result validation module 108, and a load data module 110, which together are configured to discover, classify, and label data fields of data received from the data source 112. A general description of the labeling process (also referred to as the semantic discovery process) performed by the execution system 102 is now provided. The profile data module 104 receives the source data 112 and generates profile data. As described below, the profile data includes metadata describing statistical attributes of the source data and the fields of the source data. The profile data may also include data specifying relationships (e.g., statistical correlations, dependencies, and so on) between fields of the source data.

[0038] The classification module 105 classifies each field as having a certain data type (e.g., date field, number field, string, etc.). The classification module checks the reference database 116 for a label index data structure (also referred to herein as a "label index") to identify whether the field (or other fields with the same field name) has already been labeled with a semantic label. The label index data structure may also be referred to herein as a "label index."

[0039] Generally, each field in a dataset is associated with an identifier, such as a field name. The identifier may also include a field address, which specifies the location of the field within the dataset. For example, the identifier may specify which table the field is in and which column in that table represents the field. In some embodiments, the identifier is part of a label index, which associates the identifier for each field with the semantic label for those fields. The identifier may be the name of the field, a key value, or any other value that identifies the field.

[0040] The test module 106 includes multiple classifiers, each configured to operate on field data values, field names, and profile data for each field. Each classifier (also called a test) in the test module 106 uses a different method to propose labels to be associated with a field, as described in more detail below. A test can use a dictionary of labels from the label dictionary database 114 as a label library, from which it can propose one or more labels to identify the semantic meaning of the field being tested. The tests may propose different labels from each other because they use different methods, or the proposed labels from all tests may be the same. The proposed labels (and their scores, if applicable) from each test in the test module 106 are sent to the result validation module 108.

[0041] The result validation module 108 buckets the proposed labels for a field into categories based on the amount of similarity between the proposed labels for the test. If the proposed labels are all the same, the amount of similarity is high and a match is found. The result validation module 108 can also categorize the group of proposed labels for each field from the test module 106 according to recommendations (one of the proposed labels is the recommended label). The result validation module 108 can determine that there is little similarity between the proposals and suggest that the user investigate the results further. The result validation module 108 can also categorize proposed labels from the test module 106 as to be ignored, as described below. The execution system 102 can receive validation data (e.g., from a user through a user interface) that confirms that the proposed label correctly identifies the semantic meaning of the field. Once validated, the proposed label is assigned to the field, and the label index is updated to indicate that the field is associated with the assigned label. The load data module 110 transmits the label index to a reference database 116 for access by future iterations of the labeling process (also referred to as the semantic discovery process) or downstream applications. The functions of modules 104, 105, 106, 108, and 110 are described in further detail below with respect to Figures 2A-3G.

[0042] In some embodiments, the execution system 102 may be used to discover semantic meaning for data stored in a large number (e.g., thousands, tens of thousands, hundreds of thousands, or millions) of fields in a dataset. To do so, the execution system 102 may identify, for each of the fields, a label that indicates (e.g., identifies) the semantic meaning of the field. Labeling the fields may include various operations described herein for identifying one or more candidate labels for the field and assigning the candidate labels to the field to indicate its semantic meaning. For example, the execution system 102 may present one or more candidate labels in a user interface to request user input identifying the candidate labels that indicate the semantic meaning of the field. In another example, the execution system 102 may perform an operation (e.g., categorization) to automatically assign candidate labels to the field to indicate the semantic meaning of the field.

[0043] In some embodiments, after identifying a label for a field, the execution system 102 may identify the labeling process for the field as complete. In other words, the execution system 102 may identify the field as “closed.” In some embodiments, the execution system 102 may identify labeling for a field as complete or “closed” in one of the following situations: (1) when a label is assigned to the field (e.g., automatically or based in part on user input), (2) when all proposed labels for the field are rejected (e.g., by the user), or (3) when no proposed labels were generated for the field (e.g., such that the user was never even presented with proposed labels). In some embodiments, once the execution system 102 completes labeling for a field, the execution system 102 may bypass certain actions for the field in future iterations of labeling. For example, the execution system 102 may bypass presenting candidate labels in a user interface to request user input identifying candidate labels that indicate the semantic meaning of the field. By bypassing certain operations for fields, the system reduces the calculations that are performed to complete the labeling of the fields.

[0044] However, after the labeling process for a particular field is completed by the execution system 102, various factors may change that potentially affect the identification and assignment of labels. For example, the data stored in the field may change, the tests performed by the execution system 102 to identify proposed labels may be modified, and / or the execution system 102 may be updated with new labels that more accurately indicate the semantic meaning of the field. To accommodate these factors, the execution system 102 typically iterates the labeling for the field to ensure that the best possible labels are assigned to the field to indicate its semantic meaning. Labeling a field includes (1) applying label proposal tests to the data profile, (2) using the results of the label proposal tests to identify a set of one or more candidate labels and corresponding candidate label scores, and (3) generating an interface through which a user can provide input regarding the candidate labels, when user input is required.

[0045] Repeating the entire labeling process for all fields is computationally expensive because it requires continuous identification and evaluation of candidate labels. Repeating the labeling process for all fields often requires repeating the labeling process performed in previous iterations of labeling. For example, the execution system 102 repeatedly profiles the data values ​​of the fields, runs label proposal tests to identify candidate labels, and generates one or more user interfaces to present the candidate labels for the fields in a user interface to request the user to review the candidate labels and specify whether a new label for the field should be selected. These processes consume computational resources. The execution system 102 may repeatedly perform the labeling process for hundreds or thousands of fields and repeatedly obtain the same labeling results for most of the fields to accommodate dynamic factors (e.g., updated data, updated label proposal tests, and / or new labels). The repeated labeling process increases the time and computational resources required to complete semantic discovery for a dataset without significantly affecting the labeling results.

[0046] Therefore, the inventors have developed a technique that more efficiently accommodates the dynamic nature of data and the execution system 102. Instead of repeating the entire labeling process for each field, this technique allows the execution system 102 to intelligently identify whether a particular labeling process needs to be performed. The technique may identify whether to perform data profiling, perform label suggestion testing, and / or generate a user interface for a field in which a user can provide input regarding a candidate label. The execution system 102 makes the identification based on whether a stronger labeling result potentially exists for the field. When the execution system 102 determines that it can potentially identify a stronger labeling result, the system 102 may repeat the particular labeling process. For example, the execution system 102 may bypass profiling the data and / or performing label suggestion testing if the execution system 102 is sufficiently confident in a previously assigned label (e.g., if the score corresponding to the previously assigned label is sufficiently high). As another example, the system may generate a user interface in which a user can provide input regarding a candidate label for a field only when it detects that there is another candidate label for the field that is sufficiently better than the previously assigned label.

[0047] In other words, the execution system 102 determines to “reopen” the field. The execution system 102 may then evaluate one or more candidate labels for the field (e.g., by performing label suggestion tests to identify candidate labels and / or presenting the candidate labels in a user interface for user input). When the execution system 102 determines that it cannot identify a stronger labeling result, the system 102 does not repeat the labeling process (e.g., performing label suggestion tests and / or generating a user interface), thereby conserving computational resources. In this way, when a different label assignment result for the field is unlikely, the execution system 102 can update the labels assigned to the field to indicate their semantic meaning without having to repeat the entire labeling process for the field.

[0048] As an illustrative example, during one labeling iteration of the execution system 102, the "Address ID" field of the dataset may be labeled with the term "Address." Prior to a subsequent labeling iteration of the execution system 102, one or more tests used by the execution system 102 to identify proposed labels may be modified. In a next iteration of the execution system 102, the execution system 102 may identify a new candidate label, "Zip Code," that better indicates the semantic meaning of the "Address ID" field (e.g., as reflected by the corresponding score for the label). In response to identifying a new candidate label that better indicates the semantic meaning of the field, the execution system 102 may generate a user interface in which a user can provide input indicating whether the candidate label should replace the previously assigned label to the field.

[0049] In some embodiments, the labels may indicate the semantic meaning of the data in the respective fields. Accordingly, the labels may also be referred to herein as "semantic labels." The labels may be, for example, recognized terms or sets of terms that may appear in a data dictionary. In some embodiments, the labels may associate fields with sets of attributes (e.g., in addition to or instead of indicating the semantic meaning of the data in the field). For example, a label index may associate fields with data objects that store one or more attributes.

[0050] In some embodiments, one or more attributes associated with a field by a semantic label may be used in processing data values ​​in the field. In some embodiments, the attribute may be used by a running software application. For example, the attribute may indicate that the field stores personally identifiable information (PII). When the attribute indicates that the field stores PII, the software application may mask data values ​​from the field to protect the PII. As another example, the software application may anonymize personal data from the field. In some embodiments, the software application may use the attribute associated with the field to perform data quality enforcement. For example, the attribute may indicate that values ​​in the field must adhere to a particular format or standard (e.g., a standard format for phone numbers, addresses, social security numbers, dates of birth, or other types of information). As part of its processing, the software application may determine whether values ​​in the field meet a particular format or standard based on the attribute.

[0051] Some embodiments provide a data processing system for discovering the semantic meaning of data in fields included in one or more datasets. The data processing system may be configured to: (1) identify a first field (e.g., a row in a table) having a previously assigned label that indicates the semantic meaning of the first field (e.g., the semantic meaning of a data value stored in the row), the previously assigned label having a corresponding previously determined label score; (2) identify a set of one or more candidate labels and a corresponding set of candidate label scores for possible assignment to the first field in place of the previously assigned label (e.g., by applying a label proposal test to a profile of the first field to obtain the candidate labels and corresponding candidate label scores), the set of candidate labels including a first candidate label that corresponds to a first candidate label score in the set of candidate label scores; and (3) evaluate whether to assign the first candidate label to the first field using the previously determined label scores and the first candidate label score. The evaluating includes (1) presenting the first candidate label to the user by generating an interface (e.g., a graphical user interface (GUI)) in which the user can provide input indicating whether the first candidate label should be assigned to the first field instead of the previously identified label when the first candidate label score is greater than the previously identified label score by at least a first threshold amount, and (2) bypassing presentation of the first candidate label to the user (e.g., by bypassing generation of the interface in which the user can provide input indicating whether the first candidate label should be assigned to the first field instead of the previously identified label) when the first candidate label score is not greater than the previously identified label score by at least a threshold amount.

[0052] In some embodiments, the data processing system may be configured to: (1) identify a second field that is different from the first field and that does not have an assigned label that indicates the semantic meaning of the second field, wherein one or more previously identified candidate labels were previously identified for possible assignment to the second field along with a corresponding set of previous candidate label scores, and none of the previously identified candidate labels have been assigned to the second field; (2) identify a second set of one or more candidate labels for possible assignment to the second field and a corresponding second set of candidate label scores, wherein the second set of candidate labels includes a second candidate label that corresponds to a second candidate label score in the second set of candidate label scores; and (3) evaluate whether to assign a second candidate label to the second field using the previous candidate label scores and the second candidate label score. The evaluating includes (1) presenting the second candidate label to the user by generating an interface in which the user can provide input indicating whether the second candidate label should be assigned to the second field when the second candidate label score is greater than each of the previous candidate label scores by at least a second threshold amount, and (2) bypassing presentation of the second candidate label to the user (e.g., by bypassing generation of the interface in which the user can provide input indicating whether the second candidate label should be assigned to the second field) when the second candidate label score is not greater than each of the previous candidate label scores by at least a threshold amount.

[0053] In some embodiments, the data processing system may be configured to: identify a third field that does not have an assigned label indicative of the semantic meaning of the third field, where the candidate label has not been previously identified for possible assignment to the third field (e.g., because previous label suggestion tests did not result in any candidate labels or all candidate labels were rejected by the user); identify a third set of one or more candidate labels for possible assignment to the third field and a corresponding third set of candidate label scores, where the third set of candidate labels includes a third candidate label corresponding to a third candidate label score in the third set of candidate label scores; and evaluate whether the third candidate label should be assigned to the third field. The evaluating includes presenting the third candidate label to a user by generating an interface in which the user can provide input indicating whether the third candidate label should be assigned to the third field.

[0054] In some embodiments, the multiple label proposal tests are associated with multiple weight values, and identifying the set of candidate label scores includes identifying the set of candidate label scores using the multiple label proposal scores and the multiple weight values.

[0055] In some embodiments, the data processing may be configured, in response to receiving user input indicating assignment of the first candidate label to the first field, to store, in the data store, an identifier of the first field in association with the first candidate label to indicate a semantic meaning of the first field.

[0056] In some embodiments, the data processing system may be configured, after evaluating whether the first candidate label should be assigned to the first field, to: identify a label to be assigned to the first field; associate one or more attributes with the first field based on the label assigned to the first field, where the one or more attributes indicate that data in the first field includes personally identifiable information (PII); and process at least some data from the first field based on the one or more attributes (e.g., by masking the PII in at least some of the data, anonymizing the PII in at least some of the data, and / or restricting access to at least some of the data).

[0057] In some embodiments, the data processing system may be configured to, after evaluating whether a first candidate label should be assigned to the first field, execute at least one software application program that uses the label assigned to the first field to apply data quality, anonymize personal data, and / or data mask for personally identifiable information (PII) reporting.

[0058] In some embodiments, the data processing system may be configured to, after evaluating whether a first candidate label should be assigned to the first field, receive a request to process data managed by the data processing system, and in response to the request, access the label assigned to the first field, identify one or more data processing rules for processing the data based on the label assigned to the first field, and process the data in accordance with the identified one or more data processing rules. In some embodiments, the data includes data stored in the first field. In some embodiments, identifying the one or more data processing rules for processing the data based on the label assigned to the first field includes identifying the label assigned to the first field in a label index data structure and identifying from the label index data structure one or more data processing rules associated with the label identified in the label index data structure, wherein the one or more data processing rules are associated with the label in the label index data structure.

[0059] In some embodiments, the label assigned to the first field indicates that the data stored in the first field includes PII, and the one or more data processing rules include one or more data masking rules for masking the PII. In some embodiments, processing the data in accordance with the identified one or more data processing rules includes masking the PII included in the data stored in the first field without scanning the data in the first field to identify whether it includes PII.

[0060] In some embodiments, the label assigned to the first field indicates that the data stored in the first field includes PII, and the one or more data processing rules include one or more data anonymization rules for anonymizing the PII in the data stored in the first field.

[0061] In some embodiments, processing the data in accordance with the identified one or more data processing rules includes processing the data in accordance with the identified one or more data processing rules without parsing the data stored in the first field.

[0062] The data dictionary database 114 contains a list of labels, which are candidate labels for identifying the semantic meaning of a field in the source data. Each label in the data dictionary database 114 is associated with one or more attributes. The attributes describe the semantic meaning of that label and may include tags, associations, or other metadata that provide the execution system 102 with context for what the semantic meaning of each label is. For example, attributes may include statistical metadata describing the values ​​of the field, tags for specific value sets for that field (e.g., a list of city names, zip codes, etc.), a specified data format (e.g., data format), relationships between fields in a dataset, and so on. During the testing (classification) process, the execution system 102 is configured to determine how closely the attributes of the labels match the attributes of the fields in the source data. The classification test is configured to analyze the field's profile data and the content of the field itself to determine which labels in the data dictionary database 114 identify the semantic meaning of that field. The data dictionary database 114 can be populated with a list of labels and label attributes before the execution system 102 performs the semantic discovery process.

[0063] 1A , the data dictionary database 114 is extensible because new labels can be added to the data dictionary database 114 as data with new semantic meanings is processed by the execution system. In some implementations, a set of seed labels can be automatically provided to initialize the system, and additional labels can be added as needed if the existing labels in the data dictionary database 114 do not identify the semantic meaning of the field. For example, if the execution system 102 cannot find a label in the data dictionary database 114 with attributes that satisfy the classification tests (e.g., scores for each test are below a threshold), the execution system 102 can prompt the user to define a new label for that data dictionary database 114 as one with attributes that correspond to the unlabeled field. In subsequent tests, the execution system 102 is configured to suggest the newly defined label for labeling subsequent fields with similar attributes.

[0064] The execution system 102 is configured to automatically, and in response to user input, tag portions of source data with one or more labels that identify the semantic meaning of that portion of source data. The discovery, classification, and labeling of source data can be repeated to refine the classification of source data and increase the accuracy of the source data labeling. The source data labels include metadata associated with the source data as a label index, which is described below. Once the source data discovery, classification, and labeling are complete, the label index is loaded into the reference database 116 for use by one or more downstream applications.

[0065] As shown in FIG. 1B , once the profiling, classification, and labeling of source data has been performed by the execution system 102, the load data module 110 can send the data to each of the data dictionary database 114 and the reference database 116. If additional labels are defined during or as a result of the semantic discovery process, the newly defined labels are added to the data dictionary database 114 for use in future labeling by the execution system 102 (e.g., by the test module 106). Additionally or alternatively, the load data module 110 can send label index data to the reference database 116. The label index describes, for the source data, which labels are associated with each field of the source data that has already been labeled by the execution system 102. For example, for each label, the label index may include a list of each field associated with that label. Alternatively, the label index can describe, for each field of the source data, which labels are associated with that field. The label index is stored in the reference database 116. A subsequent labeling process by the execution system 102 can check the label index (e.g., by the classification module 105) to determine whether a particular field has already been labeled. This allows the execution system 102 to bypass the testing process and simply label the field with a known label, thereby reducing the amount of data processed to label the field. In some implementations, data indicating that a label has already been assigned to that field (or similar fields) can be used during the testing process to improve the accuracy of the test classifier, as described below.

[0066] In some implementations, the execution system 102 does not bypass tests for previously labeled fields. The execution system 102 performs tests to identify candidate labels and evaluates whether one of the candidate labels should be assigned to the field. Exemplary techniques for how the execution system 102 evaluates whether a candidate label should be assigned to a field are described herein with reference to Figures 6E-6I and 8.

[0067] The label index can be provided to or searched by one or more downstream applications for various uses. The label index provides a quick reference for downstream applications to identify the meaning of data values ​​in a dataset, without the downstream application having to parse the dataset. For example, an application need only refer to the label index to identify the semantic meaning of a field. The label index can indicate whether a particular field contains personally identifiable information (PII) and how the data in that field should be masked. The label index, or labels, can indicate what the schema of the dataset is (e.g., whether the field is a primary key or a foreign key). The label index can indicate, through the attributes of specified labels, what the semantic relationships between fields are (e.g., whether they are correlated with each other, whether they are dependent, etc.). Because this semantic meaning is contained in each attribute of the label, an application need only refer to the label index to search for semantic information, rather than scanning the dataset itself. Reducing the amount of data processed by downstream applications reduces processing resources for those applications because only the label index is required to identify the semantic meaning of each field in the dataset. In one example, the label index may indicate which fields in the dataset contain PII. Instead of scanning each field in the dataset, the application can mask only those fields that contain PII as needed. This allows the application to access the data store that stores the dataset less frequently, resulting in less data being transmitted (reducing bandwidth usage).

[0068] In some implementations, a data dictionary can also be provided to applications. The relatively small data footprint of the data dictionary and label index can be beneficial in situations where the size of the dataset is large. The size of the label index and data dictionary each grows slowly relative to the size of the dataset itself. Rather than having to search the entire dataset to provide the appropriate context for the application, the label index and data dictionary provide all the context an application needs to access only the relevant portions of the dataset. An application can simply reference the dataset's labels and label attributes to identify the context of a field within the dataset.

[0069] The source data of the data source 112 may include several types of data. In one example, the source data of the data source 112 includes a table of data fields. The execution system 102 is configured to discover, classify, and label the data fields of the table. For example, the execution system 102 analyzes the data content of each discovered field of the source data to determine what the data content of that data field represents. The execution system 102 classifies each data field by associating the data field with a known label (e.g., by assigning it a probability value). The labeled data of the table can be output to the data storage 114, where other applications and systems can access it to act on the labeled data. The process of discovering, classifying, and labeling data fields of a data table is described in more detail below. The data source 112 may include any type of computing system. For example, the data source 112 may include a mainframe, a database, unstructured data provided by a third party, a data lake, a personal computer, a high-scale network, and others.

[0070] In this disclosure, the process for discovering, classifying, and labeling data fields in a data table is used as an example to explain the functionality of the execution system 102 and the networked system 100. However, while data fields are an example of what the execution system 102 discovers, classifies, and detects, the execution system 102 can also operate on other types of data. For example, the execution system 102 receives application data from an application. The execution system 102 is configured to discover, classify, and label different data for the application. The data may include different files stored and accessed to run the application. The execution system 102 can be used to discover PII stored by the application, discover malware, modify application files, and so on. In another example, the execution system 102 can analyze files stored in a file system (e.g., on a personal computer). The execution system 102 can scan the file system to identify a specific subset of files defined by the user. For example, a user may want to retrieve work files from a home computer before reformatting the computer. The execution system 102 can be configured to scan the file system of a home computer and tag all work files. The execution system 102 can be configured to label data related to data subject rights such as the right to be forgotten, data erasure, subject access requests, data correction requests, data suspension, data likelihood, and consent. Many other applications are possible.

[0071] To discover, classify, and label portions of source data (e.g., data fields), the execution system 102 is configured to access a reference database of one or more files 118. The files 118 provide the execution system 102 with a context for performing source data discovery, classification, and labeling. In some implementations, the reference database 116 can store lookup tables that store relationships between values ​​found in data field entries. For example, the data storage may include a lookup table that matches codes with a glossary, which the execution system 102 can reference during source data discovery, classification, and labeling. The files 118 of the reference database 116 may include weight values ​​used in classification. For example, the weight values ​​may indicate to the execution system 102 the likelihood that two terms (e.g., business terms) are related to each other for the source data 112 being labeled. These values ​​may be generated by a user or automatically during an initial iteration of source data 118 discovery, classification, and labeling, and may be updated during subsequent iterations.

[0072] The files 118 can be defined before, during, or after the execution system 102 discovers, classifies, and labels source data, in an iterative process. A development environment (not shown) of the networked system 100 can provide a means by which a user can describe the reference database 116 or update the data storage files 118. For example, the development environment can include a user interface that provides feedback to a user of the execution system 102. For example, the user interface of the development environment can display reports that show how the execution system 102 is performing, such as which data fields have been labeled by the execution system 102 and how likely each classification is. Examples of feedback provided to the user and the user interface are described in more detail below.

[0073] Generally, the execution system 102 includes one or more processors configured to execute the logic of the profile data module 104, the classification module 105, the test module 106, the result validation module 108, and the load data module 110. The operations of each of the profile data module 104, the classification module 105, the test module 106, the result validation module 108, and the load data module 110 can be performed either by batch processing or in real time. Additionally, the execution system 102 can perform the operations of each of the modules 104, 105, 106, 108, and 110 substantially simultaneously or during different time periods. For example, in some implementations, the profile data module 104 generates profile data representing a profile of source data from the data source 112 at a first point in time. Then, once all data from data sources 112 for a period of time has been profiled by profile data module 104, classification module 105, testing module 106, result validation module 108, and load data module 110 can analyze the profile data to discover, classify, and label data fields in that source data and load the data into data storage 116 for one or more downstream applications.

[0074] 2A-2E are block diagrams illustrating how the execution system 202 profiles, classifies, and labels data fields to identify their semantic meaning. Referring to FIG. 2A, a process 200a for profiling a data source by the profile data module 104 is shown. The profile data module 104 is configured to receive (254) source data (e.g., a table, a file, etc.) and generate a data profile for the source data. The profile data module 104 discovers (256) fields in the source data (e.g., one or more data sets). The profile data module 104 can discover fields by identifying rows in a table of the source data, finding field names, referencing fields, or any similar process. The profile data module 104 identifies (258) statistical attributes of the data fields and generates (260) profile data including these statistical attributes. The profile data identifies patterns within the source data. More specifically, the profile data includes statistics about the values ​​of data fields in a table of the source data. For example, the profile data may include information specifying whether the data values ​​of the data fields include numeric data, strings of characters, etc. For example, statistics about the data values ​​may include the maximum, minimum, standard deviation, average, etc. (if the data is numeric) of the values ​​contained in each of the data fields. In some implementations, statistics about the data may include how many digits or characters there are in each entry of the data value. For example, the data profile may indicate that each data value in a data field contains seven (or ten) numeric characters, which may provide a contextual clue that the data field contains a phone number. For example, the data profile may indicate that each entry in a data field contains a value from a small set of values, which may be used to trigger a comparison with a lookup table by the test module 106.

[0075] Data ingestion for the profile data module 104 involves parsing the field names of the fields, the location of the fields within a table (or file in a file system), and parsing the schema of the data. In other words, data ingestion occurs at the field level, the dataset level, and the schema level.

[0076] At the field level, the profile data module 104 analyzes fields and the values ​​of their entries to generate profile data. The profile data module 104 can determine whether the values ​​of a field or its entries are null, blank, valid for a particular data type, etc. The profile data can include statistics on the percentage of nulls, the percentage of blanks, and the percentage of values ​​per field. The profile data module 104 can also generate data indicating changes in these percentages from a baseline percentage (which can be specified by a user through a development environment or automatically generated). In another example, the profile data can include an indication of whether an entry's data is valid for an implicit data type. For example, if a data field is known to be a string field but date data is seen, it can be inferred that the data is invalid for that entry. In another example, the profile data can include an indication that an entry's data is valid for a specified format (e.g., two decimal places are required, but no explicit type is specified). In some implementations, some of this functionality is performed by the test module 106 after the profile data is generated.

[0077] At the dataset level, the profile data module 104 can provide statistics relating portions of source data to the entire dataset of source data. For example, the profile data module 104 includes an indication of the uniqueness and cardinality of unique keys. The profile data module 104 indicates whether there are duplicates in key values ​​of the source data. The profile data module 104 indicates whether there are duplicates in approximate unique key matches. The profile data module 104 indicates a count of records that have one or more of these characteristics. In some implementations, this information is determined by the result validation module 108, for example, after fields have been identified as key values.

[0078] At the schema level, the profile data module 104 is configured to determine statistical information about data fields in the source data over time. For example, the profile data module 104 detects whether any columns have been added or removed in a new version of the source data. The profile data module 104 detects delimited fields. In some implementations, the profile data module 104 performs lineage counting to determine whether any records have been dropped from a previous version of the source data. Other schema evolutions can also be detected. In some implementations, this functionality is performed by the result validation module 108 after a data field is found in the profile data.

[0079] In some implementations, the profile data can be improved if contextual data is available in the reference database 116. For example, if a user specifies a format for one or more data fields of the source data, the profile data can generate additional statistical information about those data fields. This can be part of an iterative process. For example, if a data field is discovered in the first iteration (but not classified in that iteration), the user can view the data content of the data field and provide the execution system 102 with additional information for the profile data module 104 (and other data modules) to analyze. For example, if a user specifies that all data fields must contain only numeric data, the profile data module 104 can quickly identify which data is invalid and provide statistical measures of that information in the data profile.

[0080] The profile data module 104 generates profile data from source data by scanning the entire data set of the source data before generating the profile data. The profile data module 104 is not required to copy the entire data set from the source data, although it may do so in order to generate the profile data. Because data sets can be large (e.g., many gigabytes or even terabytes in size), copying large amounts of data to a local system for analysis can be impractical. Generally, the profile data module 104 scans the source data during periods of low activity on the data sources.

[0081] Referring to FIG. 2B, a process 200b for classifying data fields using profile data by the classification module 105 is shown. The classification module 105 is configured to receive (262) the profile data and receive (264) source data including fields. For each field of the source data, the classification module 105 is configured to look up (266) a label index containing existing labels for the discovered field of the source data (e.g., from the reference database 116). These labels may be obtained from a previous iteration of the labeling process, or a label index (e.g., an initial label index) may be manually generated, imported, or otherwise obtained. However, the label index need not exist prior to performing the labeling process.

[0082] For a field, the classification module determines (268) whether the field is already associated with a label in the label index. If the field is still unlabeled or no label index exists, the classification module 105 determines that a label is missing for the field. If necessary, the classification module 105 generates a new label index to be populated with semantic labels. The classification module 105 performs (270) a classification of the field data type. The classification can be based on the field's profile data, the field name, and the field's value. For example, the classification module 105 can identify the field as a "date" field. In some implementations, the classification module 105 can identify the field as a numeric field, a string field, or other such data type. While the classification module 105 determines the field's data type, the semantic meaning (and therefore the semantic label) of the field is determined by the test module 106, as described below. For example, the classification module 105 can identify the field as a date field, and the test module 106 can determine that the date in the date field is the customer's "date of birth." In another example, the test module 106 identifies a numeric field as a "User ID" field. Many other such examples are possible. The classification module 105 generates 272 classified data, which is sent to the test module 106 as classification output 276 for semantic meaning discovery. The classified data is tagged with the data type identified by the classification module 105.

[0083] If a label is found, the classification module generates label data (274), which may be communicated to the testing module 106 and the result validation module 108. The label data informs the testing module 106 and the result validation module 108 that the field has already been labeled. This can be used to weight the classifier applied to the field or to suggest a label. However, the field may be reclassified by the classification module 105 and retested by the testing module 106 to confirm that the label is correct, and perhaps update the label attributes for that label in the data dictionary database 114. For example, if the testing module 106 determines that the existing label is not a good fit, a new label may be suggested. If the user selects an existing label (e.g., when presented by the result validation module 108), the label data can be used as a flag to the execution system 102 that the label attributes for that label in the data dictionary database 114 do not represent the value of the data labeled by that label, and these attributes can be updated.

[0084] In some implementations, the classification module 105 can be updated using machine learning methods over multiple iterations. For example, if a discovered field is already labeled, the classifier can identify that further classification can be bypassed. In other examples, the score applied to a field can be updated based on additional data received from the data source 112 or from user input. The classification module 105 can identify that the testing module 108 should perform a different test compared to a previous iteration. For example, if fingerprint analysis was inconclusive in a previous iteration, the classifier can identify that this test should be bypassed (or replaced with another test) in subsequent iterations.

[0085] Referring to FIG. 2C , a process 200c is shown in which the test module 106 classifies a field as associated with a semantic label. The test module 106 is configured to classify the source data of the dataset using statistics in the profile data and additional contextual information (e.g., lookup table 442) provided in the reference database 116. The classification output 276 of the classification module 105 is used to provide context for the data type of each field and to provide existing labels (if any) for that field. The test module 106 is configured to receive candidate labels 440 from the data dictionary database 114. The candidate labels are a library of existing labels, each associated with one or more attributes that identify the semantic meaning of the data field (e.g., to downstream applications or users). As previously described, the attributes associated with each label in the data dictionary database 114 provide the semantic meaning of the label. The test module 106 identifies which of these candidate labels most closely associates with the attributes of the data field identified by the data profile, the field name, and the data values ​​for that field.

[0086] The test module 106 includes multiple tests (or classifiers), each run by a different test module, to associate one or more labels with each field under test. For example, after the fields of a dataset are discovered by the profile data module 104, the test module identifies how closely the attributes of the field correspond to the attributes of each candidate label. Each test utilizes different data and methods to propose one or more labels. Because different tests use different classification methods, the proposed labels from each test may not necessarily be the same. The proposed labels are validated in the result validation module 108, as described below. Using different tests to identify the semantic meaning of a field makes the identification of semantic meaning much more robust than using any one test, because it does not rely on any one element (e.g., the field name or the inclusion of a particular value or set of values ​​in that field) to indicate the semantic meaning of the field.

[0087] Testing the data in each data field may include one or more of the following: the population level of the data in the dataset (how frequently a value occurs in the data field), the discovered data type (e.g., dates or numbers are kept as strings), the domain of the data field, finding key fields, identifying whether the field is a single word or a descriptive field, and others. For example, classifying the data may include analyzing dates and times. The test module 106 thus receives profile data from the profile module 104 and performs a series of statistical functions to identify, classify, and test the field details against a set of known label types. The testing rules vary depending on the data type, which may be identified in the classification output 276 by the classification module 105 or, in some cases, included in the profile data generated by the profile data module 104.

[0088] The test module 106 is configured to perform several different classification tests on the field names and entries within the data fields to determine how to label the data fields. The test module 106 receives the classification output 276, the candidate labels 440, and any reference data 442 from the reference database 116 and subjects the data to tests. The tests include pattern analysis 410, business term analysis 414, fingerprint analysis 418, and keyword search 422. As mentioned above, while a table containing field names and field data is described as an illustrative example, the tests 410, 414, 418, and 422 of the test module 106 can be performed on other data types as well. Examples of classification tests performed on the data fields and data entries of the source data can include a fuzzy matching plan, a columnar data plan, a business term matching plan, keyword matching, fingerprinting (e.g., contextual data lookup), pattern matching, and validation.

[0089] The fuzzy matching logic of the test module 106 includes logic for fuzzy matching of field names from a dictionary of terms. Generally, fuzzy matching is configured to find matches between field names and terms in the dictionary if an exact match cannot be found. The system finds dictionary terms that are less than exact. For example, the test module 106 sets a fuzzy match threshold to a percentage less than 100, and the dictionary database (e.g., database 116) returns to its memory all matches that correspond to (e.g., greater than) that percentage. In some implementations, a probability score is assigned to each match. The probability score can be presented to the user.

[0090] The business term analysis 414 of the test module 106 includes logic for matching data field names with known business terms in a glossary. Generally, business terms can be placed into different contexts or groups of business terms. The test module 106 checks to find when specific words or terms appear within other words or terms. This may include referencing a specific specification. For example, the test module 106 receives a specification that lists different abbreviations for street names, such as "st," "ln," "ave," "pl," and "ct." The test module 106 checks to determine whether any of these abbreviations are included within the data field. If one or more of the data elements in the specification are included within the data field, the test module 106 identifies the data field as containing a street name. This piece of information can be used by itself to identify, for example, that the data field contains a street name and should therefore be labeled as such. This piece of information may also indicate that the data field contains other information, such as an address. The test module 106 may use the identification of a street name in addition to other data in a column to identify whether the data field contains an address, only a street name, or any other data. In another example, the phrase "date of birth" may be recognized and associated with such a label. Other matching techniques may include fuzzy matching, matching using synonyms, etc. Typically, the test module 106 associates a likelihood score with each match. A user may generate a specification for configuring this logic, for example, through a development environment.

[0091] The pattern matching analysis 410 of the test module 106 uses the data content of the field (in addition to or instead of the field name). The type of pattern matching that can be used for pattern matching can be identified by the test module 106 based on the results of the classification data 276. For example, the classification data 276 may identify the data type of the field, e.g., that the data is numeric. In this example, the profile data also indicates that each entry in the data field is 13-18 characters long. This may indicate to the test module 106 that the data field is likely a credit card number data field. To confirm this, one or more pattern tests can be performed by the test module 106 against the data in the suspected data field. For example, the first 4-6 digits of each entry can be checked against a table of issuer codes. The final number can include a check digit defined by a Luhn test. If a threshold percentage of entries for a data field meet each of these patterns, the test module 106 can conclude that the field holds a credit card number and associate the field name with an appropriate label and probability. With respect to pattern matching logic, both the field data itself and the pattern of data within the field (e.g., as identified in the profile data) can be used to determine which pattern tests to perform and which labels to apply to that data field.

[0092] The test module 106 identifies (408) whether to perform a pattern analysis test 410 on the source data 112. This identification may be the result of the classification data 276. The pattern match analysis 410 uses the profile data to identify whether the source data 112 matches a predetermined pattern that indicates a candidate field label. For example, if the data in a field has a particular length and composition, the pattern match analysis 410 identifies a corresponding candidate label. The pattern score applied to the label may be a function of how closely the identified pattern matches the predetermined pattern, how distinctive the pattern is, or any number of factors. The weight may be adjusted depending on the distinctiveness of the pattern. For example, a rare pattern may correspond to a higher weight value. If the discovered field value closely matches the pattern (for all or part of the value), the score may be higher than if only a small number of values ​​match the pattern.

[0093] The test module 106 may include a keyword search test 422. A keyword test (similar to a business term matching test) involves testing a database that involves searching for specific keywords within a data field. For example, to find an address, the test module 106 searches for common address words such as "street," "road," and "avenue." The test is extensible by the user, who can add new keyword files to the reference database 116 specification. A keyword test can be used to find words or parts of words within a phrase, such as an address and a company name, where there is a limited set of common words that can uniquely identify the data field.

[0094] The test module 106 can identify 420 whether to perform keyword analysis 422 on the field names. In this example, the test module 106 will perform a keyword matching test if some of the fields are still not associated with a label value. Field names (and possibly field values) are checked to see if they contain one or more keywords from a table, which may help the test module 106 associate a particular label with the field. The test module 106 performs a keyword search 422 and generates a keyword search score.

[0095] The test module 106 may include fingerprinting tests 418 for analyzing data values ​​of the source data. Fingerprinting logic generally involves a data lookup for an entire data field. The fingerprinting logic includes tests based on the data values. The fingerprinting logic for a data field may include comparing the data in the data field to a known list (e.g., from the reference database 116) to determine whether the data in the data field correlates with the data in the list. For example, data from a data field may be compared to a list of first names, a list of state names, a list of city names, and so on. Fingerprints (also known as data lists) are generally representative rather than comprehensive. In other words, a fingerprint need not include every example of a value that is part of the set of fingerprints. Generally, a fingerprint may include selected example values ​​that are approximately representative of the most common values ​​that are expected to appear in data entries for that data field. For example, a fingerprint need not include every possible first name in the United States for a table of first names; a selected group of the most common first names may suffice. For example, the top 100 names typically provide enough data to demonstrate correlation between data fields and data fingerprints. Data fingerprints can be generated from master data. For example, a system might include the top 50-100 most populous U.S. cities to check whether data fields correspond to city names. Users can add new domains to a specification to enhance the functionality of fingerprinting tests for a particular system.

[0096] The test module 106 determines 416 whether to check the fingerprint table for the field. Fingerprinting can work well in situations where there is a long list of possible values ​​for a field, but some values ​​are expected to be more common than others. For example, city names, street names, and even first and last names are good candidates for fingerprinting analysis. The test module 106 performs fingerprint analysis 418 and generates a fingerprint score.

[0097] The test module 106 determines 412 whether to perform business term analysis. Business term analysis can be beneficial when there are many unique terms in the source data 112 that can correspond to business terms for labeling fields. Once business analysis is performed, the test module 106 performs business term analysis 414 by comparing field names to business terms to find matches and generate labels and their associated probabilities. The results of the business term analysis 414 are associated with weights and scores, similar to the pattern match analysis 410.

[0098] The test module 106 can be configured to use validation logic to compare results between tests and refine the results. For example, validation logic can be used to validate a match of a business term with another classification test. For example, a test specifying that a data field contains a maiden name value should pass a classification test that is a last name field.

[0099] The test module 106 is configured to execute machine learning logic that influences which tests are selected for subsequent iterations based on similar classifications of previous datasets (e.g., from a particular source) or previous iterations of the same dataset, and how the probability values ​​of these subsequent iterations are determined. The machine learning logic is trained on the dataset and can apply weights developed using the training data to classify new data in the dataset.

[0100] Each of the tests 410, 414, 418, and 422 can output one or more proposed labels for the data field being analyzed. The tests 410, 414, 418, and 422 do not need to agree. Each proposed label can be associated with a score (not shown) and a weight value (not shown). The score and weight for each label can be used by the result validation module 108 to suggest a particular label among the proposed labels as identifying the semantic meaning of the field and to categorize (or bucket) the labels into a category. The category (described below) indicates the degree of agreement between the tests and therefore indicates the certainty that the proposed label identifies the semantic meaning of the data in that field.

[0101] To run the tests, the test module 106 receives data 276, 442, and 440 and identifies whether each test should be run. For example, if the received data type is not formatted for a particular test, the test may not be run. Any combination of tests may be run. The test module determines (408) whether to run pattern analysis 410, whether to run business term analysis 414, whether to run fingerprint analysis 418, and whether to run keyword search 422. Each test 410, 414, 418, and 422 generates one or more proposed labels and outputs the proposed labels. The test results, including all proposed labels, are combined (424) and sent to the result validation module 108 as test results 306.

[0102] In some implementations, subsequent tests can be performed if results from previous tests are still missing. In some implementations, the test module 106 creates reference data by retrieving data from the reference database 116 based on the source data 112 being analyzed. For example, the test module 106 can retrieve data from the reference database 116 based on the location of the source data 112, the name of a table in the source data, user input, and so on. The test module 106 can identify which lookup table to access and check against each field of the source data. The lookup selection can be performed based on profile data received from the profile data module 104.

[0103] In some implementations, the test module 106 identifies whether to perform schema analysis. Schema analysis can be used to analyze the source data 112 as a whole over a period of time. For example, if fields have been deleted, added, removed, and otherwise, this information can be used to label other data fields.

[0104] In some implementations, as each test is completed, the test module 106 combines the associated scores to generate a final score, which is shown along with the associated proposed labels to which it applies. In some implementations, because upstream scores influence downstream scores, the scores are not discrete from one another, but represent scores that are updated as each analysis is performed. In some implementations, each proposed label and its score are reported separately to the result validation module 108, which then determines how to categorize the test results 306.

[0105] In some implementations, the tests may be ordered so that the most processing-intensive tests are scheduled last. The most time-intensive tests may be a last resort if other tests fail. Ordering the tests in this way may reduce the processing time on the execution system 102 for labeling the source data 112.

[0106] 2D, result validation module 108 is configured to execute validation logic. Once test module 106 performs classification tests on source data and identifies likelihood values ​​for labels of data fields in the source data, result validation module 108 checks to identify whether the classified results indicate a high degree of certainty or whether further testing should be performed.

[0107] The result validation module 108 receives test results 306 of different tests performed on the source data by the test module 106 and identifies whether the results confirm or contradict each other. The test results 306 of the test module 106 are sorted by the result validation module 108 into several classification categories. The categories include a match category 290, a recommendation category 291, a research category 292, and an ignore category 293. Each category indicates the similarity between the proposed labels in the test results 306. For example, if all the proposed labels are similar, the test results have a high level of similarity. If the proposed labels are different, the test results have a low level of similarity. Similarity can be more than simply a voting mechanism for each test. Each proposed label is weighted and associated with a score value. If one proposed label of a test does not match the others but is associated with a relatively high score value and a large weight compared to the proposed labels of other tests, the similarity may be identified as lower even if three of the four tests match.

[0108] Depending on the identified category, the result validation module 108 can automatically validate that the label identifies the semantic meaning of the field, or it can prompt the user to manually validate the label. Validation can be performed through the client device on a user interface, which is described below.

[0109] The match category 290 indicates the highest level of certainty for a label (e.g., 100 matches). In some implementations, matched labels are automatically included in the metadata associated with the source data. However, the result validation module 108 can still be configured to present the results to a user for manual validation. Generally, a match categorization indicates that all of the tests performed suggested the same label. In some implementations, the match category can be selected if the labels do not all match, but any different labels are below a score threshold, indicating a general agreement between the tests.

[0110] The recommended category 291 generally indicates that at least one label has a high-quality association with the data field. However, the recommended labels are generally below a set threshold level for highest certainty, and further validation is preferred. In some implementations, the recommended category indicates that several high-quality labels are associated with the data field. In some implementations, the result validation module 108 ranks and lists the recommended labels along with the likelihood of each, which can assist the user in selecting the best label for the data field.

[0111] The Investigation category 292 indicates values ​​or data fields with some interesting statistical properties that do not pass any particular test. For example, a data field may look as if it should be meaningful, but no labels are suggested in any tests, or these or suggested labels score below some threshold. For example, a data field may contain profile attributes that indicate the data field is an important domain or field in the source data, but no labels are recommended (or are recommended with a higher than threshold probability). Generally, such results indicate that additional rules should be added to the test (e.g., the test should be expanded or modified in some way) and run again.

[0112] The Ignored category 293 indicates that a data field is designated as uninteresting or that the field did not trigger any tests or exhibit any profile attributes that would suggest the data field is important. Generally, the Ignored category is indicated when the received data has errors or no discernible pattern. The Ignored category may indicate that a new label should be defined, but it generally indicates that the field contains miscellaneous data with no particular structure.

[0113] Generally, to categorize (e.g., bucket) the test results 306 into categories, the result validation module 108 performs process 285. The result validation module 108 compares (286) the test results from the fields. For each field, the result validation module assigns (288) the test results 306 to a category based on the level of similarity.

[0114] In addition to generating a classification file that specifies classification categories, the results validation module 108 is configured to generate other files, including a profile comments file, which is a file that describes the results of the classification analysis process, and a summary of the discovery results, which describes the combined output from the business term matching process and the fingerprinting process.

[0115] The result validation module 108 is configured to identify whether there are any outliers for the data values ​​based on the totals and ranges for each data field. Outliers include data values ​​that do not match the relationships or formats identified for a data field. The result validation module 108 identifies outliers based on the clustering and prediction of relationships from the classifications of the testing module 106.

[0116] The above process can be repeated to improve classification accuracy and allow the user to update the classification tests to achieve better results. As mentioned above, in some implementations, machine learning logic can be used to facilitate this process, training the classifier during each iteration. Generally, once profile data is generated by the profile data module 104, the process of the test module 106 and result validation module 108 is repeated, and new profile data does not need to be repeatedly generated unless the source data itself is updated.

[0117] Once the profile data module 104, classification module 105, test module 106, and result validation module 108 have generated labels for each of the data fields, the load data module 110 can load the metadata, including the label index, into the reference database 116. Referring to FIG. 2E, the load data module 110 performs a process 299 to update the label index and load data into the reference database 116. The load data module 110 receives field names (295) and proposed labels for the fields (296), which are validated manually or automatically. The load data module 110 combines the field names and proposed labels (297). The load data module updates the label index by associating the labels with the position of the field within the dataset. The load data module combines the labels (298) so that the dataset can be parsed into a label index, which can be referenced for the dataset by the execution system 202 and downstream applications.

[0118] Generally, the reference database 116 can be accessed by one or more downstream computing systems for a variety of uses, for example, the generated labels of a dataset can be used for data quality enforcement, personal data anonymization, data masking, (PII) reporting, test data management, dataset annotation, and so on.

[0119] The load data module 110 is configured to package the metadata and source data into a package that can be used by one or more other computing systems. For example, once profile data is generated, the classification module 105, testing module 106, and result validation module 108 can each be configured to run multiple instances of their processes in parallel on different partitions of the source data. For example, if the source data includes multiple tables, the source data can be partitioned by table. The testing module 106 and result validation module 108 can run instances of their logic processes on multiple tables simultaneously to increase the processing throughput of the execution system 102. Once the source data is labeled, the load data module 110 can be configured to recombine the partitions and store the labeled source data in the reference database 116.

[0120] 3A-3G are block diagrams of a process for profiling, classifying, and labeling data fields to identify their semantic meaning. In the example of FIG. 3A, block diagram 300a shows that source data 302 is partitioned and each table 302a, 302b is processed in parallel by execution system 102. Source data 302 includes fields and values ​​for entries within the fields. As mentioned above, while tables and fields are shown as examples of source data, source data may also include other data types. Profile data module 104 receives source data 302 and stores index data 302c in reference database 116. Index data 302c may be a table index for source data 302. Index data 302c is then modified to include labels for each of the fields in source data 302, thus becoming a label index. As shown in FIG. 3A, index data 302c associates field names with the field locations in source data 302 for each field.

[0121] The source data 302 includes two tables, table 302a and table 302b. The first table 302a includes dates, city names, and alphanumeric codes. The second table 302b includes names, phone numbers, and dates. While it is clear to a user that a city name is indeed a city name or that a phone number is indeed a phone number, it may be more difficult for a user to identify the purpose of an alphanumeric code or what a date represents. The execution system 102 is configured to solve this problem by categorizing each of the fields in the first and second tables 302a, 302b as described above.

[0122] In general, the profile data module 104 first profiles all of the tables in the source data 302 to identify statistical characteristics for the tables. Once profile data 304a, 304b is generated for each of the tables 302a, 302b, respectively, the execution system 102 partitions the first table 302a into a first partition 304a and the second table 302b into a second partition 304b.

[0123] As previously described, profile data 304 includes field names and statistical information about the entries in each field. For example, for table 302a, profile data 304a identifies four fields: Date 1, Date 2, Location, and ID. Profile data module 104 determines that the data values ​​(also called entries) for Date 1 have the same format: ## / ## / ####. In practice, profile data module 104 may determine that nearly all values ​​have this format and discard outliers. In addition to determining the format of the Date 1 values, profile data module 104 may also determine that the first two digits of all Date 1 values ​​are between 1 and 12. This date may indicate to test module 106 that the values ​​in the Date 1 entries are dates.

[0124] Similarly, the profile data module 104 performs formatting and statistical analysis on each of the Date 2, Location, and ID fields. Date 2 also has a ## / ## / #### format for its entries. The profile data module 104 also determines that the value of the last four digits of Date 2 is always greater than the corresponding value of the last four digits of the Date 1 entry. This indicates that if both Date 1 and Date 2 contain dates, for each entry in table 302a, the date value of Date 2 is always later in time than the corresponding date value of Date 1. The profile data module 104 determines that Location is a string in which a two-character code always follows the first word. The profile data module 104 determines that ID always contains an alphanumeric string of length 7, which may provide a hint to the test module 106 that the value of ID is a code, a user ID, etc.

[0125] The profile data module 104 also profiles table 302b to generate profile data 302b. Typically, tables 302a and 302b are profiled together and then partitioned. The profile data module 104 determines that the Last Name field in table 302a contains a string of characters, with the value "Smith" being the most common entry (x2). The profile data module 104 determines that the First Name field in table 302b contains a string of characters, with the value "John" being the most common entry (x3). The profile data module 104 determines that M / F contains a single character that is one of two values, "M" or "F." The profile data module 104 determines that the number format contains (###)###-####, with the number within the parentheses (i.e., the first three digits) forming a finite set. The profile data module 104 determines that the Notes field contains a string of characters, with the first two digits being 1-12.

[0126] Once generated by the profile data module 104, the profile data 304 is sent to the test module 106. The test module 106 is configured to operate on the table in parallel on partitions 306a and 306b. The profile data module 104 sends the profile data 304 to the classification module 105.

[0127] Referring to FIG. 3B, block diagram 300b illustrates how classification module 105 processes profile data 304 and source data 302 to classify the data type of each field. Classification module 105 receives 401 profile data 304 from profile data module 104 and label index 220 from reference database 116. Classification module 105 selects 405 fields from the source data. In this example, the Date1 field is selected for classification. Classification is performed on a field-by-field basis for source data 302. Profile data 314 for the selected field is classified based on whether there is an existing label for that field. Here, the status of the Date1 field is shown as <Unlabeled>. Classification module 105 classifies the Date1 field, identifies it as a date field, and indicates any label for the field (not found in this case). This information is classification output 286, which is sent to test module 106.

[0128] Referring to FIG. 3C, diagram 300c shows how test module 106 generates a label for the Date1 field. This diagram 300c illustrates testing for one field of a dataset at a first time T1. Testing is performed field by field, although typically all fields in a dataset are tested. Test module 106 generates classification output 306 using clues provided by classification output 286, profile data 304, and data stored in reference database 116, including lookup table 442. For example, as described above, classification module 105 identifies the first field as a "date" field based on the format of Date1 in table 302a. The additional information that the first portion is a number between 1 and 12 (and that the system uses the U.S. date notation) further suggests that Date1 is a date field. Test module 106 assigns a probability of 95 that Date1 is a date field. However, without other context, test module 106 cannot determine what type of date the Date field represents. The date field in Date1 may include the contract date for a service, the date of birth, the renewal date, and others. The test module can identify the Date1 field as a "Date of Birth" field because the test uses context information from the reference database 116 and the label table 440. The label table 440 contains a list of defined labels for the dataset being analyzed. The label table 440 can be amended as needed to add additional labels, as described above. As shown in FIG. 3C , each label in the label table 440 has one or more attributes that describe what characteristics the label describes about the data. For example, the first three digits of a valid phone number are included in the set of possible numbers in the area code table in the reference database. For example, the date in the Expiration Date field is always later than the date in the Date of Birth field (if present). These attributes indicate complex relationships between field values ​​and other characteristics of the data, indicating that the data has a semantic meaning identified by the associated label.

[0129] To test the Date1 field, the test module 106 performs any combination of tests 410, 414, 418, and 422 as described above. Each test proposes one or more labels. For example, the pattern match analysis 410 proposes the proposed Date of Birth label 318 with a score of 0.80 and a weight of 1.5. For example, the business term analysis 414 proposes the proposed Expiration Date label 320 with a score of 0.75 and a weight of 1.1. For example, the fingerprinting analysis 418 proposes the proposed Date of Birth label 322 with a score of 0.60 and a weight of 1.75. For example, the keyword search analysis 422 proposes the proposed Date of Birth label 324 with a score of 0 and a weight of 0.5. The weights can be predetermined based on feedback applied by the user to the execution system 102 or can be automatically adjusted by a machine learning process. The score indicates how strong the proposal is as a result of the test. For example, a score of 0 for keyword search test 422 may be given because the available data is not properly formatted to use that analysis. Therefore, in this case, the proposed test is discarded. A score of 1, for example, indicates a high degree of certainty that the proposed label is correct for that test. For example, once all attributes for a particular label in label table 440 are identified in source data 302 and profile data 304 for the Date1 field, the test can be certain that the particular label correctly identifies the semantic meaning of that field.

[0130] The test module 106 combines 424 the proposed labels 318, 320, 322, and 324 into test results 326. The test results 326 include the proposed labels and their associated scores and weights. The result validation module 108 uses the scores and weights to determine how similar the proposed labels are to each other in order to categorize the proposals into categories.

[0131] Referring to FIG. 3D , diagram 300d shows a second time T2, after which each of the fields in the source data is tested. Test results 326 are shown for Date 1, which are combined with other test results for other fields. For example, test module 106 identifies Date 2 as an expiration date with a score of 95, similar to identifying Date 1 as a birth date. Test module 106 has the additional context that this date is always later than the date value of Date 1 for each entry. Additional information can be accessed to further define the date. In this example, the test suggests “Expiration Date” as a possible field name. Based on fingerprint analysis, test module 106 assigns Date 2 a score of 53 for the “Birth Date” label.

[0132] The test module 106 performs a lookup based on the values ​​of the location entries. As described above, the test module 106 can use a lookup table, keyword matching, fuzzy matching logic, etc. to determine that the string values ​​are city names and that the two-character string in each entry represents a state abbreviation. In some implementations, the test module 106 is configured to split each value of the location entries into two parts to determine that the entry's value represents both a city and a state name.

[0133] The test module 106 references specific lookup tables provided by the system administrator for this source data 302. Because the system is ingesting data from the Department of Motor Vehicles, one of the tables included by the user is a table of driver's license IDs. The test module 106 can determine that the format of the ID values ​​matches the format of the driver's license IDs in the lookup table. In this case, a fingerprinting algorithm is applied. There may be some matches found between the IDs and the values ​​in the lookup table, but in this example, there are only a few matches. The test module 106 assigns a score of 62 corresponding to this label. In subsequent iterations of this example, the fingerprinting algorithm may provide additional value to obtain a stronger match. However, in other examples, the fingerprinting algorithm may provide a high certainty match on the first try.

[0134] In parallel with analyzing table 302a in partition 306a, test module 106 is configured to analyze the profile data and its contents for table 302b in partition 306b. In this example, test module 106 applies one or more classifiers to the last name field in table 302b to identify that field as representing a last name. For example, the classification algorithm described above can be applied to the data in table 302b. Because the most common value ("Smith") appears only twice, test module 106 assigns the field the label "Last Name" with a low score of 35. However, based on a similar analysis of the lookup table for first names, the name "John" appears more frequently (and is identified as a first name value), so the score for first name is 85.

[0135] Test module 106 checks the value of M / F in table 302b against the one-letter code. A gender designation table is found and the value of M / F matches that specified in the table. M / F is assigned the label "Gender" with a score of 98.

[0136] For numbers, the test module 106 determines that the format of each entry (or most of the entries) corresponds to a phone number. As a second check, the first three digits are compared to an area code table. Because the first three digits of each value of the number match a value in the area code table (not shown), the test module 106 assigns the number the label "phone number" with a score of 100. For notes, the test module 106 determines that no likely labels were found for any of the tests. The test results 306 can be kept in partitions 306a, 306b based on the table being analyzed.

[0137] Referring to Figure 3E, a diagram 300e illustrates how the result validation module 108 processes the test results 326 for the Date 1 field. Figure 3E illustrates the time T1 for processing a particular field, Date 1, because each field is categorized by field. The result validation module 108 receives the test results 326 for the Date 1 field.

[0138] The result validation module 108 compares the proposed labels of the test results 326 to each other in module 286. The proposed label for Date of Birth has a count of 3, while the proposed label for Expiration Date has a count of 1. In some implementations, a simple voting mechanism can be used to identify which label should be presented to the user as the final proposed label for identifying the semantic meaning of a field such as Date 1. In some implementations, scores and weights can be used to identify which labels are proposed and which categories are associated with the proposed labels. For example, the result validation module 108 can rank the proposed labels by their weighted scores and select the top proposed label. In some implementations, all proposed labels (if there are multiple proposals) are presented to the user for validation.

[0139] The result validation module 108 assigns the proposed labels to categories based on the similarity between the proposed labels via module 288. Categories can be selected based on the weighted score, the count of each proposed label, and so on. Using a function of the count, score, and weight, the proposed label for a field can be assigned to one of categories 290, 291, 292, or 293. Here, because the proposed labels do not match for all tests, and because the non-matching tests are assigned a sufficiently high weighted score value, the recommended category 291 (rather than the match category 290) is selected. The result validation module will likely prompt the user to validate the categorized proposal 289.

[0140] The report generation module 380 can generate a report 381 showing how each of the fields is categorized and with which proposed labels. For example, a histogram including the count of each category and which fields and proposed labels are in each category may be shown. Generally, fields with fewer counts should be investigated, while fields with more counts have recommended or matched counts.

[0141] Referring to FIG. 3F, block diagram 300e illustrates the result validation module at time T2 after all fields have been assigned proposed categorized labels. The test results and assigned categories 221 are sent to a test result validation module 399. The validation module 399 can, for example, cause a user interface to be presented to a user on the validation data 111 (e.g., a client device). Validation data 309 is received via source 111, and the validated labels are assigned to fields as assigned labels 308. Each field is associated with one assigned label that identifies the semantic meaning of the data in that field.

[0142] Referring to FIG. 3G, diagram 330f illustrates how assigned labels 308 for fields in source data 302 are stored by load data module 110 and how a label index is updated / generated. Load data module 110 receives index data 302c from reference database 116. Index 302c is used to generate, for each label in data dictionary database 114, the location of each field assigned to that label. Label index 310 illustrates this association. As downstream applications manipulate source data 302, they can reference label index 310 to reduce errors in handling the data and quickly find the semantic meaning of a field without further parsing the data. This allows downstream applications to, for example, screen for personally identifiable data without actually accessing the data itself.

[0143] 4 is a flow diagram 400 illustrating the operations performed by the logic engine of the execution system to discover, classify, and label data fields by analyzing data profiles generated from the data in the data fields. As mentioned above, after the analysis is performed by the result validation module 108, the analysis can be repeated if the results are non-conclusive.

[0144] 5 shows a block diagram of an execution system 102 for discovering, classifying, and labeling data fields by analyzing data profiles generated from the data for the data fields. The execution system 102 is shown in an exemplary environment 500 in which a particular field 208 is labeled from data 202 at a source, including fields 202a, 202b, and 202c. In this example, the scalability of the system is demonstrated, with labels already associated with the fields 208.

[0145] 5, the source data 202 includes a field 208 called "Field_XYZ," which is the technical field name. The execution system 102 is configured to receive the source data 202 and label the field 208 with a business term that describes what kind of value is stored in that field. The field 208 includes a data entry, such as data entry 210, which includes the string "Austin TX."

[0146] The profile data module 104 is configured to receive source data 202 and generate profile data from the source data. While this example is constrained to a particular data entry, the source data 202 typically includes many entries. In general, the entries can be scanned to obtain statistical information. Once profile data is generated for a field (or a table in general), the test module 106 performs one or more classification tests as described above. The classification 106 can reference the reference database 116 to determine whether the field name "Field_XYZ" already has a label. In this example, the result of the direct lookup is found in the label index 154. The field name "Field_XYZ" is known (e.g., by a user-generated specification or by any other means) to represent the field name of "City Name." Because this relationship is clearly defined in the index 154, the test module 106 can score this relationship with a high degree of certainty or pass the result to the result validation module 108. Using knowledge of what "city names" are possible, test module 106 can further specify what values ​​to include in value 210.

[0147] The labels identified by the test module 106 are sent to the result validation module 108, which categorizes the labels into a category "match" 218, identifies whether the label should be automatically applied, and requests validation if necessary. The result validation module 108 supports other classification tests as described above and outputs a ranked list of recommendations for labeling the field 208. Once the label 220 is validated, the assigned label is sent and the index table 152 is updated as needed. In some implementations, the attributes of the city name label can be updated based on the attributes of the field 208.

[0148] 6A, a depiction of a user interface 500 showing the results of a data profiling process, for example, by the profile data module 104, is shown. The user interface 500 includes a header 501 and rows 502a-e and columns 504, 506, 508, and 510. Each of the rows 502a-e represents a field of the source data 112. In some implementations, different rows represent the same field (e.g., 502a and 502b), but each row indicates a different relationship detected between that field and other fields. For example, row 502a indicates that the field "Date 1" correlates with the field "Date 2," and row 502b indicates that the field "Last Name" identifies the field "ID."

[0149] Column 504 lists each of the fields in source data 112. Column 506 lists the type of relationship, as shown in column 508, that is detected between that field and source data 112. Column 510 contains a score (e.g., likelihood) of the certainty that the fields in columns 504 and 508 are related to each other as shown in column 506.

[0150] The relationships detected in the source data can be used by the test module 106 to identify label values ​​for each of the fields. For example, identifying a zip code field as related to a state field can provide clues as to what each field actually means. This is useful when the names of the fields are technical or do not otherwise indicate the type of data contained in the field.

[0151] Referring to FIG. 6B, a depiction of a user interface 520 containing the results of the classification process of the testing module 106 is shown. Window 524 shows a list of discovered fields 526a-e. In this example, fields from columns 504 and 508 of FIG. 6A are included. Box 526a indicates which fields were analyzed by the testing module 106. In this example, the field "Date 1" is analyzed. In some implementations, each field can be selected to show details of how the testing module 106 analyzed the discovered field and how a score was generated. For example, the bold outline of button 526a indicates that the button is selected.

[0152] Window 528 shows the analysis details of the selected discovered field in window 524. Here, window 528 shows that four classifier tests are applied: pattern match analysis, business term analysis, fingerprint analysis, and keyword search. Each test is associated with a corresponding weighted score. The final adjusted score is also shown in box 536.

[0153] Window 534 shows a list 530 of candidate labels and their adjusted scores. Here, the label "Date of Birth" is associated with a score of 95. A second label, "Expiration Date," is associated with a score of 33. Although the candidate labels are shown ranked according to their score, the candidate labels may be shown in any order. The score of 95 matches the adjusted score shown in window 528. The performed analysis and score shown in window 528 are associated with candidate label 532a "Date of Birth."

[0154] In some implementations, the semantic label update module 340 of the result validation module 108 uses the adjusted score of the candidate label to perform its evaluation. For example, the semantic label update module 340 may use the adjusted score associated with the candidate label to evaluate whether the candidate label should be assigned to a field as part of its update.

[0155] Window 534 shows additional details about the classification process of test module 106. The number of label proposals may be indicated. In this example, the number is 2. In another example, the number of iterations for analyzing one or more of the discovered fields may be indicated.

[0156] 6C is a depiction of a user interface 540 showing an example result of the result validation module 108. The user interface 540 includes a label suggestion pane 542 in which potential labels representing the data fields are shown. A window 552 shows which fields were discovered in the analysis as received by the result validation module 108. The discovered field 554 has the name "Location," which is included in the profile data of FIG. 6A and analyzed as described with respect to FIG. 6B. A label suggestion 558 having a value of "City Name" is recommended to the user as the label for the "Location" field.

[0157] The recommendation information pane 544 contains data describing the discovered field for which the label is recommended. For example, the current label 568 assigned to the field may be shown. In this example, the field is unlabeled (perhaps waiting for user input). The values ​​of entries in the data field may be shown in pane 574. Links between fields may be shown in menu 572. In this example, "Albany," "Boston," "Cleveland," "Detroit," "Evanston," and "Fargo" are shown.

[0158] The profile summary pane 550 shows a summary of the contents of the label. For example, the summary pane 550 may include statistical data about the field, such as the total count, missing values, distinct / unique values, minimum and maximum entry length, characters contained in the entry, the data type of the entry, and more. In this example, the field to be labeled "Genre" has 28 values, 28 of which are unique. All values ​​are between 3 and 11 characters in length and are all string types. Using these clues, along with the entry data, the test module 106 can identify whether the label "Genre" should be recommended.

[0159] The common values ​​pane 548 shows which values ​​are most common in that field and their relative proportion of values ​​in that field. In this example, all different values ​​of the field have similar numbers of occurrences.

[0160] The pattern pane 546 shows the general pattern of each of the field's entries. For example, if there was a date, the pattern might include ## / ## / #### as previously mentioned. Here, there is no specific pattern that describes the majority of fields, but values ​​that include the pattern "New_" are the most common. While these specific profile data are shown in panes 550, 548, and 546, any data that can assist the user in approving the suggested candidate names for the discovered fields can be shown.

[0161] Referring to FIG. 6D, a depiction of a user interface 560 showing recommendations for labels for discovered data fields is shown. The user interface is presented in a table format, and the table values ​​are interactive. Column 562 shows the field name received in the source data. Column 564 shows which label is assigned to the field. As previously mentioned, each field can be associated with multiple labels. Column 566 shows the proposed label for the field in column 562. Column 568 identifies which tests the testing module 106 performed on the field, how strong the results were, and what category of recommendations were made for that label. Column 570 indicates that a decision by the user regarding the field in column 562 is pending. For example, the user may be prompted to accept / reject the proposed label, retest with updated parameters, and so on.

[0162] In this example, the discovered field "Date 1" is shown in the last row, and is discovered, profiled, classified, and labeled as described with respect to Figures 6A-6C. The field "Date 1" is assigned the label "Date of Birth." As shown in column 568, the category of the label is the "recommended" category because the score associated with the candidate label is 95, as shown in Figure 6B. The analysis performed included pattern matching in addition to other analysis. The decision to assign a label to the field "Date 1" is open because the label is already assigned and no further action by the user is required.

[0163] 6E, a block diagram illustrates a semantic label updating module 340 of the result validation module 108. The semantic label updating module 340 manages the labeling of fields for which labeling was previously completed (e.g., in a previous iteration of semantic discovery performed by the execution system 102). The semantic label updating module 340 includes a field identification module 342, a candidate label identification module 344, an evaluation module 346, and a data store 348 that stores previous label information.

[0164] In some embodiments, result validation module 108 determines that labeling of a field is complete after a label indicating the semantic meaning of the field is assigned to the field. For example, result validation module 108 completes labeling of a field in response to receiving input through a user interface indicating that a user has accepted a candidate label for assignment to the field. In some embodiments, result validation module 108 determines that labeling of a field is complete when all candidate labels are rejected. For example, result validation module 108 may receive input through a user interface indicating that all candidate labels should be ignored to indicate that none are acceptable for a given field. In this example, result validation module 108 determines that labeling is complete for a field after receiving input indicating that all candidate labels for the field should be ignored. In some embodiments, result validation module 108 determines that labeling for a given field is complete when there are no candidate labels generated for the field during the labeling process. For example, the results validation module 108 may not have identified any candidate labels from applying the label suggestion tests to the profile data.

[0165] In some embodiments, the result validation module 108 may identify labeling for a given field as incomplete if it is assigned a label outside of the system 100. For example, labeling for a field may not be identified as complete if a label is assigned manually outside of the labeling process performed by the execution system 102.

[0166] In some embodiments, previous label information 348 includes an indication of fields for which labeling was previously completed and the information labeling for those fields. Information about the fields may include information about labels assigned to the fields and / or previous candidate labels for the fields. Information 348 may include scores associated with labels assigned to one or more fields and / or may include associated scores, candidate labels for one or more fields. Information 348 may include an indication of one or more fields for which no candidate label was previously identified.

[0167] In some embodiments, the semantic label updating module 340 updates the previous label information 348. The semantic label updating module 340 updates the information 348 after completing labeling for a field. After completing labeling for a field, the semantic label updating module 340 updates the previous label information 348 to store information about the labeling of the field. For example, the semantic label updating module 340 may store the label assigned to the field and the corresponding score, one or more candidate labels proposed for the field and the corresponding scores, and / or an indication that no candidate labels were generated for the field.

[0168] The field identification module 342 identifies fields for which labeling has previously been completed. The field identification module 342 may identify fields for which labeling has been completed using previous label information for the field (e.g., stored in the data store 348). In some embodiments, the field identification module 342 determines whether a field has been assigned a label by checking whether a label has been assigned to the field in a label index. In some embodiments, the field identification module 342 determines whether one or more candidate labels have been identified for the field, and if so, whether they have all been rejected.

[0169] The candidate label identification module 344 identifies one or more candidate labels and corresponding candidate label scores for the field. The score may indicate the strength of the label in identifying the semantic meaning of the field (e.g., a higher score may indicate a stronger label). The score may be determined using the scores and weights provided by the testing module 106. In some embodiments, the candidate label scores are adjusted scores as described herein with reference to FIG. 6B , and the candidate label identification module 344 identifies the candidate labels and corresponding candidate label scores using test results obtained from the testing module 106. Techniques for generating the proposed labels and corresponding scores by applying label proposal tests are described herein.

[0170] The evaluation module 346 evaluates whether a candidate label should be assigned to a field using the corresponding candidate label score and the field's previous label information. In some embodiments, the evaluation module 346 determines whether to generate an interface through which a user can provide input indicating whether a candidate label should be assigned to a field. For example, the interface may be a graphical user interface (GUI) through which a user can provide input (e.g., as shown in FIGS. 6H-6I). In some embodiments, the evaluation module 346 evaluates whether a candidate label should be automatically assigned to a field. For example, the evaluation module 346 may (1) determine whether the candidate label score is higher than the score of the assigned label by a threshold amount (e.g., 1, 2, 3, 4, 5, 10, 15, 20, or 25), and (2) automatically assign the candidate label to the field when the candidate label score is determined to be higher than the threshold amount. In another example, the evaluation module 346 may (1) determine whether the candidate label score meets a threshold score (e.g., 70, 75, 80, 85, 90, 95, 100) and (2) automatically assign the candidate label to a field when the candidate label score meets the threshold score.

[0171] In some embodiments, the evaluation module 346 evaluates whether the candidate label should be assigned to the field by (1) determining whether the candidate label score is greater than a threshold amount than the score of the label currently assigned to the field, (2) generating an interface through which a user can provide input indicating whether the candidate label should be assigned to the field when the candidate label score is greater than the assigned label score by the threshold amount, and (3) bypassing generation of the interface when the candidate label score is not greater than the assigned label score by the threshold amount. The threshold amount may be 1%, 2%, 3%, 4%, 5%, 10%, 15%, 20%, 25%, 50% or another suitable amount greater than the assigned label score. In some embodiments, the evaluation module 346 (1) determining whether the candidate label score meets a threshold score, (2) generating an interface through which a user can provide input indicating whether the candidate label should be assigned to the field when the candidate label score meets the threshold score, and (3) bypassing generation of the interface when the candidate label score does not meet the threshold score. For example, the threshold score can be 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100 or another suitable threshold score.

[0172] In some embodiments, the evaluation module 346 evaluates whether to assign a candidate label to a field by (1) determining whether the candidate label score is greater than a threshold amount than the score of a previously identified candidate label for the rejected field, (2) generating an interface in which a user can provide input indicating whether to assign the candidate label to the field when the candidate label score is greater than the score of a previously identified candidate label for the field by the threshold amount, and (3) deciding to bypass generating the interface when the candidate label score is not greater than the score of the previously identified candidate label by the threshold amount. The threshold amount may be 1%, 2%, 3%, 4%, 5%, 10%, 15%, 20%, 25%, 50%, or other suitable amount greater than the previously identified candidate label score.

[0173] In some embodiments, evaluation module 346 evaluates whether a candidate label should be assigned to a field by (1) determining whether any candidate labels have been previously identified for the field, and (2) when a candidate label has not been previously identified for the field, generating an interface in which a user can provide input indicating whether the candidate label should be assigned to the field.

[0174] The presentation module 346A of the evaluation module 346 generates an interface in which a user may provide input regarding a candidate label for a field. In some embodiments, the presentation module 346A generates an interface in which a user may provide input indicating acceptance of the candidate label as indicative of the semantic meaning of the field. In response to receiving the input, the evaluation module 346 may assign the candidate label to the field. For example, the presentation module 346A may provide the user with an option to accept or reject the candidate label for assignment. In some embodiments, the presentation module 346A generates an interface for confirming acceptance of the candidate label for the field. The interface may allow a reviewer to confirm the user's acceptance of the candidate label for the field. For example, the presentation module 346A may provide the reviewer with an option to accept or reject the acceptance.

[0175] In some embodiments, the reassigned label 350 may be stored in a label index. For example, the load data module 110 may store the reassigned label 350 in a label index. The label index may be accessed by a downstream process (e.g., a software application). In some embodiments, a label assigned to a field may associate one or more attributes with the field. Data from the field may be processed (e.g., by a downstream process) according to the attributes. For example, the attributes associated with the field may indicate that the field contains PII. A software application may identify data from the field as containing PII from the attributes associated with the field based on its assigned label. When the software application identifies data from the field as containing PII, the software application may process the data from the field by masking the PII in the data (e.g., by masking data values ​​or portions thereof).

[0176] In some embodiments, the reassigned labels 350 may be used in the execution of a software application. For example, the reassigned labels 350 may be used to execute a software application that performs data quality enforcement. As another example, one of the reassigned labels 350 may indicate that a respective field contains PII. The label may be used to execute a software application that anonymizes PII and / or execute a software application that masks PII.

[0177] In some embodiments, the reassigned labels 350 may be used to identify data processing rules for processing data from the respective fields. For example, a data processing system may receive a request to process data from a first field. In response to the request, the data processing system may access the label assigned to the field (e.g., from a label index). The data processing system may identify one or more data processing rules for processing data from the field based on the label assigned to the field. The data processing system may process the data from the field according to the data processing rules. In some embodiments, the data processing rules may be stored in the label index in association with the label assigned to the field. Thus, the data processing system may identify a label in the label index and identify a data processing rule associated with the label in the label index. For example, a label assigned to a field in the label index may indicate that data stored in the field contains PII. The label may be further associated with one or more data processing rules in the label index, where the data processing rule indicates a data masking rule for masking PII in the data from the field. The data from the field may be processed (e.g., by a software application) using the data processing rules. For example, the data may be processed by applying data masking rules to the data from the field.

[0178] In some embodiments, the reassigned label 150 may be used to process data from a field without scanning or parsing the data from the field. For example, data from a field may be processed based on a label assigned to the field that indicates that the data from the field contains PII. Given that the assigned label indicates that the data stored in the field contains PII, the data from the field may be processed without scanning or parsing the data to determine whether it contains PII. The data from the field may be processed using data processing rules (e.g., associated with the assigned label) without scanning or parsing the data (e.g., to determine whether it has particular attributes).

[0179] Figure 6F is a depiction of labeling previously labeled fields performed by semantic label update module 340 of Figure 6E. In the example of Figure 6F, field identification module 342 identifies fields 352 for which labeling was previously completed using previous label information from data store 348 (e.g., by identifying fields with assigned labels and / or identifying fields that had all rejected candidate labels). In the example of Figure 6F, the previously labeled fields 352 include an "Address ID" field, which has been assigned the label "Address," and a "Date" field, which has been assigned the label "Date of Birth."

[0180] The candidate label identification module 344 receives from the field identification module 342 an indication of the fields for which labeling has been completed. For example, the candidate label identification module 344 may receive a reference to a field 352 from the field identification module 342. For each of the fields 352, the candidate label identification module 344 identifies one or more candidate labels and corresponding candidate label scores (e.g., using test results obtained from the testing module 106). In the example of FIG. 6F , the candidate label identification module 344 identifies a proposed label 354A for the “Address ID” field and a proposed label 354B for the “Date” field. The proposed label 354A includes the following candidate labels and corresponding scores: “ZIP Code” with a corresponding score of 96, “City” with a corresponding score of 32, “Street Address” with a corresponding score of 46, and “Address” with a corresponding score of 80. Proposed label 354B includes the following candidate labels and corresponding scores: “Date of Birth” with a corresponding score of 92, “Date of Modification” with a corresponding score of 72, “Date of Creation” with a corresponding score of 21, and “Date of Expiration” with a corresponding score of 17.

[0181] The evaluation module 346 receives label suggestion information from the candidate label identification module, which may include label suggestion information 354A, 354B identified by the candidate label identification module 344.

[0182] In the example of FIG. 6F , the evaluation module 346 determines to generate an interface in which a user can provide input indicating whether the candidate label “ZIP Code” should be assigned to the “Address ID” field. For example, the evaluation module 346 may determine to generate the interface by determining that the score associated with the “ZIP Code” candidate label is sufficiently higher (e.g., at least a threshold amount greater) than the score associated with the assigned label “Address.” As another example, the evaluation module 346 may determine to generate the interface by determining that the score associated with the “ZIP Code” candidate label meets a threshold score. When the evaluation module 346 determines to generate the interface, the presentation module 346A generates a GUI 356 in which a user can provide input to assign the candidate label “ZIP Code” to the “Address ID” field, replacing the previous “Address” label. As shown in FIG. 6G , the GUI 356 includes a graphical option 356A that allows the user to accept the updated label assignment or reject the updated label assignment.

[0183] In the example of FIG. 6F , the evaluation module 346 determines to bypass generating an interface through which a user can provide input on whether a new label should be assigned to the “Date” field. For example, the evaluation module 346 may determine to bypass generating an interface by determining that none of the candidate labels in the proposed labels 354B have a score sufficiently higher than the score associated with the currently assigned label of “Date of Birth.” As another example, the evaluation module 346 may determine to bypass generating an interface by determining that none of the scores associated with candidate labels other than “Date of Birth” meet a threshold score. Thus, the evaluation module 346 does not provide an interface through which a user can provide input to approve or reject the assignment of a different label to the “Date” field.

[0184] FIG. 6G is a depiction of reassigning a label to a previously labeled field performed by the semantic label update module of FIG. 6E. As shown in FIG. 6G, a user selects one of the graphical choices 356A in GUI 356 indicating approval to assign the candidate label "ZIP Code" to the "Address ID" field instead of "Address." In response to receiving the user input, evaluation module 346 relabels the "Address ID" field with label 350A of "ZIP Code." Evaluation module 346 updates previous label information 348 to indicate the new label for the "Address ID" field. For example, evaluation module 346 may update a label index that stores the assigned labels of fields.

[0185] 6H shows a user interface 580 displaying candidate labels 582 for various fields for which labeling was previously completed. The fields are an "Address ID" field and a "Modification Date" field. The fields may have been evaluated by the semantic label update module 340 of the result validation module 108, described herein with reference to FIG. 6E. As shown in FIG. 6H, the user interface 580 shows a new candidate label, "Zip Code," for the "Address ID" field and a new candidate label, "Modification Date," for the "Modification Date" field. The user interface 580 includes workflow options 584 that allow the user to accept each of the candidate labels for each field.

[0186] FIG. 6I shows a user interface 590 for reviewing an accepted candidate label. User interface 590 may be generated after acceptance of a candidate label. In the example of FIG. 6I, the candidate label "ZIP Code" was accepted for the "Address ID" data field. User interface 590 displays information 592 about the accepted candidate field. User interface 590 provides an option 594 for approving the label assignment. User interface 590 also provides an option 596 for rejecting the label assignment. For example, a user may provide input indicating that the proposed term "ZIP Code" for the "Address ID" field should be rejected.

[0187] The semantic label update module 340 may output one or more reassigned labels 350 as a result of the evaluation performed by the evaluation module 346. For example, the evaluation performed by the evaluation module 346 may result in replacing a previous label assignment for a field with a new label assignment to indicate the semantic meaning of the field. In another example, the evaluation performed by the evaluation module 346 may result in assigning a label to a field that previously did not have any label assigned to it. In another example, the evaluation performed by the evaluation module 346 may not result in any change in the label assignment of a field.

[0188] The semantic label update module 340 may store the reassigned label 350 in a data store. In some embodiments, the semantic label update module 340 updates the data store 348 of previous label information based on the evaluation. In some embodiments, the semantic label update module updates the label index based on the evaluation.

[0189] In some embodiments, the semantic label update module 340 updates the labeling of a field for each of a plurality of label groups. A particular field may have a label assigned in each of the label groups. As illustrative examples, the plurality of label groups may be business terms, business assets, and reference code sets. Business terms may be comprised of labels that refer to words having an understood meaning related to the business. Exemplary business terms include "social security number," "bank account number," "credit score," and / or other business terms. Business assets may be comprised of labels that refer to concepts (e.g., reporting and data transformation) and / or objects (e.g., databases and reports) that are important to the business. Reference code sets may be comprised of terms that refer to various domains, such as countries, states, cities, and / or other domains.

[0190] In some embodiments, the semantic label update module 340 evaluates whether a candidate label should be assigned to a field in each of a plurality of groups. The semantic label update module 340 may update the label assignments of fields in one or more groups while not updating the label assignments in one or more other groups. Thus, the semantic label update module 340 may apply the techniques described herein to each of a plurality of groups. Continuing with the example label groups, business assets, and reference code sets described above, the semantic label update module 340 may reassign business term labels for fields but keep the label assignments of the business asset and reference code set groups the same. When relabeling fields for a particular group, the semantic label update module 340 may use candidate labels and corresponding scores as well as previous label information for the particular label group. For example, when the semantic update module 340 is evaluating whether a candidate label should be assigned to a field for a business term group, the semantic update module 340 may use candidate labels and corresponding scores identified for the business term group as well as previous label information for the business term group.

[0191] FIG. 7 shows a flow diagram of a process 600 for labeling discovered fields with candidate labels. The process 600 includes identifying (602) a field included in one or more datasets, the field having an identifier. The process 600 includes profiling (604) one or more data values ​​of the field with a data processing system to generate a data profile. The process 600 includes accessing (606) a plurality of label proposal tests. The process 600 includes generating (608) a set of label proposals based on applying at least the plurality of label proposal tests to the data profile. The process 600 includes identifying (610) similarities between the label proposals in the set of label proposals. The process 600 includes selecting (612) a classification based at least on the similarities between the label proposals in the set. The process 600 includes identifying (614) whether user input is required. If user input is required, the process 600 includes rendering (616) a graphical user interface requesting the input. The process 600 includes identifying 618 one of the proposed labels that identifies a semantic meaning. The process includes storing 620 in a data store an identifier of a field with the one identified as identifying the semantic meaning of the proposed label.

[0192] 8 shows a flow diagram of a process 800 for updating semantic labels of a field. In some embodiments, process 800 may be performed by execution system 102. For example, process 800 may be performed by semantic label update module 340 of result validation module 108 described herein with reference to FIG. 6E.

[0193] Process 800 begins at block 802, where the system identifies fields for which labeling has previously been completed (e.g., in a previous iteration of labeling). In some embodiments, the system may identify a first field that has a previously assigned label that indicates the semantic meaning of the field. The previously assigned label may have a corresponding previously identified label score. In some embodiments, the system may identify a field that does not have an assigned label that indicates the semantic meaning of the field. One or more previously identified candidate labels for the field may have been previously identified as possible assignments to the field, along with a corresponding set of previous candidate label scores. None of the previously identified candidate labels may be assigned to the field. In some embodiments, the system may identify a field that does not have an assigned label that indicates the semantic meaning of the field, and for which no candidate labels have been previously identified for possible assignment to the field.

[0194] Process 800 then proceeds to block 804, where the system identifies one or more candidate labels and corresponding candidate label scores for the field. The system may identify the candidate labels and candidate label scores using results of label proposal tests applied to the field's profile data. In some embodiments, the label proposal tests and the field's profile data may be updated relative to the last time labeling was performed for the field. In some embodiments, after labeling is performed for the field, one or more labels may be added or removed (e.g., from the label dictionary). As a result of such updates, the candidate labels and their corresponding scores may differ from the previously identified labels and / or scores.

[0195] In some embodiments, the system identifies candidate labels by applying label proposal tests to a profile of a field to obtain proposed labels and corresponding label proposal scores. In some embodiments, the system applies the label proposal tests to a previously generated profile of the field. In some embodiments, the system profiles the field to generate a new profile and apply the label proposal tests to the new profile. The system uses the proposed labels and label proposal scores to identify corresponding sets of candidate labels and candidate label scores. In some embodiments, the label proposal tests may be associated with respective weight values, and the system uses the proposed label scores and weight values ​​to identify candidate label scores. The system may use the proposed label scores and weight values, as described above, to identify weighted scores as candidate label scores. For example, the system may identify adjusted scores as candidate label scores, as shown herein with reference to FIG. 6B.

[0196] In some embodiments, the candidate label score (e.g., the adjusted score) may be a percentage value between 0 and 100 obtained using the label proposal scores and weight values ​​associated with the label proposal test. For example, the label proposal score may be a probability value between 0 and 1 output by the label proposal test. The system may identify the percentage value using the weighted probability values. To illustrate, the system may obtain the following set of label proposal scores and corresponding weights for the candidate labels: (0.8, 1.5), (0.75, 1.1), (0.9, 1), (0.8, 0.8). In this example, the system may identify an average weighted label proposal score of approximately 0.81. The system may identify the candidate label score as 81.

[0197] Next, process 800 proceeds to block 806, where the system evaluates whether the candidate label should be assigned to the field. In the example of FIG. 8, the evaluation of block 806 includes, at block 806A, identifying whether a user interface (e.g., a GUI) should be generated in which a user can provide input indicating whether the candidate label should be assigned to the field. The system may use a candidate label score associated with the candidate label to identify whether the user interface should be generated. In some embodiments, the system may identify whether the user interface should be generated by determining whether the candidate label score is greater than a threshold amount than the score of a label currently assigned to the field. In some embodiments, the system may identify whether the user interface should be generated by determining whether the candidate label score is greater than a threshold amount than the score of a candidate label previously identified for the field. In some embodiments, the system may identify whether the user interface should be generated by determining whether any candidate labels have been previously identified for the field. Exemplary techniques for identifying whether a user interface should be generated are described herein with reference to FIG. 6E.

[0198] If, at block 806A, the system determines that it will generate a user interface in which the user can provide input indicating whether the candidate label should be assigned to the field, then process 800 proceeds to block 806B, where the system generates the user interface. Exemplary user interfaces are described herein with reference to Figures 6E-6I. The system may then receive user input indicating whether the candidate label should be assigned to the field.

[0199] At block 806A, if the system determines not to generate a user interface that allows the user to provide input indicating whether the candidate label should be assigned to the field, process 800 bypasses generating the user interface and ends.

[0200] As an illustrative example, a field may have been previously labeled with the term "code." In a subsequent iteration of semantic discovery, the system may identify a candidate label for the field as "product code." The adjusted score associated with the assigned label "code" is 89, while the adjusted score associated with the label "product code" is 95. Because the score associated with the label "product code" is sufficiently greater than the score associated with the label "code," the system determines to generate a user interface for obtaining user input indicating whether "product code" should be assigned to the field.

[0201] As another example, the "Address ID" field may not be assigned a label. The maximum adjusted score among previously identified candidate labels for the field was 56. All of the previously identified candidate labels for the "Address ID" field were rejected (e.g., based on user input and / or automatically by the system). The system may identify a candidate label for "Zip Code" that was not previously identified for the "Address ID" field and has an associated adjusted score of 96. The system determines to generate a user interface when it determines that the adjusted score associated with the candidate label "Zip Code" is sufficiently greater than the maximum score of 56 among the previously identified candidate labels for the "Address ID" field.

[0202] As described herein with reference to FIG. 6E, in some embodiments, the system may perform semantic label updates for multiple groups (e.g., business assets, business terms, reference code sets). In these embodiments, the system may perform process 800 for each of the label groups.

[0203] The techniques described above can be implemented using software executed on a computer. For example, the software forms procedures within one or more computer programs executing on one or more programmed or programmable computer systems (which may be of various architectures, such as distributed, client / server, or grid), each including at least one processor, at least one data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device or port, and at least one output device or port. The software can form one or more modules of a larger program that provides other services related to the design and construction of charts and flowcharts, for example. The nodes, links, and elements of a chart can be implemented as data structures stored on a computer-readable medium or other organized data conforming to a data model stored in a data repository.

[0204] The techniques described herein can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or combinations thereof. An apparatus can also be implemented in a computer program product tangibly embodied in or stored on a machine-readable storage device (e.g., a non-transitory machine-readable storage device, a machine-readable hardware storage device, etc.) for execution by a programmable processor, and method operations can be performed by the programmable processor executing a training program to perform functions by operating on input data and generating output. The embodiments described herein, as well as the claims and other embodiments of the techniques described herein, can advantageously be implemented in one or more computer programs executable on a programmable system including at least one programmable processor, at least one input device, and at least one output device coupled to receive data and instructions from, and transmit data and instructions to, a data storage system. Each computer program can be implemented in a high-level procedural or object-oriented programming language, or in assembly or machine language as desired; in either case, the language can be a compiled or interpreted language.

[0205] Processors suitable for the execution of a computer program include, by way of example, general-purpose and special-purpose microprocessors, and one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or be operatively coupled to receive data from or transmit data to them, or both. Computer-readable media for embodying computer program instructions and data include all forms of non-volatile memory, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special-purpose logic circuitry. Any of the foregoing may also be supplemented by, or incorporated in, ASICs (application-specific integrated circuits).

[0206] To enable interaction with a user, embodiments can be implemented on a computer having a display device, such as an LCD (liquid crystal display) monitor, to display information to the user, and a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to enable interaction with a user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0207] Embodiments may be implemented in a computer system that includes back-end components such as a data server, or that includes middleware components such as an application server, or that includes front-end components such as a client computer having a graphical user interface or web browser through which a user can interact with an implementation of an embodiment, or that includes any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0208] The systems and methods, or portions thereof, may use the "World Wide Web" (Web or WWW), a collection of servers on the Internet that utilize the Hypertext Transfer Protocol (HTTP). HTTP is a well-known application protocol that provides users with access to resources, which can be information in a variety of formats, including text, graphics, images, audio, video, Hypertext Markup Language (HTML), and even programs. When a user specifies a link, the client computer makes a TCP / IP request to the Web server and receives information, which can be another Web page formatted according to HTML. The user can also access other pages on the same or other servers by following on-screen instructions, entering specific data, or clicking selected icons. It should also be noted that any type of selection device known to those skilled in the art, such as check boxes, drop-down boxes, and the like, can be used for embodiments that use Web pages to allow users to select options for certain components. The server runs on a variety of platforms, including UNIX machines, although other platforms, such as Windows 2000 / 2003, Windows NT, Sun, Linux, and Macintosh, can also be used. A computer user may view information available on a server or network over the Web through the use of browsing software such as Firefox, Netscape Navigator, Microsoft Internet Explorer, or Mosaic browser. A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0209] Other embodiments are within the scope and spirit of the description and claims. For example, due to the nature of software, the functionality described above may be implemented in software, hardware, firmware, hard writing, or any combination thereof. The features implementing the functionality may also be physically located in various places, including distributed implementations where portions of the functionality are implemented in different physical locations. The use of the term "a" herein is not used in a limiting manner throughout this application and thus does not preclude the term "a" from referring to a plurality or "one or more." Furthermore, to the extent priority to a provisional patent application is claimed, that provisional patent application should be understood to include, without limitation, examples of how the techniques described herein may be implemented.

[0210] 9 illustrates an example of a suitable computing system environment 900 on which the technology described herein may be implemented. The computing system environment 900 is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the technology described herein. Neither should the computing environment 900 be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment 900.

[0211] The technology described herein is usable with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that may be suitable for use with the technology described herein include, but are not limited to, personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.

[0212] A computing environment may execute computer-executable instructions such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The techniques described herein may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including memory storage devices.

[0213] 9, an exemplary system for implementing the techniques described herein includes a general-purpose computing device in the form of a computer 900. Components of the computer 910 may include, but are not limited to, a processing unit 920, a system memory 930, and a system bus 921 that couples various system components including the system memory to the processing unit 920. The system bus 921 may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include the Industry Standard Architecture (ISA) bus, the MicroChannel Architecture (MCA) bus, the Enhanced ISA (EISA) bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus, also known as the Mezzanine bus.

[0214] The computer 910 typically includes a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by the computer 910, including both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer-readable media can include computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVDs) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by the computer 910. Communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.

[0215] The system memory 930 includes computer storage media in the form of volatile and / or nonvolatile memory such as read-only memory (ROM) 931 and random access memory (RAM) 932. A basic input / output system 933 (BIOS), containing the basic routines that facilitate transferring information between elements within the computer 910, such as during start-up, is typically stored in ROM 931. RAM 932 typically contains data and / or program modules that are immediately accessible to and / or presently being operated on by the processing unit 920. By way of example, and not limitation, FIG. 9 illustrates operating system 934, application programs 935, other program modules 936, and program data 937.

[0216] The computer 910 may also include other removable / non-removable, volatile / non-volatile computer storage media. By way of example only, Figure 9 illustrates a hard disk drive 941 that reads from or writes to non-removable, non-volatile magnetic media, a flash drive 951 that reads from or writes to removable non-volatile memory 952, such as flash memory, and an optical disk drive 955 that reads from or writes to a removable, non-volatile optical disk 956, such as a CD-ROM or other optical media. Other removable / non-removable, volatile / non-volatile computer storage media that may be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid-state RAM, solid-state ROM, and the like. The hard disk drive 941 is typically connected to the system bus 921 through a non-removable memory interface such as interface 940, and the magnetic disk drive 951 and optical disk drive 955 are typically connected to the system bus 921 by a removable memory interface such as interface 950.

[0217] The drives and their associated computer storage media, discussed above and illustrated in FIG. 9, provide storage of computer-readable instructions, data structures, program modules, and other data for computer 910. In FIG. 9, for example, hard disk drive 941 is illustrated as storing operating system 944, application programs 945, other program modules 946, and program data 947. Note that these components can be the same as or different from operating system 944, application programs 945, other program modules 946, and program data 947. Operating system 944, application programs 945, other program modules 946, and program data 947 are given different numbers here to illustrate that, at a minimum, they are different copies. Actors may enter commands and information into computer 910 through input devices such as a keyboard 962 and pointing device 961, commonly referred to as a mouse, trackball, or touchpad. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit 920 through a user input interface 960 that is coupled to the system bus, but may be connected by other interface and bus structures, such as a parallel port, a game port, or a universal serial bus (USB). A monitor 991 or other type of display device is also connected to the system bus 921 via an interface, such as a video interface 990. In addition to the monitor, computers may also include other peripheral output devices such as speakers 997 and printer 996, which may also be connected through an output peripheral interface 995.

[0218] The computer 910 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 980. The remote computer 980 may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 910, although only a memory storage device 981 is illustrated in FIG. 9. The logical connections depicted in FIG. 9 include a local area network (LAN) 981 and a wide area network (WAN) 983, but may also include other networks. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.

[0219] When used in a LAN networking environment, the computer 910 is connected to the LAN 981 through a network interface or adapter 980. When used in a WAN networking environment, the computer 910 typically includes a modem 982 or other means for establishing communications over the WAN 983, such as the Internet. The modem 982, which may be internal or external, may be connected to the system bus 921 via the actor input interface 960 or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer 910, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation, FIG. 9 illustrates remote application programs 985 as residing on memory device 981. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between computers may be used.

[0220] Although a number of embodiments of the present invention have been described, it will be understood by those skilled in the art that various modifications may be made without departing from the spirit and scope of the claims and techniques described herein.

Claims

1. 1. A method for discovering the semantic meaning of data in fields contained in one or more datasets, comprising: a data processing system identifying a first field having a previously assigned label indicating a semantic meaning of the first field, the previously assigned label having a corresponding previously determined label score; the data processing system identifying a set of one or more candidate labels and a corresponding set of candidate label scores for possible assignment to the first field in place of the previously assigned label, the set of candidate labels including a first candidate label corresponding to a first candidate label score in the set of candidate label scores; the data processing system evaluating whether the first candidate label should be assigned to the first field using the previously identified label score and the first candidate label score; Including, The evaluating step includes: when the first candidate label score is greater than the previously identified label score by at least a first threshold amount, presenting the first candidate label to the user by generating an interface in which the user can provide input indicating whether the first candidate label should be assigned to the first field instead of the previously identified label; bypassing presentation of the first candidate label to the user when the first candidate label score is not greater than the previously identified label score by at least the threshold amount; A method comprising:

2. 2. The method of claim 1 , wherein bypassing presentation of the first candidate label to the user comprises bypassing generation of the interface in which the user can provide input indicating whether the first candidate label should be assigned to the first field instead of the previously identified label.

3. the data processing system identifying a second field different from the first field, the second field having no assigned label indicative of a semantic meaning of the second field, wherein one or more previously identified candidate labels, along with a corresponding set of previous candidate label scores, were previously identified for possible assignment to the second field, and none of the previously identified candidate labels have been assigned to the second field; the data processing system identifying a second set of one or more candidate labels for possible assignment to the second field and a corresponding second set of candidate label scores, the second set of candidate labels including a second candidate label corresponding to a second candidate label score in the second set of candidate label scores; said data processing system evaluating whether said second candidate label should be assigned to said second field using said previous candidate label score and said second candidate label score; Further comprising: The evaluating step includes: when the second candidate label score is greater than each of the previous candidate label scores by at least a second threshold amount, presenting the second candidate label to the user by generating an interface in which the user can provide input indicating whether the second candidate label should be assigned to the second field; bypassing presentation of the second candidate label to the user when the second candidate label score is not greater than each of the previous candidate label scores by at least the threshold amount; The method of claim 1 , comprising:

4. the data processing system identifying a third field that does not have an assigned label that indicates a semantic meaning of the third field, wherein no candidate labels have been previously identified for possible assignment to the third field; the data processing system identifying a third set of one or more candidate labels for possible assignment to the third field and a corresponding third set of candidate label scores, the third set of candidate labels including a third candidate label corresponding to a third candidate label score in the third set of candidate label scores; said data processing system evaluating whether said third candidate label should be assigned to said third field; Further comprising: The evaluating step includes: presenting the third candidate label to the user by generating an interface in which the user can provide input indicating whether the third candidate label should be assigned to the third field. The method according to any one of claims 1 to 3, comprising:

5. Identifying the set of one or more candidate labels for possible assignment to the first field and the corresponding set of candidate label scores includes: applying a plurality of label proposal tests to the profile of the first field to obtain a plurality of label proposals and corresponding label proposal scores; using the plurality of proposed labels and the corresponding plurality of proposed label scores to identify the one or more sets of candidate labels and corresponding sets of candidate label scores; The method of claim 1 , comprising:

6. The plurality of label proposal tests are associated with a plurality of weight values, and identifying the set of candidate label scores includes: using the plurality of label proposal scores and the plurality of weight values ​​to identify the set of candidate label scores; The method of claim 5 , comprising:

7. 2. The method of claim 1, further comprising: in response to receiving user input indicating assignment of the first candidate label to the first field, the data processing system storing, in a data store, an identifier of the first field in association with the first candidate label to indicate the semantic meaning of the first field.

8. After the data processing system evaluates whether the first candidate label should be assigned to the first field, the data processing system identifying a label to be assigned to the first field; associating one or more attributes with the first field based on the label assigned to the first field, the one or more attributes indicating that data in the first field contains personally identifiable information (PII); processing at least some data from the first field based on the one or more attributes by the data processing system; The method of claim 1 , further comprising:

9. The method of claim 1 , wherein processing the at least some data from the first field based on the one or more attributes comprises masking PII in the data from the first field.

10. After the data processing system evaluates whether the first candidate label should be assigned to the first field, the label assigned to the first field is used by the data processing system to execute at least one software application that applies data quality, anonymizes PII, and / or performs data masking of personally identifiable information (PII); The method of claim 1 , further comprising:

11. The method of claim 1 , wherein the label assigned to the first field indicates that the first field contains PII.

12. after the data processing system evaluates whether the first candidate label should be assigned to the first field; receiving, by the data processing system, a request to process data managed by the data processing system; accessing the label assigned to the first field by the data processing system in response to the request; the data processing system identifying one or more data processing rules for processing the data based on the label assigned to the first field; processing said data by said data processing system in accordance with said identified one or more data processing rules; The method of claim 1 , further comprising:

13. The method of claim 12 , wherein the data includes data stored in the first field.

14. Identifying one or more data processing rules for processing the data based on the label assigned to the first field includes: identifying the label assigned to the first field within a label index data structure; identifying from the label index data structure the one or more data processing rules associated with the labels identified in the label index data structure, wherein one or more data processing rules are associated with the labels in the label index data structure.

14. The method of claim 12 or 13, comprising:

15. 15. The method of claim 14, wherein the label assigned to the first field indicates that the data stored in the first field includes PII, and the one or more data processing rules include one or more data masking rules for masking PII.

16. 16. The method of claim 15, wherein processing the data in accordance with the identified one or more data processing rules comprises masking any PII contained in the first field without scanning the data in the first field to identify whether it contains PII.

17. 13. The method of claim 12, wherein the label assigned to the first field indicates that the data stored in the first field includes PII, and the one or more data processing rules include one or more data anonymization rules for anonymizing PII in the first field.

18. 13. The method of claim 12, wherein processing the data in accordance with the identified one or more data processing rules comprises processing the data in accordance with the identified one or more data processing rules without parsing data stored in the first field.

19. 1. A data processing system for discovering semantic meaning of data in fields contained in one or more datasets, comprising: at least one processor; at least one non-transitory computer-readable storage medium storing instructions that, when executed by the at least one processor, cause the at least one processor to perform a method; wherein the method comprises: identifying a first field having a previously assigned label indicating a semantic meaning of the first field, the previously assigned label having a corresponding previously determined label score; identifying a set of one or more candidate labels and a corresponding set of candidate label scores for possible assignment to the first field in place of the previously assigned label, the set of candidate labels including a first candidate label corresponding to a first candidate label score in the set of candidate label scores; using the previously determined label score and the first candidate label score to evaluate whether the first candidate label should be assigned to the first field; and wherein said evaluating comprises: when the first candidate label score is greater than the previously identified label score by at least a first threshold amount, presenting the first candidate label to the user by generating an interface in which the user can provide input indicating whether the first candidate label should be assigned to the first field instead of the previously identified label; bypassing presentation of the first candidate label to the user when the first candidate label score is not greater than the previously identified label score by at least the threshold amount; a data processing system including:

20. At least one non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor of a data processing system, cause the at least one processor to perform a method for discovering meaning of data in fields included in one or more datasets, the method comprising: identifying a first field having a previously assigned label indicating a semantic meaning of the first field, the previously assigned label having a corresponding previously determined label score; identifying a set of one or more candidate labels and a corresponding set of candidate label scores for possible assignment to the first field in place of the previously assigned label, the set of candidate labels including a first candidate label corresponding to a first candidate label score in the set of candidate label scores; using the previously determined label score and the first candidate label score to evaluate whether the first candidate label should be assigned to the first field; and wherein said evaluating comprises: when the first candidate label score is greater than the previously identified label score by at least a first threshold amount, presenting the first candidate label to the user by generating an interface in which the user can provide input indicating whether the first candidate label should be assigned to the first field instead of the previously identified label; bypassing presentation of the first candidate label to the user when the first candidate label score is not greater than the previously identified label score by at least the threshold amount; At least one non-transitory computer-readable storage medium, including:

Images