generating rules for data processing values for data fields from semantic tags of the data fields

Abstract

The methods and systems of the present disclosure are configured to determine a semantic meaning of data and generate data processing rules based on the semantic meaning of the data. The semantic meaning includes a syntactic meaning or a contextual meaning of the data, determined, for example, by profiling values stored in fields included in data records of one or more data sets by a data processing system; applying one or more classifiers to the profiled values by the data processing system; identifying one or more attributes indicative of logical or syntactic properties of the values of the fields based on applying the one or more classifiers, wherein each of the one or more attributes has a respective confidence level based on an output of each of the one or more classifiers. The attributes are associated with the fields and used to generate data processing rules and for processing data.

Description

[0001] Priority Statement

[0002] This application claims priority to U.S. Patent Application Serial No. 17 / 006,504, filed August 28, 2020, and U.S. Patent Application Serial No. 62 / 981,646, filed February 26, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to generating rules for processing data in a data field based on the results of classification of the data field's values. More specifically, this disclosure relates to classifying a data field by analyzing its data and metadata data profiles, and generating one or more rules for processing the data in the data field. Background Technology

[0004] Computer systems can be used to send, receive, and / or process data. For example, a server computer system can be used to receive and store resources (e.g., website content, such as web pages) and make the content available to one or more client computer systems. Upon receiving a request for content from a client computer system, the server computer system can retrieve the requested content and send it to the client computer system to fulfill the request.

[0005] A dataset can include data values ​​stored in data fields. A data field can include one or more data values. Many instances of a data field can exist in the dataset (e.g., in different fields). Summary of the Invention

[0006] The system described in this document is configured to generate one or more rules for processing data based on semantic labels on the data being processed. These labels indicate the semantic meaning of the labeled data. The semantic meaning of data is the actual meaning that the data represents. For example, semantic meaning might indicate that a particular field represents a date, and that the data represents the birth date of a user profile. The semantic meaning of data can also indicate how the data is processed by the data processing system. More specifically, the data processing system is configured to receive data values ​​of data fields labeled with semantic tags. The data processing system is configured to generate one or more rules for processing the data values ​​of a data field based on the label content of that data field. These rules may include data masking rules, data quality rules, rules for generating test data, rules for determining patterns, or other such rules. Typically, the rules applied to the labeled data depend on the semantic meaning of that data, even if an entire category of rules (e.g., data quality rules) is applied to all data. For example, a data quality rule for a birth date may differ from a data quality rule for a valid date, even though both are dates and have similar formats. The data processing system determines the semantic meaning of the data, labels the data with tags that indicate the semantic meaning, and generates one or more rules for processing the data based on those tags.

[0007] The implementation plan may include one or more of the following features.

[0008] In one aspect, a process for determining data quality rules for values ​​in fields of data records within a set of data records based on labels associated with fields, the labels indicating characteristics of the field values. This process is implemented by a data processing system. The process includes retrieving a label index that associates labels with a set of one or more fields in a data record, wherein the label identifies the expected information type in each of the set of one or more fields. The process includes accessing a data dictionary that associates the information type indicated by the label with a set of attribute values ​​representing requirements for the values ​​of the one or more fields associated with the label, the requirements including logical or syntactic characteristics of the values ​​of the one or more fields. The process includes, for a field in a specific data record: identifying a specific label associated with the field in that specific data record by accessing the label index; retrieving attribute values ​​of that specific label from the data dictionary, the attribute values ​​specifying specific requirements for that field. The process includes generating data quality rules that, when executed, are configured to: verify whether the field values ​​meet the specific requirements represented by the attribute values; and generate output data indicating whether the specific requirements are met.

[0009] In some implementations, data quality rules indicate one or more of the following: a value in a field corresponds to a required permissible deviation, one or more permissible values ​​in the field, and one or more prohibited values ​​in the field. In some implementations, one or more permissible or prohibited values ​​in the field are associated with the field name. In some implementations, the field is a first field, and one or more permissible or prohibited values ​​in the field are determined based on values ​​in a second field in a specific data record, which is related to the first field of that specific data record. In some implementations, one or more permissible or prohibited values ​​are based on a combination of values ​​in the second field and values ​​in the first field. In some implementations, one or more permissible values ​​correspond to values ​​that satisfy a numerical function, and one or more prohibited values ​​correspond to values ​​that do not satisfy that numerical function.

[0010] In some implementations, the field is a first field, where the data record includes a second field, where the data quality rule is a first data quality rule, and where the method includes: determining a relationship between the first field and the second field based on attribute values ​​of a specific label; and generating a second data quality rule for the second field based on the relationship between the second field and the first field. In some implementations, the relationship indicates the dependency of a value of the second field on a value of the first field or vice versa. In some implementations, the relationship indicates the correlation between values ​​of the first field and values ​​of the second field. In some implementations, the process includes obtaining validation data that validates the relationship between each value of the first field and the second field. In some implementations, the relationship is validated against a threshold number of values ​​of the first field and the second field. In some implementations, the second data quality rule is configured to enforce constraints on the values ​​of the second field based on the values ​​of the first field. In some implementations, determining that a relationship exists between the first field and the second field includes: determining that the values ​​of the second field include key values ​​referenced by values ​​of the first field, and where the data quality rule is configured to require each value of the second field to be a valid key value. In some implementations, the first field and the second field each include numerical values, and determining the relationship between the first field and the second field includes determining a numerical function that associates the values ​​of the first field and the values ​​of the second field. In some implementations, determining the relationship between the first field and the second field includes using at least one classifier configured by a machine learning process.

[0011] In some implementations, the process includes determining attribute values ​​associated with a specific tag that indicate a field includes the primary key value for a specific data record. In some implementations, the process includes configuring data quality rules to require that each primary key value be unique within a field.

[0012] In some implementations, the attribute value of a specific label represents at least one of the following: the average value of the field, the maximum length of the value, the minimum length of the value, the data type of the value, and the format of the value. In some implementations, the specific data record is a first data record, and the method includes applying data quality rules to another field associated with the specific label in a second data record, which is different from the first data record. In some implementations, the method includes storing data that associates the data quality rules and the specific label.

[0013] In some implementations, the process includes generating data for requesting approval of data quality rules; and approving data quality rules in response to obtaining approval data indicating approval of the data quality rules.

[0014] In some specific implementations, generating data quality rules includes: determining the historical trend of values ​​in a field; and generating requirements based on the historical trend of values ​​in that field.

[0015] In some specific implementations, generating data quality rules for this field includes: identifying the historical average of the values ​​in the field; and generating requirements based on the historical average of the values ​​in the field.

[0016] The implementation plan may include one or more of the following features.

[0017] The process is executed by a data processing system that selects test data to trigger the execution of processing rules during testing of a data processing application. The process includes retrieving a tag index that associates tags with a set of one or more fields in a data record, where the tag identifies the expected information type in each of the set of one or more fields. The process also includes accessing a data dictionary that associates the information type indicated by the tag with a set of attribute values ​​representing requirements for the values ​​of the one or more fields associated with that tag, including logical or syntactic characteristics of the values ​​of the one or more fields. The process involves, for a specific data record's field: identifying a specific tag associated with the field of that specific data record by accessing the tag index; retrieving the attribute value of that specific tag from the data dictionary, which specifies a specific requirement for that field; generating a subset rule that, when executed, specifies whether the field's value includes the attribute value; selecting a subset of fields from the specific data record according to the subset rule, where each selected field includes a value with the attribute value; and providing the selected subset to the data processing application for testing.

[0018] In some implementations, the process includes using a data processing application to process a first data record. This data processing application includes processing rules configured to manipulate values ​​of fields associated with a tag in the first data record and generate at least one output value. The process includes obtaining execution information indicating the number of times the processing rules are executed in conjunction with processing the first data record, wherein whether the processing rules are executed by the data processing application during the processing of the first data record depends directly or indirectly on the value of the field associated with the tag. In some implementations, a subset of rules is determined based on the execution information indicating the number of times the processing rules are executed in conjunction with processing the first data record. This subset of rules includes the identification of fields in the first data record.

[0019] In some implementations, a subset rule identifies the field as a key field of a specific data record, the key field including at least one key value of a data element of the specific data record. In some implementations, a subset of the fields of a specific data record includes fields with key values ​​having predetermined values. In some implementations, a subset rule identifies a list of key field names corresponding to a label, and the subset rule identifies a field as a key field by comparing the attribute values ​​of the label with the list of key fields. In some implementations, the fields of a specific data record include Personal Identification Information (PII), and the method further includes selecting a subset of the fields of the specific data record where the field does not include PII information. In some implementations, the fields of a specific data record include Personal Identification Information (PII), and the method further includes applying a masking function to the PII to generate masked data, and selecting a subset of the fields of the specific data record that includes the masked data.

[0020] The implementation plan may include one or more of the following features.

[0021] In a general sense, a process is performed by a data processing system configured to mask data for a data processing application. The process includes retrieving a tag index that associates tags with a set of one or more fields in a data record, wherein the tag identifies an expected information type in each of the set of one or more fields. The process includes accessing a data dictionary that associates the information type indicated by the tag with a set of attribute values ​​representing a requirement for the value of the one or more fields associated with the tag, the requirement including logical or syntactic characteristics of the value of the one or more fields. The process includes, for a specific data record's field: identifying a specific tag associated with the field of the specific data record by accessing the tag index; retrieving the attribute value of the specific tag from the data dictionary, the attribute value specifying a specific requirement for the field; and determining, based on the attribute value, the field of the specific data record representing sensitive data. The process includes, in response to the determination, executing a data masking function to convert the values ​​of the sensitive data including the field into masked values.

[0022] In some implementations, the field is a first field, and the label is a first label. The process includes determining that a relationship exists between the first field with the first label and the second field with the second label. The process includes, in response to determining the existence of a relationship, performing a data masking function to convert values ​​in the second field into masked values. In some implementations, determining the existence of a relationship includes determining that the first label and the second label are associated with a common data source. In some implementations, the common data source includes user profiles. In some implementations, the relationship indicates the dependency of the second field's value on the first field's value or the dependency of the first field's value on the second field's value. In some implementations, the relationship indicates the correlation between a first value of the first field and a second value of the second field. In some implementations, the relationship includes an arithmetic function. In some implementations, the process includes selecting the type of data masking function based on the type of the relationship.

[0023] In some implementations, the data masking function includes at least one of shuffling functions, data encryption, character scrambling, and data substitution. In some implementations, the process includes scanning a specific data record to determine whether one or more specific values ​​of at least one other field are converted to masked values. In some implementations, the process includes selecting the type of data masking function based on the attribute value of the label. In some implementations, a first data masking function is selected for a first field of a specific data record, the first field including numeric values, and a different second masking function is selected for a second field of the specific data record, the second field including non-numeric values.

[0024] In some implementations, the process includes selecting the type of data masking function based on a label of another field related to the field. In some implementations, sensitive data includes personally identifiable information (PII).

[0025] The implementation plan may include one or more of the following features.

[0026] In a general sense, a process is performed by a data processing system configured to determine a schema of data records. The process includes retrieving a tag index that associates tags with a set of one or more fields in a data record, wherein the tag identifies an expected information type in each of the set of one or more fields. The process includes accessing a data dictionary that associates the information type indicated by the tag with a set of attribute values ​​representing a requirement for the value of the one or more fields associated with the tag, the requirement including logical or syntactic characteristics of the value of the one or more fields. The process includes, for a first field of a particular data record: identifying a specific tag associated with the first field of the particular data record by accessing the tag index; retrieving attribute values ​​of the specific tag from the data dictionary, the attribute values ​​specifying a specific requirement for the first field; determining a schema feature representing a value included in the first field; determining, based on the schema feature, that the first field with the tag includes a key value referenced by a value of a second field; and, in response to the determination, updating schema data describing the particular data record to reference the key value referenced by the value of the second field in the first field.

[0027] In some implementations, the process includes generating a data flow diagram based on the schema data. In some implementations, the second field resides in a second data record that is different from the specific data record.

[0028] In some implementations, the process includes generating an association function configured to associate a specific data record with a second data record based on a key value. In some implementations, the second field is associated with a second label that is different from the first label.

[0029] The implementation plan may include one or more of the following features.

[0030] In a general sense, a process for determining data quality rules for values ​​in one or more datasets, implemented by a data processing system, includes the data processing system archiving values ​​stored in a field included in data records of the one or more datasets. The process includes the data processing system applying one or more classifiers to the archived values. The process includes identifying one or more attributes based on the application of the one or more classifiers, indicating logical or syntactic characteristics of these values ​​in the field, wherein each of the one or more attributes has a corresponding confidence level based on the output of each of the one or more classifiers. The process includes associating one or more of the identified attributes, wherein the confidence level satisfies the threshold level, with the field. The process includes determining one or more constraints on the values ​​included in the field based on the one or more attributes associated with the field. The process includes determining data quality rules for the field in the dataset based on the determined one or more constraints, the data quality rules indicating the relationship between the values ​​of the field and the one or more constraints.

[0031] Any of these processes can be implemented as a system comprising one or more processing devices and a memory storing instructions configured, when executed by the one or more processing devices, to cause the one or more processing devices to perform the operations of these processes. In some embodiments, one or more non-transitory computer-readable media can be configured to store instructions configured, when executed by the one or more processing devices, to cause the one or more processing devices to perform the operations of these processes.

[0032] These aspects can include one or more advantages. For example, the techniques described herein enable data processing systems to automatically generate one or more rules for processing data fields in a dataset. Once the semantic meaning of the data is understood, the data processing system determines which data processing operations to apply to the data values ​​to achieve the application's specified goals. Therefore, the data processing system can automatically determine how to process different data fields of a dataset (such as masking data values, enforcing data quality rules, identifying patterns in the dataset, and / or selecting test data for testing another application) to achieve the goals of the entire dataset. Automatically determining which data fields have which semantic meanings allows these rules to be applied to large datasets where manually labeling each field might be impractical or impossible for a user. For example, for datasets with hundreds, thousands, or even millions of tables, manually labeling each field to enforce rules such as data masking rules might be impractical. In another example, automatically masking sensitive data can be useful before any user views it. In yet another example, the data processing system can mask sensitive data stored in data fields that are not intended to include sensitive data.

[0033] The data processing system also identifies data fields that do not require processing based on data processing rules. For example, the system can determine data fields that do not need to be masked. Identifying which data fields do not require processing through data processing operations saves processing time on the entire dataset, which is used to ensure that the application's specified objectives (e.g., masking) are met.

[0034] The data processing system overcomes at least one or more of the following challenges in applying rules to datasets. Typically, data fields in a dataset are not named in a standardized way, making the field names reliable indicators of their contents. It is often important for the data processing system to understand the meaning of the data values ​​in the dataset's data fields, as the processing requirements for a data field to achieve a specified application objective often depend on the meaning of the data values ​​in that data field. For example, if the application's objective is to mask personally identifiable information (PII), the data processing system determines which data in the dataset includes PII before applying a masking function to that data. In some implementations, specific data fields may require the application of specific masking functions (e.g., to meet security requirements for the data values ​​in that data field). Furthermore, configuring the data processing system to determine the meaning of the data values ​​in a data field can be difficult, as the meaning is not always obvious from the field name, the format of the field's data values, or the data values ​​themselves.

[0035] Details of one or more embodiments are set forth in the accompanying drawings and the following description. Other features and advantages will be apparent from the description and drawings, as well as from the claims. Attached Figure Description

[0036] Figure 1 This is a block diagram of a data processing system.

[0037] Figures 2A to 7B Each includes provisions for data quality rule processing. Figure 1 A block diagram of an exemplary data processing module.

[0038] Figure 8 This is a block diagram of a data processing system.

[0039] Figures 9 to 12 Flowcharts, each including exemplary processes, are shown. Detailed Implementation

[0040] Figure 1 This is a block diagram of a data processing system 100. The data processing system 100 is configured to process data records from input data 112. The data processing system includes a semantic discovery system 602 and a data processing device 102. The semantic discovery system 602 is configured to determine the meaning (e.g., semantic meaning) of the values ​​of one or more fields of a data record. The semantic discovery system 602 can label each of these fields with a semantic tag 118, which is selected from a data dictionary database 614. The semantic tag 118 (also called a label) is associated with one or more attributes 120 in the data dictionary database 614. The attributes associated with the semantic tag 118 define the semantic meaning of that tag.

[0041] Data processing device 102 is configured to receive tag index 610 and data dictionary database 614, as well as input data 112, from semantic discovery system 602. Data processing device 102 is configured to process input data 112 based on tags in tag index 614 and tags and attributes 120 in data dictionary database 614. Tags 118 in database 614 and tag index 610 are generated by the semantic discovery process of semantic discovery system 602.

[0042] Semantic discovery system 602 is configured to receive input data 112 and determine the semantic meaning of fields in the input data. The semantic meaning of a field is a description of the actual (e.g., business) meaning of the field's value, and is subsequently described in further detail. To determine the semantic meaning of fields in input data 112, semantic discovery system 602 archives the input data 112 and performs multiple classifications that analyze the profile data and values ​​of the fields. The classification of the field profile data and values ​​enables semantic discovery system 602 to determine the attributes most associated with the fields. Attribute 120 includes metadata (or other data) indicating the characteristics of a given field. For example, attribute 120 may indicate a specific format, a specific relationship between the field and one or more other fields, allowed or prohibited values ​​for the field, associated key terms or business terms, statistical criteria for the values ​​of associated fields (individually relative to other values ​​of the field, or as a group of field values), and so on.

[0043] The semantic discovery system 602 is configured to generate a tag index 610 that associates fields of data records of input data 112 with tags 118 in the data dictionary database 614. When a field of a data record is processed by the data processing device 102, the tag index 610 can be used to retrieve the semantic tag 118 (if any) associated with that field.

[0044] (For example, the semantic label 118) includes a term or set of terms (typically strings or alphanumeric characters) that indicates the semantic meaning of the data in the field tagged by that semantic label. Semantic label 118 is a recognized term or set of terms appearing in the data dictionary associated with the data processing system. Each semantic label 1118 is associated with one or more fields in the dataset. For example, a table can associate a semantic label 118 with individual fields (or each instance of an individual field) in the dataset that have data values ​​that carry the meaning of the semantic label.

[0045] In some implementations, the semantic discovery system 602 does not necessarily generate actual labels to identify the semantic meaning of a field, but instead associates the field with a corresponding attribute 120 that indicates the semantic meaning. For example, the label index 610 can associate a field with entries in a data dictionary database 614 that include one or more attributes 120 but do not include specified labels for the field. In other implementations, the labels are used as abbreviations that indicate a set of one or more attributes 120 in the data dictionary database 614 that represents the semantic meaning of the field.

[0046] Typically, the semantic meaning of a field can correspond to either the actual meaning or the contextual meaning of the field's value. The actual meaning (e.g., business meaning) or contextual meaning refers to one or more syntactic features of the data value and / or contextual indicators of the data value in a data record. When interpreted together, these one or more syntactic features convey how these values ​​are processed by one or more modules of the data processing device 102, which use the values ​​of data records for various applications. These syntactic features are indicated by attributes 120 associated with each tag 118 in the data dictionary database 614. For example, for data values ​​corresponding to dates, the semantic meaning could indicate that these data values ​​represent (e.g., the date of birth of a customer or user associated with these data records). In another example, the semantic meaning could indicate that a numeric identifier represents a Social Security number, which is a specific type of numeric identifier and has specific characteristics unique to Social Security numbers. In yet another example, the semantic meaning could indicate that the values ​​of two different fields convey a business meaning when grouped together. For example, if a number is identified as a postal code, and that number is paired with other fields that have already been identified as representing a city name and a country name, then the semantic discovery system 602 is configured to determine that the semantic meaning of all three fields combined is an address. Then, further examples of semantic meaning are described in detail.

[0047] In some implementations, the semantic meaning of a field includes the actual meaning of the data in that field (e.g., business meaning or real-world meaning). Semantic meaning can be more specific than simply indicating what type of data a field includes. For example, a field that includes a date can be identified from a data profile. Data processing system 100 can determine that a field includes a date based on the length of the values ​​in the field (e.g., the average length of the values). Data processing system 100 can determine that a field includes a date based on the format (e.g., values ​​are all numbers or follow a format like ##\##\####, etc.). Other statistics can indicate to data processing system 100 that a field includes a date. However, the semantic meaning of a field could be that the field represents the birth date of a user of the application, which can be distinguished from other dates associated with that user (such as the date the user profile was created).

[0048] As previously described, semantic label 118 is associated with one or more attributes 120 in the data dictionary database 116. Attribute 120 indicates the meaning of data values ​​in a data field. Attribute 120 may indicate that these data values ​​have specific characteristics. For example, attribute 120 may indicate that these data values ​​include personally identifiable information (PII). Attribute 120 may indicate that the data values ​​of this data field have a specific relationship with other data fields. For example, attribute 120 may indicate that the data field includes database keys. Attribute 120 may indicate the specific meaning of the data values ​​in the field, implied by the format of the data values, by the data values ​​themselves, by the name of the data field, and by comparison of the data values ​​of the data field with other values ​​of other data fields in the dataset. For example, attribute 120 may indicate that the data values ​​of the data field include a date of birth (rather than just a date or a date for some other purpose). See below for further details. Figure 8 The semantic discovery process of the semantic discovery system 602 is described in more detail.

[0049] In some implementations, semantic labels (or attribute 120 associated with the labels) can be associated with a set of fields rather than a single field. For example, if fields in a set are determined to represent street name, street number, postal code, and city, the semantic discovery process can determine that the set of fields together represents an address, and can associate address attributes with that set of fields. For example, attribute 120 can indicate that all these fields should be included as a group in a given set of data (e.g., for data quality checks targeting missing parts of addresses). In some implementations, attribute 120 can indicate that these fields together represent a PII, but that these fields do not individually represent a PII. This can be used for data masking checks. For example, if the dataset includes all fields, the data masking module 108 can determine that data masking is needed, where data masking is not needed if the dataset includes only one of these fields. This can also be used for pattern analysis, where these attributes indicate that each field in the group is related to the other fields. Other such examples can be used for multi-field analysis, which is described in detail below.

[0050] like Figure 1 As shown, when data processing system 100 labels a field with semantic tag 118, the field is associated with attribute 120 of the semantic tag. Rule modules 104, 106, 108, and 110 of data processing device 102 use attribute 120 to determine rules to be applied to a given field of the dataset or rules to be generated for a given field of the dataset. Data processing device 102 is capable of processing data for each field in a unique way that automatically applies different rules to each field. Data processing device 102 is capable of generating rules for processing a given field of the dataset.

[0051] Modules 104, 106, 108, and 110 of data processing device 102 are configured to process input data according to one or more processing rules. Typically, rules for processing data values ​​in a dataset can include one or more data processing operations performed on data values ​​when they are in a field associated with a specific semantic label. The rule can include a single operation or a list of more than one operation. Semantic label 118 can indicate to the data processing system (e.g., a rule from a rule base) which rules should be applied to data values ​​in a given data field of the application, or whether any rule should be applied to data values ​​in a given field. For example, the rule can include a data quality rule that imposes data requirements on each data value in a given field. These data requirements could be format requirements, range requirements, etc. The rule can include a data masking rule that requires the data value to be converted to a different format before a downstream application can process it. Many other types of rules are also possible, as described below.

[0052] Data processing device 102 is configured to receive data from data dictionary 614 and tag index 610 for processing input data 112. Data processing device 102 is configured to use semantic tags 118 for one or more applications among several different applications. Data processing device 102 includes a data quality rules module 104 (also referred to as data quality rules module 104). Data quality rules module 104 is configured to generate data quality rules for fields of data records (e.g., input data 112) based on semantic tags associated with each field in the fields. In an example, a first set of data quality rules can be automatically generated for a first field having a first semantic tag (and this first set of data quality rules can be used to process the first field having the first semantic tag). For example, if the first field is tagged as a Social Security number, data quality rules module 104 can be configured to generate a data quality rules model 104, which can be configured to generate a first set of data quality rules configured to test whether the value of the field is actually a valid Social Security number. In another example, a second field can be tagged as a birth data field. The data quality rules module 104 can be configured to generate a different second set of data quality rules, which are configured to test whether the value of the second field is actually a valid birth date (and not other dates or other types of data). About Figures 2A to 3C The data quality rules module 104 is described in further detail.

[0053] Data processing device 102 includes a test dataset module 106. Test dataset module 106 is configured to generate test datasets for testing data processing applications. Typically, to test an application, actual test data is used to ensure that all features of the application function as expected. Generating test data for all functionalities of the application can be difficult and time-consuming because there may be many processing permutations, case structures, etc., that need to be tested. Furthermore, test data should function in a manner similar to real data because it should adhere to referential integrity and internal logical consistency. For example, links in the test data should point to valid data that can also be tested. Test dataset module 106 is configured to ensure that the test data meets these requirements. For example, test dataset module 106 is configured to generate test data that satisfies referential integrity (e.g., pointers to valid fields), includes correct data (e.g., the test data includes a Social Security number field if necessary), excludes foreign data (e.g., fields not needed in the test data are excluded from the test data), and so on. (See reference...) Figures 4A to 5B The test dataset module 106 is described in further detail.

[0054] Data masking module 108 is configured to generate rules for masking data fields in input data 112. Data masking can be used to ensure that sensitive data (such as personally identifiable information (PII)) is masked so that the data no longer represents a real person or actual entry in a data record. For example, data masking can involve the values ​​of fields in anonymizing data records. Data masking can include encrypting the values ​​of specific fields. As described later, many other methods for masking data are also possible. Data masking module 106 is configured to mask fields using specific masking functions. A specific masking function can be selected for a field based on its semantic tag. For example, a semantic tag associated with a field can specify how the data will be masked (e.g., specifying the masking function or masking technique to be used for the field). Thus, a semantic tag can specify not only which fields should be masked, but also how those fields should be masked.

[0055] The dataset pattern analysis module 110 is configured to identify relationships among the fields of the input data 112 and determine patterns in the input data 112 based on these relationships. A pattern indicates the structure of the input data 112. For example, the pattern of the input data 112 may indicate key-value relationships among fields, field dependencies on other fields, value correlations between fields, and so on. One or more of the other modules 104, 106, and 108 of the data processing device 102 process the input data 112 based on the identified patterns, including generating or applying data quality rules, generating or applying data masking strategies, and generating test datasets. In some implementations, the pattern of the input data 112 is determined in parallel with the process of the semantic discovery system 602. In some implementations, the dataset pattern analysis module 110 uses semantic labels generated by the semantic discovery system 602 to determine the patterns of the input data 112.

[0056] Typically, data processing device 102 receives a dataset from input data repository 112 and outputs processed data to output data repository 114. In some embodiments, input data repository 112 and output data repository 114 are included in the same data repository. In some embodiments, input data repository 112 and output data repository 114 are databases. The dataset received by data processing device 102 may include data records, data tables, or other data, where groups of one or more values ​​are associated with names or labels. For example, the dataset may include one or more fields, each including one or more values. Each field may include a field name. Field names may be technical names, code names, business terms, numeric values, or any other values. In some embodiments, these fields do not have field names. Typically, one or more fields in the input data 112 are associated with corresponding semantic labels. Each semantic label indicates the syntactic or semantic meaning of the field, as previously described.

[0057] Data processing system 100 includes a reference data repository 116. Reference data repository 116 stores data used by data processing system 100 to operate modules 104, 106, 108, and 110. For example, if data quality rules module 104 requires data quality rules from a data quality rules repository, data processing device 102 retrieves data quality rules from reference data repository 116 based on the criteria described below. Reference data repository 116 stores information such as masking functions used during the data masking process. Reference data repository 116 may store tag index 610 and data dictionary database 614 data. Reference data repository 116 may include an in-memory data repository or a persistent data repository. Reference data repository 116 may be a single data repository or a data repository on a distributed network. Essentially, reference data repository 116 is a place for storing data used in various processes of data processing system 100.

[0058] Figures 2A to 7B Each is a description Figure 1 A block diagram of the data processing module of the data processing device 102. (Go to...) Figures 2A to 3C The data quality rules module 104 is configured to generate data quality rules for the dataset or apply data quality rules to the dataset (such as input data 112) using the semantic meaning of the fields in the dataset. The semantic meaning of the fields is determined by the semantic discovery system 602, as previously discussed. Figure 1 As described, and as subsequently discussed Figure 8 As described in detail.

[0059] Figure 2A The process 200a, performed by the attribute analysis module 204 of the data quality rule engine 104, is illustrated. The attribute analysis module 204 is configured to determine attributes associated with fields having values ​​processed by the data quality rule module 104, such that one or more data quality rules for that field can be generated by module 104. The attribute analysis module 204 receives a dataset 201 that includes fields associated with attributes of the data dictionary 614 via label index 610. As previously indicated, semantic labels 228 can associate attributes with fields, but no actual labels are required. For simplicity, subsequent examples include semantic labels for associating the semantic meaning indicated by the attributes with fields, but in each of the subsequent examples, no actual labels (e.g., descriptive words or terms) are required.

[0060] The attribute analysis module 204 determines the attributes of a given field for which data quality rules will be generated or applied. The attribute analysis module 204 is configured to receive (250) dataset 201. The attribute analysis module selects (252) the fields of dataset 201. Figure 2A In the example, the selected field 218 is named "SSN", although a field name is not required. The value 220 of the field "SSN" is "55-555-5555".

[0061] Once a field is selected, the attribute analysis module uses tag index 610 to look up (254) semantic tag 228 in tag index 610. In some implementations, tags are not received, but entry identifiers (or some other indicators in which the attribute is associated with the selected field) in data dictionary database 614 are received instead of tags. Attribute analysis module 204 performs a lookup (256) to determine the attribute associated with tag 228a “Social Security Number”. Here, attribute 230 is associated with tag 228a in data dictionary database 614. Attribute 230 is then stored (e.g., in memory) for use in generating or applying data quality rules by rule generation module 206 or rule application module 208 of data quality rule generation module 104.

[0062] Go to Figure 2B In process 200b, rule generation module 206 is configured to determine one or more data quality rules for a field based on the semantic meaning of the field (e.g., indicated by attribute 230). For example, a specific attribute may indicate one or more rules that can be applied to a selected field. In this context, attribute 230 may indicate a set or range of allowed values, the relationship between the field's value and the values ​​of one or more other fields (e.g., always greater than, always less than, equal to, functionally related, etc.), data type, expected value length, or other characteristics of the field's value. For example, as described later, an attribute may indicate allowed values ​​in the field and disallowed values ​​in the field. For example, a given range indicated in an attribute associated with a field's label may also be a data quality rule for that field. However, the attributes associated with a field and the rules generated for that field do not necessarily match. For example, a data quality rule for a field may require the absence of duplicate values ​​or null values, while attribute 230 does not need to indicate such a rule. Instead, an attribute may indicate that the selected field meets a specific requirement, but the generated data quality rule does not need to enforce that requirement.

[0063] Rule generation module 206 is configured to receive (260) attributes 230 associated with a field. Rule generation is configured to generate (262) data quality rules for the selected field based on the attributes as previously described. Any combination of attributes 230 can be used to generate any number of data quality rules. These rules can enforce requirements for each value of the field independently of any other field or value in dataset 201. For example, data quality rule 214a might require that the values ​​of a field associated with the label “Social Security Number” be formatted as “XXX-XX-XXXX”, where X can be any number in the range 0 to 9. This requirement is based on each individual value and does not require further analysis. In another example, rule 214b might check the type of each character in the value to ensure that all X values ​​are indeed numbers in the range 0 to 9 and not other characters. In another example, data quality rules might require a check on the entire selected field. For example, data quality rule 214c requires that each value in the selected field be unique. This requires checking the values ​​in the selected field in addition to the currently processed values ​​to determine if each value is unique.

[0064] Other examples of data quality rules are possible. In some examples, a rule used to characterize the quality of a set of data can indicate allowed or prohibited attributes of the profiles of the data records in that set. Attributes of a profile can be values ​​or ranges of values. When a profile includes allowed attributes, the rule indicating the allowed attributes of that profile is satisfied. Examples of allowed attributes for a field can be the maximum and minimum allowed values ​​for that field; if the average of the field falls between the maximum and minimum allowed values, the rule is satisfied. Rules indicating prohibited attributes of a profile are satisfied as long as the profile does not include prohibited attributes. Examples of prohibited attributes for a field can be a list of prohibited values ​​for that field; if the field includes any of these prohibited values, the rule is not satisfied.

[0065] Data quality rules can indicate the permissible deviation between one or more values ​​of a field and the field's profile. A deviation between a profile and a field's value that exceeds the permissible deviation indicated by the corresponding rule can be an indication of data quality problems in the dataset, and therefore indicate that the dataset may be the root cause of existing or potential data quality problems in downstream datasets. In some examples, the permissible deviation can be specified as a range of values ​​(such as a maximum and minimum permissible value). In some examples, the permissible deviation can be specified as the standard deviation from a single value, which could be the mean (e.g., the mean or median of values ​​in a past dataset).

[0066] In some examples, rules used to characterize the quality of a set of data can be, for example, based on the validity of values ​​in a field, indicating allowed or prohibited characteristics of values ​​in each of one or more fields of a data record. A rule indicating a allowed characteristic of a field is satisfied when a value in a field satisfies that characteristic. A rule indicating a prohibited characteristic of a field is satisfied as long as a value in a field does not satisfy that characteristic. Values ​​that satisfy a rule are sometimes referred to as valid values; values ​​that do not satisfy a rule are sometimes referred to as invalid values. Various characteristics of values ​​in a field can be indicated as allowed or prohibited characteristics by rules. Exemplary rules can indicate allowed or prohibited characteristics of field content, such as a range of allowed or prohibited values, a maximum allowed value, a minimum allowed value, or a list of one or more specific values ​​that are allowed or prohibited. For example, a birth_year field with values ​​less than 1900 or greater than 2016 might be considered invalid. Exemplary rules can indicate allowed or prohibited characteristics of the data type of a field. Exemplary rules can indicate whether the absence of a value (or the presence of a null value) in a field is allowed or prohibited. For example, a surname field that includes a string value (e.g., "Smith") might be considered valid, while a surname field that is blank or includes a numeric value might be considered invalid. Exemplary rules can indicate allowed or prohibited relationships among two or more fields in the same data record. For instance, a rule could specify a list of values ​​for a postal code field corresponding to each possible value of the surname field, and could specify that any combination of values ​​for the postal code and country fields that is not supported by that list is invalid.

[0067] In some examples, rules can be generated based on the analysis of historical data. Rules generated by rule generation module 206 can indicate allowed or prohibited attributes of profiles of data records in a set of data. For example, a rule for a profile can indicate the allowed deviation between a profile of a field in a particular dataset and a defined historical profile of that field. The historical profile of a dataset can be based on historical data; for example, a historical profile can be a profile of the same dataset from a previous day, an average profile of the same dataset from multiple previous days (e.g., within the past week or month), or a lifetime average profile of the same dataset. More generally, the profile can retain various reference information to utilize various statistical analyses. For example, the profile can include information about the standard deviation or other indications of the value distribution. For the purposes of the following examples, and without limiting the generality of this application, the profile can include the numerical mean of previous datasets and may also include the standard deviation.

[0068] The generated rules can indicate specific allowed or prohibited characteristics for values ​​in fields of data records. In the example, the generated rules for a field can indicate the maximum or minimum allowed value for that field based on analysis of the field's historical maximum or minimum values. In the example, the generated rules for a field can indicate a list of allowed values ​​for that field based on analysis of values ​​that have previously occurred for that field.

[0069] In some examples, machine learning techniques are employed to generate data quality rules. For instance, data can be analyzed over a learning period so that user preferences or the application understands which characteristics affect the application's data quality and thus requires data quality rules to enforce those characteristics. For example, if the application typically fails due to incorrectly formatted field values, the rule generation module generates rules to enforce specific formats. If the application fails due to values ​​exceeding acceptable ranges, rule generation module 206 generates data quality rules to enforce specific value ranges for that field. Other such examples are possible. The learning period can be a specified time interval or the amount of time until the average or expected value converges to a stable value.

[0070] Rule generation module 206 is configured to generate one or more data quality rules based on the expected characteristics of the data records to be processed by the system. In a specific example, the source data is credit card transaction records for transactions occurring in the United States. The source data is streaming data processed incrementally on an hourly basis. Based on attribute 230 identified for this field and data from the application (which indicates the actions to be performed when processing credit card transaction records), the user can identify the transaction identifier field, card identifier field, country field, date field, and amount field as key data elements to be documented.

[0071] In a specific example where the source data is credit card transaction records, rule generation module 206 receives an attribute indicating that the country field contains only fifty allowed values. If the profile of the source data set identifies more than fifty values ​​in the country field, rule generation module 206 can create a rule that causes an alert flag to be used, regardless of the standard deviation of the profile of the source data set relative to a reference. Rule generation module 206 can also receive an attribute indicating credit card transaction records that were completed only on the same day, as the processing should exist in the source data set. If any source data record has a date that does not match the processing date, rule generation module 206 can create a rule that causes an alert message to be sent.

[0072] In some examples, rule generation module 206 can specify one or more rules generated via a user interface. Through the user interface, users can provide feedback on rules for one or more fields, or approve pre-populated default rules for fields. Further description of the user interface can be found in U.S. Patent Application Serial No. 13 / 653,995, filed October 17, 2012, the entire contents of which are incorporated herein by reference. Other specific implementations of the user interface are also possible.

[0073] In some examples, if a potential data quality issue is detected in a set of data (such as in a new version of a set of reference data or in a set of source data), the identifier of that set of data with the potential data quality issue is placed on a list of root cause datasets stored in the database. If a data quality issue is later detected with a set of output data, the database can be queried to identify the upstream data successor elements of that set of output data and to determine which of those upstream data successor elements (if any) are included on the root cause dataset list.

[0074] In some examples, user notifications can be enabled if a potential data quality issue is detected in a set of data (such as in a new version of a set of reference data or in a set of source data). In some examples, alert flags can be stored to indicate data quality issues. For example, if a potential data quality issue is detected in a new version of a set of reference data, the alert flags can be stored together with a profile of that new version of the reference data. If a potential data quality issue is detected in a set of source data, the alert flags can be stored together with a profile of that set of source data. In some examples, an alert message can be sent to the user to indicate the presence of a potential data quality issue. This alert message can be, for example, a message, icon, or pop-up on a user interface; an email or Short Message Service (SMS) message; or another form.

[0075] In some examples, rules can specify one or more threshold deviations from the profile used for alarm tags or alarm messages. For instance, if the deviation between the current set of data's profile and the profile of that set of data is small (such as between one and two standard deviations), an alarm tag can be stored; and if the deviation is greater than two standard deviations, an alarm message can be delivered. The threshold deviation can be specific to each set of source and reference data.

[0076] In some examples, such as if the bias is severe (e.g., more than three standard deviations compared to the profile), further processing by the data processing system can be stopped until user intervention. For example, stopping any further processing that would be affected by source or reference data with severe bias. The transformation to be stopped can be identified by referencing the data of downstream data successor elements of the affected source or reference data.

[0077] In some examples, profile data is determined automatically. For instance, the profile data for a given set of data can be automatically updated to the historical average of past profile data for that set of data (e.g., by recalculating the profile data whenever new profile data for a set of data is determined). In some examples, the user can provide initial profile data, for example, by profiled a set of data with desired characteristics.

[0078] continue Figure 2B Data quality rules 214a through 214c can be generated once and associated with a specific semantic label for later reuse. Not all constraints indicated by attribute 230 need to be enforced as explicit data quality rules 214, although this can be done if needed. Instead, a set of data quality rules 214 for enforcing constraints can be generated as needed for a given application. For example, the semantic label attribute might indicate that the length of all data in a field with the semantic label is 9. Data quality rules associated with the attribute might require that the length of each data value in the field is exactly 9. The length requirement can be specified in profile 216. Associated data quality rules can be generated based on this information. For example, data quality rule 214 could include a data quality rule specifying a length requirement of 1. The rule generation module can generate a data quality rule that requires a length of exactly 9 digits based on the available rules and after querying profile 216.

[0079] Rule generation module 206 generates at least one data quality rule for the field. The generated data quality rule can be specific to the field or common to more than one field. The data quality rule can indicate one or more of the following: the permissible deviation between the characteristics of the values ​​in the field of the dataset and one or more characteristics included in the field's profile, one or more permissible values ​​in the field, and one or more prohibited values ​​in the field.

[0080] As previously indicated, the data quality rule module 104 is configured to automatically generate data quality rules using attributes associated with semantic label 228. To this end, the rule generation module 206 converts one or more attributes into logical rules. For example, if an attribute indicates that all values ​​of a field are between 1 and 100, a data quality rule requiring all values ​​to be between 1 and 100 can be generated. However, it is not necessary to generate such a data quality rule. For example, another attribute associated with semantic label 228 could indicate that the last digit of each value in the field ends in 7. Therefore, a data quality rule requiring values ​​to be between 1 and 100 and ending in 7 can be generated. These could also be represented as two different rules. In another example, an attribute could indicate a specific format of a value, such as a value having a format like ###-##-####, which is the format of a Social Security number. The data quality rule generator can generate data quality rules requiring values ​​to follow this exact format, a portion of that exact format, or common variations (e.g., nine digits without any dashes).

[0081] Semantic label 228 can indicate the relationship between the value of this field and the value of another field. Rule generation module 206 can generate data quality rules based on these relationships. For example, if this field is a first field and the dataset includes a second field, the rule generation module can be configured to generate data quality rules for the first field based on the relationship between the second field and the first field. For example, the second field can be associated with a semantic label indicating that the field is filled with usernames. Data quality rules may require that a valid name in the second field be associated with each value in the first field to pass a data quality test for the first field. Combinations of rules 214a to 214c can be used. For example, a data quality rule output of "pass" can be generated specifying that for the SSN field, if the format is ###-##-#### and the value of each Social Security number 218 in the SSN field 220 is unique. This combines two of the three proposed rules (which may also be referred to as constraints), including constraints 214a and 214c.

[0082] In some implementations, these rules can be based on how the data values ​​of a field change over time. For example, a profile of a field can be selected based on the historical trend of the values ​​in the field. In another example, a profile of a field can be determined based on the historical average of the values ​​identified in the field.

[0083] Rule generation module 206 can generate data for requesting approval of data quality rules. For example, a user can approve or reject rules in a user interface. Rule generation module 206 can approve data quality rules for use in an application in response to obtaining approval data indicating approval of data quality rules. In some implementations, approval data is not required to apply rules. Once rule generation module 206 generates rules (e.g., rules 214a to c), rules 214 are stored (264) in a data repository (such as data quality rule data repository 250).

[0084] Turn Figure 2C In process 200c, rule application module 208 is configured to apply generated data quality rules 214 to any received data 112 based on the semantic meaning associated with the received data (such as attribute 230 associated with a field using semantic label 228).

[0085] Rule application module 208 is configured to receive (270) dataset 201. The rule application selects (272) fields of dataset 201, such as field 218 named “SSN”, which has a position 221 in the dataset at column 1 of Table 1. Rule application module 208 references (274) tag index 610 and finds the rule associated with the field in dataset 201 at column 1 of Table 1. Here, semantic tag 228 “Social Security Number” is used to represent the field. Rule application module 208 accesses (276) the data quality rule data repository 250 and retrieves data quality rule 214 for the field tagged “Social Security Number”. Rule application module 208 applies (278) these data quality rules to the value of field 218. In the example, the value 555-55-5555 passes the format rule, character restriction rule, and unique value rule. The result of all rules 214a through c is PASS. Application module 208 stores (280) the results 224 of the data quality rule application in the output data repository 114.

[0086] Data quality rules generated for fields associated with label 228 can be applied to any field associated with the same label in a second instance of dataset 201 or in a new, different dataset. For example, label 228 is the Social Security Number tag, for which rule generation module 206 has generated one or more data quality rules. These generated data quality rules can be applied to any field with the same label 228 because these fields have the same semantic meaning. This can also be the case for different fields labeled with label 228 across datasets. For example, if a new field in a new dataset has label 228 (Social Security Number), the data quality rules generated for data record 202 will still apply to that new field in the new dataset.

[0087] The examples above illustrate rules generated from analysis based on a single field. Rules can also be generated based on analysis across multiple tables (e.g., multiple fields). Go to... Figure 3A In one aspect, data quality rules can be generated based on relationships between multiple fields or data records (such as data records 202 and 203). The relationships between fields can indicate the dependency of a second field's value on a first field's value, or vice versa. In another aspect, the relationship can indicate the correlation between the values ​​of the first and second fields. In some examples, the data quality rule module 104 verifies relationships between or among fields by obtaining verification data that confirms the relationship. This verification data can be user input or attributes based on semantic labels of the relevant fields. For example, during rule generation, the relationship can be verified against a threshold number of values ​​for the first and second fields to test the generated rules. For example, if the first and second fields each include numerical values, determining the existence of a relationship between the first and second fields can include determining a numerical function that associates the values ​​of the first and second fields. In some implementations, the generated data quality rules may require one or more allowed values ​​to correspond to values ​​that satisfy the numerical function, and one or more prohibited values ​​to correspond to values ​​that do not satisfy the numerical function.

[0088] In relation to Figure 2A In a process similar to the described process 300a, the attribute analysis module 204 determines the attributes of a given field for which data quality rules are to be generated or applied. The attribute analysis module 204 is configured to receive (282) dataset 201. The attribute analysis module selects (284) fields from dataset 201. Figure 3A In the example, the selected field 218 is named "SSN", although a field name is not required. The value 220 of the field "SSN" is "55-555-5555".

[0089] Once a field is selected, the attribute analysis module uses tag index 610 to look up (286) semantic tag 228 in tag index 610. In some implementations, tags are not received, but entry identifiers (or some other indicators in which the attribute is associated with the selected field) in data dictionary database 614 are received instead of tags. Attribute analysis module 204 performs a lookup (288) to determine the attribute associated with tag 228a “Social Security Number”. Here, attribute 230 is associated with tag 228a in data dictionary database 614. Attribute 231 is then stored (e.g., in memory) for use by rule generation module 206 or rule application module 208 of data quality rule generation module 104 to generate or apply data quality rules. In this example, attribute 231 indicates (e.g., in record 202) that field 218 named “SSN” in table 1 references (in record 203) a field named “User ID” in table 2. Data quality rule generation module 206 can use this information (which can be discovered during schema analysis, regarding…) Figures 7A to 7B The described method generates data quality rules based on the relationships between the fields in records 202 and 203.

[0090] Figure 3B The process 300b, in which module 206 generates data quality rules, is shown, and is similar to... Figure 2B Process 200b. Data quality rules can be configured by rule generation module 206 to enforce constraints on the values ​​of a second field (e.g., record 201) based on the values ​​of a first field (e.g., record 203). For example, determining that a relationship exists between the first and second fields can include determining that the values ​​of the second field include key values ​​referenced by the values ​​of the first field. Data quality rule generation module 206 is configured to generate data quality rules that require each value of the second field to be a valid key value.

[0091] Rule generation module 206 can be configured to generate rules through the output of a semantic discovery process performed by system 602 or through information about... Figures 7A to 7B The described pattern analysis determines a relationship between the first and second fields. In the example, at least one classifier configured by a machine learning process can be used to determine the relationship among the fields of records 202 and 203. For example, one or more tests (described later) used to determine the semantic labels of fields can also determine the relationship among fields. The relationship among fields is included in attribute 231 associated with the semantic label of each of these fields. For example, these attributes may indicate that the semantic label determining the field indicates that the field includes the primary key value of the dataset. In the example, rule generation module 206 configures data quality rules to require that each primary key value is unique within the field.

[0092] continue Figure 3B This illustrates an exemplary data quality rule 233 generated by the rule generation module 206 using multi-table analysis. As previously described, attribute 231 includes multi-table constraints, which are different from those regarding... Figures 2A to 2C The described single-field constraint. Multiple records 202 and 203 are analyzed by attribute analysis module 204 to determine multi-table attributes 231 associated with records 202 and 203. In some specific implementations, schema analysis is performed by dataset schema analysis module 110 (subsequently discussed). Figures 7A to 7B The described () is used to determine multi-table attribute 231. Similar to attribute 230, multi-table attribute 231 can be defined in relation to Figure 8 The semantic discovery process is determined during the described procedure. Rule generation module 206, in conjunction with previous references... Figure 2A The rule generation module described uses attribute 230 in a similar way to multi-table attribute 231.

[0093] Multi-table constraints (attribute 231 in this example) are found by rule generation module 206. Multi-table constraints can include any constraints applied to multiple fields. Based on the multi-table constraints determined from multi-table attribute 231, rule generation module 206 can be configured to generate data quality rules 233b that examine the relationships between and within multiple fields. Here, multi-table attribute 231 shows that the SSN field references the User ID field. Rule generation module 206 generates data quality rule 233b. For example, data quality rule 233b checks to determine whether a corresponding value exists in the User ID field (for each value of the selected field) related to the selected field. This data quality rule 233b requires determining that the corresponding User ID field is related to the selected field, and that each value has a corresponding existing value in that User ID field. If no value exists in the User ID field, data quality rule 233 for the SSN field fails.

[0094] This exemplary rule only checks the existence of values ​​in the relevant field. However, more complex multi-table data quality rules are possible. For example, rule generation module 206 could check for the existence of a valid corresponding user ID in the user ID field for each value in the SNN field. This could be done by checking the data quality rules associated with the user ID field, such as whether the ID is unique within a given numerical range. If any data quality rule applied to the user ID field fails, in this example, the data quality rule for the SSN field also fails.

[0095] exist Figure 3CThe process 300c of applying multi-field data quality rule 233b is illustrated. For the value of the SSN field, which includes 555-55-5555, the user ID field is checked by the rule application module 208. The rule application module 208 is configured to receive (271) records 202 of Table 1. The rule application selects (273) fields of datasets 202 to 203, such as field 218 named “SSN”, which has a position 221 in the dataset at column 1 of Table 1. The rule application module 208 references (275) the tag index 610 and finds the rule associated with the field in the first column of Table 1 in dataset 201. Here, the semantic tag 228 “Social Security Number” is used to represent the field. The rule application module 208 accesses (277) the data quality rule data repository 250 and retrieves the data quality rule 214 for the field tagged “Social Security Number”. Rule application module 208 applies these data quality rules (279) to the value of field 218. In the example, the value 555-55-5555 is governed by format rule 233a, character restriction rule, and unique value rule.

[0096] When the application includes a second data quality rule 233b that requires multiple fields, the rule application module 208 accesses and checks the second field in order to apply rule 233b. In this example, because the value of the corresponding user ID field in Table 2 of data record 203 is <empty>, indicating that a user ID does not exist, rule 233b fails. Therefore, although field 202 passes all the individual field data quality rules, it fails the multi-table data quality rule 233b. This provides a clearer picture to the user of how the system failed the data quality checks compared to simply checking each individual field. The result of all rules 233a through 233c is FAIL. The application module 208 stores (281) the result 224 of the data quality rule application in the output data repository 114.

[0097] Data quality rules generated for fields associated with label 228 can be applied to any field associated with the same label in a second instance of dataset 201 or in a new, different dataset. For example, label 228 is the Social Security Number tag, for which rule generation module 206 has generated one or more data quality rules. These generated data quality rules can be applied to any field with the same label 228 because these fields have the same semantic meaning. This can also be the case for different fields labeled with label 228 across datasets. For example, if a new field in a new dataset has label 228 (Social Security Number), the data quality rules generated for data record 202 will still apply to that new field in the new dataset.

[0098] As previously described, additional data quality rules can be generated based on the attribute 231 associated with the field. For example, the attribute may indicate that the value is based on at least one of the values ​​in the corresponding field, such as the average value in the field, the maximum length of the value, the minimum length of the value, the data type of the value, and the format of the value. For example, one or more allowed or prohibited values ​​in the field are associated with the field name that labels the field. For example, one or more allowed or prohibited values ​​in the field are determined based on values ​​in a second field in the dataset, which is related to the first field of the dataset. For example, the one or more allowed or prohibited values ​​are based on a combination of values ​​in the second field and values ​​in the first field.

[0099] Figures 4A to 4B The procedures 400a and 400b for generating test data using the results of the semantic discovery process of system 602 are illustrated. The test data is generated by test dataset module 106. Test dataset module 106 includes test processing module 304 and test data generation module 306.

[0100] Figure 4A A test processing module 405 is shown, configured to generate test data from a dataset (e.g., dataset 401) for testing the application's logic using the semantic meaning of the dataset fields. The semantic meaning is represented by attribute 430 and, in some respects, by semantic labels, as previously described. A test dataset module 106 is configured to generate a set of test data for the application. The test data is generated such that it represents real data to be processed by the application, tests as many functions of the application as possible (e.g., all functions of the application), and ensures the test data is valid (e.g., all keys are valid, fields include valid values, etc.). The selection of fields to be included in the test data and the validation of the test data can be performed based on the semantic meaning of each field in dataset 401.

[0101] Typically, to generate test data, test dataset module 106 receives (402) dataset 401, which includes fields labeled with semantic tags (or fields otherwise associated with attribute 430 indicating the semantic meaning of each field in dataset 401). In this example, test processing module 405 selects (404) one or more fields of dataset 401, such as field 418 named “SSN”, which includes value 420 as “12-22-1960”. Test processing module 405 performs a lookup (406) of semantic tag 428 in index 610. In response to retrieving semantic tag 428, tag 428a is found as the tag field 418. Attribute 430 of tag 428a is received by test processing module 405 (408). Attribute 430 indicates that field 418 includes numeric characters with 10 digits and MM / DD / YYYY format.

[0102] Once attribute 430 is received, test processing module 405 retrieves (410) test configuration data 407 from test configuration data repository 403. Test configuration data repository 403 may be associated with the application to be tested. Test configuration 407 includes a specification of the requirements for the test data to be generated. For example, test configuration 407 may include a list of fields to be ignored. Fields to be ignored may be based on the semantic meaning of the field, as the application may not have data indicating the field name before receiving dataset 401. For example, when test configuration 407 is generated, the user can specify social security fields that should be ignored for testing purposes. Test dataset module 106 can determine the fields representing social security numbers and remove those fields from the test data. This can reduce the data footprint of the test data, thereby saving storage space. In another example, test configuration data 407 may indicate that social security numbers in the test data should be unique. Test dataset module 106 can find the fields representing social security numbers in dataset 401 and remove duplicate entries in the fields to reduce the size of the test data. In another example, test configuration data 407 can indicate the fields that should be key values ​​and which fields in other datasets they should point to. Test dataset module 106 can execute the structure in the generated test data, as described below. Any other similar requirements can be included in test configuration data 407, and these requirements can include schema requirements, data quality requirements, field inclusion requirements, and so on.

[0103] After performing a lookup (410) of test configuration data 407, test processing module 405 executes the requirements for the fields of dataset 401 indicated in the test configuration data based on the semantic meaning of each of the identified fields. For example, Figure 4ARequirement 412 includes ignoring the date of birth field in dataset 401 and implementing a specific Social Security number requirement. Test processing module 405 associates attribute 430 with requirement 412 of test configuration data 407.

[0104] In some implementations, attribute 430 of semantic label 428 may include the relationship that a field with a semantic label should have with another field with another specific semantic label. For example, attribute 430 of field 418 labeled "Date of Birth" may indicate that the value of the Date of Birth field for a given data entry should always be earlier in time than another value in the field labeled "Date of Registration". Attribute 430 of semantic label 428 may indicate the format of the field's data values, the allowed values ​​(e.g., the attribute may reference a lookup table that includes all possible values ​​for the field). Attribute 430 may indicate the value of the field, the expected range of the field's values, and the degree of commonality of other such characteristics of the field's data values. Attribute 430 is used during the test generation process 400b.

[0105] Figure 4B The process 400b of generating test data 422 by test data generation module 415 is illustrated. Test data generation module 415 receives (440) attributes 430 associated with the selected field. If applicable, test data generation module 415 may also determine how the field was processed by processing rules in previous tests (if applicable). For example, processing rule data indicating the number of times a processing rule is executed on a field marked with semantic tag 428 may be received. In some embodiments, this data is indicated in test configuration data 407. In some embodiments, for example, if the field has not been tested in previous tests, the field may be removed from the test data.

[0106] Test data generation module 415 receives (442) test configuration data 407 and generates (446) test data based on the requirements of test configuration data 407 and attribute 430. In the example, the two values ​​included in the SSN field 418 are each unique and conform to the format requirement 412 of test configuration data 407. The two values ​​in the SSN field are “555-55-5555” associated with the first user ID value (value 1) and “666-66-6666” associated with the second user ID value (value 2). Test data generation module 415 stores (448) test data 422 in test data storage repository 450 for use by one or more applications.

[0107] In some implementations, test configuration data 407 may indicate subset rules for test data 422. In this example, once attribute 430 is identified, test data generation module 415 receives the attribute value and process rule data, and determines the subset rules that should be applied to that field. Test data generation module 415 determines the subset rules for that field based on the semantic label 428 of the field. Subset rules indicate whether all or a subset of the values ​​of a field should be included in the test data for testing the application. For example, a specific semantic label may be used to determine whether the value of a field will be available for testing in a given test or whether the field should be ignored during testing. For example, when testing an application, fields that are not of interest to a particular application may be ignored. As previously described, semantic labels are associated with one or more attributes in a data dictionary. Attributes indicate the characteristics that a field should have. For example, these attributes together may indicate the real-world meaning of the field in the data. When configuring tests, the user can indicate real-world data associated with a larger dataset for testing. When a dataset is received, test dataset module 106 may automatically select relevant fields and ignore irrelevant fields.

[0108] The test data generation module 415 selects subset rules based on the attributes of semantic tag 428. In some implementations, subset rules can be directly associated with semantic tag 428. For example, subset rules can indicate whether a specific field should be included or excluded from the test data based on values ​​included in configuration data 407. For instance, if configuration data 407 indicates that the execution count of an operation performed on a field is below a threshold number, that field can be removed from the test data.

[0109] Other data can be included in configuration data 407. For example, data 407 can specify that a field can be removed from the test data if it fails one or more data quality rule tests. For instance, null values, values ​​with incorrect formatting, or other such data can be removed from the field. In another example, if attribute 430 indicates that a field includes a PII, that field can be removed from the test data. In another example, values ​​falling outside a specific range can be removed from the test data. In yet another example, if a field includes several categories, each with duplicate values, a subset rule can extract a set of data records with unique values. Ensuring the test data has as few duplicate values ​​as possible, while representing a complete selection of possible test inputs for the system, ensures that the test data occupies as little space as possible.

[0110] In some implementations, the test data generation module 415 can select one or more subset rules from the subset rules to apply to fields. The selection of subset rules can be verified by the user (e.g., via a user interface). The selection of subset rules can be for any purpose, but typically the goal is to generate a test dataset 422 that provides full operational coverage of the application under test with the smallest possible data footprint. Therefore, subset rules are selected to remove foreign values ​​from the dataset, such as duplicate values, erroneous values, incorrectly formatted values, etc. In some implementations, the list of selected subset rules can be sent to an output database 114 or a test configuration data repository 403.

[0111] In one aspect, test data 422 can be sent to a data processing application that includes processing rules configured to manipulate the values ​​of fields in the test dataset and generate at least one output value, the field being labeled with a tag proposal. A test dataset generation module is configured to obtain execution information indicating the number of times the processing rules are executed in conjunction with processing the first dataset. Whether a processing rule is executed by the data processing application during the processing of the first dataset can depend directly or indirectly on the value of the field with the tag proposal. Subset rules can be determined based on the execution information indicating the number of times the processing rules are executed in conjunction with processing the first dataset. In this example, the subset rule includes the identification of fields in the dataset. For example, the subset rule can identify a field as a key field of the dataset, which includes at least one key value of a data element of the dataset. In some implementations, a subset of the data records in the dataset includes data records with key values ​​having predetermined values. In some implementations, the subset rule identifies a list of key field names corresponding to the tag proposals. The subset rule identifies a field as a key field by comparing the tag proposal of the field with the list of key fields.

[0112] In one aspect, as previously stated, when the dataset's fields include Personal Identification Information (PII), the subset rule generation module can generate subset rules configured to select a subset of data records from the dataset that do not include PII information. Alternatively or additionally, when the dataset's fields include Personal Identification Information (PII), the test dataset generation module can be further configured to apply a masking function to the PII to generate masked data, and select a subset of data records from the dataset that includes the masked data.

[0113] Figures 5A to 5BProcesses 500a and 500b are illustrated, in which the test dataset module 106 is configured to verify that test data 422 correctly represents real data used to test a given system, which may include verification of patterns in the test data 422. For example, an application may expect certain kinds of data during operation. Therefore, the test data should approximate the real data to be processed by the application. To generate test data that approximates real data, the test processing module 405 receives (452) dataset 401 and selects (454) field 418 and the value 420 of that field. Field 418 is labeled with a semantic tag 428 called Social Security Number. Module 405 receives (456) the tag and retrieves (458) attributes 430 describing field 418. The test processing module 405 associates attribute 430 with the SSN field according to the semantic tag 428. Attribute 430 may indicate the characteristics of the field in real-life data. For example, associating an actual Social Security Number with a real person can represent that real person in a user ID table. In this specific context, all actual Social Security numbers are associated with real people, but not all real people have Social Security numbers. The test data generation module 408 is configured to generate test data tables that satisfy these metrics.

[0114] Processing module 405 retrieves pattern data 409 (460) from test configuration data repository 403. This can be used in the semantic discovery process (regarding...). Figure 8 During or after the described period, the dataset pattern analysis module 110 (reference) Figures 7A to 7B The described method determines schema data 409. Schema data 409 can indicate which fields in the dataset are related keys, which are primary keys, which are foreign keys, etc. For example, schema data 409 can include field names for keys. Therefore, the test dataset module 106 is configured to generate test data that is of reference value because it correctly references other tables in the test data as expected. More specifically, if in the real data, a table with Social Security Numbers includes references to a User ID table, the test dataset generation module is configured to generate a table with a Social Security Number field that includes a key that correctly refers to an existing User ID table in the dataset. When the application uses the test data, the test data approximates the real data that the application expects. Users testing the application can determine that errors in the results of processing the test data are the result of errors in the application itself, rather than errors in the test data.

[0115] Figure 5BThe process 500b of validating test data using test data validation module 417 is illustrated. Validation module 417 verifies whether test data 423 satisfies pattern rule 409. Test data 423 includes tables 411 and 413, which have been updated according to tables 402 and 403 to remove duplicate or incorrect values ​​and records. For example, table 402 includes entries 402a to c, each including a foreign key that references the primary key of the corresponding entries 403a to c in table 403. However, as... Figure 5A As shown, the primary key of table 403 is duplicated with the entry for the key "BBB". Additionally, a missing user ID exists for SSN 444-44-4444 for entries 402a / 403a. Test data generation module 415 removes entries 402a and 403a from tables 402 and 403, and corrects the key value of entry 403c to "CCC" instead of "BBB" to satisfy the primary key requirements of test configuration data 407 (e.g., each primary key is unique). For verification performed by module 417, the primary key of table 413 is verified as valid, and all entries in tables 411 and 413 are verified to include a valid user ID value. The verified test data 423 is stored (470) in the test dataset data repository 450.

[0116] refer to Figures 6A to 6C The diagram illustrates processes 600a, 600b, and 600c for masking data using data masking module 108. Data masking module 108 is configured to determine which fields of dataset 501 (e.g., from input data 112) should be masked, and if fields are determined to be masked, to determine which masking functions should be applied to those fields. Data masking module 108 uses attributes 530 and semantic labels 528 of dataset 501 to associate masking functions with fields.

[0117] Typically, to generate masked data, the attribute analysis module 504 of the data masking module 108 receives (550) a dataset 501 including fields (e.g., field 518) with values ​​(e.g., value 520). The attribute analysis module 504 selects a given field, such as field 518. The attribute analysis module then retrieves (554) associated semantic tags in index 610 and determines the attribute 530 to be retrieved (556) from the data dictionary database 614.

[0118] Semantic labels 528 are stored in a data dictionary that associates each available semantic label with its attribute 530. When a field is tagged, the semantic label 528 is selected from the data dictionary, and the field is associated with the attribute 530 of that semantic label. An index is stored that indicates, for each semantic label 528 in the data dictionary, that the field is associated with a semantic label (if any) in the dataset processed by the data processing device 102.

[0119] Each semantic label 528 is associated with one or more attributes 530 that indicate the semantic meaning of that semantic label. Attributes 530 of a semantic label 528 may include the relationships that a field with a semantic label should have with another field having another specific semantic label. For example, attribute 530 of a field labeled “Date of Birth” may indicate that the value of the “Date of Birth” field in a given data entry should always be earlier in time than another value in a field labeled “Date of Registration”. Attributes 530 of a semantic label 528 may indicate the format of the field’s data values, the allowed values ​​(e.g., the attribute may reference a lookup table that includes all possible values ​​for the field). Attributes 530 may indicate the degree of commonality that should exist between the field’s values, the expected range of the field’s values, and other such characteristics of the field’s data values.

[0120] Attributes indicate the characteristics that a masked field should possess. For example, these attributes together can indicate the real-world meaning of a field in the data. When masking data, the user can indicate real-world data relevant to the masking. When a dataset is received, the data masking module 108 can automatically select relevant fields for masking and ignore irrelevant fields.

[0121] Data masking module 108 determines a masking function for a field based on field attribute 530. The masking function indicates whether all or a subset of a field's values ​​should be masked for a given application. For example, specific semantic tags can be used to determine whether a field's values ​​include a PII or whether the field should be ignored during masking. For instance, values ​​of fields including names can be scrambled, mixed, associated with different indexes, or otherwise anonymized for use in subsequent applications.

[0122] Figure 6BProcess 600b is illustrated, through which masking module 506 applies a masking function to fields of dataset 501. Once attributes 530 are received (560), masking module 506 is configured to obtain (562) a masking function based on these attributes. The value of attribute 530 can indicate whether field 518 should be masked and how it should be masked. For example, a list of masking functions can be referenced by the attribute, or a specific masking function can be specified. In some implementations, attribute 530 may simply indicate that the data for the selected fields includes PII, and masking module 506 determines how to mask the values ​​of the fields. In some implementations, the masking function is based on a set of fields such that when the fields are rendered together, the masking function is applied to one or more fields in the set. This may occur when PII is displayed only when the fields are rendered together, but not when any field is rendered individually.

[0123] The data masking module 108 may use one or more masking functions 514 to mask data. Masking functions 514a to 514c may include regrouping values ​​in a field, replacing values ​​in a field with alternative values, applying variance to numeric values ​​in a field, encryption functions, removing masked values, or other masking functions. In some implementations, masking functions may be combined with the previously described test dataset generation and / or data quality processes such that the masked data is valid test data and / or still conforms to one or more data quality rules for the field. Semantic labels 528 for the field can be used to relate the different requirements of these processes.

[0124] The masking module 506 determines (564) the masking function to be applied to the field (if any). The selected function 514a and the selected field are stored (566) in the masking function data repository 590. Therefore, when an example of a field 518 for masking is received, the masking application module 508 can determine the masking function to be used for masking, or can determine which of these masking functions has been used (so that the data can be demasked if applicable).

[0125] Figure 6CA masking application module 508 is shown, which is configured during process 600c to mask or demask data according to a masking function associated with a field of the received dataset 501. The masking application module 508 is configured to receive (570) dataset 501. The masking module 508 selects (572) a field (e.g., field 518) having position 520 in dataset 510. Module 508 looks up (574) a semantic tag in index 610 and determines an attribute 530 associated with field 518. Attribute 530 indicates the masking function to be used, as represented in the masking function data repository 590. Module 508 retrieves (576) masking function data 516 from the data repository 590 and applies (578) the masking function to the field value of field 518. For example, the function F(x) converts the field value 521, which is “555-55-5555”, into a mask value 522, which is “W2YSQP4LKED” (or another value, depending on the masking function used). Module 508 stores (580) the masked result 524 to the output data repository 114.

[0126] In one aspect, data masking module 108 is configured to determine that a relationship exists between a field with a semantic label and a second field with a different semantic label. In response to determining the existence of the relationship, rule application module 408 executes a data masking function to convert the value in the second field into a masked value. In some specific implementations, a different function than the one used to mask the first field may be used to mask the second field.

[0127] In one aspect, determining the existence of a relationship includes determining that the first label and the second label are associated with a public data source. For example, the public data source may include user profiles. The relationship may indicate the dependency of the value of the second field on the value of the first field, or the dependency of the value of the first field on the value of the second field. In some implementations, the relationship indicates the correlation between a first value of the first field and a second value of the second field. In yet another example, the relationship includes an arithmetic function. The masking module 508 can be configured to select the type of data masking function based on the type of the relationship. The types of data masking functions described above may include one or more of shuffling functions, data encryption, character scrambling, and data substitution.

[0128] In one aspect, module 508 can be configured to scan dataset 501 to determine whether one or more specific values ​​of at least one other field are converted to masked values. In some specific implementations, module 508 can be configured to select a masking function for a first field of a dataset that includes numeric values, and wherein a different second masking function is selected for a second field of a dataset that includes non-numeric values.

[0129] Figures 7A to 7BThe processes 700a to 700b illustrate determining the schema of dataset 701 based on semantic labels associated with fields in the dataset. A dataset schema analysis module 110 is configured to determine the schema of the dataset. For example, the dataset schema analysis module 110 is configured to determine fields that are key values ​​or indexes, how these fields reference each other, and so on.

[0130] Typically, to determine the schema of a dataset, the attribute analysis module 704 receives a dataset 701 comprising one or more tables 701a to b and fields labeled with semantic tags. The dataset schema analysis module 110 determines relationships such as SSN fields and index fields, including relationships between and within field values, based on the semantic tags of the fields in tables 701a to b. The schema of dataset 701 can indicate which fields are indexed and which values ​​refer to other values ​​within the dataset. This information can be used by downstream applications, such as for merging datasets, transforming datasets for storage in a data warehouse, etc. As previously described, semantic tags are associated with one or more attributes in the data dictionary 614. Attributes 730 indicate the characteristics a field should possess. For example, attributes 730 together can indicate the real-world meaning of a field in the data. When determining the schema of a dataset, the user can indicate real-world data associated with the schema. When a dataset is received, the dataset schema analysis module 110 can automatically select relevant fields as indexes and ignore irrelevant fields.

[0131] In one aspect, the analysis module 705 receives (702) data records 701 that include the fields of the dataset. Each field typically includes a field name and one or more data values. For a specific field 718, module 705 selects (704) the field and determines (706) the semantic label 728 associated with that field. For a dataset, the values ​​of different fields can be related by placing them in common data records. A data record can be a row of a table in the dataset, or it can simply be a collection of data values ​​from one or more fields. The following is about Figure 8 The steps for determining the semantic label 728 for the field are described.

[0132] Module 705 determines attribute 730 of field 718 (708). Attribute 730 can indicate the relationship between the Social Security Number and the User ID field. Module 705 sends the discovered relationships in the dataset to schema update module 508.

[0133] Figure 7BThe process 700b of determining or updating the schema of a dataset using schema update module 707 is illustrated. Module 707 can update the values ​​of the dataset to conform to the schema referenced in attribute 730. In some specific implementations, module 707 updates the schema describing a particular dataset 701 to explicitly include discovery relationships between and within its fields. Module 707 receives (712) attributes 730 of dataset 701 and determines (714) relationships by scanning fields based on the information given by attribute 730. For example, module 707 can determine that the key field of table 1 points to the index field of table 2 in data 701. Module 707 updates (716) the schema describing dataset 701 to include that relationship with schema data 735. For example, the schema data can indicate that "the index field of table 2 is a primary key. The key field is a foreign key to table 1 for table 2." The generated schema 735 is stored (721) in schema data repository 750. Downstream applications can access data repository 750 to use schema data 735.

[0134] In one aspect, the schema update module 707 is configured to generate a data flow graph based on schema data. This data flow graph can operate on the datasets based on the relationships in the schema data 735. For example, the schema update module can be configured to generate an association function that associates a first dataset with a second dataset based on key values.

[0135] refer to Figure 8A networked system 600 is illustrated for discovering, classifying, and labeling data fields by analyzing data profiles generated from data fields. The networked system 600 includes an execution system 602, one or more data sources 612, a reference database 216, a development environment 620, and a production environment 614. The execution system 602 is configured to receive source data from data sources 612a, 612b (collectively referred to as data sources 612) in the networked system 600. The execution system 602 is configured to archive the source data received from data sources 612 to generate data profiles representing the source data of data sources 612. The execution system 602 is configured to analyze the profile data to discover, classify, and label portions of the source data. More specifically, the execution system 602 uses the profile data to classify portions of the source data. Classifying the source data involves associating a probability with a portion of the source data. This probability specifies the likelihood that a portion of the source data corresponds to a label known by the execution system 602. The execution system 602 is configured to automatically or in response to user input label portions of the source data with one or more labels. The discovery, classification, and labeling of source data can be iteratively performed to improve the classification of source data and increase the accuracy of labeling source data. The tags for source data include metadata associated with that source data. Once the discovery, classification, and labeling of source data are complete, the tags are loaded into the production environment 614 for use by one or more downstream applications. The semantic discovery process is described in detail in U.S. Patent Application Serial No. 16 / 794,361, filed February 19, 2020, entitled “Discovering a Semantic Meaning of Data Fields From ProfileData of the Data Fields,” which is incorporated herein by reference in its entirety.

[0136] The source data of data source 612 may include several different types of data. In one example, the source data of data source 612 includes a table with data fields. Execution system 602 is configured to discover, classify, and label the data fields of the table. For example, execution system 602 analyzes the data content of each discovery field of the source data and determines what the data content of the data field represents. Execution system 602 classifies each data field by associating it with a known label (e.g., by assigning it a probability value). The labeled data of the table can be output to data storage device 614, which can be accessed by other applications and systems that operate on the labeled data. The process of discovering, classifying, and labeling the data fields of the data table is then described in detail. Data source 612 may include any type of computing system. For example, data source 612 may include data from third parties, data lakes, personal computers, hosts provided by high-scale networks, databases, unstructured data, and so on.

[0137] In this disclosure, the process of discovering, classifying, and tagging data fields of a data table is used as an example to illustrate the functionality of execution system 602 and networking system 600. However, while data fields are one example of something that execution system 602 is configured to discover, classify, and detect, execution system 602 can operate on other types of data. For example, execution system 602 receives application data from an application. Execution system 602 is configured to discover, classify, and tag different data of the application. This data may include different files that are stored and accessed for operating the application. Execution system 602 can be used to discover PIIs stored by the application, discover malware, changes to application files, and so on. In another example, execution system 602 can analyze files stored in a file system (e.g., on a personal computer). Execution system 602 can scan the file system to identify a specific subset of files defined by the user. For example, a user may want to remove working files from a home computer before reformatting it. Execution system 602 can be configured to scan the home computer's file system and tag all working files. The execution system 602 can be configured to mark data subject rights (such as rights for forgetting, data erasure, subject access requests, data correction requests, data suspension, data migration, and consent). Many other applications are possible.

[0138] To discover, classify, and label portions of source data (such as data fields), execution system 602 is configured to access a reference database 216 of one or more files 618. Files 618 provide execution system 602 with the context for performing the discovery, classification, and labeling of the source data. In some implementations, reference database 216 may store lookup tables that store relationships between values ​​found in entries of data fields. For example, the data storage device may include a lookup table that matches codes to a terminology that can be referenced by execution system 602 during the publication, classification, and labeling of source data. Files 618 of reference database 216 may include weight values ​​for classification. For example, weight values ​​may indicate to execution system 602 the probability that two terms (e.g., business terms) are related to each other for the labeled source data 612. These values ​​may be generated during the initial iteration of discovering, classifying, and labeling source data 618 and updated by the user or automatically during subsequent iterations.

[0139] During the iteration process, file 618 can be defined before, during, or after the source data is discovered, classified, and labeled by execution system 602. The developer environment 620 of network system 600 can provide means through which users can write to reference database 216 or update file 618 on data storage devices. For example, development environment 620 may include a user interface that provides feedback to users of execution system 602. For example, the user interface of development environment 620 may display a report showing how execution system 602 performed, such as labeled data fields and the probability of each classification performed by execution system 602. Examples of feedback and user interfaces provided to users are then described in detail.

[0140] Typically, the execution system 602 includes one or more processors configured to execute the logic of the profile data module 604, classification module 605, testing module 606, result verification module 608, and data loading module 610. The operations of each of the profile data module 604, classification module 605, testing module 606, result verification module 608, and data loading module 610 can be executed in batches or in real time. Furthermore, the execution system 602 can execute the operations of each of modules 604, 605, 606, 608, and 610 approximately simultaneously or at different time intervals. For example, in some implementations, the profile data module 604 generates profile data representing a profile of source data from the data source 612 at the first moment. At a later time, once all data from data source 612 within a given time period has been archived by profile data module 604, classification module 605, testing module 606, result verification module 608, and data loading module 610 can analyze the profile data to discover, classify, and label data fields of the source data, and load the data into reference database 216 for use in one or more downstream applications, as previously discussed. Figures 2A to 7B As described.

[0141] Profile data module 604 is configured to receive source data (e.g., tables, documents, etc.) and generate a data profile of the source data. Profile data module 604 identifies fields in the source data (e.g., one or more datasets). Profile data module 604 can identify fields by recognizing rows in tables within the source data, finding field names, finding field references, or using any similar process. Profile data module 604 determines statistical properties of the data fields and generates profile data that includes those statistical properties. Profile data identifies patterns in the source data. More specifically, profile data includes statistics on the values ​​of data fields in the source data tables. For example, profile data may include information specifying whether the data values ​​of a data field include numeric data, strings, etc. For example, statistics on data values ​​may include the maximum, minimum, standard deviation, average, etc., of the values ​​included in each data field (if the data is represented numerically). In some implementations, statistics on data may include the number of digits or characters in each entry of the data values. For example, a data profile may indicate that each data value of a data field includes seven (or ten) digits, which can provide a contextual clue indicating that the data field includes phone numbers. For example, a data profile can indicate that each entry in a data field includes a value from a smaller set of values, which can be used to trigger a comparison with a lookup table via test module 606.

[0142] Data ingestion in the Brief Data Module 604 includes analyzing the field names of the fields, the location of the fields in the table (or file in the file system), and the analysis of the data's patterns. In other words, data ingestion occurs at the field level, the dataset level, and the pattern level.

[0143] At the field level, the profile data module 604 analyzes the field values ​​and field entries to generate profile data. The profile data module 604 can determine whether, for a specific data type, the field value or field entry is empty, blank, valid, etc. The profile data can include statistics on the percentage of empty fields, blank fields, and the percentage of values ​​per field. The profile data module 604 can also generate data indicating how these percentages change compared to a baseline percentage (which can be specified by the user through the development environment or automatically generated). In another example, the profile data can include an indication of whether data for implicit data type entries is valid. For example, if a data field is known to be a string field, but date data is found, it can be inferred that the data is invalid for that entry. In another example, the profile data can include an indication that the data for an entry is valid for a specified format (e.g., requiring two decimal places, but without specifying an explicit type). In some implementations, after the profile data is generated, some functions of this feature are performed by the test module 606.

[0144] At the dataset level, the profile data module 604 can provide statistics relating a portion of the source data to the entire dataset. For example, the profile data module 604 includes indications of natural key uniqueness and key cardinality. The profile data module 604 indicates whether duplicates exist on key values ​​in the source data. The profile data module 604 indicates whether duplicates exist on approximate natural key matches. The profile data module 604 indicates the count of records having one or more of these characteristics. In some implementations, this information is determined by the result verification module 608 after, for example, determining that a field is a key value.

[0145] At the schema level, the profile data module 604 is configured to determine statistics about data fields over time by referencing source data. For example, the profile data module 604 detects whether columns have been added or removed in a new version of the source data. The profile data module 604 detects delimiter fields. In some implementations, the profile data module 604 performs a succession count to determine if any records have been removed from previous versions of the source data. Other schema evolutions can be detected. In some implementations, this function is performed by the result confirmation module 608 after data fields are found in the profile data.

[0146] In some implementations, the profile data can be improved if contextual data is available in reference database 216. For example, if the user specifies the format of one or more data fields in the source data, the profile data can generate additional statistics about those data fields. This can be part of an iterative process. For example, once a data field is found in the first iteration (but the iteration fails to classify it), the user can view the data content of the data field and provide execution system 602 with additional information for analysis by profile data module 604 (and other data modules). For example, if the user specifies that all data fields should include only numerical data, profile data module 604 can quickly identify invalid data and provide statistical measurements of the information in the data profile.

[0147] Profile data module 604 generates profile data from the source data by scanning the entire dataset of the source data before generating the profile data. Profile data module 604 does not require copying the entire dataset from the source data, although this can be done to generate profile data. Because datasets can be very large (e.g., several gigabytes or even terabytes), wholesale copies of the data to the local system for analysis may be impractical. Typically, profile data module 604 scans the source data during periods of low activity in the data source.

[0148] The classification module 605 is configured to receive profile data and source data including fields. For each field of the source data, the classification module 605 is configured to look up a tag index that includes existing tags (e.g., from reference database 216) for the discovery field of the source data. These tags may come from previous iterations of the tagging process, or the tag index (e.g., the initial tag index) may be manually generated, imported, or otherwise obtained. However, the tag index does not need to exist before the tagging process is performed.

[0149] For a field, the classification module determines whether the field is already associated with a tag in the tag index. If the field has not yet been tagged or if a tag index does not exist, the classification module 605 determines that no tag has been found for the field. If necessary, the classification module 605 generates a new tag index populated with semantic tags. The classification module 605 performs a classification of the field's data type. This classification can be based on the field's profile data, field name, and field value. For example, the classification module 605 may determine that the field is a "Date" field. In some specific implementations, the classification module 605 may determine that the field is a numeric field, a string field, or another such data type. While the classification module 605 determines the data type of the field, the semantic meaning of the field (and therefore the semantic tag) is determined by the testing module 606, as described below. For example, the classification module 605 may determine that the field is a Date field, and the testing module 606 determines that the date in the Date field is the customer's "Date of Birth". In another example, the testing module 606 determines that the numeric field is a "User ID" field. Many other such examples are possible. The classification module 605 generates classification data to be sent to the testing module 606 as classification output for finding semantic meaning. The classification data is labeled with a data type determined by the classification module 605.

[0150] If a label is found, the classification module generates label data that can be verified by the testing module 606 and the result verification module 608. This label data informs the testing module 606 and the result verification module 608 that the field has been labeled. This can be used to weight the classifier applied to the field or to suggest labels. However, the field can be reclassified by the classification module 605 and retested by the testing module 606 to confirm that the label is accurate and potentially update the label's label attributes in the data dictionary database 614. For example, if the testing module 606 finds an existing label unsuitable, it can suggest a new label. If the user selects an existing label (e.g., as presented by the result verification module 608), the label data can be used as a marker for the execution system 602, indicating that the label attributes in the data dictionary database 614 do not represent the data value labeled by that label, and these attributes can be updated.

[0151] In some implementations, machine learning methods can be used to update the classification module 605 through multiple iterations. For example, if the discovery field has already been labeled, the classifier can determine that further classification can be bypassed. In another example, the score applied to the field can be updated based on additional data received from data source 612 or from user input. The classification module 605 can determine that a different test should be performed by the testing module 608 compared to previous iterations. For example, if the fingerprint analysis was indeterminate in a previous iteration, the classifier can determine that the test should be bypassed (or replaced with another test) in subsequent iterations.

[0152] Test module 606 is configured to classify the source data of the dataset using statistical data from the profile data and additional contextual information (such as lookup table 442) provided in reference database 216. The classification output of classification module 605 is used to provide context for the data type of each field and to provide existing labels (if any) for the fields. Test module 606 is configured to receive candidate labels 440 from data dictionary database 614. Candidate labels are a library of existing labels, each associated with one or more attributes that identify the semantic meaning of a data field (e.g., for a downstream application or user). As previously described, the attributes associated with each label in data dictionary database 614 provide the semantic meaning of the label. Test module 606 determines those candidate labels that are most closely associated with the attributes of the data field determined by the field's data profile, field name, and data value.

[0153] Test module 606 includes multiple tests (or classifiers), each executed by a different test module, to associate one or more labels with each field being tested. For example, after fields in the dataset are discovered by profile data module 604, the test modules determine how closely the attributes of that field correspond to the attributes of each of these candidate labels. Each test uses different data and methods to propose one or more labels. Because different tests use different classification methods, the proposed labels from each test may not necessarily be the same. The proposed labels are validated in result validation module 608, which is described subsequently. Using different tests to identify the semantic meaning of a field results in a more robust determination of the semantic meaning compared to using any single test, because a single factor (e.g., the field name or including a specific value or a set of values ​​in a field) does not depend on the semantic meaning indication of that field.

[0154] Testing the data in each data field may include determining one or more population levels (frequency of values ​​appearing in the data field) of the dataset's data, discovering data types (e.g., dates or numbers stored as strings), identifying the fields of the data field, discovering key fields, determining whether a field is a word or a descriptive field, and so on. For example, data classification may include date and time analysis. Therefore, the testing module 606 receives profile data from the profile data module 604 and executes a series of statistically based functions to identify, classify, and test field details for a set of known label types. The testing rules vary depending on the data type, which may be identified by the classification module 605 in the classification output or, in some cases, included in the profile data generated by the profile data module 604.

[0155] Test module 606 is configured to perform multiple different classification tests on field names and entries in data fields to determine how the data fields are labeled. Test module 606 receives classification output, candidate labels, and any reference data from reference database 216 and provides this data to the tests. These tests include pattern analysis, business terminology analysis, fingerprint analysis, and keyword search. As previously stated, although the table with field names and field data is described as an illustrative example, the tests performed by test module 606 can be performed on other data types. Examples of classification tests performed on data fields and data entries of the source data may include fuzzy matching schemes, column data schemes, business terminology matching schemes, keyword matching, fingerprinting schemes (e.g., contextual data lookups), pattern matching, and confirmation.

[0156] The fuzzy matching logic of test module 606 includes logic for fuzzy matching field names against a terminology dictionary. Typically, fuzzy matching is configured to find matches between field names and terms in the dictionary if no exact match is found. The system finds the dictionary terms to be somewhat inaccurate. For example, test module 606 sets a fuzzy matching threshold to a percentage value less than 100, and the dictionary database (e.g., database 216) will return any matches in its memory that correspond to (e.g., are greater than) that percentage. In some implementations, a probability score is assigned to each match. This probability score can be presented to the user.

[0157] The business terminology analysis of test module 606 includes logic for matching data field names with known business terms in a glossary. Typically, business terms can be placed in different contexts or groups of business terms. Test module 606 performs checks to find the occurrence of a specific word or term within another word or term. This can include references to specific specifications. For example, test module 606 receives specifications indicating different abbreviations of street names (such as “st,” “ln,” “ave,” “pl,” “ct,” etc.). Test module 606 performs checks to determine whether any of those abbreviations is included in a data field. If one or more data elements from the specification's data elements are included in the data field, test module 606 determines that the data field includes a street name. This information can be used on itself, such as to determine that the data field includes a street name and therefore should be labeled. This information can also indicate that the data field includes other information, such as an address. Test module 606 uses the determination that a street name exists in the column, among other data, to determine whether the data field includes only an address, a street name, or some other data. In another example, the phrase “date of birth” can be identified and associated with such a label. Other matching strategies may include using fuzzy matching, synonym matching, and so on. Typically, the test module 606 associates a probability score with each match. Users can generate instructions to configure this logic, such as through a development environment.

[0158] The pattern matching analysis of test module 606 uses the data content of the fields (in addition to or in lieu of the field name). The pattern matching type used for pattern matching can be determined by test module 606 based on the results of categorical data. For example, categorical data can identify the data type of a field, such as the data being represented by numbers. In this example, the profile data also indicates that each entry in the data field is 13 to 18 characters long. This can indicate to test module 606 that the data field could be a credit card number data field. To confirm this, test module 606 can perform one or more pattern tests on the data of the suspected data field. For example, the first 4 to 6 digits of each entry in a table of issue codes can be checked. The last digit can include the check digits defined by the Luhn test. If a threshold percentage of the entries in the data field satisfies each of these patterns, test module 606 can infer that these fields hold credit card numbers and associate the field name with appropriate labels and probabilities. For pattern matching logic, both the data itself of a given field and the patterns of the data in the field (e.g., identified in the profile data) can be used to determine the pattern test to be run and the label to be applied to the given data field.

[0159] Test module 606 determines whether to perform a pattern analysis test on source data 612. This determination can be the result of categorical data. Pattern matching analysis uses profile data to determine whether source data 612 conforms to a predetermined pattern that indicates candidate field labels. For example, if the data in a field has a specific length and composition, pattern matching analysis identifies corresponding candidate labels. The pattern score applied to the label can be a function of how closely the identified pattern matches the predetermined pattern, the uniqueness of the pattern, or any number of factors. The weights can be adjusted based on the uniqueness function of the pattern. For example, a very unusual pattern may correspond to a higher weight value. If the values ​​of a field are found to closely match the pattern (more than all values ​​or a subset of values), the score can be higher than the score for a pattern that matches only a small number of values.

[0160] Test module 606 may include keyword search tests. Keyword testing (similar to business term matching tests) includes data-based tests that involve searching for specific keywords within a data field. For example, to find an address, test module 606 searches for common address terms such as "street," "road," "main street," etc. The tests can be extended by the user, who can add new keyword files to the specification of reference database 216. Keyword testing can be used to find words or parts of words in phrases, such as those used for addresses and company names, where a limited set of common words exists that can uniquely identify the data field.

[0161] Test module 606 can determine whether to perform keyword analysis on field names. In this example, if some of these fields still do not have a tag value associated with them, test module 606 will perform a keyword matching test. Checking whether the field name (and possibly the field value) includes one or more keywords from the table helps test module 606 associate specific tags with the field. Test module 606 performs a keyword search and generates a keyword search score.

[0162] Test module 606 may include fingerprinting tests for analyzing data values ​​in the source data. The fingerprinting logic typically involves a data lookup across the entire data field. The fingerprinting logic includes tests based on data values. The logic for fingerprinting a data field involves comparing the data in the data field with a known list (e.g., from reference database 216) to determine if the data in the data field is related to the data in the list. For example, data from a data field may be compared with a list of names, a list of country names, a list of city names, and so on. The fingerprint (the known list of data) is typically representative, not exhaustive. In other words, the fingerprint does not need to include every example of a value that is part of the fingerprint set. Typically, the fingerprint may include selected exemplary values ​​that roughly represent the most common values ​​that should appear in the data entries of the data field. For example, for a list of names, the fingerprint does not need to include all possible names for the United States; a selected group of the most popular names is sufficient. For example, the top 100 names typically provide enough data to demonstrate the correlation between the data field and the data fingerprint. Data fingerprints can be generated from master data. For example, the system may include 50 to 100 of the most populous U.S. cities to check if a data field corresponds to a city name. Users can add new domains to the manual to enhance fingerprint recognition testing capabilities for specific systems.

[0163] Test module 606 determines whether to examine the fingerprint table for that field. Fingerprint recognition works well when a long list of possible values ​​for a field exists, but some values ​​are expected to be more common than others. For example, city names, street names, and even first and last names are suitable candidates for fingerprint analysis. Test module 606 performs fingerprint analysis and generates a fingerprint score.

[0164] Test module 606 determines whether to run business terminology analysis. Business terminology analysis can be useful when there are many specific terms in the source data 612 that may correspond to the business terms used to label the fields. If business terminology analysis is performed, test module 606 performs it by comparing field names with business terms to find matches and generate labels and their associated probabilities. The results of business terminology analysis are correlated with weights and scores, similar to pattern matching analysis.

[0165] Test module 606 can be configured to compare results across tests to refine the results using validation logic. For example, validation logic can be used to verify business term matching using another classification test. For instance, a test specifying that a data field includes a premarital name value should also pass a classification test for the surname field.

[0166] Test module 606 is configured to execute machine learning logic that remembers classifications of previous datasets (e.g., from a specific source) or previous iterations of the same dataset, and influences which tests are selected for subsequent iterations and how the probability values ​​for those subsequent iterations are determined. The machine learning logic is trained on the dataset, and weights developed using the training data can be applied to classify new data on the dataset.

[0167] Each of these tests can output one or more proposed labels for the data field being analyzed. These tests do not need to be consistent. Each proposed label can be associated with a score (not shown) and a weight value (not shown). The score and weight of each label can be used by the results verification module 608 to indicate a specific label for the proposed label when the semantic meaning of the field is identified, and to classify (or store) that label into a category. The category (described later) indicates the degree of consistency among the tests and thus indicates the confidence level of the proposed label when the semantic meaning of the field data is identified.

[0168] To perform tests, test module 606 receives profile data, classification data, and other reference data, and determines whether to execute each test. For example, if the received data type is not formatted for a specific test, the test may not be executed. Any combination of tests can be executed. The test module determines whether to perform pattern analysis, business terminology analysis, fingerprint analysis, and keyword search. Each test generates one or more proposed tags and outputs these proposed tags. The test results, including all proposed tags, are correlated and sent to result verification module 608 as test results.

[0169] In some implementations, if no results are found from earlier tests, subsequent tests can be performed. In some implementations, test module 606 prepares reference data by retrieving data from reference database 216 based on the source data 612 being analyzed. For example, test module 606 can retrieve data from reference database 216 based on the location of source data 612, table names in the source data, user input, etc. Test module 606 can determine the lookup table to access and to examine for each field of the source data. The selection of the lookup can be based on the profile data received from profile data module 604.

[0170] In some implementations, test module 606 determines whether to run pattern analysis. Pattern analysis can be used to analyze source data 612 over time and as a whole. For example, if fields are missing, added, or deleted, this information can be used to flag other data fields.

[0171] In some implementations, once the test is complete, the test module 606 combines the associated scores to generate a final score, which is displayed as an associated proposal label with which the score is applied. In some implementations, upstream scores influence downstream scores, making the scores distinct from each other, but representing scores that are updated with each analysis. In some implementations, the label and score for each proposal are reported separately to the results verification module 608, which then determines how to classify the test results 306.

[0172] In some implementations, the testing order may be such that more processing-intensive tests are scheduled last. If other tests fail, more time-intensive tests may be used as a last resort. Arranging the tests in this way can reduce the processing time on execution system 602 for marking source data 612.

[0173] Once the test module 606 has run a classification test on the source data and has determined the probability values ​​of the labels of the data fields of the source data, the result confirmation module 608 performs a check to determine whether the classification result indicates a high confidence level or whether further testing should be performed.

[0174] The result verification module 608 receives the test results of different tests performed on the source data by the test module 606 and determines whether the results confirm or conflict with each other. The result verification module 608 categorizes the test results of the test module 606 into several classification categories. These categories include matching, recommendation, research, and ignore categories. Each of these categories indicates the similarity among the label proposals in the test results. For example, if all label proposals are identical, the test results have a high level of similarity. If each label proposal is different, the test results have a low level of similarity. Similarity can be determined not only by a voting mechanism performed on each test. Each label proposal is associated with a weighted score value. If a label proposal in a test does not match the others, but has a relatively high score value and a large weight associated with it compared to other proposals in other tests, then even though three out of four tests are consistent, the similarity can be identified as low.

[0175] Depending on the category being identified, the result verification module 608 can automatically verify the label while identifying the semantic meaning of the field, or it can prompt the user to manually verify the label. Verification can be performed via a client device on the user interface, as described below.

[0176] The match category indicates the highest level of confidence for the label (e.g., 100 matches). In some implementations, matching labels are automatically included in the metadata associated with the source data. However, the result verification module 608 can still be configured to present the result to the user for manual verification. Typically, the match category indicates that all performed tests proposed the same label. In some implementations, a match category can be selected if not all labels match, but any distinct label is below a score threshold, indicating general consistency between tests.

[0177] Recommended categories typically indicate that at least one label has a high-quality association with the data field. However, recommended labels are often below a threshold level set for maximum confidence and are preferably further validated. In some implementations, recommended categories indicate several high-quality labels associated with the data field. In some implementations, the results verification module 608 sorts and lists the recommended labels, each with a probability score, which can help the user select the best label for the data field.

[0178] Research categories indicate values ​​or data fields that possess a certain statistically interesting property that failed any particular test. For example, a data field might seem to have meaning, but no test has a proposed label, or the proposed label has a score below a given threshold. For instance, a data field might include a profile attribute indicating that the data field is an important domain or field in the source data, but no recommended label (or no recommended label with a probability above a threshold). Typically, such results indicate that additional rules should be added to the test (e.g., the test should be expanded or modified in some way) and run again.

[0179] The ignore category indicates that a data field is designated as uninteresting, or that the field does not trigger tests and does not display profile attributes indicating the importance of the data field. Typically, the ignore category is displayed when received data has errors or lacks a discernible pattern. While the ignore category can indicate that a new label should be defined, it usually indicates that the field includes miscellaneous data without a specific structure.

[0180] Typically, to categorize test results (e.g., buckets) into categories, the result verification module 608 performs the following process. The result verification module 608 compares the test results with fields. For each field, the result verification module assigns the test result to a category based on a similarity level.

[0181] In addition to generating classification files for specified categories, the results verification module 608 is configured to generate other files. These other files include a profile annotation file, which describes the results of the classification analysis process. Other files include a findings overview, which describes the process of combining outputs to form business term matching and fingerprinting processes.

[0182] The results confirm that module 608 is configured to determine the existence of outliers in data values ​​based on the aggregation and range of each data field. Outliers include data values ​​that do not conform to the relationships or formats identified for the data fields. The results confirm that module 608 determines outliers based on clustering and prediction of relationships with the classification of test module 606.

[0183] The process described above can be iterated to improve classification accuracy and allow users to update classification tests for better results. As previously stated, in some implementations, machine learning logic can be used to train the classifier during each iteration to facilitate the process. Typically, once profile data is generated by profile data module 604, the process and results of testing module 606 are iterated to confirm the results of module 608; new profile data does not need to be generated repeatedly unless the source data itself is updated.

[0184] Once the profile data module 604, classification module 605, testing module 606, and result verification module 608 have the generated labels for each of these data fields, the data loading module 610 can load metadata, including the label index, into the reference database 216. The data loading module 610 performs the process of updating the label index and loading data into the reference database 216. The data loading module 610 receives field names and proposed labels for those fields, which have been manually or automatically verified. The data loading module 610 associates the field names with the proposed labels. The data loading module updates the label index by associating the labels with the fields' positions in the dataset. The data loading module associates the labels of the dataset being analyzed with the label index, which can be referenced by the dataset through the execution system 602 and downstream applications.

[0185] Typically, the reference database 216 is accessible to one or more downstream computing systems for use in various applications. For example, the generated labels from the dataset can be used for data quality enforcement, personal data anonymization, data masking, (PII) reporting, test data management, dataset annotation, and so on.

[0186] The data loading module 610 is configured to package metadata and source data into a package that can be used by one or more other computing systems. For example, once profile data is generated, the operations of the classification module 605, testing module 606, and result verification module 608 can each be configured to run multiple instances of their processes in parallel on different partitions of the source data. For example, if the source data includes multiple tables, the source data can be partitioned by table. The testing module 606 and result verification module 608 can run instances of their logical processes on multiple tables simultaneously to increase the throughput of the processes of execution system 602. Once the source data is tagged, the data loading module 610 can be configured to merge these partitions back together and store the tagged source data in reference database 216.

[0187] Figures 9 to 12 Flowcharts, each including exemplary processes, are shown. Figure 9 A process 900 for determining data quality rules for values ​​in one or more datasets is shown. Process 900 can be implemented by a data processing system such as those previously described. Process 900 includes receiving (902) one or more datasets by the data processing system, wherein the one or more datasets include fields and wherein values ​​are stored in the fields. Process 900 includes archiving (904) the values ​​stored in the fields by the data processing system, the fields being included in the one or more datasets. The process includes applying (906) one or more classifiers by the data processing system to the archived values. Process 900 includes identifying (908) one or more label proposals that identify the semantic meaning of the field, wherein each of the one or more label proposals has a calculated confidence level. Process 900 includes labeling (910) the field with a label proposal having a calculated confidence level that satisfies a threshold level. Process 900 includes determining (912) one or more features representing values ​​included in the fields with label proposals, based on the label proposals for the labeled fields and the field profiles. Process 900 includes determining (914) data quality rules for the fields of the dataset based on the determined profile.

[0188] Figure 10A process 1000 is illustrated for selecting test data to induce the execution of processing rules during testing of a data processing application. Process 1000 can be implemented by a data processing system such as those previously described. Process 1000 includes receiving (1002) one or more datasets by the data processing system, wherein the one or more datasets include fields and wherein values ​​are stored in the fields. Process 1000 includes archiving (1004) the values ​​stored in the fields by the data processing system, the fields being included in the one or more datasets. The process includes applying (1006) one or more classifiers by the data processing system to the archived values. Process 1000 includes identifying (1008) one or more label proposals and labeling the field with label proposals having a calculated confidence level that satisfies a threshold level, the one or more label proposals identifying the semantic meaning of the field, wherein each of the one or more label proposals has a calculated confidence level. Process 1000 includes determining (1010) subset rules based on the label proposals for the labeled fields. Process 1000 includes selecting (1012) a subset of data records from the dataset according to the subset rule, the selection of which is based on the dataset's label proposal tag field. Process 1000 includes providing (1014) the selected subset to a data processing application for testing the data processing application.

[0189] Figure 11 A process 1100 for masking data in a data processing application is illustrated. Process 1100 can be implemented by a data processing system such as those previously described. Process 1100 includes receiving (1102) one or more datasets by the data processing system, wherein the one or more datasets include fields and wherein values ​​are stored in the fields. Process 1100 includes archiving (1104) the values ​​stored in the fields by the data processing system, the fields being included in the one or more datasets. The process includes applying (1106) one or more classifiers by the data processing system to the archived values. Process 1100 includes identifying (1108) one or more label proposals and labeling the field with label proposals having a calculated confidence level that satisfies a threshold level, the one or more label proposals identifying the semantic meaning of the field, wherein each of the one or more label proposals has a calculated confidence level. Process 1100 includes determining (1110) sensitive data representations of the fields in the datasets based on the label proposals. Process 1100 includes, in response to the determination, executing (1112) a data masking function to convert the values ​​of sensitive data, including the field, into masked values.

[0190] Figure 12A process 1200 for determining patterns in a dataset is illustrated. Process 1200 can be implemented by a data processing system such as those previously described. Process 1200 includes receiving (1202) one or more datasets by the data processing system, wherein the one or more datasets include fields and wherein the fields store values. Process 1200 includes archiving (1204) the values ​​stored in the fields by the data processing system, the fields being included in the one or more datasets. The process includes applying (1206) one or more classifiers by the data processing system to the archived values. Process 1200 includes identifying (1208) one or more label proposals and labeling the field with label proposals having a calculated confidence level that satisfies a threshold level, the one or more label proposals identifying the semantic meaning of the field, wherein each of the one or more label proposals has a calculated confidence level. Process 1200 includes determining (1210) one or more pattern features of the values ​​included in the field, the field having label proposals, based on the label proposals for the labeled field. Process 1200 includes determining (1212) a key value in a first field with a first label proposal, which is referenced by a value in a second field with a second label proposal, based on the one or more pattern features. Process 1200 includes updating (1214) the pattern data describing the dataset in response to the determination, to reference the key value in the first field, which is referenced by a value in the second field.

[0191] Some specific embodiments of the subject matter and operations described in this specification can be implemented in digital electronic circuits, or in computer software, firmware, or hardware. These embodiments include the structures disclosed in this specification and their structural equivalents, or combinations thereof. For example, in some embodiments, modules of the data processing device 102 may be implemented using digital electronic circuits or in computer software, firmware, or hardware, or combinations thereof. In another example, processes 900, 1000, 1100, and 1200 may be implemented using digital electronic circuits, or in computer software, firmware, or hardware, or combinations thereof.

[0192] Some of the specific implementations described in this specification (e.g., data quality rule module 104, test dataset module 106, data masking module 108, dataset pattern analysis module 110, etc.) can be implemented as one or more groups or modules of digital electronic circuits, computer software, firmware, or hardware, or combinations thereof. Although different modules may be used, each module need not be different, and multiple modules may be implemented on the same digital electronic circuit, computer software, firmware, or hardware, or combinations thereof.

[0193] Some specific embodiments described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by a data processing apparatus or for controlling the operation of the data processing apparatus. The computer storage medium may be, or may be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination thereof. Furthermore, although the computer storage medium is not a propagating signal, it may be a source or destination of computer program instructions encoded in an artificially generated propagating signal. The computer storage medium may also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, discs, or other storage devices).

[0194] The term "data processing apparatus" encompasses all kinds of apparatus, devices, and machines for processing data, including, for example, programmable processors, computers, systems-on-a-chip, or a combination of the foregoing. In some specific implementations, data quality rules module 104 and / or data structure module 106 include the data processing apparatus as described herein. The apparatus may include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may also include code that creates an execution environment for the associated computer program, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or combinations thereof. The apparatus and execution environment can implement various different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0195] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages. A computer program may, but is not required to, correspond to a file in a file system. A program may be stored as a part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinating files (e.g., a file storing one or more modules, subroutines, or portions of code). Computer programs can be deployed to execute on one computer or on multiple computers located at a site or distributed across multiple sites and interconnected via a communications network.

[0196] Some of the processes and logic flows described in this specification can be executed by one or more programmable processors that execute one or more computer programs to perform actions by manipulating input data and generating outputs. These processes and logic flows can also be executed by special-purpose logic circuitry such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and devices can also be implemented as such special-purpose logic circuitry.

[0197] Processors suitable for executing computer programs include, for example, both general-purpose microprocessors and special-purpose microprocessors, as well as processors of any kind of digital computer. Typically, a processor receives instructions and data from read-only memory or random access memory, or both. A computer includes a processor for performing actions according to instructions and one or more memory devices for storing instructions and data. A computer may also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or operatively coupled to receive data from or transfer data to one or more mass storage devices, or both. However, a computer does not need to have such devices. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, flash memory devices, etc.), magnetic disks (e.g., internal hard disks, removable disks, etc.), magneto-optical disks, and CD-ROM and DVD-ROM discs. The processor and memory may be supplemented or incorporated therein by dedicated logic circuitry.

[0198] To provide interaction with the user, operation can be implemented on a computer having a display device (e.g., a monitor or another type of display device) for displaying information to the user and a keyboard and pointing device (e.g., a mouse, trackball, tablet, touchscreen, or another type of pointing device) that the user can use to provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Additionally, the computer can interact with the user by sending and receiving documents from the device the user is using; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0199] Computer systems may include a single computing device or multiple computers operating close to or substantially far from each other and typically interacting via a communication network. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), inter-network networks (e.g., the Internet), networks including satellite links, and peer-to-peer networks (e.g., autonomous peer-to-peer networks). Client-server relationships may arise from computer programs running on the respective computers, and these programs may have client-server relationships with each other.

[0200] An exemplary computer system includes a processor, memory, storage devices, and input / output devices. Each of these components may be interconnected, for example, via a system bus. The processor is capable of processing instructions for execution within the system. In some implementations, the processor is a single-threaded processor, a multi-threaded processor, or another type of processor. The processor is capable of processing instructions stored in memory or on storage devices. Memory and storage devices may store information within the system.

[0201] Input / output devices provide input / output operations for the system. In some implementations, input / output devices may include one or more of network interface devices (e.g., Ethernet cards), serial communication devices (e.g., RS-232 ports), and / or wireless interface devices (e.g., 802.11 cards, 3G wireless modems, 4G wireless modems, 5G wireless modems, etc.). In some implementations, input / output devices may include driver devices configured to receive input data and send output data to other input / output devices (e.g., keyboards, printers, and display devices). In some implementations, mobile computing devices, mobile communication devices, and other devices may be used.

[0202] Although this specification contains numerous details, these details should not be considered as limiting the scope of the claims, but rather as descriptions of features specific to particular examples. Certain features described herein may also be combined in the context of individual embodiments. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable combination in multiple embodiments.

[0203] Several embodiments have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of the data processing system described herein. Therefore, other embodiments are also within the scope of the following claims.

Claims

1. A method for generating one or more rules from labels of a data field for processing data values ​​of the data field, the method being implemented by a data processing system, and the method comprising: Generating a tag index that associates tags with a set of one or more fields in a data record, wherein the tags identify the expected information type in each of the set of one or more fields, the generation including: Perform multiple different classification tests on the fields of the data records; and The label of the field is determined by comparing the results of the cross-classification test using confirmation logic. Access a data dictionary that associates the information type indicated by the label with a set of attribute values ​​representing a requirement for the values ​​of the one or more fields associated with the label, the requirement including logical or syntactic characteristics of the values ​​of the one or more fields; and Fields for a specific data record: Specific tags associated with the field of the specific data record are identified by accessing the tag index; Retrieve attribute values ​​for the specific tag from the data dictionary, the attribute values ​​specifying specific requirements for the field; and Based on the attribute value, apply one or more rules to the field to process the data value of the field.

2. The method of claim 1, wherein at least one of the classification tests specifies fuzzy matching logic for fuzzy matching field names with a terminology dictionary, wherein the fuzzy matching logic is configured to find a match between the name of the field and a term in the terminology dictionary if no exact match is found.

3. The method according to claim 1 or 2, wherein at least one of the classification tests specifies logic for determining whether any term or word from a set of predetermined terms or words is included in the field.

4. The method of claim 1 or 2, wherein the classification test includes at least one pattern test, the pattern test specifying pattern matching logic for identifying one or more patterns in the data of the field.

5. The method of claim 1 or 2, wherein at least one of the classification tests specifies logic for comparing a known list of data with the data of the data field to determine whether the data of the field is related to the data in the list, wherein the known list represents possible values ​​that should appear in the data of the field.

6. The method of claim 1 or 2, wherein the method is used to select test data to induce the execution of processing rules during testing of the data processing application, and applying the one or more rules comprises: Generate subset rules, which, when executed, specify whether the value of the field includes the attribute value; A subset of fields is selected from the specific data record according to the subset rule, wherein each selected field includes a value having the attribute value; as well as The selected subset is provided to the data processing application for testing.

7. The method of claim 1 or 2, wherein the method is used to mask data of a data processing application, and applying the one or more rules includes: Based on the attribute values, it is determined that the field of the specific data record represents sensitive data; as well as In response to the determination, a data masking function is executed to convert the value of the sensitive data, including the field, into a masked value.

8. The method of claim 1 or 2, wherein the method for determining a pattern of data records, and applying the one or more rules, comprises: The attribute value is determined to represent the pattern feature of the value included in the first field; Based on the pattern features, the first field with the label is determined to include a key value referenced by the value of the second field; and In response to the determination, the schema data describing the specific data record is updated to reference the key value referenced by the value of the second field in the first field.

9. The method of claim 1 or 2, wherein the method is used to determine data quality rules for values ​​in a field of a data record in a set of data records based on a label associated with the field, the label indicating a characteristic of the value of the field, and applying the one or more rules includes: Generate data quality rules, which are configured to be executed as follows: Verify whether the value of the field meets the specific requirement represented by the attribute value, and Generate output data indicating whether the specific requirements are met.

10. The method of claim 9, wherein the data quality rule indicates one or more of the following: The value in the field is subject to the allowable deviation from the requirement. One or more allowed values ​​in the field, and One or more prohibited values ​​in the field.

11. The method of claim 9, wherein the field is a first field, the data record includes a second field, the data quality rule is a first data quality rule, and the method comprises: A relationship is determined between the first field and the second field based on the attribute value of the specific tag; as well as A second data quality rule for the second field is generated based on the relationship between the second field and the first field.

12. The method of claim 9, comprising: Determining the attribute value associated with the specific tag indicates that the field includes the primary key value of the specific data record; as well as Configure the data quality rules to require that each primary key value is unique within the field.

13. The method of claim 9, wherein the attribute value of the specific tag represents at least one of the average value of the values ​​in the field, the maximum length of the values, the minimum length of the values, the data type of the values, and the format of the values.

14. The method of claim 9, wherein the specific data record is a first data record, and wherein the method includes applying the data quality rule to another field associated with the specific tag in a second data record, the second data record being different from the first data record.

15. The method of claim 9, further comprising: Generate data for requesting approval of the data quality rules; as well as The data quality rule is approved in response to the receipt of approval data indicating the approval of the data quality rule.

16. The method of claim 9, wherein generating the data quality rules comprises: Determine the historical trend of the values ​​in the field; as well as The historical trend generation requirement is based on the values ​​in the field.

17. The method of claim 9, wherein generating the data quality rule for the field comprises: Identify the historical average value of the values ​​in the field; as well as The historical average value generation requirement is based on the value in the field.

18. The method of claim 9, comprising: The data processing system archives the values ​​stored in the fields, which are included in the data records of one or more datasets; The data processing system applies one or more classifiers to the archived values; Based on the application of the one or more classifiers to identify one or more attributes that indicate the logical or syntactic characteristics of the value of the field, each of the one or more attributes has a corresponding confidence level based on the output of each of the one or more classifiers; Associate one or more of the identified attributes whose confidence level meets the threshold level with the field; One or more constraints on the values ​​included in the field are determined based on the one or more attributes associated with the field; as well as Based on one or more constraints, data quality rules are determined for the field of the dataset, the data quality rules indicating the relationship between the value of the field and the one or more constraints.

19. The method of claim 4, wherein the at least one pattern test comprises a plurality of different pattern tests, and the method comprises selecting a pattern test to be performed on the data in the field based on the data in the field.

20. A data processing system, the data processing system comprising: At least one processing device; and At least one memory, the at least one memory communicating with the at least one processing device, the at least one memory storing instructions that, when executed by the at least one processing device, cause the at least one processing device to perform the method according to any one of claims 1 to 19.

21. One or more non-transitory computer-readable media storing instructions configured to cause the at least one processing device to perform the method according to any one of claims 1 to 19 when executed by at least one processing device.

Images