Intelligent tagging of structured and unstructured data

Abstract

In some examples, a system inputs information related to a data set of data into a trained model that is trained to determine tags related to the data set. The system may receive, as an output of the trained model, a plurality of values, each value associated with a candidate tag. The system infers tags for the data set based on comparing the plurality of values of the candidate tags with other values of the plurality of values of the candidate tags. Based on the inference, the system selects one or more of the candidate tags as tags for the data set, and adds the one or more selected tags to metadata associated with data that includes the data set.

Description

INTELLIGENT TAGGING OF STRUCTURED AND UNSTRUCTURED DATATECHNICAL FIELD

[0001] This disclosure relates to the technical field of storing, classifying, and accessing data, such as in systems that store large amounts of data.BACKGROUND

[0002] A data catalog may include an assemblage of metadata that can provide an organization with additional information about data sources and other data assets of the organization. The metadata in the data catalog may include tags that describe data assets or that otherwise provide information about the data assets. Automatically tagging large amounts of data requires that the tagging be accurate for the data to be useful. Some conventional techniques for tagging data may include use of machine learning models. However, traditional methods of tagging may involve gathering statistics of patterns in the data and performing an all-to-all comparison of the patterns to statistically estimate the correct tags. This may result in an output that is merely approximate, and the learning may be arbitrary in that the weights of the features are altered by an arbitrary amount based on user acceptance or rejection of the results. Further, there may be no notion of probability or formal optimization in the methodology as there is no notion of convergence. Additionally, in existing techniques the features may be designed manually and, to a certain extent, may be arbitrarily chosen, which may result in inaccuracies since some critical features might be missed, causing incorrect tagging or giving rise to false negatives.SUMMARY

[0003] Some examples herein include a system that inputs information related to a data set of data into a trained model that is trained to determine tags related to the data set. The system may receive, as an output of the trained model, a plurality of values, each value associated with a candidate tag. The system infers tags for the data set based on comparing the plurality of values of the candidate tags with other values of the plurality of values of the candidate tags. Based on the inference, the system selects one or more of the candidate tags as tags for the data set, and adds the one or more selected tags to metadata associated with data that includes the data set.BRIEF DESCRIPTION OF THE DRAWINGS

[0004] The detailed description is set forth with reference to the accompanying figures. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different figures indicates similar or identical items or features.

[0005] FIG. 1 illustrates an example architecture of a system able to tag data sets according to some implementations.

[0006] FIG. 2 includes a flow diagram illustrating an example process for training one or more models according to some implementations.

[0007] FIG. 3 is a flow diagram illustrating an example process for tagging data sets according to some implementations.

[0008] FIG. 4 is a flow diagram illustrating an example process for determining statistics based on the output of the deep learning model according to some implementations.DESCRIPTION OF THE EMBODIMENTS

[0009] Some implementations herein are directed to techniques and arrangements for tagging of structured and unstructured data by training and utilizing an artificial intelligence (Al) model, such as an artificial neural network. The examples herein provide an improvement over conventional techniques for tagging structured and unstructured data and can either work together with conventional tagging techniques in a complementary fashion or may be used as a replacement for the conventional techniques. The examples herein may be extended for document classification. Additionally, the examples herein may be extended to process context automatically, unlike existing techniques that may need to source business entity rules from a user or define the same arbitrarily.

[0010] Some examples herein may combine a deep learning neural network and an autoencoder to alleviate the above-discussed disadvantages with existing techniques. The extent of pattern learning by the network might only be limited by the architecture. Therefore, the techniques herein may include an architecture that improves the tagging accuracy. Further, the machine learning in the examples herein may be based on sound optimization principles that are applied to the neural networks and autoencoders. Additionally, feature selection may be automatic and may only be limited by the number of nodes and layers in the neural network, which reduces the risk of missed features.

[0011] Some examples herein use a feed forward neural network with encoded characters (say, via an embedding layer) with a classification layer (with a softmax or a sigmoid layer) at thetop. Feed many rows of the column as an embedded vector of the seeds as a fixed length vectors (randomized across rows of different seeds) as training data, using the seed as the ground-truth. Once trained, the trained model may receive the candidate columns as input to determine tags for the candidate columns. An autoencoder model may also be trained on the seed columns and a loss associated with the autoencoder may be measured and stored. To potentially reduce false positives, the autoencoder model may be used on the candidate columns, and if the loss is higher than the seed loss by a threshold amount, the tags determined by the deep learning train model may be discarded.

[0012] Additionally, to perform context sensitive disambiguation, neighboring columns may be input into an attention-based neural network, such as a transformer, to automatically learn discriminants in the neighboring columns, if present. This technique may be used to automatically disambiguate the terms. This process only relies on the seed data and, accordingly, no specific rules for disambiguation need to be provided by the user.

[0013] In addition, conventional fingerprint-based methods are typically not suitable for unstructured text. On the other hand, deep learning based tagging according to the examples herein is able to determine tags for unstructured text and other unstructured data since the techniques herein are based on encoding the input entirely. Further, the conventional fingerprint-based methods typically do not take into account context processing across rows, while the deep learning based tagging herein can be extended to do so.

[0014] For discussion purposes, some example implementations are described in the environment of one or more service computing devices in communication with one or more storages and one or more client devices for receiving and tagging data. However, implementations herein are not limited to the particular examples provided, and may be extended to other types of computing systems, other types of storage environments, other system architectures, and so forth, as will be apparent to those of skill in the art in light of the disclosure herein.

[0015] FIG. 1 illustrates an example architecture of a system 100 able to tag data sets according to some implementations. The system 100 includes one or more service computing devices 102 that are able to communicate with one or more storages 104 through one or more networks 106. In addition, the service computing device(s) 102 may also be able to communicate over the one or more networks 106 with a plurality of client devices 108(l)-108(m), such as user devices or other devices that may communicate with the service computing devices 102. For example, the system 100 may store, classify, and manage data for the client devices 108, e.g., as a data storage, data repository, database, data warehouse, or the like.

[0016] In some examples, the service computing devices 102 may include a plurality of physical servers or other types of computing devices that may be embodied in any number of ways. For instance, in the case of a server, the modules, programs, other functional components, and a portion of data storage may be implemented on the servers, such as in a cluster of servers, e.g., at a server farm or data center, a cloud-hosted computing service, and so forth, although other computer architectures may additionally or alternatively be used. In the illustrated example, each service computing device 102 may include, or may have associated therewith, one or more processors 116, one or more communication interfaces 118, and one or more computer-readable media 120. Further, while a description of one service computing device 102 is provided, the other service computing devices 102 may have the same or similar hardware and software configurations and components.

[0017] Each processor 116 may be a single processing unit or a number of processing units, and may include single or multiple computing units or multiple processing cores. The processor(s) 116 can be implemented as one or more central processing units, microprocessors, microcomputers, microcontrollers, system-on-chip processors, digital signal processors, state machines, logic circuitries, and / or any devices that manipulate signals based on operational instructions. For instance, the processor(s) 116 may be one or more hardware processors and / or logic circuits of any suitable type specifically programmed or configured to execute the algorithms and processes described herein. The processor(s) 116 can be configured to fetch and execute computer-readable instructions stored in the computer-readable media 120, which can program the processor(s) 116 to perform the functions described herein.

[0018] The computer-readable media 120 may include volatile and nonvolatile memory and / or removable and non-removable media implemented in any type of technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. For example, the computer-readable media 120 may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, optical storage, solid state storage, magnetic tape, magnetic disk storage, storage arrays, network attached storage, storage area networks, cloud storage, or any other medium that can be used to store the desired information and that can be accessed by a computing device. Depending on the configuration of the service computing device 102, the computer-readable media 120 may be a tangible non-transitory medium to the extent that, when mentioned, non-transitory computer-readable media exclude media such as energy, carrier signals, electromagnetic waves, and / or signals per se. In some cases, the computer-readable media 120 may be at the same location as the service computing device102, while in other examples, the computer-readable media 120 may be separate or partially remote from the service computing device 102.

[0019] The computer-readable media 120 may be used to store any number of functional components that are executable by the processor(s) 116. In many implementations, these functional components comprise instructions, applications, or other programs that are executable by the processor(s) 116 and that, when executed, specifically program the processor(s) 116 to perform the actions attributed herein to the service computing device 102. Functional components stored in the computer-readable media 120 may include a service application 122 that may include, or that may execute an inference program 123, each of which may include one or more computer programs, applications, executable code, computer-readable instructions, or portions thereof. For example, the service application 122 may be executed by the processors(s) 116 for performing various data tagging tasks, as well as data storage and retrieval tasks. For instance, the service application 122 may be executed for interacting with the client devices 108, such as for storing data 124 and metadata 125 for the client devices 108 as stored data 130 in the storage 104, retrieving data 124 and metadata 125 for the client devices 108, and / or for providing the client devices 108 with access to data and data structures stored in the storage 104 as the stored data 130. The service application 122 may also receive instructions 126 from the client devices 108 for performing various operations. Thus, the service application 122 may configure the service computing device(s) 102 to provide one or more services to the client computing devices 108. In addition, the inference program 123 may be executed by the processor(s) 116 in some examples to determine information for performing the tagging functions described herein, as discussed additionally below. In some cases, the functional component(s) may be stored in a storage portion of the computer-readable media 120, loaded into a local memory portion of the computer-readable media 120, and executed by the one or more processors 116.

[0020] In addition, the computer-readable media 120 may store data and data structures used for performing the functions and services described herein. For example, the computer-readable media 120 may store data, metadata, data structures, and / or other information generated by and / or used by the service application 122. For instance, the service computing device(s) 102 may store and manage one or more deep learning models 127, trained weights 128, and one or more autoencoder models 129, as described additionally below. Additionally, or alternatively, the deep learning model(s) 127, trained weights 128, and / or the autoencoder model(s) 129 may be stored on the storage(s) 104. In addition, the service computing device(s) 102 may store various other data structures such as metadata that corresponds to at least a portion of stored data 130, which may include a glossary 132 having classification information for the stored data 130.

[0021] Each service computing device 102 may also include or maintain other functional components and data, which may include an operating system, programs, drivers, etc., and other data used or generated by the functional components. Further, the service computing device(s)102 may include many other logical, programmatic, and physical components, of which those described above are merely examples that are related to the discussion herein. Additionally, numerous other software and / or hardware configurations will be apparent to those of skill in the art having the benefit of the disclosure herein, with the foregoing being merely one example provided for discussion purposes.

[0022] The communication interface(s) 118 may include one or more interfaces and hardware components for enabling communication with various other devices, such as over the network(s) 106. Thus, the communication interfaces 118 may include, or may couple to, one or more ports that provide connection to the one or more network(s) 106 for communication with the storage(s) 104 and the client device(s) 108. For example, the communication interface(s) 118 may enable communication through one or more of a FAN (local area network), WAN (wide area network), the Internet, cable networks, cellular networks, wireless networks (e.g., Wi-Fi) and wired networks (e.g., Fibre Channel, fiber optic, Ethernet), direct connections, as well as closerange communications such as BEUETOOTH®, and the like, as additionally enumerated elsewhere herein. In addition, for increased fault tolerance, the communication interfaces 118 of the service computing device(s) 102 may include redundant network connections to each of the network(s) 106 to which the service computing device(s) 102 is coupled.

[0023] The network(s) 106 may include any suitable communication technology, including a WAN, such as the Internet; a FAN, such as an intranet; a wireless network, such as a cellular network, a local wireless network, such as Wi-Fi, and / or a short-range wireless communications, such as BEUETOOTH®; a wired network including Fibre Channel, fiber optics, Ethernet, or any other such network, a direct wired connection, or any combination thereof. Thus, the network(s) 106 may include wired and / or wireless communication technologies. Components used for the network(s) 106 can depend at least in part upon the type of network, the environment selected, desired performance, and the like. The protocols for communicating over the network(s) 106 herein are well known and will not be discussed in detail. Accordingly, the service computing device(s) 102 is able to communicate with the storage(s)104 and the client device(s) 108 over the network(s) 106 using wired and / or wireless connections, and combinations thereof.

[0024] Each client device 108 may be any suitable type of computing device such as a desktop, workstation, server, laptop, tablet computing device, mobile device, smart phone, wearablecomputing device, or any other type of computing device able to send data over a network. For instance, the client device(s) 108 may generate data 124, metadata 125 and / or instructions 126 that are sent to the service computing device(s) 102 for performing data storage, backup storage, long term remote storage, or any other sort of data storage. In some cases, the client device(s) 108 may include hardware configurations similar to that described for the service computing device 102, but with different data and functional components to enable the client device(s) 108 to perform the various functions discussed herein. In some examples, a user may be associated with a respective client device 108, such as through a user account, user login credentials, or the like. Each client device 108(l)-108(m) may access one or more of the service computing devices 102 through a respective instance of a client application 136(l)-136(m), such as a browser, web application, or other application executed on the respective client device 108. For instance, the client application 136 may provide a graphical user interface (GUI), a command line interface, and / or may employ an application programming interface (API) for communicating with the service application 122 on a service computing device 102. Furthermore, while one example of a client-server configuration is described herein, numerous other possible variations and applications for the computing system 100 herein will be apparent to those of skill in the art having the benefit of the disclosure herein.

[0025] The storage(s) 104 may provide storage capacity for the system 100 for storage of data, such as file data or other object data, and which may include data content and metadata about the content. The storage(s) 104 may include storage arrays such as network attached storage (NAS) systems, storage area network (SAN) systems, cloud storage, storage virtualization systems, or the like. Further, the storage(s) 104 may be co-located with one or more of the service computing devices 102, or may be remotely located or otherwise external to the service computing device(s) 102.

[0026] In the illustrated example, the storage(s) 104 includes one or more storage computing devices referred to as storage controller(s) 138, which may include one or more servers or any other suitable computing devices, such as any of the examples discussed above with respect to the service computing device 102. The storage controller(s) 138 may each include one or more processors 142, one or more computer-readable media 144, and one or more communication interfaces 146. For example, the processor(s) 142 may correspond to any of the examples discussed above with respect to the processors 116, the computer-readable media 144 may correspond to any of the examples discussed above with respect to the computer-readable media 120, and the communication interfaces 146 may correspond to any of the examples discussed above with respect to the communication interfaces 118.

[0027] Further, the computer-readable media 144 of the storage controller 138 may be used to store any number of functional components that are executable by the processor(s) 142. In many implementations, these functional components comprise instructions, modules, or programs that are executable by the processor(s) 142 and that, when executed, specifically program the processor(s) 142 to perform the actions attributed herein to the storage controller 138. Functional components stored in the computer-readable media 144 may include a storage management program 148, which may include one or more computer programs, applications, executable code, computer-readable instructions, or portions thereof. For example, the storage management program 148 may control or otherwise manage the storage of the stored data 130 in a plurality of storage devices 150 coupled to the storage controller 138.

[0028] In some cases, the storage devices 150 may include one or more arrays of physical storage devices. For instance, the storage controller 138 may control one or more arrays, such as for configuring the arrays in a RAID (redundant array of independent disks) configuration or other desired storage configuration. In some examples, the storage controller 138 may present logical units based on the physical devices to the service computing devices 102, and may manage the data stored on the underlying physical devices. The storage devices 150 may include any type of storage device, such as hard disk drives, solid state devices, optical devices, magnetic tape, and so forth, or combinations thereof. Alternatively, in other examples, one or more of the service computing devices 102 may act as the storage controller, and the storage controller 138 may be eliminated.

[0029] In the illustrated example, the service computing device(s) 102 and storage(s) 104 may be configured to act as a data storage system for the client devices 108. The service application 122 on the service computing device(s) 102 may be executed to receive and store data 124 from the client devices 108 and / or subsequently retrieve and provide the data 124 to the client devices 108. The system 100 may be scalable to increase or decrease the number of service computing devices 102 in the system 100, as desired for providing a particular operational environment. The amount of storage capacity included within the storage(s) 104 can also be scaled as desired. Further, the service computing devices 102 and the client devices 108 may include any number of distinct computer systems, and implementations disclosed herein are not limited to a particular number of computer systems or a particular hardware configuration.

[0030] In some examples, the stored data 130 may include a huge amount of data, at least some of which may be configured as data sets 152 and metadata 154. For instance, a data set 152 may include a collection of data and one or more corresponding data fields. The data field may be associated with the data of the data set in a structured or semi-structured data resource, such as ina table, comma separated value (csv) file, json, xml, parquet, or other data structure. As one example, a column in a csv file may be a field, and may accompany, correspond to, or otherwise be associated with a particular data set 152.

[0031] In some examples, the terms “classification” and “tag” may be used interchangeably. For example, suppose that a given data field is “classified” as a social security number. The data field may also be referred to as being “tagged” as a social security number. Accordingly, in some examples, a tag may be, or may include a descriptive phrase of one or more words that describes the content and / or context of the data with which the tag is associated, such as a column of data or other data set 152. As discussed below, the tags herein may be inferred by the inference program 123 to determine a plurality of inferred tags 156 that may be incorporated into the metadata 154, such as into the glossary 132.

[0032] Furthermore, in implementations herein, a data field or a data file may be classified and represented by one or more associated classifications in a data catalog referred to herein as the glossary 132. For instance, the glossary 132 may include metadata about each data file or other data set 152 stored in the stored data 130. In some cases, the glossary 132 may allow annotations provided by users to be retained as part of the metadata content. These annotations can then be used to enable searching and data understanding.

[0033] The service application 122 may include a tagging algorithm, as discussed additionally below, that may be executed by the processor(s) 116 to automatically determine tags to associate with the data sets 152. The result of the tagging algorithm is that one or more tags are associated with one or more data sets 152 (also referred to as “fields” or “columns” in some examples). The service application 122 may be executed to perform systematic organization of the stored data 130 so that the users of the client devices 108 can reliably and efficiently access desired data without requiring the users to have knowledge of the underlying data organization. This may include performing an automatic inventory of all files or other data sets 152, which may include capturing the lineage, format, and profile of each file or other data set 152 and storing this information in the metadata 154.

[0034] According to the techniques herein, the one or more deep learning models 127 and the one or more autoencoder models may be employed in a synergistic manner to perform pattern learning on the data sets 152 based on sound optimization principles applied in neural networks and autoencoders. Further, the feature selection may be automatic and is only limited by the number of nodes and layers in the network, so that there is a reduced risk of missed features.

[0035] In some examples, the deep learning model 127 is a feed-forward neural network with encoded characters. For example, the encoded characters may be provided in an embedding layerof the deep learning model 127. An input layer of the deep learning model 127 may have a predefined maximum size, and the number of intermediate layers may be variable and configurable by the user. Additionally, the number of nodes in each layer may also be variable and configurable by the user. In addition, deep learning model 127 may include a classification layer (with a softmax or a sigmoid function layer) as the final layer of the deep learning model 127. For example, the softmax function or sigmoid function may be used to normalize the output of deep learning model 127 to a probability distribution over predicted output classes. The size of the output layer may be the same size as the size of a one hot encoded vector that is used as the input for the model.

[0036] During training of the deep learning model 127, a configurable maximum size of the training data to use as input is defined. Additionally, seed columns or other seed data sets are received from a user along with a respective tag corresponding to each seed data set. For example, the tags associated with the seed data sets may have been assigned or otherwise confirmed by the user to correspond to its respective seed data set. Thus, the seed data sets may provide a set of unique tags that may be one-hot encoded. In some examples, all training data input may be treated as a string type. If the training data to be input is not already a string type, then it may be converted to a string. In addition, each word (seed cell) of the training data may be parsed into its characters, and all the unique characters are collected into a vocabulary by passing through the entire training data once. Padding and unknown characters may be added to the vocabulary as the training data is processed. The unique characters may be sorted and then the ordinal of each character may be used as the encoding of that character. The vocabulary ordinal is used to encode the characters to a numerical vector of size max size. If the length of the word is longer than the max size configured, the word is trimmed to the max size. On the other hand, if the length of the word is shorter than the max size, the word is post-padded with a padding character which is also part of the vocabulary.

[0037] The algorithm feeds into the deep learning model 127 as many rows of the training data columns as there are rows in the one-hot encoded vector of the seeds determined as fixed length vectors (e.g., randomized across rows of different seeds) and using the seed as the groundtruth. A mapping is made to each encoded seed cell and the one hot encoding of the ground-truth tag for that seed cell. The order of the seed cells and their associated mappings may be randomized. The encoded seed cell is passed to the input layer and is fed forward across the deep learning neural network layer in turn. The output for each input is compared to the corresponding one-hot- encoded ground truth of the corresponding tag and the cross-entropy loss is calculated. The loss is back propagated through the neural network adjusting the weights in proportion to the loss. Thisprocess may be performed for the entire training data input and for a user configurable maximum number of epochs. The training may be stopped if the decrease in loss saturates for a configurable number of epochs.

[0038] In addition, the example algorithm herein may also include training of the autoencoder model 129 using the same seed columns, and measuring and storing the losses obtained. The autoencoder model herein may be or may include a type of artificial neural network used to learn efficient codings of unlabeled data. The autoencoder may include a learned encoding function that transforms the input data, and a decoding function that recreates the input data from an encoded representation. For example, the trained autoencoder model 129 may be used to potentially reduce false positives on the data sets 152 (e.g., columns) that are to be tagged. For instance, if the autoencoder model’s loss is greater than a threshold proportion of the seed loss, the process herein may discard the determined tags, even if the classifier says that the tags correspond to that data set 152.

[0039] Once the deep learning model 127 and the autoencoder model 129 are trained, the deep learning model 127 and autoencoder model 129 may be used to process selected candidate data sets 152 for tagging the selected candidate data sets 152. As one example, suppose that the candidate data sets 152 are a plurality of candidate columns that have been selected for tagging. For example, each column may have a plurality of rows (also referred to as “cells”) of the same type of data, e.g., social security numbers of a plurality of users might be an example of one column, with each different social security number corresponding to a different row in the column. Street addresses might be an example of the type of data in another column, phone numbers might be an example of yet another type of data in another column, usernames might be an example of yet another type of data in another column, and so forth. Implementations herein are not limited to any particular type of data for the columns or other data sets 152.

[0040] The selected candidate columns may be sampled up to a user configurable sample size. Additionally, each cell of the sampled candidate columns to be tagged may be encoded in a manner similar to that discussed above with respect to the training phase. The encoded information may be fed forward through the trained deep learning model 127. For each column, statistics are calculated on the output of the sampled rows of the candidate column. Details of determining the statistics are discussed additionally below. Based on the determined statistics, the most probable set of tags, and a set of alternative likely tags are determined for each respective column of the selected candidate columns. Those inputs whose outputs all have a likelihood that is less than a user configurable threshold of the sigmoid value are discarded as not belonging to the respective column. The likely tags determined for each column are listed as the possible tags for that column.

[0041] In addition, in order to perform context sensitive disambiguation, the neighboring columns that neighbor the selected candidate columns may be determined, and an attention-based neural network, such as a transformer neural network, may be used as at least one of the deep learning models 127, and may be used to automatically learn discriminants in the neighboring columns, if present. For example, a transformer neural network is a deep learning architecture based on a multi-head attention mechanism. Transformer neural networks typically do not include any recurrent units, and therefore require less training time than many other types of recurrent neural architectures. In some examples herein, the transformer neural network may also be used to automatically disambiguate terms included in the data sets 152 being processed for tagging. This function only requires seed data and does not require the user to provide any specific rules for disambiguation.

[0042] In the examples herein, users are able to cause the system 100 to execute the inference program 123 to have tags added to their data based on the seeds alone, and with no need for the users to make a manual determination of features. Furthermore, implementations herein eliminate the need for calculation of computationally expensive statistics. Instead, the training and the inference performed according to the implementations herein may merely include a series of matrix multiplications which can be significantly accelerated by a graphics processing unit or a tensor processing unit. Furthermore, large storage space for storing large properties such as bloom filters are not needed, and the training learns upfront from differences between the tags as opposed to learning features of only one tag. In addition, the process herein is easily amenable to context processing by passing in, during training, the adjacent columns and using a transformer-based attention mechanism. This adds further intelligence automatically to the inferring of the tags. Additionally, the examples herein are suitable for processing unstructured documents using neural-network-based natural language processing techniques.

[0043] The system 100 is not limited to the particular configuration illustrated in FIG. 1. This configuration is included for the purposes of illustration and discussion only. Various examples herein may utilize a variety of hardware components, software components, and combinations of hardware and software components that are configured to perform the processes and functions described herein. In addition, in some examples, the hardware components described above may be virtualized. For example, some or all of the service computing devices 102 may be virtual machines operating on the one or more hardware processors 116 or portions thereof, and / or other service computing devices 102 may be separate physical computing devices, or may be configured as virtual machines on separate physical computing devices, or on the same physical computing device. Numerous other hardware and software configurations will be apparent to those of skill inthe art having the benefit of the disclosure herein. Thus, the scope of the examples disclosed herein is not limited to a particular set of hardware, software, or a combination thereof.

[0044] FIGS. 2-4 include flow diagrams illustrating example processes according to some implementations. The processes are illustrated as collections of blocks in logical flow diagrams, which represent a sequence of operations, some or all of which may be implemented in hardware, software or a combination thereof. In the context of software, the blocks may represent computerexecutable instructions stored on one or more computer-readable media that, when executed by one or more processors, program the processors to perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures and the like that perform particular functions or implement particular data types. The order in which the blocks are described should not be construed as a limitation. Any number of the described blocks can be combined in any order and / or in parallel to implement the process, or alternative processes, and not all of the blocks need be executed. For discussion purposes, the processes are described with reference to the environments, frameworks and systems described in the examples herein, although the processes may be implemented in a wide variety of other environments, frameworks and systems.

[0045] FIG. 2 includes a flow diagram illustrating an example process 200 for training one or more models according to some implementations. For example, the process 200 may be executed by one or more of the service computing devices 102 or other suitable computing devices, such as by execution of the service application 122. For example, through execution of the service application 122, the system may receive and tag data.

[0046] At 202, the computing device may receive seed data for use in training the deep learning model(s) 127 and / or the autoencoder model(s) 129. For example, when selecting training data sets, a user may attempt to select data sets with the relevant and representative data (referred to herein as “seed data sets”) and may provide one or more tags with each seed data set. It is desirable for the tags to represent their associated seed data sets as precisely as possible.

[0047] At 204, the computing device may build a vocabulary and one hot encode the tags received with the seed data. For example, the seed data sets may provide a set of unique tags that may be one-hot encoded. In some examples, all training data input may be treated as a string type. If the training data to be input is not already a string type, then it may be converted to a string. In addition, each word (seed cell) of the training data may be parsed into its characters, and all the unique characters are collected into a vocabulary by passing through the entire training data once. Padding and unknown characters may be added to the vocabulary as the training data is processed.The unique characters may be sorted and then the ordinal of each character may be used as the encoding of that character.

[0048] At 206, the computing device may vectorize the sampled seed data. For example, the vocabulary ordinal is used to encode the characters to a numerical vector of size “max size”, which may be a default vector size, or which may be a size that was configured by the user. If the length of a word is longer than the max size configured, the word is trimmed to the max size. On the other hand, if the length of the word is shorter than the max size, the word is post-padded with a padding character that is also part of the vocabulary.

[0049] At 208, the computing device may merge and randomize the vectors. For example, a mapping is made to each encoded seed cell and the one-hot encoding of the ground-truth tag for that seed cell. The order of the seed cells and their associated mappings may be randomized.

[0050] At 210, the computing device may forward the vectors through the deep learning model for training the deep learning model. The encoded seed cell is passed to the input layer and is fed forward across the deep learning neural network layer in turn. The output for each input is compared to the corresponding one-hot-encoded ground truth of the corresponding tag and the cross-entropy loss is calculated.

[0051] At 212, the computing device may back-propagate and adjust the weights of the deep learning model. For example, the loss may be back-propagated through the neural network of the deep learning model by adjusting the weights of the deep learning model in proportion to the loss. Thus, the back-propagation may improve the weights and biases of the deep learning model so that the deep learning model makes more accurate predictions.

[0052] At 214, the computing device may determine whether the loss is saturated, i.e., the point at which additional back-propagation does not satisfy a threshold amount of improvement in the amount of loss. For example, the back-propagation and weight adjustment may be performed for the entire input and for a user configurable maximum number of epochs. The training is stopped if the decrease in loss saturates for a configurable number of epochs. Accordingly, if the loss is not yet saturated, the process returns to 212 to perform more back propagation and weight adjustment. On the other hand, if the loss has saturated, the process goes to 216.

[0053] At 216, when the loss is determined to be saturated, the computing device may save the trained model and / or provide the train the model for data tagging such as at C of FIG. 3.

[0054] At 218, the computing device may also train one or more autoencoder models using the same seed data sets. For example, the trained autoencoder model 129 may be usedsubsequently to potentially reduce false positives on the data sets 152 (e.g., data columns) that are to be tagged.

[0055] At 220, the computing device may measure and store losses obtained through autoencoder model testing during the training process. The trained autoencoder model 129 may be applied at B in FIG. 3 during the tagging process. For example, if the autoencoder model’s loss is greater than a threshold proportion of the seed loss, the process herein may discard the determined tags, even if the deep learning model 127 performing the tagging says that the tags are predicted to correspond to a data set 152 that is being processed.

[0056] FIG. 3 is a flow diagram illustrating an example process 300 for tagging data sets according to some implementations. In some examples, the process 300 may be executed by one or more of the service computing devices 102 or other suitable computing devices. For instance, the process 300 may be performed at least in part by the computing device 102 executing the inference program 123 of the service application 122 discussed above with respect to FIG. 1.

[0057] At 302, the computing device may receive data sets for tagging. For example, the received data sets may be new data sets received from a client device 108 or may be data sets received from any of various other data sources such as storages 104 or other storage systems, servers, databases, computer programs, or the like. The received data sets may be structured data or semi- structured data, such as in tables, csv files, JSON data, XML data, PARQUET data, or other data structures. Additionally, in some examples, the data may be unstructured data, such as any unstructured alphanumeric data.

[0058] At 304, the computing device may sample and vectorize the received data sets. For example, each data set may have a plurality of rows (also referred to as “cells”) of the same data type. The data sets may be sampled up to a user configurable sample size. Additionally, each cell of the data sets may be encoded in a manner similar to that discussed above with respect to the training phase, e.g., parsed into its characters, and all the unique characters are collected into a vocabulary by passing through the data set once. The unique characters may sorted and then the ordinal of each character may be used as the encoding of that character. The vocabulary ordinal is used to encode the characters to a numerical vector of size max size.

[0059] At 306, the computing device may retrieve the trained deep learning model 127 for use in tagging the received data sets.

[0060] At 308, the computing device may forward the encoded and vectorized data through the deep learning model 127.

[0061] At 310, the computing device may also input the encoded and vectorized data into the autoencoder model 129 for use in potentially reducing false positives.

[0062] At 312, the computing device may analyze the output of the deep learning model. For example, the output of the trained deep learning model may include a plurality of values, each value associated with a candidate tag for describing the content and / or context of the corresponding data set. As discussed additionally below, statistics may be calculated on the values of the candidate tags for the sampled rows of the data set being processed, and one or more of the candidate tags may be selected as a suggested tag (or an alternative tag) for the corresponding data set, such as based on a statistical comparison of the respective values of the candidate tags with one another.

[0063] At 314, the computing device may determine if the autoencoder model loss is greater than the deep learning model by more than a threshold amount. If so, the process goes to 318. If not, the process goes to 320.

[0064] At 316, if the loss of the autoencoder model is greater than the deep learning model loss by more than the threshold amount the computing device may discard the tag results.

[0065] At 318, if the difference between the autoencoder model loss and the deep learning model loss is not greater than the threshold amount, the computing device may determine statistics based on the output of the deep learning model to be used for inferring one or more tags from a plurality of candidate tags. For instance, those inputs whose outputs all have a likelihood that is less than a user configurable threshold of the sigmoid value are discarded as not belonging to the respective data set being processed. An example of determining the statistics for the output of the deep learning model 127 is illustrated and described below with respect to FIG. 4.

[0066] At 320, the computing device may infer one or more tags for the corresponding data set based on the determined statistics. For example, based on the determined statistics, the most probable set of tags, and, in some cases, a set of alternative likely tags are determined for each respective data set input to the deep learning model. Additional details are discussed below with respect to FIG. 4.

[0067] At 322, the computing device may ingest the one or more inferred tags into the metadata of the system for association with the corresponding data set. As one example, the tags may be incorporated into a glossary 132 of a data catalog, such as discussed above with respect to FIG. 1.

[0068] FIG. 4 is a flow diagram illustrating an example process 400 for determining statistics based on the output of the deep learning model 127 according to some implementations. In some examples, the process 400 may be executed by one or more of the service computing devices 102 or other suitable computing devices, such as by execution of the inference program 123 of theservice application 122. In some examples, the process 400 may be included, at least in part, in blocks 318 and 320 of FIG. 3.

[0069] At 402, for calculating the statistics, the computing device may initialize one or more dictionaries 121 (not shown in FIG. 4) for use during the statistics calculations. The dictionaries 121 may be used to keep track of various values. In some examples, six dictionaries may be initialized, and which may include a first dictionary (sigmoid value) that may be initialized to keep track of the sigmoid value. A second dictionary (toprank counts) may be initialized to keep track of counts of the tags occurring in the maximum position of the output across the rows. A third dictionary (maxrank alternates) may be initialized to keep count of tags that occur in the n- maximum position of the output across the rows. A fourth dictionary (predicted count) may be initialized to count the output of indices occurring in the match position across the rows. A fifth dictionary (name match) may be initialized for maintaining a fuzzy match score of the seed data set with a field name for that seed data set. A sixth dictionary (tag name match) may be initialized for maintaining a fuzzy match score of seed tags with field names. Furthermore, while a plurality of dictionaries are described herein as one example of a technique for tracking values related to the determined tags, numerous other techniques for accomplishing the same result will be apparent to those of skill in the art having the benefit of the disclosure herein.

[0070] At 404, after the dictionaries have been initialized the computing device may round off the output of the deep learning model for a selected candidate data set. For example, the number of digits for the output may be configurable by the user.

[0071] At 406, the computing device may traverse each output value for each row of the selected data set and may increment the sigmoid value of the output as a key in the sigmoid value dictionary.

[0072] At 408, the computing device may track counts of the tags occurring in the maximum position of the output across the rows in the top rank counts dictionary.

[0073] At 410, the computing device may track counts of tags occurring in the n-maximum position of the output across the rows in the maxrank alternates dictionary.

[0074] At 412, the computing device may track counts of output indices occurring in the maximum position across the rows in the predicted count dictionary.

[0075] At 414, the computing device may store a fuzzy match score of the seed data set with a field name for the seed data set in the name match dictionary and keyed by the seed tag.

[0076] At 416, the computing device may store a fuzzy match score of a seed tag with the field the name in the tag name match dictionary and keyed by the seed tag.

[0077] At 418, the computing device may determine a primary set of suggested labels that are in a suggested labels list and that correspond to the n largest values in the predicted count dictionary.

[0078] At 420, the computing device may determine a “Max Label” that corresponds to a label that has occurred with a maximum frequency in the largest position in the predicted count dictionary.

[0079] At 422, the computing device may save, as one or more “Alternate Labels”, those labels that have occurred within the largest four frequencies in the maximum position of prediction.

[0080] At 424, the computing device may store as “Max Name Match” the maximum across all the name matches between field names and seed data set names, and may store as “Max Tag Name Match” the maximum across all the name matches between the field names and the seed tag names.

[0081] At 426, the computing device may perform thresholding on the fuzzy match scores to determine a filtered list of Max Name Match and Max Tag Name Match, which are combined to generate a Suggested Tags List.

[0082] At 428, the computing device may determine, as Top Ranked Alternative tags, the tags that have occurred in the maximum value positions in predictions at least a threshold number of times, and may determine Max Ranked Alternative tags based on the m-maximum frequencies found in the maxrank alternates dictionary.

[0083] At 430, the computing device may create a list of Suggested Tags from a union of the Max Label determined at 420, and the Suggested Tags List determined at 426.

[0084] At 432, the computing device may determine a list of Alternative Tags from a union of the Top Ranked Alternative tags and the Max Ranked Alternative tags.

[0085] At 434, the computing device may prune the list of Suggested Tags and the list of Alternative Tags by thresholding on a confidence that may be calculated for the tags. As one example, the confidence may be calculated according to the following pseudocode:Retrieve the dictionary for the tag from sigmoid value dictionary set total to zero set confidence to zero for each key-value pair in the sigmoid value dictionary for the tag: set confidences confidence + key*value set totals total + value if total >0return confidence / total else: return 0 .

[0086] The confidence / total values may be associated with the respective tags, and the tags with the lower confidence / total values may be removed from the respective lists of Suggested Tags or Alternative Tags. Additionally, in some examples, the list of Suggested Tags and the list of Alternative Tags may be further pruned based on thresholding on the key for the respective tag.

[0087] At 436, the computing device may proceed with block 322 of FIG. 3 based on the determined Suggested Tags and Alternative Tags (if any).

[0088] The example processes described herein are only examples of processes provided for discussion purposes. Numerous other variations will be apparent to those of skill in the art in light of the disclosure herein. Further, while the disclosure herein sets forth several examples of suitable frameworks, architectures and environments for executing the processes, the implementations herein are not limited to the particular examples shown and discussed. Furthermore, this disclosure provides various example implementations, as described and as illustrated in the drawings. However, this disclosure is not limited to the implementations described and illustrated herein, but can extend to other implementations, as would be known or as would become known to those skilled in the art.

[0089] Various instructions, processes, and techniques described herein may be considered in the general context of computer-executable instructions, such as program modules stored on computer-readable media, and executed by the processor(s) herein. Generally, program modules include routines, programs, objects, components, data structures, executable code, etc., for performing particular tasks or implementing particular abstract data types. These program modules, and the like, may be executed as native code or may be downloaded and executed, such as in a virtual machine or other just-in-time compilation execution environment. Typically, the functionality of the program modules may be combined or distributed as desired in various implementations. An implementation of these modules and techniques may be stored on computer storage media or transmitted across some form of communication media.

[0090] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

Claims

CLAIMS1. A system comprising: one or more processors configured by executable instructions to perform operations comprising: inputting information related to a data set of data into a trained model that is trained to determine tags related to the data set; receiving, as an output of the trained model, a plurality of values, each value associated with a candidate tag; inferring tags for the data set based on comparing the plurality of values of the candidate tags with other values of the plurality of values of the candidate tags; based on the inferring, selecting one or more of the candidate tags as tags for the data set; and adding the one or more selected tags to metadata associated with data that includes the data set.

2. The system as recited in claim 1, wherein the trained model is trained based at least on receiving, as training data, a plurality of training data sets, each training data set including at least one associated tag corresponding to the training data set.

3. The system as recited in claim 2, wherein the trained model is further trained based at least on: parsing words in the training data sets into respective characters; assigning respective different values to different unique characters; and encoding the words into a numerical vector based on the respective different values of the respective characters.

4. The system as recited in claim 3, wherein the trained model is further trained based at least on: one-hot encoding the tags corresponding to the plurality of training data sets; and determining a mapping between cells of the training data sets and respective ones of the one-hot encoded tags.

5. The system as recited in claim 4, wherein the trained model is further trained based at least on: determining a loss based on comparing an output of the trained model with an input based on a corresponding one-hot encoded ground truth of a corresponding tag; and back-propagating the loss by adjusting one or more weights of the trained model.

6. The system as recited in claim 2, further comprising training an autoencoder model based at least on the plurality of training data sets.

7. The system as recited in claim 6, the operations further comprising inputting the information related to the data set into the autoencoder model to check for one or more false positive tags in the output of the trained model.

8. The system as recited in claim 1, wherein the trained model comprises a neural network including at least one of a sigmoid function layer or a softmax function layer as a final layer.

9. The system as recited in claim 1 , wherein inferring tags for the data set based on comparing the plurality of values of the candidate tags with other values of the plurality of values of the candidate tags further comprises: traversing output values for individual cells of the data set corresponding to the information related to the data set input into the trained model; determining a label that has occurred with a maximum frequency among the individual cells; and including the label as one of the selected tags.

10. The system as recited in claim 9, further comprising deselecting at least one of the one or more selected tags based at least on determining a confidence value for each of the one or more selected tags.

11. The system as recited in claim 1, wherein selecting the one or more selected tags includes selecting at least one selected tag and at least one alternative tag.

12. A method comprising: inputting, by one or more processors, information related to a data set of data into a trained model that is trained to determine tags related to the data set; receiving, as an output of the trained model, a plurality of values, each value associated with a candidate tag; inferring tags for the data set based on comparing the plurality of values of the candidate tags with other values of the plurality of values of the candidate tags; based on the inferring, selecting one or more of the candidate tags as tags for the data set; and adding the one or more selected tags to metadata associated with data that includes the data set.

13. The method as recited in claim 12, wherein the trained model is trained based at least on receiving, as training data, a plurality of training data sets, each training data set including at least one associated tag corresponding to the training data set.

14. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, configure the one or more processors to perform operations comprising: inputting, by one or more processors, information related to a data set of data into a trained model that is trained to determine tags related to the data set; receiving, as an output of the trained model, a plurality of values, each value associated with a candidate tag; inferring tags for the data set based on comparing the plurality of values of the candidate tags with other values of the plurality of values of the candidate tags; based on the inferring, selecting one or more of the candidate tags as tags for the data set; and adding the one or more selected tags to metadata associated with data that includes the data set.

15. The one or more non-transitory computer-readable media as recited in claim 14, wherein the trained model is trained based at least on receiving, as training data, a plurality of training data sets, each training data set including at least one associated tag corresponding to the training data set.

Images