Automatically infer data relationships in datasets

Abstract

Described herein are systems and methods for inferring data relationships of multiple data sets. The data content (and optional metadata) of multiple data sets are scanned to extract features of each of the data sets. Features may be related to the structure of the data, the profile of the data within the data set, and / or the metadata of the data set. Each feature has an associated weight. Based on at least some of the weighted features (e.g., based on a sim-hash or min-hash of the data set), the data sets may be clustered into clusters. Based on the weighted features of the data sets in each cluster, an exact similarity measure between the data sets in each cluster is calculated. Data sets with an exact similarity measure above a threshold number are inferred to be possibly related. Information about the inferred possibly related data sets is provided.

Description

Background Art

[0001] Large organizations such as companies, governments, etc. can store more and more data. Heterogeneous big data systems can have hundreds of millions of datasets. Many datasets may be related. For example, one dataset may be an exact copy of another dataset, but stored in different underlying storage; one dataset may be an approximate copy of another dataset; one dataset may be derived from another dataset by filtering, projection, or aggregation; two or more datasets may have been joined, etc. Summary of the invention

[0002] Described herein is a system for inferring data relationships among multiple data sets, comprising: a processing system including a processor and a memory having computer executable instructions stored thereon, wherein when the computer executable instructions are executed by the processor, the processing system is caused to: algorithmically scan the data contents of the multiple data sets to extract features of each of the data sets, wherein each feature has an associated weight; cluster the data sets into clusters based on at least some of the weighted features; calculate precise similarity measures between the data sets in each cluster based on the weighted features of the data sets in each cluster; infer data sets having precise similarity measures above a threshold number as possibly related; and provide information about the inferred possibly related data sets.

[0003] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 is a functional block diagram illustrating a system for inferring data relationships of data sets.

[0005] Figure 2 is a flow chart of a method for inferring data relationships among multiple data sets.

[0006] Figure 3 is a flow chart of a method for inferring data relationships among multiple data sets.

[0007] Figure 4 is a functional block diagram illustrating an exemplary computing system. DETAILED DESCRIPTION

[0008] Referring now to the accompanying drawings, various techniques for automatically inferring data relationships of data sets are described, wherein the same reference numerals are used to refer to the same elements throughout. In the following description, for the purpose of explanation, a large number of specific details are set forth to provide a thorough understanding of one or more aspects. However, it is apparent that these (multiple) aspects may be practiced without these specific details. In other instances, for ease of describing one or more aspects, well-known structures and devices are shown in the form of block diagrams. In addition, it should be understood that the functions described as being performed by a particular system component may be performed by multiple components. Similarly, for example, a component may be configured to perform the functions described as being performed by multiple components.

[0009] The subject matter discloses various products and processes that support performing or are configured to perform various actions related to automatically inferring data relationships of data sets. The following are one or more exemplary systems and methods.

[0010] Aspects of the subject disclosure relate to the technical problem of identifying related data sets. Technical features associated with solving the problem include algorithmically scanning data content of multiple data sets to extract features of each data set in the data set, wherein each feature has an associated weight; clustering the data sets into clusters based on at least some of the weighted features; calculating precise similarity measures between the data sets in each cluster based on the weighted features of the data sets in each cluster; inferring data sets with precise similarity measures above a threshold number as possibly related; and providing information about the inferred possibly related data sets. Therefore, aspects of these technical features exhibit more efficient technical effects, and effectively reduce (multiple) computing resources and / or bandwidth used to identify related data sets.

[0011] Furthermore, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless specified otherwise or clear from context, "X employs A or B" is intended to mean any of the natural inclusive permutations. That is, "X employs A or B" satisfies any of the following: X employs A; X employs B; or X employs both A and B. Furthermore, the articles "a" and "an" as used in this application and the appended claims should generally be construed to mean "one or more" unless the context indicates otherwise or clearly directs to a singular form.

[0012] As used herein, the terms "component" and "system" and its various forms (e.g., component, system, subsystem, etc.) are intended to refer to a computer-related entity, whether hardware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an instance, an executable file, an execution thread, a program, and / or a computer. By way of example, both an application running on a computer and a computer can be a component. One or more components can reside within a process and / or execution thread, and a component can be located on one computer and / or distributed between two or more computers. In addition, as used herein, the term "exemplary" is intended to mean an illustration or example of something, and is not intended to indicate a preference.

[0013] Large organizations such as companies, governments, etc. can store more and more data. Heterogeneous big data systems can have hundreds of millions of datasets. Many datasets may be related. For example, one dataset may be an exact copy of another dataset, but stored in different underlying storage; one dataset may be an approximate copy of another dataset; one dataset may be derived from another dataset through filtering, projection, or aggregation; two or more datasets may have been joined, etc. Determining these relationships usually requires a deep understanding of the tools or processes that created the datasets; however, there may be hundreds or even thousands of such tools, making it impractical to understand a significant portion of them to determine dataset relationships.

[0014] Described herein are systems and methods for automatically inferring data relationships of data sets. Features can be automatically computed by algorithmically scanning the data content and metadata of data sets, and features are used to automatically infer relationships between these data sets in a manner that is scalable to large data sets (e.g., billions of data sets). For example, relationships can be inferred without access to the query(s) used to generate the data set.

[0015] refer to Figure 1 , a system for inferring data relationships of data sets 100 is illustrated. The system 100 can automatically infer data relationships (eg, replicated, derived, aggregated, connected) of a plurality of data sets 110 storing a collection of data.

[0016] In some embodiments, data set 110 includes a relational database that includes one or more tables (e.g., (multiple) relationships) of column(s) (e.g., (multiple) attributes, (multiple) fields) and row(s) (e.g., (multiple) records). The (multiple) relationships can logically connect the tables to each other. In some embodiments, data set 110 includes an object-oriented data structure, a hierarchical data structure, and / or a network data structure that stores data according to (multiple) schemas. In some embodiments, the data content of data set 110 is stored in a variety of formats.

[0017] The system 100 includes a feature extraction component 120 that algorithmically scans data and / or metadata content of the plurality of data sets 110 to extract features of each of the data sets. Each feature is of significance in inferring relationships between various data sets of the plurality of data sets 110.

[0018] In some embodiments, a feature may include one or more aspects (e.g., a schema) related to the structure of the data within a data set. Features may include a name (e.g., a column name, an object name), a data type (e.g., an integer, a floating point, a string), and / or a format (e.g., a five-digit number, a string in a specific format).

[0019] In some embodiments, features can be calculated or determined based on one or more aspects related to the profile of the data in the data set. For example, features can include data profiles (e.g., masks), length histograms, value histograms, minimum values, maximum values, minimum-hash, sim-hash, data sketches, pattern matching, and / or data samples. "Minimum-hash" refers to a value calculated using a minimum-hash function (e.g., a minimum independent permutation local sensitive hashing scheme). "Sim-hash" refers to a value calculated using a sim-hash function. Minimum-hash and sim-hash are two techniques for evaluating the similarity of two sets. In some embodiments, sim-hash and / or minimum-hash can be calculated based on a pattern name (e.g., a column name), a pattern format (e.g., a column format), a type (e.g., a column type), (multiple) data samples, and / or (multiple) resource uniform resource identifiers (URIs).

[0020] In some embodiments, features may include one or more aspects of metadata related to a data set. For example, features may include a timestamp, a data set name, one or more column names, one or more column types, and / or labels. In some embodiments, one or more of the extracted and / or calculated features are normalized before being utilized. For example, a pattern name may be stemmed.

[0021] Each feature has an associated weight, and the associated weight is based on the importance of the inferred relationship. That is, features with higher associated weights are more important when inferring a relationship, and features with lower associated weights are less important when inferring a relationship. Zero weights indicate that the associated features are not used for clustering purposes. In some embodiments, as described below, using a machine learning algorithm, the associated weights are determined and / or adaptively modified. In some embodiments, the associated weights are manually determined (e.g., based on user input). In some embodiments, one or more features can be determined.

[0022] For example, text frequency-inverse document frequency (TF-IDF) can be employed to weight unique pattern parts (e.g., column names, dataset names, masks) more heavily than more frequently occurring names. In this way, two datasets with a column with a unique name ("Microsoft Account Payable Identifier") are more likely to be related than two datasets with a column with a common name ("Name"). Similarly, two datasets with a column with a unique mask, such as three uppercase letters followed by a period and three digits ("ABC.123"), may be weighted higher than a column with a more frequently occurring mask ("123").

[0023] For a big data system with hundreds of millions of data sets, it is not practical to compare the features of the first data set with each of the remaining hundreds of millions of data sets to identify similar data sets. In order to significantly reduce the complexity of the comparison, the system 100 first clusters the data sets into candidate clusters based on the weighted features, and the data set members of each cluster may be related to each other.

[0024] Therefore, the system 100 further includes a clustering component 130, which clusters the data sets (e.g., approximately) into clusters based on the weighted features using a streaming approximate clustering algorithm. In some embodiments, for each data set, the minimum-hash values ​​of at least some of the weighted features in the weighted features are generated (e.g., in parallel) and used to create clusters. In some embodiments, for each data set, the sim-hash values ​​of at least some of the weighted features in the weighted features are generated (e.g., in parallel) and used to create clusters. In some embodiments, the clustering component 130 can classify the data sets into clusters using a similarity metric calculation component 140, which compares adjacent or nearby data sets to infer data sets that may be related. In some embodiments, approximate clustering can be performed iteratively, for example, on the sim-hash values ​​of different weighted features or weighted feature groups, and the results of each iteration are combined together for higher accuracy in determining meaningful clusters.

[0025] Once the approximate clusters have been created, the similarity measure calculation component 140 calculates an exact similarity measure between the data sets in each cluster based on at least some of the weighted features of the data sets in each cluster. In some embodiments, the exact similarity measure is based on one or more similarity functions, such as Jaccard similarity, cosine similarity, edit distance, etc. Based on the exact similarity measure, the relationship inference component 150 can infer that data sets with an exact similarity measure above a threshold number are likely to be related. In some embodiments, the threshold number is determined based on user input. For example, the user can indicate that the threshold number is 95%, that is, data sets with an exact similarity measure of 95% or higher probability are identified as likely to be related.

[0026] The system 100 also includes an output component 160 that provides information about the inferred possibly related data sets. In some embodiments, the information may include identification of the related data sets. In some embodiments, the information may include metrics about the basis for the inferred relationship based at least in part on the extracted (or calculated) features. For example, the column name(s) are unique, the minimum values ​​are the same, the maximum values ​​are the same, the data masks are the same, etc.

[0027] In some embodiments, system 100 may optionally include input component 170 and adaptation component 180. Input component 170 may receive user feedback (e.g., positive and / or negative) regarding the information provided regarding the inferred possibly related data sets. For example, a user may provide feedback that two data sets identified as possibly related data sets are not related (e.g., negative feedback) and / or that two data sets identified as possibly related data sets are determined to be related (e.g., positive feedback).

[0028] The input component 170 can provide the received user feedback to the adaptation component 180. The adaptation component 180 can modify the weight(s) associated with the feature(s) used by the feature extraction component 120 based on at least some of the information provided about the inferred potentially relevant data set and the received user feedback.

[0029] In some embodiments, the modified weight(s) may be determined using an algorithm that is initially trained using a machine learning process that utilizes various features present in a data set, the algorithm representing associations among the features. In some embodiments, the algorithm is trained using one or more machine learning algorithms, including a linear regression algorithm, a logistic regression algorithm, a decision tree algorithm, a support vector machine (SVM) algorithm, a naive Bayes algorithm, a K-nearest neighbor (KNN) algorithm, a K-means algorithm, a random forest algorithm, a dimensionality reduction algorithm, an artificial neural network (ANN), and / or a gradient boosting & adaptive boosting algorithm. The algorithm may be trained in a supervised, semi-supervised, and / or unsupervised manner.

[0030] Figure 2 and Figure 3 An exemplary method related to inferring data relationships of multiple data sets is shown. Although the method is shown and described as a series of actions performed in sequence, it should be understood and appreciated that the method is not limited by the order of the sequence. For example, some actions may occur in an order different from that described herein. Additionally, an action may occur simultaneously with another action. In addition, in some cases, all actions may not be required to implement the method described herein.

[0031] In addition, the actions described herein may be computer executable instructions, which may be implemented by one or more processors and / or stored on one or more computer readable media. Computer executable instructions may include routines, subroutines, programs, execution threads, etc. In addition, the results of the actions of the method may be stored in a computer readable medium, displayed on a display device, etc.

[0032] refer to Figure 2 , a method of inferring data relationships of a plurality of data sets 200 is illustrated. In some embodiments, the method 200 is performed by the system 100.

[0033] At 210, data content and / or metadata of a plurality of data sets are algorithmically scanned to extract features of each of the data sets. Each feature has an associated weight.

[0034] At 220, the data sets are clustered into clusters based on at least some of the weighted features. At 230, precise similarity measures between the data sets in each cluster are calculated based on the weighted features of the data sets in each cluster. At 240, data sets having precise similarity measures above a threshold number are inferred to be possibly related. At 250, information about the inferred possibly related data sets is provided.

[0035] Go to Figure 3, a method of inferring data relationships of a plurality of data sets 300 is shown. In some embodiments, the method 300 is performed by the system 100.

[0036] At 310, data content and / or metadata content of a plurality of data sets are algorithmically scanned to extract features of each of the data sets. Each feature has an associated weight. At 320, a sim-hash of each data set is calculated based on at least some of the weighted features. At 330, the data sets are clustered into clusters based at least in part on the calculated sim-hashes.

[0037] At 340, based on the weighted features of the data sets in each cluster, the exact similarity measures between the data sets are calculated. At 350, the data sets with the exact similarity measures above a threshold number are inferred to be possibly related. At 360, information about the inferred possibly related data sets is provided.

[0038] Described herein is a system for inferring data relationships among multiple data sets, comprising: a processing system including a processor and a memory having computer executable instructions stored thereon, wherein when the computer executable instructions are executed by the processor, the computer executable instructions cause the processing system to: algorithmically scan at least one of the data content or metadata content of the multiple data sets to extract features of each of the data sets, wherein each feature has an associated weight; cluster the data sets into clusters based on at least some of the weighted features; calculate precise similarity measures between the data sets in each cluster based on the weighted features of the data sets in each cluster; infer data sets having precise similarity measures above a threshold number as possibly related; and provide information about the inferred possibly related data sets.

[0039] The system may also include a memory having additional computer executable instructions stored thereon, which, when executed by the processor, causes the processing system to: scan metadata content of multiple data sets to extract features of the data sets, wherein the features include one or more of the following: a timestamp, a data set name, one or more column names, one or more column types, or a label.

[0040] The system may also include a memory having additional computer executable instructions stored thereon, which, when executed by the processor, cause the processing system to: normalize the extracted features. The system may also include, wherein clustering the data set into clusters based on the weighted features is performed using: a min-hash or a sim-hash of at least some of the weighted features. The system may also include, wherein the features include one or more aspects related to the structure of the data within a particular data set, including one or more of: a name, a data type, or a data format.

[0041] The system may further include, wherein the features include one or more aspects related to a profile of the data within the data set, including one or more of: a data profile, a length histogram, a value histogram, a minimum value, a maximum value, a min-hash, a sim-hash, a data sketch, a pattern match, or a data sample. The system may also include, wherein the weighted features include at least one of a min-hash or a sim-hash calculated on at least a portion of the data content or metadata of the data set. The system may also include, wherein the exact similarity measure is calculated using a Jaccard similarity function or a cosine similarity function of at least some of the weighted features.

[0042] The system may also include, wherein clustering the data set into clusters based on at least some of the weighted features is performed iteratively, and the results of each iteration are combined together to define clusters. The system may also include, wherein Text Frequency - Inverse Document Frequency (TF-IDF) is utilized to determine the associated weights of at least some of the features.

[0043] This article describes a method for inferring data relationships of multiple data sets, including: algorithmically scanning at least one of the data content or metadata content of the multiple data sets to extract features of each data set, wherein each feature has an associated weight; calculating a sim-hash for each data set based on at least some of the weighted features; clustering the data sets into clusters based at least in part on the calculated sim-hashes; calculating an exact similarity measure between the data sets in each cluster based on the weighted features of the data sets in each cluster; inferring data sets with an exact similarity measure above a threshold number as possibly related; and providing information about the inferred possibly related data sets.

[0044] The method may also include receiving user feedback in response to the provided information about the inferred possibly related data sets; and modifying at least some of the weights associated with the features based on the received user feedback. The method may also include, wherein the features include one or more aspects related to the structure of the data within the particular data set, including one or more of: a name, a data type, or a data format.

[0045] The method may further include one or more aspects of a profile of the data within the data set, including one or more of: a data profile, a length histogram, a value histogram, a minimum value, a maximum value, a min-hash, a sim-hash, a data sketch, a pattern match, or a data sample. The method may also include wherein an exact similarity measure is calculated using a Jaccard similarity function or a cosine similarity function of at least some of the weighted features.

[0046] Described herein is a computer storage medium storing computer-readable instructions that, when executed, cause a computing device to: algorithmically scan at least one of data content or metadata content of a plurality of data sets to extract features of each of the data sets, wherein each feature has an associated weight; calculate a sim-hash for each data set based on at least some of the weighted features; cluster the data sets into clusters based at least in part on the calculated sim-hashes; calculate an exact similarity measure between the data sets in each cluster based on the weighted features of the data sets in each cluster; infer data sets having an exact similarity measure above a threshold number as being possibly related; and provide information about the inferred possibly related data sets.

[0047] The computer storage medium may further include, wherein the features include, one or more aspects related to the structure of the data within a particular data set, including one or more of the following: name, data type, or data format. The computer storage medium may further include, wherein the features include, one or more aspects related to the profile of the data within the data set, including one or more of the following: data profile, length histogram, value histogram, minimum value, maximum value, min-hash, sim-hash, data sketch, pattern match, or data sample.

[0048] The computer storage medium may also include wherein an exact similarity measure is calculated using a Jaccard similarity function or a cosine similarity function of at least some of the weighted features. The computer storage medium may also include wherein the information provided about the inferred possibly related data sets includes a metric about a basis for the inferred relationship based at least in part on the extracted features.

[0049] refer to Figure 4 , an example general purpose processing system, computer or computing device 402 (e.g., a mobile phone, desktop computer, laptop computer, tablet computer, watch, server, handheld, programmable consumer or industrial electronic device, set-top box, gaming system, computing node, etc.) is shown. For example, computing device 402 can be used in a system for inferring data relationships of data set 100.

[0050] The computer 402 includes one or more processors 420, a memory 430, a system bus 440, (multiple) mass storage devices 450, and one or more interface components 470. The system bus 440 communicatively couples at least the above-mentioned system components. However, it should be understood that in its simplest form, the computer 402 may include one or more processors 420 coupled to the memory 430, which execute various computer-executable actions, instructions and / or components stored in the memory 430. The instructions may be, for example, instructions for implementing the functions described as being performed by one or more components discussed above, or instructions for implementing one or more of the methods described above.

[0051] Processor(s) 420 may be implemented using a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof to perform the functions described herein. A general purpose processor may be a microprocessor, but alternatively, the processor may be any processor, controller, microcontroller, or state machine. Processor(s) 420 may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, a multi-core processor, a combination of one or more microprocessors with a DSP core, or any other such configuration. In one embodiment, processor(s) 420 may be a graphics processor.

[0052] Computer 402 may include or otherwise interact with various computer-readable media to support control of computer 402 to implement one or more aspects of the claimed subject matter. Computer-readable media can be any available media that can be accessed by computer 402, and includes both volatile and nonvolatile media and removable and non-removable media. Computer-readable media can include two different and mutually exclusive types, namely computer storage media and communication media.

[0053] Computer storage media include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include storage devices such as storage devices (e.g., random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), etc.). Magnetic storage devices (e.g., hard disks, floppy disks, magnetic tapes, tapes, etc.). Optical disks (e.g., compact disks (CDs), digital versatile disks (DVDs), etc.), and solid-state devices (e.g., solid-state drives (SSDs), flash drives (e.g., cards, sticks, key drives), etc.), or any other similar media that store the desired information accessible to the computer 402, rather than for transmission or communication. Therefore, computer storage media do not include modulated data signals and what is described with respect to communication media.

[0054] Communication media embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" refers to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media.

[0055] Memory 430 and mass storage device(s) 450 are examples of computer-readable storage media. Depending on the exact configuration and type of computing device, memory 430 may be volatile (e.g., RAM), non-volatile (e.g., ROM, flash memory, etc.), or some combination of the two. For example, the basic input / output system (BIOS), including the basic routines for transferring information between elements within computer 402, such as during startup, may be stored in non-volatile memory, while volatile memory may be used as external cache memory to support processing by processor(s) 420, etc.

[0056] The mass storage device(s) 450 include removable / non-removable, volatile / non-volatile computer storage media for storing large amounts of data relative to the memory 430. For example, the mass storage device(s) 450 include, but are not limited to, one or more devices such as a magnetic or optical disk drive, a floppy disk drive, a flash memory, a solid state drive, or a memory stick.

[0057] Memory 430 and mass storage device(s) 450 may include or have stored thereon an operating system 460, one or more applications 462, one or more program modules 464, and data 466. Operating system 460 acts to control and allocate resources of computer 402. Application 462 includes one or both of system and application software and may utilize management of resources by operating system 460 through program modules 464 and data 466 stored in memory 430 and / or mass storage device(s) 450 to perform one or more actions. Thus, application 462 may turn general purpose computer 402 into a special purpose machine based on logic provided by general purpose computer 402.

[0058] All or part of the claimed subject matter may be implemented using standard programming and / or engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computer to implement the disclosed functionality. By way of example and not limitation, the system 100 or a portion thereof may be an application 462 or form part of an application 462, and include one or more modules 464 and data 466 stored in memory and / or mass storage device(s) 450, the functionality of which may be implemented when executed by one or more processors 420.

[0059] According to a specific embodiment, the processor(s) 420 may correspond to a system on a chip (SOC) or similar architecture, or in other words, both hardware and software are integrated on a single integrated circuit substrate. Here, the processor(s) 420 may include one or more processors and memory, at least memory similar to the processor(s) 420 and memory 430, etc. Conventional processors include a minimal amount of hardware and software and rely extensively on external hardware and software. In contrast, a SOC implementation of a processor is more powerful because it has hardware and software embedded therein, enabling specific functions to be minimally dependent or non-dependent on external hardware and software. For example, the system 100 and / or associated functions may be embedded in hardware in a SOC architecture.

[0060] The computer 402 also includes one or more interface components 470 that are communicatively coupled to the system bus 440 and support interaction with the computer 402. By way of example, the interface component 470 can be a port (e.g., serial, parallel, PCMCIA, USB, firewall, etc.), or an interface card (e.g., sound, video, etc.), etc. In one example implementation, the interface component 470 can be implemented as a user input / output interface to enable a user to enter commands and information to the computer 402, for example, through one or more input devices (e.g., such as a mouse, trackball, stylus, touchpad, keyboard, microphone, joystick, game pad, satellite dish, scanner, camera, pointing device of other computers, etc.). In another example implementation, the interface component 470 can be implemented as an output peripheral interface to provide output to a display (e.g., LCD, LED, plasma, etc.), a speaker, a printer, and / or other computers, etc. In addition, the interface component 470 can be implemented as a network interface to enable communication with other computing devices (not shown), such as over a wired or wireless communication link.

[0061] What is described above includes examples of aspects of the claimed subject matter. Of course, it is not possible to describe every possible combination of components or methods for the purpose of describing the claimed subject matter, but one of ordinary skill in the art will recognize that many further combinations and permutations of the disclosed subject matter are possible. Therefore, the disclosed subject matter is intended to encompass all such changes, modifications, and variations that fall within the spirit and scope of the appended claims. In addition, to the extent that the term "comprising" is used in the detailed description or claims, the term is intended to be included in a manner similar to how the term "comprising" is interpreted when "comprising" is adopted as a transitional word in a claim.

Claims

1. A system for inferring data relationships of a data set, comprising: processor; as well as a memory having computer executable instructions stored thereon, which, when executed by the processor, cause the system to: Scanning at least one of data content or metadata content of a set of data sets based on an algorithm to extract features in the data sets, wherein the features have associated weights; clustering the set of data sets into a plurality of clusters based on at least some of the weighted features, the plurality of clusters comprising at least a first cluster and a second cluster, the first cluster comprising a first subset of the data sets, the second cluster comprising a second subset of the data sets; calculating an exact similarity measure among a first subset of the data sets in the first cluster based on the weighted features of the first subset of the data sets in the first cluster; inferring related data sets within the first subset of the data sets based on the exact similarity measure being above a threshold amount; as well as Provides information about the data set in question.

2. The system of claim 1 , the memory having further computer executable instructions stored thereon that, when executed by the processor, cause the system to: The metadata content of the set of data sets is scanned to extract the features of the data sets, wherein the features include one or more of the following: a timestamp, a data set name, one or more column names, one or more column types, or a label.

3. The system of claim 1 , the memory having additional computer executable instructions stored thereon that, when executed by the processor, cause the system to: normalize the features.

4. The system of claim 1, wherein clustering the data set into clusters based on the weighted features is performed using: a min-hash or a sim-hash of at least some of the weighted features.

5. The system of claim 1, wherein the features include one or more aspects related to the structure of data within a particular data set, the one or more aspects including one or more of: name, data type, or data format.

6. The system of claim 1, wherein the features comprise one or more aspects related to a profile of the data within the data set, the one or more aspects comprising one or more of: a data profile, a length histogram, a value histogram, a minimum value, a maximum value, a min-hash, a sim-hash, a data sketch, a pattern match, or a data sample.

7. The system of claim 1, wherein the exact similarity measure is calculated using a Jaccard similarity function or a cosine similarity function of at least some of the weighted features.

8. The system of claim 1, wherein clustering the data set into the plurality of clusters is performed iteratively, wherein results of each iteration are combined together to define the plurality of clusters.

9. The system of claim 1, wherein Text Frequency - Inverse Document Frequency (TF-IDF) is utilized to determine the associated weights of at least some of the features.

10. The system of claim 1, wherein the set of data sets is clustered using a machine learning algorithm trained in an unsupervised manner.

11. The system of claim 1, wherein the computer executable instructions cause the system to: avoid computing the exact similarity measure between the first subset of the data set and the second subset of the data set.

12. A method for inferring data relationships of a data set, comprising: Scanning at least one of data content or metadata content of a set of data sets based on an algorithm to extract features in the data sets, wherein the features have associated weights; calculating a sim-hash of the set of data sets based on at least some of the weighted features; Based at least in part on the calculated sim-hash, clustering the set of data sets into a plurality of clusters, the plurality of clusters comprising at least a first cluster and a second cluster, the first cluster comprising a first subset of the data sets, the second cluster comprising a second subset of the data sets; calculating an exact similarity measure among a first subset of the data sets in the first cluster based on the weighted features of the first subset of the data sets in the first cluster; inferring related data sets within the first subset of the data sets based on the exact similarity measure being above a threshold amount; as well as Provides information about the data set in question.

13. The method according to claim 12, further comprising: receiving user feedback in response to said provided information regarding said data set; as well as At least some of the weights associated with features are modified based on the received user feedback.

14. The method of claim 12, wherein the features include: One or more aspects relating to the structure of data within a particular data set, including one or more of name, data type, or data format; or One or more aspects related to a profile of the data within the particular data set, the one or more aspects comprising one or more of: a data profile, a length histogram, a value histogram, a minimum value, a maximum value, a min-hash, a sim-hash, a data sketch, a pattern match, or a data sample.

15. The method of claim 12, wherein clustering the set of data sets comprises: Use machine learning algorithms that are trained in an unsupervised manner.

16. The method of claim 12, wherein computing the exact similarity measure among a first subset of the data set avoids computing the exact similarity measure between the first subset of the data set and a second subset of the data set.

17. A computer storage medium storing computer readable instructions that, when executed, cause a computing device to: Scanning at least one of data content or metadata content of a set of data sets based on an algorithm to extract features in the data sets, wherein the features have associated weights; calculating a sim-hash of the set of data sets based on at least some of the weighted features; Based at least in part on the calculated sim-hash, clustering the set of data sets into a plurality of clusters, the plurality of clusters comprising a first cluster and a second cluster, the first cluster comprising a first subset of the data sets, the second cluster comprising a second subset of the data sets; calculating an exact similarity measure between the first subset of the data sets in the first cluster based on the weighted features of the first subset of the data sets in the first cluster; inferring related data sets within the first subset of the data sets based on the exact similarity measure being above a threshold amount; as well as Provides information about the data set in question.

18. The computer storage medium of claim 17, wherein the information provided about the data set in question comprises: The exact similarity measure or the feature.

19. The computer storage medium of claim 17, wherein the set of data sets is clustered using a machine learning algorithm trained in an unsupervised manner.

20. The computer storage medium of claim 17, wherein computing the exact similarity measure among a first subset of the data sets avoids computing the exact similarity measure between the first subset of the data sets and a second subset of the data sets.

Images