Linking Data Elements Based on Similarity Data Values and Semantic Annotations

Abstract

Data elements from data sources and having a data value set are linked by using hash functions to determine a dimensionally reduced instance signature for each data element based on all data values associated with that data element to yield a plurality of dimensionally reduced instance signatures of equivalent fixed size such that similarities among the data values in the data value sets across all data elements is maintained among the plurality of instance signatures. Candidate pairs of data elements to link are identified using the plurality of instance signatures in locality sensitive hash functions, and a similarity index is generated for each candidate pair using a pre-determined measure of similarity. Candidate pairs of data elements having a similarity index above a given threshold are linked.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS[0001]The present application is a continuation of co-pending U.S. patent application Ser. No. 13 / 491,724 filed Jun. 8, 2012. The entire disclosure of that application is incorporated herein by reference.FIELD OF THE INVENTION[0002]The present invention relates to data management and data linkingBACKGROUND OF THE INVENTION[0003]The semantic web is an extension of the world wide web that incorporates semantics into the data or web pages that are accessed and downloaded across the internet. Discovering links between data elements of different data sources is a fundamental problem for the emerging semantic web, as well as traditional data integration systems. These links are the building blocks for searching, querying and other higher level services. Link discovery techniques span syntactic methods, many derived from similarity measures developed by Information Retrieval (IR). These techniques include structural ones, e.g., using foreign key relat...

AI Technical Summary

Benefits of technology

This patent text describes a method for identifying similar data elements in a database using a technique called Locality Sensitive Hashing (LSH). This approach reduces the cost of comparing pairs of data by assigning a higher likelihood to similar data elements. Only a small portion of the data needs to be compared, resulting in a more efficient search. The method also uses semantic annotations to construct signatures that better represent the content of each data element. By using multiple signatures, the accuracy of the similarity predictions is increased. Additionally, this method allows for the use of both data value instances and semantic annotations to create more complex signatures.

Problems solved by technology

Discovering links between data elements of different data sources is a fundamental problem for the emerging semantic web, as well as traditional data integration systems.

Therefore, users need a reasonably deep understanding of the data sources and their elements, limiting both which data sources will be linked (only those with which a user is familiar) and the number of data sources that will be linked, since these specifications take time to create.

Furthermore, the ability of existing linking systems to scale to a large number of medium to large size data sources is limited due to both the quadratic number of comparisons that need to be performed to exhaustively check for links between pairs of elements of different data sources.

The result is that some standard analytic techniques that assume complete access to the whole data are not applicable.

Very few existing methods even attempt to use instance values for matching data elements, and such instance-based methods have been described as useful but prohibitively expensive.

Examples of methods that use instance value include methods that use values only for validation, methods that look at the distribution and other properties of the values to infer meta-data about the elements, methods that use only a selected sample of instance values, methods that are expensive and do not avoid the quadratic number of comparisons and methods that rely on particular properties of instance values that work only in a limited set of applications.

Images