Linking Data Elements Based on Similarity Data Values and Semantic Annotations

Abstract

Data elements from data sources and having a data value set are linked by using hash functions to determine a dimensionally reduced instance signature for each data element based on all data values associated with that data element to yield a plurality of dimensionally reduced instance signatures of equivalent fixed size such that similarities among the data values in the data value sets across all data elements is maintained among the plurality of instance signatures. Candidate pairs of data elements to link are identified using the plurality of instance signatures in locality sensitive hash functions, and a similarity index is generated for each candidate pair using a pre-determined measure of similarity. Candidate pairs of data elements having a similarity index above a given threshold are linked.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS[0001]The present application is a continuation of co-pending U.S. patent application Ser. No. 13 / 491,724 filed Jun. 8, 2012. The entire disclosure of that application is incorporated herein by reference.FIELD OF THE INVENTION[0002]The present invention relates to data management and data linkingBACKGROUND OF THE INVENTION[0003]The semantic web is an extension of the world wide web that incorporates semantics into the data or web pages that are accessed and downloaded across the internet. Discovering links between data elements of different data sources is a fundamental problem for the emerging semantic web, as well as traditional data integration systems. These links are the building blocks for searching, querying and other higher level services. Link discovery techniques span syntactic methods, many derived from similarity measures developed by Information Retrieval (IR). These techniques include structural ones, e.g., using foreign key relat...

AI Technical Summary

Benefits of technology

[0006]Exemplary embodiments of systems and methods in accordance with the present invention are directed to an automated method for choosing data elements from data sources to link based on similarity of semantic annotations, similarity between data value instances or a combination of both semantic annotations and data instances. Well known measures of similarity from Information Retrieval and data mining are used, e.g., Jaccard similarity and cosine similarity. Any type of annotations about semantic features of data elements can be used; however, the present invention it is not dependent on the method used to create the annotations. Unlike existing techniques that require user or human interaction to choose candidate data elements for linking, the present invention is completely automatic, because all pairs of data elements across and within all data sources are considered for linking Systems and methods in accordance with the present invention are computationally feasible in both space and time through the significant reduction in search space achieved through the use of signatures. For each data element, a summary of the set of data values or annotations for that element is computed. This summary is called a signature. These signatures are constructed in a manner that preserves similarity, i.e., the similarity of two signatures is very close to the similarity of their underlying sets. In addition, these signatures have a fixed size regardless of the size of the sets of annotations and data values.

Problems solved by technology

Discovering links between data elements of different data sources is a fundamental problem for the emerging semantic web, as well as traditional data integration systems.

Therefore, users need a reasonably deep understanding of the data sources and their elements, limiting both which data sources will be linked (only those with which a user is familiar) and the number of data sources that will be linked, since these specifications take time to create.

Furthermore, the ability of existing linking systems to scale to a large number of medium to large size data sources is limited due to both the quadratic number of comparisons that need to be performed to exhaustively check for links between pairs of elements of different data sources.

The result is that some standard analytic techniques that assume complete access to the whole data are not applicable.

Very few existing methods even attempt to use instance values for matching data elements, and such instance-based methods have been described as useful but prohibitively expensive.

Examples of methods that use instance value include methods that use values only for validation, methods that look at the distribution and other properties of the values to infer meta-data about the elements, methods that use only a selected sample of instance values, methods that are expensive and do not avoid the quadratic number of comparisons and methods that rely on particular properties of instance values that work only in a limited set of applications.

Images