Method for performing efficient similarity search

Abstract

The present invention provides systems and methods for performing efficient k-NN approximate similarity search on a database of objects. The invention is based on the definition of an index data structure that enables to have fast searches and very good scalability with respect to the database size. Such index makes efficient use of both the main and secondary memory of the computer, taking advantage of the specific properties of both kinds of memories.A prefix tree is built on all the sequences assigned to the database objects by a sequence generation function. The prefix tree is stored in the main memory.The information required to identify each database object and to compute the similarity between database objects and query objects are stored in a data storage kept in the secondary memory.Given a query object and the request for the k nearest neighbors, the search functionality of the invention uses the prefix tree to quickly identify a set of candidate objects. The organization of the data storage is then used to efficiently retrieve the information relative to the candidate objects. Such information is used to compute the similarity of candidate object with the query, in order to select the k most similar ones, which are thus returned as the result.

Description

1 PROVISIONAL LINKRelated U.S. Application Data[0001]Provisional application No. 61 / 108,943, filed 28 Oct. 2008, by the same inventors of the present application.2 FIELD OF THE INVENTION[0002]This invention relates generally to methods for performing similarity searches in a collection of objects. In particular the invention performs approximate k nearest neighbors analysis using a particular data index structure that permits to execute efficient and fast searches.3 BACKGROUND[0003]In a lot of modern applications is required to find, in a database, some objects similar to a given one, on the base of a degree of similarity. This problem can be solved with many advantages with similarity search methods. In these methods, to determine if an object is similar to another, a distance function is used: the smaller is the distance between two objects, the higher is their relative similarity.[0004]More formally the problem can be expressed in the following way:[0005]a database D contains obj...

AI Technical Summary

Benefits of technology

[0032]The main contribution of the invention is the definition of an index data structure that enables to have fast searches and very good scalability with respect to the database size. Such index makes efficient use of both the main and secondary memory of the computer, taking advantage of the specific properties of both kinds of memories. The main memory is a relatively small but very fast random-access memory that allows fast access and navigation through complex data structures. The secondary memory is a permanent storage that allows to store large amounts of data. It is orders of magnitude slower than the main memory but it still guarantees good I / O performance for sequential accesses.

Problems solved by technology

This simple data organization results in a limited scalability to large collection of objects, due to the large amount of main memory required to store the sequences, and a limited efficiency, due to the non-optimized pattern of accesses to disk in order to retrieve the objects to be compared with the query.

In [1], the authors propose two optimizations that improve the efficiency of the search process, marginally affecting the accuracy of the produced ranking.

Images