Document similarity scoring and ranking method, device and computer program product

Abstract

A device, computer program product and a method for searching, navigating or retrieving documents in a set of electronic documents, including performing a link analysis of the set of electronic documents. The link analysis includes one of analyzing at least two of the set of documents with at least a portion of a similarity graph constructed among the set of documents and analyzing the at least two of the set of documents with the at least a portion of the similarity graph and at least a portion of a hyperlink graph constructed from hyperlinks between the set of documents. Also described is a method for building a similarity matrix.

Description

CROSS REFERENCE TO RELATED APPLICATIONS [0001] The present invention contains subject matter related to co-pending application Ser. No. 10 / 687,602 filed on Oct. 20, 2003; co-pending application Ser. No. 10 / 918,713 filed on Aug. 16, 2004; and co-pending application Ser. No. 11 / 227,495 filed on Sep. 16, 2005, the contents of each being incorporated herein by reference.BACKGROUND OF THE INVENTION [0002] 1. Field of Use [0003] A computer implemented method and device for intelligent on-line document searching, similarity scoring and retrieval. In particular, an efficient method of calculating similarity scores between electronic documents. Also, a plurality of embodiments for combining a similarity graph and a hyperlink structure graph for ranking hit lists from searches over a set of electronic documents. [0004] 2. Description of the Background [0005] The following discusses methods both for computing similarity scores of electronic documents and for performing link-based analysis of e...

AI Technical Summary

Benefits of technology

[0030] The present invention includes a device, computer program product and a method for computing the similarity of a set of documents that is much more efficient than the straightforward, exhaustive method described above. The present invention avoids the large, wasted computational effort involved in calculating very small similarity scores. The present invention uses thresholds to stop the similarity calculation between documents, thus ensuring that, with high probability, all document pairs with higher similarity than the thresholds have been found.

Problems solved by technology

However, finding similarity scores for all document pairs (i.e., calculating the entire similarity matrix) of a large document set is not part of the state of the art, since conventional (brute force) methods are not able to cope with the computational burden engendered by a large document set.

However, normalization will keep documents with many words from overwhelming documents with fewer words.

This step is non-conventional, since to our knowledge no method has been presented for performing this determination efficiently.

However, this goal is unattainable for large document sets, unless some good method for streamlining the calculation is found.

That is, for large document sets, both the calculation time and the storage requirement grow as the square m2 of the number of documents m. Hence, when the number of documents in the collection reaches millions or even billions, it is not feasible to calculate the similarity matrix using conventional methods.

Thus, using known methods, it is practically impossible to use the information contained in the full similarity matrix, unless the document set is sufficiently small.

As noted above, known methods for calculating similarity scores among a set of documents are very computationally intensive.

Due to the fact that many of these similarity scores will also be zero (or very small), a lot of computational time is wasted on just calculating very small numbers (including zeros).

This is clearly highly undesirable.

Good text relevance analysis is difficult to achieve using a machine—it involves asking a machine to estimate the relevance and / or quality of a given document, with respect to a given set of keywords.

However, a major shortcoming of the conventional similarity analysis technology (apart from the present inventors' co-pending application Ser. No. 11 / 227,495) is the lack of any application of link analysis to the similarity matrix.

Images