Automated satisfaction measurement for web search

Abstract

Context-based user behavior data is collected from a search mechanism. This data includes, for a given query, user feedback (implicit and explicit) on the query and context information on the query. A predictive pattern is applied to the context-based user behavior data in order to produce predicted user satisfaction data. Data mining techniques may be used to create and improve one or more predictive patterns. Predicted user satisfaction data can be used to monitor or improve search mechanism performance, via a display reporting the performance or identification of any queries with a shared characteristic and sub-par user satisfaction. A dynamically-improving search mechanism uses the predicted user satisfaction data to improve the performance of the search mechanism.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS [0001] This application is a continuation-in-part of U.S. patent application Ser. No. 10 / 727,444, filed Dec. 3, 2003, entitled “Search System Using User Behavior Data”.FIELD OF THE INVENTION [0002] This invention relates in general to the field of data search and retrieval. More particularly, this invention relates to the collection and use of user data for search result evaluation. BACKGROUND OF THE INVENTION [0003] Data on one or more computer systems may contain data useful for a user. However, the data may be too large for the user to find the data by direct examination. Additionally, some parts of the data repository may contain information that is not accessible to the user. In many cases, in order to allow the user useful access to the data, a search mechanism is provided. The search mechanism allows a user to issue a search request (also termed a search query). The results are then returned for the user. [0004] For example, a web-based...

AI Technical Summary

Benefits of technology

[0021] In addition, where repeating queries or similar queries are presented multiple times with different contexts, such as a different search result order being presented to the user, it can be determined that one such different search result order is preferable to another. Thus, the search engine can use the predicted user satisfaction data to dynamically improve the quality of search results over time.

Problems solved by technology

However, the data may be too large for the user to find the data by direct examination.

Additionally, some parts of the data repository may contain information that is not accessible to the user.

Generally, search engines in the prior art use non-scalable methods for evaluating the quality of search results.

However, this presents at least three major problems.

First, as noted, this method is non-scalable with respect to the number of judgments provided for individual queries.

While 300 results may be judged by a reviewer, it is hard to generalize the satisfactoriness of 300 judged results to over 3,000,000 results.

Second, the method is non-scalable with respect to the number of unique queries that can be judged.

A search engine may perform in an unsatisfactory way on searches of a specific type or with a given characteristic.

If only a small subset of the all searches performed are judged, such a problem may be difficult to diagnose.

Thus, where only a small number of queries judged, a sufficient accumulation of such unsatisfactory queries may never be gathered.

A last problem is that the opinion of judges on user satisfaction may not be equivalent to the opinion of actual users on their satisfaction.

Thus, substituting the opinion of judges for the opinion of actual users may not result in a correct assessment of satisfaction.

In addition, explicit feedback techniques require that users engage in activities beyond their intended searching behavior, and this may influence the search outcome.

Finally, since the costs to the user are high, and the benefits not immediately obvious, it can be difficult to collect data in a reliable fashion from a large, representative sample of users.

While this technique does gather user feedback, it has limited utility in situations in which users may have different needs for a page.

However, another user who is looking for information on books about traveling cheaply may evaluate the same page and give it a low score.

Thus the technique described will have limited utility in the wide variety of situations in which different users may have different needs, or even where a single user may have different needs for information at different times. In other words, the usefulness of this technique is limited because evaluation of each page is completely independent of the context in which the user arrived at the page.

Thus, this technique is not useful for evaluating the quality of a search engine.

In general, this technique is not useful for evaluations which are context-based, but only for evaluating the quality of individual data items, independent of the context in which a user arrived at the data items.

However, while such data has been gathered, the raw data does not contain explicit user satisfaction data which can replace the judged user satisfaction data from a reviewer, which judged data suffers from the drawbacks described above.

Images