Automated satisfaction measurement for web search

Abstract

Context-based user behavior data is collected from a search mechanism. This data includes, for a given query, user feedback (implicit and explicit) on the query and context information on the query. A predictive pattern is applied to the context-based user behavior data in order to produce predicted user satisfaction data. Data mining techniques may be used to create and improve one or more predictive patterns. Predicted user satisfaction data can be used to monitor or improve search mechanism performance, via a display reporting the performance or identification of any queries with a shared characteristic and sub-par user satisfaction. A dynamically-improving search mechanism uses the predicted user satisfaction data to improve the performance of the search mechanism.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS [0001] This application is a continuation-in-part of U.S. patent application Ser. No. 10 / 727,444, filed Dec. 3, 2003, entitled “Search System Using User Behavior Data”.FIELD OF THE INVENTION [0002] This invention relates in general to the field of data search and retrieval. More particularly, this invention relates to the collection and use of user data for search result evaluation. BACKGROUND OF THE INVENTION [0003] Data on one or more computer systems may contain data useful for a user. However, the data may be too large for the user to find the data by direct examination. Additionally, some parts of the data repository may contain information that is not accessible to the user. In many cases, in order to allow the user useful access to the data, a search mechanism is provided. The search mechanism allows a user to issue a search request (also termed a search query). The results are then returned for the user. [0004] For example, a web-based...

AI Technical Summary

Benefits of technology

[0020] The predicted user satisfaction data may also be used to isolate groups of queries which should be examined for a possible problem with the search mechanism. For example, a group of queries with one or more related characteristics may be examined to see if the predicted user satisfaction is lower for the group than for the rest of the queries being performed. If so, an analysis may allow the search engine to b

Problems solved by technology

However, the data may be too large for the user to find the data by direct examination.

Additionally, some parts of the data repository may contain information that is not accessible to the user.

Generally, search engines in the prior art use non-scalable methods for evaluating the quality of search results.

However, this presents at least three major problems.

First, as noted, this method is non-scalable with respect to the number of judgments provided for individual queries.

While 300 results may be judged by a reviewer, it is hard to generalize the satisfactoriness of 300 judged results to over 3,000,000 results.

Second, the method is non-scalable with respect to the number of unique queries that can be judged.

A search engine may perform in an unsatisfactory way on searches of a specific type or with a given characteristic.

If only a small subset of the all searches performed are judged, such a problem may be difficult to diagnose.

Thus, where only a small number of queries judged, a sufficient accumulation of such unsatisfactory queries may never be gathered.

A last problem is that the opinion of judges on user satisfaction may not be equivalent to the opinion of actual users on their satisfaction.

Thus, substituting the opinion of judges for the opinion of actual users may not result in a correct assessment of satisfaction.

In addition, explicit feedback techniques require that users engage in activities beyond their intended search

Images