Method of evaluating learning rate of recommender systems

Abstract

A recommender system is analyzed to determine various performance characteristics, such as a learning rate for new items, or a learning rate for new subscriber tastes. Comparisons of different recommenders are presented to assist consumers and marketers in selecting appropriate e-commerce sites for purchasing, advertising, etc.

Description

RELATED APPLICATION DATA[0001]The present application claims the benefit under 35 U.S.C. 119(e) of the priority date of Provisional Application Ser. No. 60 / 473,994 filed May 28, 2003, which is hereby incorporated by reference.FIELD OF THE INVENTION[0002]The present invention relates to testing, evaluating and measuring learning rates and other performances of electronic recommendation systems and other related systems employed by online content service providers.BACKGROUND[0003]Recommender systems are well known in the art. In one example, such systems can make recommendations for movie titles to a subscriber. In other instances they can provide suggestions for book purchases, or even television program viewing. Such algorithms are commonplace in a number of Internet commerce environments, including at Amazon, CDNOW, and Netflix to name a few, as well as programming guide systems such as TiVO. While the details of such algorithms are often proprietary, the latter typically use a num...

AI Technical Summary

Benefits of technology

"The present invention provides a method for testing, rating, and reporting on the performance of a recommender system used in an online content service provider. The method involves setting up a policy for the recommender system, interacting with a plurality of proxy accounts to generate recommendations for items, and analyzing the recommendations to determine if they meet the policy. The method can also be used to test the shipping and returns management system of the content service provider, and to measure the behavior of the recommender system. The invention provides a valuable tool for evaluating the performance of the recommender system and optimizing its performance."

Problems solved by technology

CF algorithms nonetheless may not be entirely “neutral, and may include subtle unintended (or even intended) bias in their recommendations.

In some cases they may not recommend items that are “new” because CF systems tend to lag in their learning capabilities.

This problem is treated as one of “noise” which can affect the reliability and reputation of recommender systems.

Notably, however, Kushmerick fails to consider the possibility of an internal “bias” which is intentionally introduced by the recommender system operator, or how to detect / measure the same.

Since such bias is introduced by the operator, it is extremely challenging to detect from the outside.

One important parameter, for example, may be the issue of how quickly a recommender system for a particular vendor is able to assimilate and give recommendations on new items.

The lack of data for new items is a known limitation of recommender systems, and yet the prior art does not describe any mechanism for comparing the performance of recommender systems in this respect.

In addition, the prior art does not consider how to determine whether a recommender system is complying with a particular preference policy which might be specified for recommendations.

Finally, the prior art does not indicate how the effects of advertising can be correlated with recommender system behavior, or even how recommender system recommendations can be mined and exploited to improve online advertising campaigns.

Images