Real-time recommendation system and method based on Spark

Abstract

The invention discloses a real-time recommendation system and method based on Spark. A stacked recommendation system framework based on Spark is constructed, and comprises a data collecting module, an offline recommendation module, an online recommendation module and a recommendation module. The offline recommendation module is used for selecting a corresponding recommendation algorithm from an offline recommendation algorithm library according to user configuration parameters to train user behavior data, and a user feature model is obtained; the online recommendation module is used for sending the user behavior data to corresponding algorithms in an online recommendation algorithm library for training, and an increment user feature model is obtained; an online model training engine is used for adopting the user feature model obtained through training as a basic model, carrying out incremental updating through a current increment recommendation algorithm and the newly received user behavior data, and a newest user feature model is obtained. The recommendation module is used for updating a user recommendation list in combination with an inertia updating mechanism according to the user feature model. The accuracy and timeliness of the recommendation result can be effectively improved.

Description

Technical field [0001] The invention relates to a real-time recommendation system and method, in particular to a real-time recommendation system and method based on Spark. Background technique [0002] The real-time recommendation system refers to the content that can perceive changes in user interest in real time, and adjust the content recommended by the user in real time according to the change in user interest. Traditional recommendation systems generally use Hadoop MapReduce to perform offline training on user behavior data every day through a cluster, and then use the trained user characteristic model to generate a recommendation list for users. This obviously cannot meet the real-time recommendation requirements, and cannot sense changes in user preferences in real time, resulting in a decrease in recommendation accuracy. At present, the existing real-time recommendation system framework mainly includes Netflix's real-time recommendation system framework. [0003] The Netf...

AI Technical Summary

Problems solved by technology

[0004] The problems of Netflix’s real-time recommendation system are: (1) The online computing module mainly interacts with users, and lacks the latest user behavior data to update the user characteristic model, so it cannot perceive changes in user interests in real time. (2) Each module adopts different Type systems process data separately, and the processing results cannot be seamlessly shared between modules; (3) It is difficult to coordinate and allocate resources required by each system in the same cluster

Images