Real-time recommendation system and method based on Spark

A real-time recommendation and recommendation system technology, applied in special data processing applications, instruments, electronic digital data processing, etc., can solve problems such as the inability to perceive changes in user interests in real time, the inability to seamlessly share processing results, and the lack of user feature model updates. , to achieve the effect of improving model update efficiency, reducing system development and maintenance costs, and reducing data sharing overhead

Active Publication Date: 2016-11-16
UNIV OF SCI & TECH OF CHINA
View PDF3 Cites 42 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The problems of Netflix’s real-time recommendation system are: (1) The online computing module mainly interacts with users, and lacks the latest user behavior data to update the user characteristic model, so it cannot perceive changes in user interests

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Real-time recommendation system and method based on Spark
  • Real-time recommendation system and method based on Spark
  • Real-time recommendation system and method based on Spark

Examples

Experimental program
Comparison scheme
Effect test

Embodiment

[0052] Such as figure 1 Shown is a schematic diagram of the framework of a real-time recommendation system based on Spark, which mainly includes a data collection module, an offline recommendation module, an online recommendation module, a model fusion module, and a recommendation module. It uses Spark Software Analysis Stack (BDAS) to build a Spark-based one-stack recommendation system framework. The system divides the recommendation system framework into offline batch processing module, close to online recommendation module and online recommendation module. The offline batch processing module uses the most efficient memory computing framework Spark to replace the traditional batch processing framework Hadoop Map Reduce to train user behavior data , The proximity online recommendation module and the online recommendation module use Spark Streaming to receive user behavior data from the Kafka cluster in real time, and use incremental algorithms such as user-based collaborative f...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a real-time recommendation system and method based on Spark. A stacked recommendation system framework based on Spark is constructed, and comprises a data collecting module, an offline recommendation module, an online recommendation module and a recommendation module. The offline recommendation module is used for selecting a corresponding recommendation algorithm from an offline recommendation algorithm library according to user configuration parameters to train user behavior data, and a user feature model is obtained; the online recommendation module is used for sending the user behavior data to corresponding algorithms in an online recommendation algorithm library for training, and an increment user feature model is obtained; an online model training engine is used for adopting the user feature model obtained through training as a basic model, carrying out incremental updating through a current increment recommendation algorithm and the newly received user behavior data, and a newest user feature model is obtained. The recommendation module is used for updating a user recommendation list in combination with an inertia updating mechanism according to the user feature model. The accuracy and timeliness of the recommendation result can be effectively improved.

Description

Technical field [0001] The invention relates to a real-time recommendation system and method, in particular to a real-time recommendation system and method based on Spark. Background technique [0002] The real-time recommendation system refers to the content that can perceive changes in user interest in real time, and adjust the content recommended by the user in real time according to the change in user interest. Traditional recommendation systems generally use Hadoop MapReduce to perform offline training on user behavior data every day through a cluster, and then use the trained user characteristic model to generate a recommendation list for users. This obviously cannot meet the real-time recommendation requirements, and cannot sense changes in user preferences in real time, resulting in a decrease in recommendation accuracy. At present, the existing real-time recommendation system framework mainly includes Netflix's real-time recommendation system framework. [0003] The Netf...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
CPCG06F16/9535
Inventor 陈航周学海庄航
Owner UNIV OF SCI & TECH OF CHINA
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap