System for realizing personalized recommendation based on collaborative filtering algorithm

Abstract

The invention discloses a personalized recommendation system based on a collaborative filtering algorithm. The personalized recommendation system comprises a data acquisition layer, an offline recommendation layer and an online recommendation layer, the data acquisition layer adopts a Flume log collection multi-layer architecture; the offline recommendation layer is used for carrying out SparkALS recall, Spark is used for carrying out collaborative filtering recall based on articles, and sorting is carried out based on GBDT + LR; and the online recommendation layer writes feature variables to the HBase, analyzes user behaviors in real time by using streaming computation, and updates the model weight through the FTRL. The problem of accurate sending of a large amount of information in a large-scale user scene is solved, and the effect of a recommendation result is continuously evaluated by collecting user behavior data offline and in real time, establishing a user behavior model and performing personalized recommendation.

Description

Technical Field

[0001] This invention relates to the field of personalized recommendation, and in particular to a personalized recommendation system based on a collaborative filtering algorithm. Background Technology

[0002] The recommendation system consists of two parts: an online module and an offline module. The online module addresses the real-time nature of recommendations, while the offline module is used for data mining and model building. Collected user browsing and click behavior data is synchronously stored in a database. A computing engine analyzes and mines user preference characteristics to derive user models. When a user visits a product, the system checks the database to see if a model of that user exists. If so, products are recommended, placing the most probable purchase items at the top. If no such model exists, popular products are recommended.

[0003] The offline component collects logs through a multi-level Flume proxy. From Flume, the log collection tool, to the Kafka Channel message queue, the collected business logs are stored in the HDFS distributed file system. After data cleaning, formatting, and standardization, they are stored in corresponding Hive tables, following a three-layer data warehouse structure. One part of the data acquisition layer collects logs, passes them to the message queue, and then performs stream processing, ultimately storing the data in HBBase to implement application services. Another part collects logs and uses the Kafka message queue to store data in the HDFS distributed file system. The data storage layer involves the HDFS distributed file system, a Hive user behavior log data warehouse, and HBase to store recommendation models and historical recommendation results. The offline recommendation layer uses SparkALS for recall, Spark-based item-based collaborative filtering for recall, and GBDT+LR-based ranking. The online recommendation layer writes feature variables to HBase, then uses stream processing to parse user behavior in real time and updates model weights via FTRL. Finally, Spark is used to perform offline data mining, create models, and store them in a real-time database, ultimately achieving personalized recommendations. In real time, the business log data collected by Flume is fed into Storm via Kafka for real-time computation, then retrieved and sorted, and finally used to achieve real-time personalized recommendations.

[0004] Disadvantages of existing technology:

[0005] Common problems with recommender systems are mostly concentrated in data processing and algorithm optimization. Recommender systems are about mining potential users, and common problems are in mining human nature, such as cold start, data sparsity, constantly changing user preferences, and unpredictable events.

[0006] To address the cold start problem, users are categorized based on their registration information, such as phone number, email address, and region. In addition, a popular recommendation leaderboard can be implemented, which can be based on a deep learning semantic understanding model. Reducing matrix dimensionality can mitigate data sparsity, and data imputation can address the sparsity problem. For users with different preferences, user behavior storage needs to be real-time, considering both recent and long-term user behavior, and continuously mining user interests and hobbies. Summary of the Invention

[0007] In view of the above problems, the present invention is proposed to provide a personalized recommendation system based on collaborative filtering algorithm that overcomes or at least partially solves the above problems.

[0008] According to one aspect of the present invention, a personalized recommendation system based on a collaborative filtering algorithm is provided, the personalized recommendation system comprising: a data acquisition layer, an offline recommendation layer, and an online recommendation layer;

[0009] The Flume log collection layer features a multi-layer architecture.

[0010] The offline recommendation layer includes Spark ALS recall, Spark item-based collaborative filtering recall, and GBDT+LR-based sorting.

[0011] The online recommendation layer writes feature variables to HBase, then uses streaming computation to parse user behavior in real time and updates model weights via FTRL.

[0012] Optionally, the data acquisition layer collects user data and item data; the user data includes: user information, user behavior, and user relationship data.

[0013] Optionally, the Flume log collection multi-layer architecture of the data acquisition layer specifically includes:

[0014] From log collection to message queues and streaming computing, data is fed into HBbase to enable application services.

[0015] Data is collected from logs and then written to the HDFS distributed file system via the message queue Kafka.

[0016] The data storage layer consists of the HDFS distributed file system, the Hive user behavior log data warehouse, and HBase for storing recommendation models and historical recommendation results.

[0017] Optionally, the user behavior data is divided into two categories: explicit user behavior and implicit user behavior.

[0018] Explicit user behaviors include purchasing, saving, browsing, and following.

[0019] Implicit user behaviors include dwell time, redirection, and clicks.

[0020] Optionally, the user relationship data includes explicit user relationships and implicit user relationships;

[0021] The explicit user relationships are classmates, relatives, and colleagues;

[0022] The implicit user relationships are following / being followed, adding as a friend, forwarding, liking, and interacting.

[0023] Optionally, the offline recommendation layer collects logs through a multi-level broker of Flume, from the log collection tool Flume to the message queue Kafka Channel;

[0024] The collected business logs will be stored in the distributed file system HDFS, and after data cleaning, formatting and standardization, they will be stored in the corresponding Hive tables;

[0025] The data is stored according to a three-tier data warehouse structure. Spark is used to perform offline data mining, create models, and store the data in a real-time database to achieve personalized recommendations.

[0026] Optionally, after implementing personalized recommendations, the method further includes: real-time business log data collected by Flume is fed into Storm via Kafka for real-time calculation, then recalled and sorted, and finally real-time personalized recommendations are implemented.

[0027] This invention provides a personalized recommendation system based on a collaborative filtering algorithm. The personalized recommendation system includes: a data acquisition layer, an offline recommendation layer, and an online recommendation layer. The data acquisition layer uses a multi-layered architecture for Flume log collection. The offline recommendation layer employs SparkALS recall, Spark item-based collaborative filtering recall, and GBDT+LR-based ranking. The online recommendation layer writes feature variables to HBase, then uses streaming computation to parse user behavior in real time and updates model weights via FTRL. This addresses the problem of accurately sending large amounts of information in large-scale user scenarios by collecting user behavior data offline and in real time, building a user behavior model, performing personalized recommendations, and continuously evaluating the effectiveness of the recommendation results.

[0028] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a composition diagram of a personalized recommendation system based on a collaborative filtering algorithm, provided as an embodiment of the present invention. Detailed Implementation

[0031] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0032] The terms "comprising" and "having," and any variations thereof, in the specification, embodiments, claims, and drawings of this invention are intended to cover non-exclusive inclusion, such as including a series of steps or units.

[0033] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0034] like Figure 1 As shown, a recommendation system is a complex system engineering project consisting of four parts: algorithm, UI, data, and domain knowledge. The first step is data collection, which is divided into user data and item data. User data includes user information, user behavior, and user relationship data. User behavior data is further divided into explicit and implicit user behaviors. Explicit user behaviors include purchasing, saving, browsing, and following. Implicit user behaviors include dwell time, jumping, and clicking. User relationship data is also divided into explicit and implicit relationships. Explicit user relationships include classmates, relatives, and colleagues. Implicit user relationships include following / being followed, adding as friends, forwarding, liking, and interacting. Data is the cornerstone of the entire recommendation system. Data extracted from various data sources must be cleaned and preprocessed to form a unified format before being placed in a data warehouse. User behavior can realistically reflect each user's preferences and habits. Explicit user data is relatively sparse, while implicit feedback data contains a wealth of information. The algorithm is a collaborative filtering-based recommendation algorithm, divided into user-based recommendation, item-based recommendation, and model-based recommendation. Model-based recommendation is further divided into SVD / SVD++ models, probability-based matrix factorization, and implicit semantic models.

[0035] The Flume log collection layer employs a multi-layered architecture. One part involves collecting logs, processing them via message queues and streaming computation, and ultimately storing the data in HBBase to provide application services. Another part uses the Kafka message queue to store data in the HDFS distributed file system. The data storage layer includes the HDFS distributed file system, a Hive user behavior log data warehouse, and HBase for storing recommendation models and historical recommendation results.

[0036] The offline recommendation layer utilizes Spark ALS for recall, Spark item-based collaborative filtering for recall, and GBDT+LR for ranking. The online recommendation layer writes feature variables to HBase, then uses streaming computation to analyze user behavior in real time and updates model weights via FTRL. Offline and online recommendation are the core of the entire system methodology.

[0037] The offline component collects logs through a multi-level Flume broker. From the Flume log collection tool to the Kafka Channel message queue, the collected business logs are stored in the HDFS distributed file system. After data cleaning, formatting, and standardization, they are stored in corresponding Hive tables according to a three-layer data warehouse structure. Spark is then used for offline data mining, model creation, and storage in a real-time database, ultimately enabling personalized recommendations. The real-time component, on the other hand, uses Flume to collect business log data, which is then fed into Storm for real-time computation, retrieval, and sorting, ultimately achieving real-time personalized recommendations.

[0038] Beneficial effects: Recommender systems address the problem of accurately delivering massive amounts of information to large-scale user scenarios. They collect user behavior data offline and in real-time, build user behavior models, provide personalized recommendations, and continuously evaluate the effectiveness of the recommendations. A complete recommender system architecture is formed based on algorithms, translating from real-world scenarios into model applications. It processes large amounts of data, analyzes user preference characteristics in real-time and accurately, and pushes potentially interesting products to users, improving the accuracy of product recommendations. It collects user data, extracts necessary data from massive datasets, builds models to describe user preferences, and ensures real-time performance and accuracy.

[0039] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A personalized recommendation system based on collaborative filtering algorithm, characterized in that, The personalized recommendation system includes: a data collection layer, an offline recommendation layer, and an online recommendation layer; The Flume log collection layer features a multi-layer architecture. The offline recommendation layer includes Spark ALS recall, Spark item-based collaborative filtering recall, and GBDT+LR-based sorting. The online recommendation layer writes feature variables to HBase, then uses streaming computation to parse user behavior in real time and updates model weights via FTRL.

2. The personalized recommendation system based on collaborative filtering algorithm according to claim 1, characterized in that, The data acquisition layer collects user data and item data; The user data includes: user information, user behavior, and user relationship data.

3. A personalized recommendation system based on collaborative filtering algorithm according to claim 1, characterized in that, The Flume log collection multi-layer architecture specifically includes: From log collection to message queues and streaming computing, data is fed into HBbase to enable application services. Data is collected from logs and then written to the HDFS distributed file system via the message queue Kafka. The data storage layer consists of the HDFS distributed file system, the Hive user behavior log data warehouse, and HBase for storing recommendation models and historical recommendation results.

4. A personalized recommendation system based on collaborative filtering algorithm according to claim 2, characterized in that, The user behavior data is divided into two categories: explicit user behavior and implicit user behavior. Explicit user behaviors include purchasing, saving, browsing, and following. Implicit user behaviors include dwell time, redirection, and clicks.

5. A personalized recommendation system based on collaborative filtering algorithm according to claim 2, characterized in that, The user relationship data includes explicit user relationships and implicit user relationships; The explicit user relationships are classmates, relatives, and colleagues; The implicit user relationships are: following / being followed, adding as a friend, forwarding, liking, and interacting.

6. A personalized recommendation system based on collaborative filtering algorithm according to claim 1, characterized in that, The offline recommendation layer collects logs through Flume's multi-level proxy, from the log collection tool Flume to the message queue KafkaChannel; The collected business logs will be stored in the distributed file system HDFS, and after data cleaning, formatting and standardization, they will be stored in the corresponding Hive tables; The data is stored according to a three-tier data warehouse structure. Spark is used to perform offline data mining, create models, and store the data in a real-time database to achieve personalized recommendations.

7. A personalized recommendation system based on collaborative filtering algorithm according to claim 6, characterized in that, The process of implementing personalized recommendations also includes: real-time business log data collected by Flume is fed into Storm via Kafka for real-time computation, then recalled and sorted, and finally real-time personalized recommendations are achieved.

Images