Parallel implementation method and system of user-based collaborative filtering

Abstract

The invention relates to a method and a system for realizing concurrent cooperated filtering based on users. The method comprises the following steps of firstly, storing to-be-processed data into a distributed file system according to the size of a fixed block; performing data standardizing processing, user similarity computing and recommended object computing; in the data standardizing processing, arranging the data into concurrent processing type; in the user similarity computing and recommended object computing, arranging the data, and merging the data of the same file system; sending the processing results to a center node in main key and value pair ways; enabling the center node to calculate to-be-merged nodes of each main key according to the main key; finally, merging the data of the nodes in a crossing-node way, so as to obtain the final recommended result of the user. The method has the advantages that the characteristics of the file blocking storage of the distributed file system are sufficiently utilized, the cost of cyclic traversing is reduced by the concurrent operation, and the requirement of cooperated recommending to massive users is met.

Description

technical field [0001] The present invention relates to the field of computers. Specifically, it relates to a parallel implementation method and system of user-based collaborative filtering. Background technique [0002] Collaborative filtering is a commonly used method for building recommendation systems. Unlike traditional content-based filtering systems that directly analyze content for recommendation, collaborative filtering combines all users' evaluations of a certain information, and searches for information in user groups based on recommendation targets. Users who are similar to it will finally form the system's prediction of the degree of preference of the specified user for this information. Collaborative filtering is currently widely used in commercial applications. Systems such as Amazon, CDNow, and MovieFinder all use this method to improve the service quality of applications. [0003] There are two types of collaborative filtering: item-based collaborative fil...

AI Technical Summary

Problems solved by technology

[0004] As the number of items and the number of users continue to increase, the current collaborative filtering algorithm mainly faces efficiency problems. Since the number of items in existing recommendation systems is relatively fixed, most application websites tend to use item-based collaborative filtering methods to avoid this problem. However, this method can only reflect the relationship between items and does not consider the similarity between users, so this method has strong limitations for the application of specific user groups.

Existing patents also consider using distributed hash table routing algorithm to improve the efficiency of user-based collaborative filtering. This method requires each client to run an agent program, and users only keep their own scoring results for items. The advantage is that customers The terminal only takes the intersection information with its own evaluation item set, thereby reducing the calculation amount of the user itself, but this method only avoids the problem of calculation amount by reducing the calculation objects, and does not fundamentally solve the problem, and this method requires Ensure that all client agents are online and connected to get accurate recommendation results

[0005] From the above analysis, it can be seen that a new idea is needed to fundamentally solve the efficiency problems of collaborative filtering algorithms for massive users

Images