Noise filtering method for recommendation system based on information entropy and fuzzy c-means clustering

Abstract

The invention discloses a recommendation system noise filtering method based on information entropies and fuzzy C-means clustering. The method comprises steps that first, user historical scoring dataof a target recommendation system is collected and arranged; second, Monte Carlo stochastic simulation is utilized to construct sub data sets of the user scoring data, a recommendation algorithm is utilized to acquire information entropies and recommendation precision of different sub data sets; third, the information entropies are classified according to uncertainty levels, recommendation precision is classified according to recommendation precision levels, and an empirical model is constructed to determine the potential natural noise data proportion; fourth, fuzzy clustering analysis on allthe user scoring data sets is carried out, and noise data is identified and deleted; and fifth, a recommendation algorithm operates for all the scoring data sets, and a recommendation precision indexis utilized to evaluate recommendation quality. The method is advantaged in that quantization measurement of the user scoring information can be realized, and the proposed natural noise data filteringtechnology has certain universality and portability.

Description

technical field [0001] The invention belongs to the technical field of artificial intelligence and machine learning, and relates to a noise data elimination method from massive user data, in particular to a noise data filtering method using information entropy and fuzzy C-means clustering. Background technique [0002] As of 2017, the number of monthly active users on Sina Weibo has reached 297 million, Taobao users have exceeded 800 million, stores have exceeded 5 million, and products have exceeded 800 million. According to incomplete statistics, 98% of products have the opportunity to be accepted by users with different hobbies. Relevant technicians systematically counted the sales records of electronic platforms such as Google, Amazon, eBay, Netflix, etc., arranged all products in reverse order of sales volume to form a long tail shape, and found that the total sales of products with low overall sales far exceeded that of major popular products. Total sales. This "long...

AI Technical Summary

Problems solved by technology

[0010] (1) Search engines can help users obtain information resources at the keyword level, but this method is less intelligent and cannot describe users' information needs at the knowledge level, resulting in a very large space for searched information resources

Although the existing information filtering can handle dynamically changing user needs and allow users to modify and adjust the needs, the processing method is relatively simple and cannot identify natural noise data caused by user behavior diversity

[0011] (2) The scores of some natural noise data are very similar to the scores of normal users, so when calculating user similarity, it is easy to enter the user's neighbor circle, and the recommendation scores of normal users for the target item will deviate

If only part of the user data is used as input, the amount of evaluation data for some items may be insufficient, and the recommendation system may not be able to provide high-quality recommendation services for some users.

[0013] The deficiencies in the natural noise filtering technology in the existing recommendation systems mentioned above have brought great difficulties in the design, development, deployment and operation of different e-commerce platforms, especially in the network platform of massive project information, resulting in the service quality of the recommendation system. decline, which in turn affects the sales performance of the e-commerce system

Images