A method for detecting spam

Abstract

The invention discloses a garbage webpage detection method. The method comprises the steps of 1: performing a K-Means algorithm on a data set and storing all objects n by using a data set D; 2: performing IPR calculation on the data set D, and sorting IPR values according to a sequence from high to low; 3: selecting webpages with the maximum and minimum IPR values in the data set as initial clustering centers C; 4: calculating a distance dist(xi,cj) between xi and cj in the data set D, and dividing dist(xi,cj) in a cluster where the center closest to dist(xi,cj) is located; 5: viewing a clustering center when aggregation is finished, and obtaining a new cj expression; and 6: repeating the steps 4-6, representing a target function with SSE, stopping the algorithm until a minimum value of SSE is achieved, and obtaining a final clustering result, thereby identifying garbage webpages. According to the method, the shortcoming of ignoring of webpage importance during link weight allocation in a conventional recommendation technology can be overcome; and in combination with personalized webpage sorting, the purpose of detecting the garbage webpages in an aggregation form is achieved.

Description

technical field [0001] The invention relates to the fields of data mining, natural language processing and information retrieval, relates to garbage webpage detection technology and webpage clustering technology, in particular to a garbage webpage detection method based on webpage authority. Background technique [0002] Currently, in related technologies, recommendation technologies are mainly divided into two types: the first type of recommendation technology is link-based recommendation, such as the PageRank algorithm. Its advantage is that the authority is expressed in numerical form, and then arranged in order from high to low. It is precisely because of the digitalization of webpage quality that it is widely used to find spam webpages, forming a very good criterion for judging the authority of webpages. [0003] The defects of the PageRank algorithm are mainly manifested in two aspects. On the one hand, it ignores the relevance of web page content. For example: if sp...

AI Technical Summary

Problems solved by technology

Often we cannot determine in advance the number of clusters that need to be clustered.

Second, the randomness of the center selection during the initial clustering may lead to a polarized aggregation effect

Third, noise points have a serious impact on clustering

Fourth, repeated calculations make the method have a higher time complexity

Images