SVM (Support Vector Machine)-based Web partitioning method

Abstract

The invention provides an SVM (Support Vector Machine)-based Web partitioning method, which comprises the steps of: partitioning all Websites into N groups; taking K=1, 2, 3, ......,N, for the value of each K, selecting Website samples in the 1st-(K-1)th, (K+1)th-Nth groups, and initializing LibSvm training; training LibSvm; storing a trained SVM model; selecting a Website sample in a Kth group to perform a Web partitioning test; and saving the Web partitioning test result. According to the method provided by the invention, the generalization capability of the SVM is strong, fault tolerance and classification can be better performed when louder noise data are processed. The accuracy rate of coordinates established by a network coordinate system is approximately 80 percent, the problem of nonlinear classification can be solved by the SVM, the number of classification by the SVM is fixed, the extreme condition that no crawler crawls on a website is avoided, and the uncertainty of partitioning the number of a set in the clustering algorithm is overcome by using the classification algorithm.

Description

technical field [0001] The invention relates to a Web partitioning method based on a support vector machine, and belongs to the technical field of Web partitioning methods for vector machines. Background technique [0002] First of all, the chainsaw algorithm, a random web partition method, can make the load of the entire system very balanced, but it does not consider the location of nodes at all, and the network distance overhead is too large. [0003] Secondly, the clustering-based Web partitioning method HONet is suitable for situations where the intra-class distance is small and the inter-class distance is large. Otherwise, the division result is related to the order of sampling, which generally results in too many internal samples in the first division set. [0004] Finally, the IWAP algorithm based on the iterative self-organizing Web partition strategy cannot guarantee that each partition set contains crawler nodes, or contains a sufficient number of crawler nodes, w...

AI Technical Summary

Problems solved by technology

Otherwise, the division result is related to the order of sampling, which generally causes too many internal samples in the first division set

[0004] Finally, the IWAP algorithm based on the iterative self-organizing Web partition strategy cannot guarantee that each partition set contains crawler nodes, or contains a sufficient number of crawler nodes, which will cause no corresponding crawler to crawl the Web sites in the non-crawler partition set, and the scheduling fails

Images