A support vector machine training method based on spark framework

Abstract

The invention provides a support vector machine training method based on the Spark framework, comprising: obtaining a training sample set, and storing all sample vectors in the training sample set in the data nodes of the Spark framework in a distributed manner; The sample vector V 2 , and select the sample vector V at the same time 2 The sample vector V with the largest difference between the centers of the spheres 1 ; for sample vector V 1 and V 2 Perform iterative optimization calculations to obtain the updated sample vector V 1 new and V 2 new ; the sample vector V 1 new and V 2 new Broadcast to Spark's data nodes, and calculate the sample vector V in each data node 1 and V 2 The resulting difference is calculated to obtain the updated center of the sphere; then update the center distance and sphere radius of each sample vector in the data node. The method provided by the present invention disperses the intensive calculation work of a single computer to each working node by applying the Spark distributed computing framework, and can expand horizontally when the data increases, and the storage space is not limited by a single computer.

Description

technical field [0001] The present invention relates to the field of computer technology, more specifically, to a support vector machine training method based on the Spark framework. Background technique [0002] Since the appearance of Support Vector Machine (SVM), it has been widely used in information security, image processing, pattern recognition, fault diagnosis, anomaly detection and other fields. In 1999, Tax, Scholkopf and Duin et al. proposed two One Class SVM algorithms, namely the Hyperplane-based and Hypersphere-based One Class SVM. Among them, support vector data description (support vector data description, SVDD) is a single-class classification method using hyperspheres, and its goal is to use training data to describe a hypersphere as a discriminant model for classification. [0003] The current commonly used software packages for SVM pattern recognition and regression are python's scikit-learn and Taiwan's LIBSVM of Professor Lin Zhiren. Among them, Sciki...

AI Technical Summary

Problems solved by technology

[0004] However, with the exponential growth of the amount of data, the requirements for the memory and CPU of the stand-alone version can no longer meet the demand, and the demand for the algorithm parallelization solution method is becoming more and more urgent

The SMO algorithm needs to calculate multiple quadratic programming problems to solve the support vector data description (SVDD), which has high computational complexity, and the running time of SVDD will increase sharply with the increase of the number of training samples.

The memory required to store the kernel matrix Kii increases rapidly with the number of training points N in the training set. The size of the kernel matrix is ​​the square of the number of samples. Directly applying SVDD to data anomaly detection will lead to excessive calculation and memory overflow problems.

Images