Improved hierarchical cascade support vector machine parallelization method

Abstract

The invention provides an improved hierarchical cascade support vector machine parallelization method. Optimization is obtained through an improved Cascade SVM algorithm. Firstly, a c is introduced in the improved algorithm to be used to measure a change situation of the number of model support vectors obtained in training of each layer in a hierarchical training process; and then through adjusting a merging strategy and a hierarchical structure in the model training process, a merging mode of the support vectors obtained by training of each layer is optimized and adjusted from merging of every two to average segmentation after merging of all, and thus a defect, existing in the merging mode of every two, that filtering non-boundary samples is insufficient is avoided. Under the premise of ensuring that the classification accuracy is not reduced, the method can effectively shorten the model training time and improve the training and classification efficiency of a model by virtue of a current-mainstream parallel framework of Spark.

Description

technical field [0001] The invention relates to an improved Casade SVM parallelization method. Background technique [0002] In the era of big data, with the rapid development and wide application of the Internet and information technology, various massive data closely related to people's lives have been generated. Among these messy data, unstructured text information accounts for the main part. Faced with such huge text data, it is difficult for people to quickly obtain their own useful information. How to quickly process and discover these data information has become a major problem, which has also promoted the in-depth research and extensive application of text classification related technologies. Text classification is a very important and popular technology in the field of data mining. Text classification technology can calculate a huge amount of unstructured text data in a way that computers can understand, so as to help people better and quickly obtain the informati...

AI Technical Summary

Problems solved by technology

Assuming that the model training is performed on a cluster containing N nodes, a large number of node resources will be idle during the model training process, which leads to low resource utilization of the cluster

[0007] (2) During the whole layer training process, after the first layer of SVM training, most of the non-support vectors are eliminated, and the number of non-support vectors filtered out in the subsequent layer training is small, but it consumes the calculation. lot of time

Images