SMO parallel processing method orientated at multi-core cluster

Abstract

The invention relates to an SMO parallel processing method orientated at a multi-core cluster. The SMO parallel processing method orientated at the multi-core cluster comprises the steps that an initial value is assigned to a local problem parameter according to a global parameter, and an initial value is assigned to an algorithm parameter; a local first boundary and a local second boundary of the local problem parameter are calculated according to the initial value of the local problem parameter; a global first boundary and a global second boundary are obtained according to local first boundary and the local second boundary; when the difference between the global first boundary and the global second boundary is not smaller than preset accuracy, a first multiplier corresponding to the global first boundary and a second multiplier corresponding to the global second boundary are calculated in an iterative model; the local problem parameter is updated in a multithreading mode after each time of iteration; when the iteration reaches a preset iteration frequency, a local solution of data to be classified is calculated according to a local sample multiplier, a global solution is obtained according to the local solution, and data classification is finished. The SMO parallel processing method orientated at the multi-core cluster resolves the traditional problems of high data classification cost, a high error rate and low response speed.

Description

technical field [0001] The invention relates to data classification, in particular to a multi-core cluster-oriented SMO parallel processing method. Background technique [0002] The problem to be solved by the present invention is to realize the parallelization of text classification, which not only improves the speed of text classification, but also distributes and stores data more reasonably and effectively. The technical problem comes from Sogou Company. Sogou Search is an interactive Chinese search engine, and classification technology is one of its core technologies. Due to the ever-increasing data scale of the modern Internet, the volume of samples that need to be classified is huge. However, the solution of storing these data in the same place and on the same disk is obviously difficult to cope with the changes in the future data world. Therefore, there is an urgent need for a technical solution to realize the distributed storage and parallel classification of these ...

AI Technical Summary

Problems solved by technology

According to the current actual needs of Sogou in terms of classification business, we need to complete the classification of the following sample size: a sample file with 10,000 lines, 2 sample files (2 categories) per day, and the script is set to study sample files for 10 days, that is, 200,000 samples, the size of the kernel matrix is ​​the square of the number of samples, that is, 4*1010 elements (but the kernel matrix), assuming 4 bytes per character, considering the symmetry needs to be greater than 4*4*1010 / 2byte=80Gbyte, when storing the sample needs The space is large, the cost is high, and when dealing with large-scale data, the storage reflection speed is slow

Images