Parallel k-means algorithm used for high-dimensional text data

Abstract

The invention belongs to the intersection field of natural language processing and machine learning, and provides a parallel k-means algorithm used for high-dimensional text data. According to the algorithm, firstly, the text data are vectorized, then a dimension reduction model is established for a matrix after vectorization, and the high-dimensional data are converted into low-dimensional data with highly efficient features; and then algorithm accuracy is further improved through the optimized k-means clustering algorithm, then clustering is carried out on the data after dimension reduction,algorithm parallelization is realized through GPU and MPI technology, and finally, improvement of accuracy and efficiency of the high-dimensional text data is realized. The algorithm can greatly improve accuracy of text clustering, and improve running speed and portability of the algorithm.

Description

technical field [0001] The invention belongs to the cross field of natural language processing and machine learning, and relates to a parallel k-means algorithm (pk-means) for medical text data. Background technique [0002] The medical text data mentioned in the present invention refers to the medical orders issued by doctors, and the medical orders are short text data, so the medical orders in the entire medical order corpus have high-dimensional and sparse characteristics. The present invention is in the preprocessing part of the whole project, and intends to use an efficient method to perform initial clustering of all medical orders and prepare for other follow-up work of the project. In the following, we will use the term "text data" to refer to "medical text data". [0003] At this stage, clustering for text data can be divided into two steps: data preprocessing and clustering. [0004] In terms of data preprocessing, the idea based on dimensionality reduction is mai...

AI Technical Summary

Problems solved by technology

For high-dimensional text data, if the classic k-means proposed by Lloyd is used, the method of randomly selecting the initial clustering seeds will make the randomness of the clustering results too large; in addition, in the clustering process of the algorithm, the Distance-based data similarity measurement method, but for high-dimensional document data, using distance to measure similarity is not suitable

Images