Text classification screening method using LDA

Abstract

The invention provides a text classification screening method using LDA, and the method comprises the steps: obtaining a data set which comprises a plurality of short sentences; preprocessing the data by using a natural language processing method, and cleaning and sorting the data; determining a subject, and manually selecting text sentences suitable for the subject; establishing a corresponding text vector matrix by using the selected text sentences and a bag-of-words model; training a first LDA model by using the vector matrix; screening residual sentences in the text by using the first LDA model, calculating the correlation between the text set and a plurality of topic words obtained by calculating a first LDA topic, and taking the correlation as a threshold for evaluating whether one sentence meets a selected topic model or not; adding a text screened through topic correlation, and training a second LDA model; using the second LDA model for judging and screening remaining sentences in the text through cosine similarity; and taking the sentences screened for three times in total as text data conforming to a screening target.

Description

technical field [0001] The invention relates to the field of natural language processing, which can effectively screen sentences matching selected topics, prepare data sets for various machine learning algorithms, or perform text classification. Background technique [0002] At present, machine learning has been widely used in various fields. However, for models that need to process natural language, it is often necessary to preset a special topic to train the model. Training a model requires a human-labeled dataset to ensure the quality of the model. But in many cases, in the absence of readily available labeled data, how to provide the model with data of the highest possible quality becomes a concern. [0003] Training models are inseparable from data, but in many cases there is not enough data (data quality is too low or the cost of labeling money is too high), so the industry has proposed so-called unsupervised learning, but it is still rarely used in practice, and mor...

AI Technical Summary

Problems solved by technology

But in many cases, in the absence of readily available labeled data, how to provide the model with data of the highest possible quality becomes a concern

[0003] Training models are inseparable from data, but in many cases there is not enough data (data quality is too low or the cost of labeling money is too high), so the industry has proposed so-called unsupervised learning, but it is still rarely used in practice, and more often it is add more training samples

Images