Chinese text category recognition system and method for unbalanced data sampling

Abstract

The invention discloses a Chinese text category recognition system and method for unbalanced data sampling. The method comprises the following steps: firstly employing a text encoder to encode a Chinese text, and forming sentence vector representation; secondly, redundant and noise data are removed by adopting a multi-time DBSCAN clustering algorithm for negative sample data, and a deep learning sample selector is constructed for positive sample data to filter samples and select high-quality sample data to supplement positive samples while the samples are randomly generated; and finally, decoding the selected sample data through a text decoder to form text data, and forming a balanced data set together with the processed negative sample data so as to be applied to a text classification model. According to the method, a mixed sampling method is adopted to process data, particularly redundant data processing is performed on negative samples, a deep learning sample selector is constructedfor positive samples to filter generated samples, a high-quality sample training classifier is selected, and the text classification performance is improved.

Description

【Technical field】 [0001] The invention belongs to the technical field of natural language processing, and relates to a method for unbalanced data processing and text classification, in particular to a Chinese text category recognition system and method for unbalanced data sampling. 【Background technique】 [0002] In recent years, the rapid development of science and technology has promoted the explosive growth of data in various fields. The phenomenon of the number of data categories and the uneven distribution of features exists in almost every practical data set in real life. It is of great significance to effectively classify unbalanced data in reality. For example, enterprises can use classified user review information for different departments to make targeted improvements to related products and services. Specifically, there are the following two states in the real unbalanced data: one is that the amount of negative sample data is large, and there are certain noise da...

AI Technical Summary

Problems solved by technology

Among them, the undersampling method refers to screening some representative samples so that the positive and negative samples reach a proportional balance; however, the current two types of undersampling methods based on clustering and integration only consider the selection of important features, and feature loss exists to a certain extent. The risk, and the redundancy problem in the negative sample data also needs to be considered

Oversampling refers to generating new positive samples from a small number of existing positive samples through the model to balance the positive and negative samples in the data set; currently there are two types of methods: filter-based sample generation and limited sample space to generate data. When using space, the diversity of sample generation is more limited, so that the features cannot be evenly distributed in the sample space, and the filtering-based method is often not enough to construct a good sample due to the low granularity of feature extraction and single use of positive samples. Selector to select better samples to complement the data

In the process of mixed sampling, the relationship between positive and negative sample features is not fully utilized, and the undersampling and oversampling methods are simply stacked.

Images