Data cleaning method and device based on deep reinforcement learning model

Abstract

The invention belongs to the technical field of data communication and data processing, and particularly relates to a data cleaning method based on a deep reinforcement learning model. The method comprises the steps of obtaining a to-be-cleaned data set with a label; deleting the content-free data, the label data not in a label set and the label contradictory data in the to-be-cleaned labeled data set by adopting a pre-screening algorithm to obtain a to-be-classified data set; inputting the to-be-classified data set into a pre-trained deep reinforcement learning model to obtain different types of delay rewards; and discarding the biased data, retaining the effective data and updating a state list S according to the obtained different types of delay rewards and an action set in the pre-trained deep reinforcement learning model, and maximizing a delay reward value of each type; and taking the training data set with the label corresponding to the maximum delay reward value of each type as a cleaned training data set with the label, thereby completing data cleaning.

Description

technical field [0001] The invention belongs to the technical field of data communication and data processing, and in particular relates to a data cleaning method and device based on a deep reinforcement learning model. Background technique [0002] With the rapid development of computer technology and communication technology, people can obtain more and more digital information, but at the same time, they need to invest more time in organizing and sorting out the information. In order to alleviate this burden, people began to study the use of computers to automatically classify data. [0003] In the research of text classification technology, the usual data cleaning method is to use labeled text data to train a deep neural network classifier to achieve the purpose of identifying text categories. In this process, the credibility and validity of the data directly affect the performance of the system. Therefore, it is necessary to clean the data to eliminate abnormal data. A...

AI Technical Summary

Problems solved by technology

[0007] The purpose of the present invention is to solve the above-mentioned defects in the existing data cleaning methods. The present invention proposes a data cleaning method based on a deep reinforcement learning model, which overcomes the problem of noise in the training data of text classification, and adopts the method of reinforcement learning. , discard the abnormal label data, keep the correct label data, so as to achieve the purpose of data cleaning

Images