Data filtering method, system and equipment based on generative adversarial network and medium

Abstract

The invention relates to a data filtering method and system based on a generative adversarial network, electronic equipment and a storage medium, and the method comprises the steps: obtaining named entity data, inputting the named entity data into a pre-trained generator in the generative adversarial network, and generating a preset number of synthetic data, under the condition that the generative adversarial network model comprises a discriminator, respectively inputting a preset number of synthetic data and named entity data into the discriminator for encoding, classifying the encoded data to obtain a classification result, if the classification result is judged to be negative-class synthetic data, filtering out the negative-class synthetic data, and if the classification result is judged to be negative-class synthetic data, outputting the negative-class synthetic data; according to the method, the problem that the performance of the model is damaged due to the fact that a large amount of dirty data exists in a large amount of related synthesized data is solved, the enhancement effect of the model is improved, good training data is provided for training of a downstream model, and the overall quality of the synthesized data is also improved.

Description

technical field [0001] The present application relates to the technical field of machine learning, and in particular, to a data filtering method, system, electronic device and storage medium based on generative adversarial networks. Background technique [0002] Data augmentation techniques have been widely used to alleviate the data hunger problem of deep learning models under low resource conditions. The so-called low resource problem is that in some scenarios, only a small amount of data is available for model training. Based on a small amount of existing actual data, a large amount of synthetic data can be obtained through data augmentation techniques. [0003] Existing rule-based implementation methods and deep learning-based methods have shortcomings when implementing data synthesis, specifically: First, the rule-based implementation method is weak in generalization, and may have anomalies such as incorrect keyword deletion , From the point of view of data distributi...

AI Technical Summary

Problems solved by technology

[0005] The embodiment of the present application provides a data filtering method, system, electronic device, and storage medium based on a generative confrontation network, so as to at least solve the problem that there is a large amount of dirty data in a large amount of related synthetic data, which leads to the destruction of model performance

Images