Data filtering method, system, device and medium based on generative adversarial network

Abstract

The application relates to a data filtering method and system based on a generative adversarial network, an electronic device and a storage medium, the method comprising the following steps: acquiring named entity data; inputting the named entity data into a pre-trained generator of a generative adversarial network to generate a preset number of synthetic data; in the case that the generative adversarial network model comprises a discriminator, inputting the preset number of synthetic data and the named entity data into the discriminator for coding respectively, and classifying the coded data to obtain a classification result; if the classification result is negative synthetic data, filtering out the negative synthetic data to obtain filtered synthetic data. The application solves the problem that a large amount of dirty data exists in a large amount of related synthetic data, and the model performance is damaged, improves the enhancement effect of the model, provides good training data for the training of a downstream model, and improves the overall quality of the synthetic data.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a data filtering method, system, electronic device, and storage medium based on a generative adversarial network. Background Art

[0002] Data augmentation techniques have been widely used to alleviate the data-hungry problem of deep learning models in low-resource environments. The so-called low-resource problem refers to scenarios where only a small amount of data is available for model training. Data augmentation techniques can generate large amounts of synthetic data based on the small amount of real-world data available.

[0003] Existing rule-based and deep learning-based implementation methods both have shortcomings when implementing data synthesis. Specifically, the following are: First, rule-based implementation methods have weak generalization and may have anomalies such as incorrect deletion of keywords. From the perspective of data distribution, the data generated by rule-based methods is relatively concentrated in the feature space. At the same time, since the rules are hard-set and often have no correction mechanism for correction, the generated data may be sparsely distributed and have more concentrated outliers than the original data feature space. If such data is used to train downstream models, the model performance may be damaged. Second, when generating data based on deep learning methods, the generated data has the advantage of being widely distributed, but due to the lack of training data in low-resource scenarios, and the training of deep learning models themselves relies on a certain scale of training data, there is a tendency to generate a lot of dirty data (i.e., dirty data in machine learning refers to unreliable data, missing data, or abnormal data). A large amount of dirty data will lead to a large difference between the overall distribution of the generated data and the reasonable distribution of the original data. At the same time, due to the poor interpretability of the deep learning model itself, it is difficult to obtain fundamental improvements and constraints from the model itself; but since data quality has a great impact on the performance of deep learning models, the quality of synthetic data sometimes not only fails to bring an enhancement effect to the model, but may even reduce the model performance, and even worse, cause the model to crash and become unusable.

[0004] Currently, no effective solution has been proposed to the problem that a large amount of dirty data exists in the large amount of synthesized data in related technologies, which leads to the destruction of model performance. Summary of the Invention

[0005] The embodiments of the present application provide a data filtering method, system, electronic device and storage medium based on a generative adversarial network to at least solve the problem that a large amount of dirty data exists in a large amount of related synthetic data, resulting in the destruction of model performance.

[0006] In a first aspect, an embodiment of the present application provides a data filtering method based on a generative adversarial network, the method comprising the following steps:

[0007] Get named entity data;

[0008] Inputting the named entity data into a pre-trained generator in a generative adversarial network to generate a preset number of synthetic data; wherein the number of the preset number of synthetic data is greater than the number of the named entity data, and the generative adversarial network model includes a discriminator;

[0009] Inputting the preset amount of synthetic data and the named entity data into the discriminator for encoding respectively, and classifying the encoded data to obtain a classification result;

[0010] If the classification result is determined to be synthetic data of the negative class, the synthetic data of the negative class is filtered out to obtain filtered synthetic data.

[0011] In some embodiments, when the discriminator includes a pre-trained language model and a classifier, the preset amount of synthetic data and the named entity data are respectively input into the discriminator for encoding, and the encoded data is classified, and the classification results obtained include:

[0012] Inputting the preset amount of synthetic data and the named entity data into the pre-trained language model for encoding respectively to obtain encoded data;

[0013] The encoded data is input into a classifier to obtain a classification result.

[0014] In some embodiments, the pre-trained language model is a Bert model.

[0015] In some embodiments, when the classifier is a feedforward neural network, the training process of the discriminator includes:

[0016] Merging the preset amount of synthetic data with the named entity data, and using the merged data as training data for the discriminator;

[0017] Inputting the training data into the pre-trained language model for encoding to obtain encoded data;

[0018] Inputting the encoded data into a feedforward neural network to obtain a classification result, and if the classification result is determined to be synthetic data of a negative class, filtering out the synthetic data of the negative class to obtain filtered synthetic data;

[0019] Calculating the probability error of the classification result based on the Softmax function of the last layer of the feedforward neural network;

[0020] When the probability error is minimum, a trained discriminator is obtained.

[0021] In some embodiments, before merging the preset amount of synthesized data with the named entity data, the method further includes:

[0022] The preset amount of synthetic data is marked as a negative class, and the named entity data is marked as a positive class.

[0023] In some embodiments, the training process of the discriminator further includes: connecting the discriminator to an end-to-end training process.

[0024] In some embodiments, when the pre-trained generator in the generative adversarial network is a deep recurrent neural network, inputting the named entity data into the pre-trained generator in the generative adversarial network to generate a preset amount of synthetic data includes:

[0025] The named entity data is input into the deep recurrent neural network to generate the preset amount of synthetic data.

[0026] In a second aspect, an embodiment of the present application provides a data filtering system based on a generative adversarial network, the system comprising:

[0027] Acquisition module, used to obtain named entity data;

[0028] A generation module, configured to input the named entity data into a pre-trained generator in a generative adversarial network to generate a preset number of synthetic data; wherein the number of the preset number of synthetic data is greater than the number of the named entity data;

[0029] an encoding module, configured to, when the generative adversarial network model includes a discriminator, input the preset amount of synthetic data and the named entity data into the discriminator for encoding respectively;

[0030] The classification module is used to classify the encoded data and obtain the classification results;

[0031] The filtering module is used to filter out the synthetic data of the negative class if the classification result is determined to be synthetic data of the negative class, so as to obtain filtered synthetic data.

[0032] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to implement the data filtering method based on the generative adversarial network as described in the first aspect above when running the computer program.

[0033] In a fourth aspect, an embodiment of the present application provides a storage medium storing a computer program, wherein the computer program is configured to execute the data filtering method based on the generative adversarial network as described in the first aspect above when running.

[0034] Based on the above technical solution, an embodiment of the present application first obtains named entity data, and then inputs the named entity data into a pre-trained generator in a generative adversarial network to generate a preset number of synthetic data to alleviate the data hunger problem of deep learning models under low-resource conditions. Then, when the generative adversarial network model includes a discriminator, the preset number of synthetic data and named entity data are respectively input into the discriminator for encoding, and the encoded data is classified to obtain a classification result, which facilitates the judgment of whether the input data is synthetic data or named entity data (i.e., real data). Finally, if the classification result is judged to be synthetic data of the negative class, the synthetic data of the negative class is filtered out to obtain the filtered synthetic data, which solves the problem that a large amount of dirty data exists in the relevant synthetic data, resulting in the destruction of model performance. The present application not only improves the enhancement effect of the model, provides good training data for the training of downstream models, but also improves the overall quality of the synthetic data. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0036] Figure 1 This is a first flow chart of a data filtering method based on a generative adversarial network according to an embodiment of the present application;

[0037] Figure 2 This is a structural diagram based on a generative adversarial network in an embodiment of the present application;

[0038] Figure 3 is a structural block diagram of a data filtering system based on a generative adversarial network according to an embodiment of the present application;

[0039] Figure 4 Schematic diagram of the internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for ordinary technicians in the field related to the contents disclosed in the present application, some changes such as design, manufacturing or production based on the technical contents disclosed in the present application are only conventional technical means and should not be understood as the contents disclosed in the present application being insufficient.

[0041] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it refer to independent or alternative embodiments that are mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments unless there is a conflict.

[0042] Unless otherwise defined, the technical or scientific terms used in this application should have the ordinary meaning understood by a person of ordinary skill in the technical field to which this application belongs. The words "one", "a", "the" and the like used in this application do not indicate a limit on quantity and may indicate the singular or plural. The terms "include", "comprise", "have" and any variations thereof used in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units that are inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The word "multiple" used in this application means greater than or equal to two. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The terms "first", "second", "third" and the like involved in this application are merely used to distinguish similar objects and do not represent a specific ordering of the objects.

[0043] This application provides a data filtering method based on Generative Adversarial Networks (GANs). Figure 1 This is a first flow chart of the data filtering method based on the generative adversarial network according to an embodiment of the present application. Figure 1 As shown, in this embodiment, the method includes the following steps:

[0044] Step S101, obtaining named entity data; wherein, named entity data refers to names of people, organizations, places, and all other entities identified by names. Of course, a wider range of entities also includes numbers, dates, currencies, addresses, etc. For example, in the sentence "Xiao Ming is a stockbroker", "Xiao Ming" and "stockbroker" are named entities, the entity category of "Xiao Ming" is "name", and the entity category of "stockbroker" is "occupation".

[0045] Step S102: Input the named entity data into a pre-trained generator in a generative adversarial network to generate a preset amount of synthetic data. This alleviates the data hunger problem of deep learning models under low-resource conditions. The amount of the preset amount of synthetic data is greater than the amount of named entity data. The generative adversarial network model includes a discriminator. The specific amount of the preset amount of synthetic data is set according to user needs and is not specifically limited here.

[0046] Step S103: Input a preset amount of synthetic data and named entity data into the discriminator for encoding, and classify the encoded data to obtain a classification result; in this way, it is convenient to determine whether the input data is synthetic data or named entity data (i.e., real data);

[0047] In step S104, if the classification result is determined to be synthetic data of the negative class, the synthetic data of the negative class is filtered out to obtain filtered synthetic data. In other words, if the discriminator determines that the classification result is synthetic data of the negative class, it proves that the synthetic data is low-quality data or dirty data. This embodiment filters out the synthetic data of the negative class to obtain filtered synthetic data, thereby improving the enhancement effect of the model and providing good training data for the training of the downstream model. Repeat the above steps S101 to S104 until the adversarial network model generates a preset number of synthetic data that reaches the user's final preset number. For example, when the user's final data is 1,000, the final preset number is 1,000. The final preset number is set according to the actual needs of the user and is not specifically limited here. Since those skilled in the art know that dirty data in machine learning refers to data such as unreliable data, missing data, or abnormal data, they will not be described in detail here.

[0048] Through the above steps S101 to S104, first, the named entity data is obtained, and then the named entity data is input into the pre-trained generator in the generative adversarial network to generate a preset number of synthetic data to alleviate the data hunger problem of the deep learning model under low resource conditions. Then, in the case where the generative adversarial network model includes a discriminator, the preset number of synthetic data and named entity data are respectively input into the discriminator for encoding, and the encoded data is classified to obtain a classification result, which is convenient for judging whether the input data is synthetic data or named entity data (i.e., real data). Finally, if the classification result is judged to be synthetic data of the negative class, the synthetic data of the negative class is filtered out to obtain the filtered synthetic data, which solves the problem that a large amount of dirty data exists in the relevant synthetic data, resulting in the destruction of model performance. It not only improves the enhancement effect of the model, provides good training data for the training of the downstream model, but also improves the overall quality of the synthetic data.

[0049] In some embodiments, when the discriminator includes a pre-trained language model and a classifier, inputting a preset amount of synthetic data and named entity data into the discriminator for encoding, and classifying the encoded data to obtain a classification result includes the following steps:

[0050] Input a preset amount of synthetic data and named entity data into the pre-trained language model for encoding to obtain encoded data;

[0051] The encoded data is input into the classifier to obtain the classification result. This embodiment first uses the pre-trained language model as the feature encoder of the discriminator, making full use of the rich semantic information provided by the pre-trained language model to make up for the problem that the language model is not fully trained in a low-resource state and cannot capture the semantic distribution of real data. Then, the encoded data is input into the classifier to obtain the classification result, thereby realizing the judgment of whether the input data is synthetic data or named entity data (i.e., real data).

[0052] In order to make up for the problem of insufficient language model training in low-resource conditions and the inability to capture the semantic distribution of real data, in some embodiments, the pre-trained language model is a BERT model (Bidirectional Encoder Representations from Transformer). Since the BERT model uses a Transformer model (i.e., a transformer or transformer), the Transformer model adopts an attention mechanism, which allows the machine to consider not only the original word but also other words related to it when translating, and obtain its meaning in the context, so that more feature information is available when translating the word, making the final result more accurate.

[0053] Figure 2 This is a structural diagram of an embodiment of the present application based on a generative adversarial network (ie, GANs), such as Figure 2 As shown, the generative adversarial network consists of a generator G(z), a discriminator D(x) and real data distribution data P data It consists of three parts, among which the prior distribution data P z The real data distribution data P data Belong to the same distribution and come from the same feature space, it can also be simply considered that the prior distribution data P z From the real data distribution data P data The operation principle of the generative adversarial network is as follows: First, the generator G(z) is obtained by sampling the prior distribution data P z Learning to approximate the real data distribution data P data The discriminator is a binary classifier whose input data comes from either the generator or real data. The discriminator needs to determine whether the input data comes from the generator or real data. Intuitively, the generator can be seen as an imitator, constantly imitating real data and attempting to produce high-quality synthetic data to deceive the discriminator. The discriminator can be seen as an authentic identification expert, improving its ability to judge the authenticity of data by constantly being exposed to real data and synthetic data and judging the authenticity. Y is the classification result obtained by the discriminator. Under ideal conditions, the two will eventually reach a Nash equilibrium, that is, the data generated by the generator is very similar to the real data, and the discriminator can no longer accurately judge the authenticity and can only rely on guesswork to draw conclusions. The guesswork accuracy is 50%. When the generator converges with the discriminator, a trained generative adversarial network is obtained.

[0054] In some embodiments, when the classifier is a feedforward neural network, the training process of the discriminator includes the following steps:

[0055] First, a preset amount of synthetic data and named entity data are merged, and the merged data is used as the training data for the discriminator;

[0056] Next, the training data is input into the pre-trained language model for encoding to obtain the encoded data;

[0057] Then, the encoded data is input into the feedforward neural network to obtain the classification result. If the classification result is determined to be synthetic data of the negative class, the synthetic data of the negative class is filtered out to obtain filtered synthetic data;

[0058] Next, the probability error of the classification result is calculated based on the Softmax function of the last layer of the feedforward neural network; after this step, the generator training can be continued, and the training step of the generator also includes: obtaining the first error between the generator synthetic data and the named entity data (i.e., real data); merging the first error and the probability error of the classification result to obtain merged data, and using the merged data as the training data of the generator to improve the training effect of the generator; it should be noted that merging the first error and the probability error of the classification result includes simple addition or weighted merging using a "multi-task mechanism", etc., thereby providing diversity for the error calculation of the generator and optimizing the learning process of the generator model;

[0059] Finally, when the probability error is minimized, the trained discriminator is obtained.

[0060] In some embodiments, before merging the preset amount of synthesized data with the named entity data, the method further includes the following steps:

[0061] A preset amount of synthetic data is labeled as negative, and named entity data is labeled as positive. This makes it easier to distinguish synthetic data from real data.

[0062] To facilitate the training of the discriminator, in some embodiments, the training process of the discriminator further includes: connecting the discriminator to an end-to-end training process; wherein, those skilled in the art know that the end-to-end (i.e., End to End) neural network model is a common neural network training structure, which often means that from the original input to the actual output, there is only one model in the middle, and does not rely on multiple models or components.

[0063] In some embodiments, when the pre-trained generator in the generative adversarial network is a deep recurrent neural network (RNN), inputting the named entity data into the pre-trained generator in the generative adversarial network to generate a preset amount of synthetic data includes:

[0064] The named entity dataset is input into a deep recurrent neural network to generate a preset amount of synthetic data. Of course, in other embodiments, the generator can also be replaced with other models, which are not specifically limited here.

[0065] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0066] This embodiment also provides a data filtering system based on a generative adversarial network, which is used to implement the above-mentioned embodiments and preferred embodiments. Details that have already been described will not be repeated. As used below, the terms "module," "unit," "subunit," etc. may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0067] Figure 3 is a structural block diagram of a data filtering system based on a generative adversarial network according to an embodiment of the present application. Figure 3 As shown, the system includes:

[0068] An acquisition module 31 is used to acquire named entity data;

[0069] A generating module 32 is configured to input the named entity data into a pre-trained generator in a generative adversarial network to generate a preset number of synthetic data; wherein the number of the preset number of synthetic data is greater than the number of the named entity data;

[0070] an encoding module 33 for inputting a preset amount of synthetic data and named entity data into the discriminator for encoding, respectively, when the generative adversarial network model includes a discriminator;

[0071] The classification module 34 is used to classify the encoded data and obtain a classification result;

[0072] The filtering module 35 is used to filter out the synthetic data of the negative class if the classification result is judged to be synthetic data of the negative class, and obtain the filtered synthetic data. In this embodiment, first, named entity data is obtained, and then the named entity data is input into the pre-trained generator in the generative adversarial network to generate a preset amount of synthetic data to alleviate the data hunger problem of the deep learning model under low resource conditions. Then, when the generative adversarial network model includes a discriminator, the preset amount of synthetic data and named entity data are respectively input into the discriminator for encoding, and the encoded data are classified to obtain the classification result, which facilitates the judgment of whether the input data is synthetic data or named entity data (i.e., real data). Finally, if the classification result is judged to be synthetic data of the negative class, the synthetic data of the negative class is filtered out to obtain the filtered synthetic data, which solves the problem that a large amount of dirty data exists in the relevant synthetic data, resulting in the destruction of model performance. It not only improves the enhancement effect of the model, provides good training data for the training of the downstream model, but also improves the overall quality of the synthetic data.

[0073] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0074] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0075] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0076] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0077] Step S101, obtaining named entity data;

[0078] Step S102: inputting the named entity data into a pre-trained generator in a generative adversarial network to generate a preset number of synthetic data; wherein the number of the preset number of synthetic data is greater than the number of the named entity data, and the generative adversarial network model includes a discriminator;

[0079] Step S103: Input a preset amount of synthetic data and named entity data into the discriminator for encoding, and classify the encoded data to obtain a classification result;

[0080] Step S104: If the classification result is determined to be synthetic data of the negative class, the synthetic data of the negative class is filtered out to obtain filtered synthetic data.

[0081] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be repeated here.

[0082] In addition, in conjunction with the data filtering method based on a generative adversarial network in the above embodiments, embodiments of the present application may provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, it implements any of the data filtering methods based on a generative adversarial network in the above embodiments.

[0083] In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, memory, a network interface, a display screen, and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When executed by the processor, the computer program implements a data filtering method based on a generative adversarial network. The display screen of the computer device may be a liquid crystal display or an electronic ink display screen. The input device of the computer device may be a touch layer covering the display screen, or may be buttons, a trackball, or a touchpad provided on the computer device housing, or may be an external keyboard, touchpad, or mouse.

[0084] In one embodiment, Figure 4 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, such as Figure 4 As shown, an electronic device is provided, which may be a server, and its internal structure diagram may be as shown in FIG. Figure 4 As shown. The electronic device includes a processor, a network interface, an internal memory, and a non-volatile memory connected via an internal bus, wherein the non-volatile memory stores an operating system, a computer program, and a database. The processor is used to provide computing and control capabilities, the network interface is used to communicate with external terminals via a network connection, the internal memory is used to provide an environment for the operation of the operating system and the computer program, and when the computer program is executed by the processor, it implements a data filtering method based on a generative adversarial network. The database is used to store data.

[0085] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0086] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, which can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0087] Those skilled in the art should understand that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0088] The above embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A data filtering method based on generative adversarial network, characterized in that: The method comprises the following steps: Acquire named entity data, wherein the named entity data is entity data identified by a name; Inputting the named entity data into a pre-trained generator in a generative adversarial network to generate a preset number of synthetic data; wherein the number of the preset number of synthetic data is greater than the number of the named entity data, and the generative adversarial network model includes a discriminator; In a case where the discriminator includes a pre-trained language model and a classifier, the preset amount of synthetic data and the named entity data are respectively input into the pre-trained language model for encoding to obtain encoded data; and the encoded data is input into the classifier to obtain a classification result; If the classification result is determined to be synthetic data of the negative class, the synthetic data of the negative class is filtered out to obtain filtered synthetic data; The preset amount of synthetic data is marked as a negative class, and the named entity data is marked as a positive class; wherein, when the classifier is a feedforward neural network, the training process of the discriminator includes: merging the preset amount of synthetic data and the named entity data, and using the merged data as training data for the discriminator; inputting the training data into the pre-trained language model for encoding to obtain encoded data; inputting the encoded data into the feedforward neural network to obtain a classification result, and if the classification result is judged to be synthetic data of the negative class, filtering out the synthetic data of the negative class to obtain filtered synthetic data; calculating the probability error of the classification result based on the Softmax function of the last layer network of the feedforward neural network; when the probability error is minimized, obtaining a trained discriminator.

2. The method according to claim 1, characterized in that The pre-trained language model is a Bert model.

3. The method according to claim 1, characterized in that The training process of the discriminator further includes: connecting the discriminator to an end-to-end training process.

4. The method according to claim 1, characterized in that In a case where the pre-trained generator in the generative adversarial network is a deep recurrent neural network, inputting the named entity data into the pre-trained generator in the generative adversarial network to generate a preset amount of synthetic data includes: The named entity data is input into the deep recurrent neural network to generate the preset amount of synthetic data.

5. A data filtering system based on generative adversarial networks, characterized in that: The system is used to perform the method according to any one of claims 1 to 4, and the system includes: Acquisition module, used to obtain named entity data; A generation module, configured to input the named entity data into a pre-trained generator in a generative adversarial network to generate a preset number of synthetic data; wherein the number of the preset number of synthetic data is greater than the number of the named entity data; an encoding module, configured to, when the generative adversarial network model includes a discriminator, input the preset amount of synthetic data and the named entity data into the discriminator for encoding respectively; The classification module is used to classify the encoded data and obtain the classification results; The filtering module is used to filter out the synthetic data of the negative class if the classification result is determined to be synthetic data of the negative class, so as to obtain filtered synthetic data.

6. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the data filtering method based on a generative adversarial network according to any one of claims 1 to 4.

7. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the data filtering method based on a generative adversarial network according to any one of claims 1 to 4 when running.

Images