Text classification method and system based on fusion constraint confrontation mechanism

Abstract

The invention relates to a text classification method and system based on a fusion constraint confrontation mechanism, and belongs to the technical field of text classification. Under the condition of limited supervision with labels and without labels, feature representations of multiple different channels are obtained through multiple encoders; and a feature fusion module is utilized to mine richer semantic information contained in the text content. On the basis, an adversarial constraint module is constructed to promote a generator to generate text data representation which is more similar to real data and more reasonable. In addition, a category discriminator and an authenticity discriminator are combined, so that the classification performance and the authenticity discrimination capability of the model are improved. According to the whole text classification method and device based on the fusion constraint confrontation mechanism, a confrontation constraint module and a feature fusion module are introduced into a generative confrontation learning framework, and the training process of a generator and a discriminator is improved; therefore, the performance and stability of the classification model in processing a small amount of annotated data and a large amount of unannotated data are improved.

Description

Technical Field

[0001] The invention relates to a method and system for text classification based on a fusion constraint adversarial mechanism, and belongs to the technical field of text classification. Background Art

[0002] Text classification is a basic research in the field of natural language processing and an important means for computers to understand human language. Text classification is the process of classifying texts of unknown categories into known categories according to the semantic content of the text. It has great application potential in many intelligent fields such as intelligent question answering, sentiment classification, topic classification and spam filtering. With the continuous development of deep learning technology and computer hardware, text classification technology has been significantly improved, and has been comparable to or surpassed the human ability to distinguish and classify on standard text datasets. However, in real application scenarios, in some special fields, facing the constantly generated text data, most models have poor understanding and representation capabilities and low classification accuracy. For example, in intelligent sentiment analysis systems, questioners usually provide text languages ​​that do not meet grammatical standards, such as spelling errors, misuse of words, and disordered word order, which often lead to the system's inability to accurately judge and identify the emotions contained in them. In order to promote the application of expanded text classification in practical scenarios, high-quality semantic representation capabilities and accurate classification and recognition capabilities are particularly critical.

[0003] The text classification problem is an important foundation for the research of multiple natural language processing tasks such as information retrieval and text mining. The description of this problem is to determine the category to which the text description belongs based on the content of the text description among multiple text description sentences and a pre-set set of categories. The excellent classification performance of the model is often inseparable from a large amount of high-quality annotated data, but in many practical applications, annotated data is often scarce, and manual annotation is time-consuming and expensive, which also leads to poor generalization ability of the model and is prone to overfitting problems, affecting the classification performance of the model in practical applications and failing to meet the analysis accuracy requirements corresponding to actual scenarios.

[0004] As multimedia data is growing explosively, the constantly updated text data will also cause problems for model classification. For example, in intelligent question-answering systems, people often enter text content with grammatical errors or incorrect characters. This content may not affect human reading judgment, but it can easily cause classification errors in the model, limiting the application of text classification in actual scenarios.

[0005] In recent years, generative adversarial networks have been widely used in text generation and data augmentation tasks, and can generate adversarial samples with perturbations. However, these adversarial samples may generate semantically unreasonable or structurally damaged samples without reasonable constraints, causing the classification model to perform poorly when facing these adversarial samples, or even be vulnerable to adversarial attacks. Therefore, how to design a method that can improve text classification performance with limited annotated data while ensuring the rationality of adversarial samples has become a problem that needs to be solved urgently.

[0006] In summary, in order to promote the application of text classification technology in practice and improve productivity, it is necessary to conduct in-depth research on the above problems and propose reasonable solutions. This invention focuses on solving the problems of low quality of generated samples, unstable model training, low utilization rate of unlabeled data, insufficient generalization ability, etc. in the classification method, and proposes a text classification method based on a fusion constraint adversarial mechanism to improve the classification performance and robustness of the model, which are all of great significance in promoting the development and implementation of cutting-edge technologies in this field.

[0007] With the development of deep learning and machine learning technologies, text classification methods have gradually evolved from traditional feature engineering-based models to complex neural network-based models.

[0008] The earliest text classification methods mainly rely on manually designed features, focus on the representation and extraction of text features, and combine traditional machine learning algorithms for classification. The text classification method based on the bag-of-words model represents the text as a word frequency vector, but ignores the word order information in the text and cannot capture the semantic relationship and contextual information between words. The method based on word frequency-inverse document frequency determines the importance of a word by measuring the frequency of a word in the document and the occurrence of the word in the entire corpus, which improves the filtering effect of high-frequency common words, but also faces the problem of missing contextual information. The above methods are combined with common traditional machine learning classification methods, such as naive Bayes, support vector machines, random forests, etc., and perform well in small data sets and low-feature spaces, but have limited classification performance on complex, high-dimensional text data sets.

[0009] With the rapid development of deep learning in various fields, many research works apply deep models such as convolutional neural networks, recurrent neural networks, and long-short time neural networks to text classification. Text classification based on convolutional neural networks Text classification methods regard text as a one-dimensional sequence, effectively extract features through a sliding window mechanism, and use convolution kernels to capture local features in the text to complete classification. Recurrent convolutional neural networks are suitable for processing sequence data. They can capture contextual dependencies in text. This method is good at processing long text classification tasks, especially in sentiment analysis, topic classification and other tasks. However, the gradient disappearance problem occurs when processing long texts. Classification methods based on attention mechanisms enhance the model's ability to capture key text information. By introducing the attention mechanism, the model can assign different importance weights to words in different positions, which greatly improves the classification effect and is particularly suitable for long text classification tasks. In addition, there are pre-trained language models based on the Transformer architecture, which can obtain deep semantic information through large-scale unsupervised training and fine-tune it in downstream tasks.

[0010] In addition, adversarial learning-based text classification methods have gradually attracted more and more attention due to their superiority in improving model robustness, enhancing data expansion capabilities, and improving model generalization capabilities. In the field of text classification, adversarial samples are small perturbations added to the input text, making the classification model prone to misclassification when facing these samples. Adversarial training enhances the robustness of the model under adversarial attacks by adding these adversarial samples to the training set. In text classification, some researchers have proposed to directly replace or delete words in the text, or introduce perturbations in the word embedding space to effectively improve the classification performance of the model. Generative adversarial networks have shown great potential in text generation and classification tasks. The basic idea of ​​generative adversarial networks is to generate samples similar to the real data distribution through the game between the generator and the discriminator. It is mainly used to generate adversarial samples or enhance data expansion capabilities. This method can also help text classification models improve performance by generating high-quality enhanced samples. Text classification methods based on adversarial learning have made significant progress in recent years, especially in improving model robustness, semi-supervised learning capabilities, and performance in the face of adversarial sample attacks. However, current adversarial learning methods still face challenges in semantic plausibility and quality of generated text when generating adversarial samples.

[0011] Text classification based on adversarial learning is mainly achieved in two different ways: one is based on adversarial samples, which achieves data enhancement by adding small perturbations to the input text, and uses the expanded training set to improve the robustness of the model; the other is based on generative adversarial networks, which uses the game between generation and adversary to encourage the generator to generate samples with similar distribution to the real data, and use the generated high-quality samples to help the model improve classification performance.

[0012] Although existing adversarial learning methods can enhance the classification performance of the model, there are still many problems that need to be solved.

[0013] 1) Unstable model training. In the process of adversarial training between the generator and the discriminator, training crash problems are prone to occur. Due to the complexity of text representation and the fact that the data may contain noise or information irrelevant to the task, the feature information learned by the generator is biased. In the initial stage of training, it is difficult for the generator to fully learn the text representation, resulting in confusion in the generated distribution representation and training crash problems, making the model easily unstable during training.

[0014] 2) The quality of generated samples is unstable. Due to the high-dimensional sparsity and semantic complexity of text data, the samples generated by the generator do not necessarily conform to the characteristic distribution of the original data only under the constraints of the discriminator. At the same time, in real applications, some specific fields do not contain much labeled data. In the training of adversarial generation, they cannot provide enough data, resulting in insufficient model training, which directly affects the generation ability of the generator. The generated sample representation distribution is prone to semantic irrationality. When there is less real text data, the generated sample representation distribution lacks constraints, making the generation of sample representation irregular. Its distribution does not conform to the data distribution of real samples, so that the expanded training data set contains erroneous samples with deviations, resulting in a decrease in the accuracy of the classification model training and reasoning stages.

[0015] 3) The utilization rate of unlabeled data is low. Previous methods used samples generated by generators to expand data sets, and most of them were completed on labeled data sets. New text samples are obtained by introducing small perturbations on text sentences or their embedded representations. Or the generator is trained to generate samples that are close to the distribution of real sample data using a generative adversarial approach. All of the above methods obtain new samples based on labeled data and expand the training data set without considering unlabeled data. In real applications, a large amount of text data generated on the Internet, especially new data generated in specific fields in reality, is mostly unlabeled data, including text data with unreasonable semantics or grammatical errors. Existing adversarial learning-based methods are difficult to fully explore the potential information in unlabeled data, thereby limiting the classification ability of the model. At the same time, it is difficult for the model to make correct judgments and identifications for text data with some erroneous information. Summary of the invention

[0016] In view of the shortcomings of the prior art, in order to improve the quality of samples generated in text classification tasks using adversarial learning, improve the stability of model training, and mine potential information in unlabeled data, and improve the overall classification performance of the model, the present invention proposes a method and device for text classification based on a fusion constraint adversarial mechanism. Specifically, the method of the present invention is an improved architecture of generative adversarial learning, which adds certain constraints to the game between model generation and adversarial, prompting the generator to generate higher quality sample representations. In order to mine richer semantic information in text content, we construct a multi-encoder module to provide multiple hidden layer representations of text samples. Then, the representation features of multiple semantic information are fused, and the data distribution representation of the generated samples is constrained by the generator. At the same time, a combination module of a category discriminator and a true-false discriminator is constructed, which can assign category labels to text samples and distinguish between real samples and generated samples. In addition, during the entire iterative training process of the model, the generated data samples do not need to be labeled and expanded into the training data set. For unlabeled data, it can also participate in the overall training of the model, and use the feature information carried by the unlabeled data to constrain the generated sample data distribution representation, so that it generates samples with more realistic sample distribution representation. Finally, the combined classifier is used to effectively improve the classification and recognition ability of the model.

[0017] The present invention focuses on the construction of a text classification method and system based on a fusion constraint adversarial mechanism. Under limited supervision with or without labels, feature representations of multiple different channels are obtained through multiple encoders, and the feature fusion module is used to mine the richer semantic information contained in the text content. On this basis, an adversarial constraint module is constructed to prompt the generator to generate a text data representation that is closer to real data and more reasonable. In addition, the category discriminator and the authenticity identifier are combined to improve the classification performance and authenticity identification ability of the model. The entire text classification method and device based on the fusion constraint adversarial mechanism described are focused on protection.

[0018] The technical solution of the present invention is as follows:

[0019] The present invention provides a text classification method and system based on a fusion constraint adversarial mechanism, which improves the training process of the generator and the discriminator by introducing an adversarial constraint module and a feature fusion module into a generative adversarial learning framework, thereby improving the performance and stability of the classification model when processing a small amount of labeled data and a large amount of unlabeled data.

[0020] A method for text classification based on a fusion constraint adversarial mechanism includes the following steps:

[0021] (1) Data preprocessing

[0022] In the data preprocessing part, the generator input and original text data are mainly processed; among them, for the generator input, the input is a 100-dimensional representation vector, and the noise randomly sampled from the normal distribution is expressed as follows:

[0023]

[0024] In the processing of raw text data, different preprocessing operations are used for different languages. In foreign text sentences, extra spaces, stop words, and special characters are removed, where special characters include: !, @, #, $, %, &, ~, <, >; in Chinese text sentences, Chinese word segmentation operations must be performed. The word segmentation operation is a known tool that uses existing software, such as the publicly available jieba Chinese classification tool, whose link is https: / / github.com / fxsjy / jieba.

[0025] (2) Feature Fusion Module

[0026] Construct multiple encoders and perform average pooling on the output representations of multiple encoders to capture feature information from different channels;

[0027] Preferably, the basis of feature fusion is the construction of a multi-encoder, the purpose of which is to mine richer semantic information contained in labeled data and unlabeled data; multiple output representations of text samples are obtained through the multi-encoder, that is, different text information is obtained from different encoder channels; wherein the multi-encoder is constructed by stacking multiple Transformer models with the same structure. The present invention takes C encoders as an example, and the formula is expressed as follows:

[0028]

[0029] in, Represents multiple encoders, is the cth encoder, C is the number of encoders;

[0030] For labeled sample data (x i ,y i ), the output after the multi-encoder is expressed as:

[0031]

[0032] For the unlabeled sample data (x j ), the output after the multi-encoder is expressed as:

[0033]

[0034] The feature fusion module mainly includes two implementation methods: one is to use the average pooling method; the other is to use the maximum pooling method. In this invention, we use the average pooling method for feature fusion to capture feature information from different channels. For all text data (x), the output obtained after the feature fusion module is expressed as:

[0035]

[0036] (3) Adversarial Constraint Module

[0037] Construct an adversarial constraint module and use the expected value calculation method to perform constraint calculation on the output representation of the generator and the output representation after feature fusion, in order to make the distribution of the output representations of the two tend to be consistent;

[0038] For the data distribution representation of the real samples obtained after feature fusion and the data representation of the generated samples obtained by the generator, feature matching is used for constraint, that is, the expected value method is used to promote the approximate consistency of the distribution of the two; the formula is as follows:

[0039]

[0040] in, represents the expected calculation, To combat constraint loss, is the expectation of the output representation obtained by the generator, is the expected output representation after the feature fusion module, is the expectation of the output obtained by the random noise generator, It is the expectation of the output representation of the real text sample after passing through multiple encoders and feature fusion modules;

[0041] (4) Construction of combined discriminator

[0042] A combination module of category discriminator and authenticity discriminator is constructed, in which the category discriminator is mainly used to correctly classify real labeled samples and identify the samples generated by the generator; the authenticity discriminator is used as an auxiliary classifier to only distinguish between real samples and generated samples, with the aim of further helping the category discriminator improve its ability to recognize generated samples.

[0043] The combined discriminator includes a category discriminator and a true-false discriminator, wherein the category discriminator is a two-layer multilayer perceptron, whose input is a multi-dimensional representation vector of real data and generated data, and whose output is a k+1-dimensional logit representation, wherein k is the number of original categories, and the k+1th dimension is the category of the sample generated by the generator; the true-false discriminator is a one-layer multilayer perceptron, whose input is the representation of all data, including the output representation of real text samples after passing through multiple encoders and feature fusion modules, and the output representation of random noise after passing through the generator, and the output is a 2-dimensional logit representation, which only identifies whether the sample data is true or false;

[0044] When training the category discriminator in the model, on the one hand, the labeled text samples should be correctly assigned to one of the kth categories to which they originally belonged, and on the other hand, the generated samples should be identified as the k+1th category; the loss calculation formula is as follows:

[0045]

[0046] Among them, θ is the parameter of the model, is the computational loss of the class discriminator, is the cross entropy loss calculation on the real labeled samples, p is the probability distribution representation, is the cross entropy loss calculation on the generated samples obtained by the generator, is the predicted category, θ is the parameter of the model;

[0047] When training the true and false discriminator in the model, it is only responsible for correctly identifying whether it is a real sample; for real samples identified as generated samples, and for samples that are not identified as generated, the loss calculation formula is as follows:

[0048]

[0049] Among them, fake and real represent the categories of generated samples and real samples respectively. is the loss of real samples being identified as generated samples, The computational loss for the true and false discriminator;

[0050] (5) Adversarial training with integrated constraint mechanism

[0051] In the overall adversarial game training, the generator loss is also included, which mainly includes the output representation of the generated sample, the discriminator's successful recognition, and the adversarial constraint loss; the total loss calculation formula of the generator is as follows:

[0052]

[0053] is the generator loss, P is the distribution G Expectations, P G is the distribution representation of random noise x after passing through the generator, x is the sample generated by the random noise after passing through the generator, and y is the generated sample label;

[0054] In summary, during the overall training of the model, we need to calculate and The above three losses are used to gradually update the parameters of the generator, category discriminator, multi-encoder and authenticity discriminator through alternating iterations; through the joint optimization of the generator and category discriminator, a small amount of labeled data and a large amount of unlabeled data are used to train the text classification model, and its generalization ability is gradually improved.

[0055] (6) Model deployment and application

[0056] After the overall training of the model is completed, the trained generator, category discriminator, multi-encoder and authenticity discriminator are obtained; in the recognition application stage, the generator and authenticity discriminator parts are not used, and the multi-encoder and category discriminator parts are retained as the final text classification model. The data to be classified is input into the multi-encoding module, and then passes through the feature fusion module and the category discriminator in turn to obtain the predicted category label.

[0057] A text classification system based on a fusion constraint adversarial mechanism comprises: a preprocessing module for processing the input of a generator and original text data; a feature fusion module for constructing a multi-encoder and performing average pooling on the output representations of multiple encoders; an adversarial constraint module for using a calculation expectation method to perform constraint calculation on the output representation of the generator and the output representation after feature fusion, so that the distribution of the output representations of the two tends to be consistent; a combined discriminator module for constructing a combination of a category discriminator and a true-false discriminator, with the category discriminator as the main method, correctly classifying real labeled samples and identifying samples generated by the generator; the true-false discriminator, as an auxiliary classifier, only distinguishes between real samples and generated samples; after the overall training of the model is completed, the multi-encoder and the category discriminator parts are retained as the final text classification model.

[0058] A computer-readable storage medium stores a program, which, when executed by a processor, implements the steps in the above-mentioned method for text classification based on a fusion constraint adversarial mechanism.

[0059] An electronic device comprises a memory, a processor and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in the method for text classification based on a fusion constraint adversarial mechanism as described above are implemented.

[0060] Deploy the optimized text classification model to actual application scenarios, such as intelligent sentiment analysis, intelligent medical analysis, spam detection, intelligent retrieval systems, etc., and perform accurate classification.

[0061] The present invention can be applied to tasks related to intelligent text information understanding. For example: in an intelligent sentiment analysis system, it can help the model better understand sentiment expression and provide strong support for sentiment analysis; in an intelligent question-answering system, it can help judge the integrity of the conversation and help the model distinguish the rationality of text sentences; in a retrieval system, it can assist in screening and filtering the retrieval content and improve retrieval efficiency; in an intelligent medical analysis system, it can help doctors obtain a large amount of medical case information and assist in making certain case diagnoses. In the spam detection task, the cost of labeling spam is high and the data distribution is uneven. The present invention generates virtual email samples covering different categories through category balance constraints, improves the discriminator's ability to detect spam, and reduces the false alarm rate.

[0062] Compared with the prior art that does not consider the fusion of multi-channel information, this application uses a multi-encoder construction and a corresponding feature fusion module to capture feature information from the multi-channel encoder. On this basis, an adversarial constraint module is designed to constrain the fused output representation and the output representation of the generator, so that the generator can further generate a sample representation that is closer to the real one. The prior art uses a single discriminator for classification, while this case constructs a combination module of a category discriminator and a true-false discriminator, and uses multiple discriminators to improve the overall category classification ability of the model and the ability to distinguish between real samples and generated samples.

[0063] The beneficial effects of the present invention are:

[0064] The present invention solves three problems that cannot be solved in most existing text classification methods using adversarial learning:

[0065] 1. Model training instability problem.

[0066] Most existing methods that use generative adversarial learning for text classification are prone to training instability. The present invention obtains multi-channel text semantic features by constructing multiple encoders, and further mines the rich semantic information contained in the text content using a feature fusion module, providing more feature information for model training. In addition, the constraints of the fused feature representation are used to improve the stability of model training.

[0067] 2. The utilization rate of unlabeled data is low.

[0068] The present invention improves the situation that the original generative adversarial model can only distinguish the authenticity by constructing a combination of a category discriminator and an authenticity discriminator. Under this combined architecture, the model can not only classify categories, but also distinguish between authentic and fake samples. At the same time, the unlabeled samples participate in the constraint calculation of the generator and the loss calculation of authenticity identification, and more text semantic information can be obtained by utilizing the unlabeled samples.

[0069] 3. The quality of generated samples is unstable.

[0070] Compared with the original generative network, which only uses the discriminator to improve the generated samples, the present invention improves the original architecture of generative adversarial learning, introduces the adversarial constraint module and the feature fusion module, and constructs a multifunctional discriminator part. Then, the collaborative training of the above modules further promotes the improvement of the quality of the samples generated by the generator. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 Schematic diagram of the text classification method based on the fusion constrained adversarial mechanism. DETAILED DESCRIPTION

[0072] The present invention will be further described below by way of embodiments in conjunction with the accompanying drawings, but is not limited thereto.

[0073] Embodiment 1:

[0074] A method for text classification based on a fusion constraint adversarial mechanism includes the following steps:

[0075] (1) Data preprocessing

[0076] In the data preprocessing part, the generator input and original text data are mainly processed; among them, for the generator input, the input is a 100-dimensional representation vector, and the noise randomly sampled from the normal distribution is expressed as follows:

[0077]

[0078] In the processing of raw text data, different preprocessing operations are used for different languages. In foreign text sentences, redundant spaces, stop words, and special characters are removed. Special characters include: !, @, #, $, %, &, ~, <, >. Stop words refer to the existing vocabulary or are customized according to actual conditions. In Chinese text sentences, Chinese word segmentation operations should be performed well. The word segmentation operation is a known tool using existing software, such as the publicly available jieba Chinese classification tool, whose link is https: / / github.com / fxsjy / jieba.

[0079] (2) Feature Fusion Module

[0080] Construct multiple encoders and perform average pooling on the output representations of multiple encoders to capture feature information from different channels;

[0081] The basis of feature fusion is the construction of multiple encoders, which aims to mine richer semantic information contained in labeled data and unlabeled data; multiple output representations of text samples are obtained through multiple encoders, that is, different text information is obtained from different encoder channels; wherein, the multi-encoder is constructed by stacking multiple Transformer models with the same structure. The present invention takes C encoders as an example, and the formula is expressed as follows:

[0082]

[0083] in, Represents multiple encoders, is the cth encoder, C is the number of encoders;

[0084] For labeled sample data (x i ,y i ), the output after the multi-encoder is expressed as:

[0085]

[0086] For the unlabeled sample data (x j ), the output after the multi-encoder is expressed as:

[0087]

[0088] The feature fusion module mainly includes two implementation methods: one is to use the average pooling method; the other is to use the maximum pooling method. In this invention, we use the average pooling method for feature fusion to capture feature information from different channels. For all text data (x), the output obtained after the feature fusion module is expressed as:

[0089]

[0090] (3) Adversarial Constraint Module

[0091] Construct an adversarial constraint module and use the expected value calculation method to perform constraint calculation on the output representation of the generator and the output representation after feature fusion, in order to make the distribution of the output representations of the two tend to be consistent;

[0092] For the data distribution representation of the real samples obtained after feature fusion and the data representation of the generated samples obtained by the generator, feature matching is used for constraint, that is, the expected value method is used to promote the approximate consistency of the distribution of the two; the formula is as follows:

[0093]

[0094] in, represents the expected calculation, To combat constraint loss, is the expectation of the output representation obtained by the generator, is the expected output representation after the feature fusion module, is the expectation of the output obtained by the random noise generator, It is the expectation of the output representation of the real text sample after passing through multiple encoders and feature fusion modules;

[0095] (4) Construction of combined discriminator

[0096] A combination module of category discriminator and authenticity discriminator is constructed, in which the category discriminator is mainly used to correctly classify real labeled samples and identify the samples generated by the generator; the authenticity discriminator is used as an auxiliary classifier to only distinguish between real samples and generated samples, with the aim of further helping the category discriminator improve its ability to recognize generated samples.

[0097] The combined discriminator includes a category discriminator and a true-false discriminator, wherein the category discriminator is a two-layer multilayer perceptron, whose input is a multi-dimensional representation vector of real data and generated data, and whose output is a k+1-dimensional logit representation, wherein k is the number of original categories, and the k+1th dimension is the category of the sample generated by the generator; the true-false discriminator is a one-layer multilayer perceptron, whose input is the representation of all data, including the output representation of real text samples after passing through multiple encoders and feature fusion modules, and the output representation of random noise after passing through the generator, and the output is a 2-dimensional logit representation, which only identifies whether the sample data is true or false;

[0098] When training the category discriminator in the model, on the one hand, the labeled text samples should be correctly assigned to one of the kth categories to which they originally belonged, and on the other hand, the generated samples should be identified as the k+1th category; the loss calculation formula is as follows:

[0099]

[0100] Among them, θ is the parameter of the model, is the computational loss of the class discriminator, is the cross entropy loss calculation on the real labeled samples, p is the probability distribution representation, is the cross entropy loss calculation on the generated samples obtained by the generator, is the predicted category, θ is the parameter of the model;

[0101] When training the true and false discriminator in the model, it is only responsible for correctly identifying whether it is a real sample; for real samples identified as generated samples, and for samples that are not identified as generated, the loss calculation formula is as follows:

[0102]

[0103] Among them, fake and real represent the categories of generated samples and real samples respectively. is the loss of real samples being identified as generated samples, The computational loss for the true and false discriminator;

[0104] (5) Adversarial training with integrated constraint mechanism

[0105] In the overall adversarial game training, the generator loss is also included, which mainly includes the output representation of the generated sample, the discriminator's successful recognition, and the adversarial constraint loss; the total loss calculation formula of the generator is as follows:

[0106]

[0107] is the generator loss, P is the distribution G Expectations, P G is the distribution representation of random noise x after passing through the generator, x is the sample generated by the random noise after passing through the generator, and y is the generated sample label;

[0108] In summary, during the overall training of the model, we need to calculate and The above three losses are used to gradually update the parameters of the generator, category discriminator, multi-encoder and authenticity discriminator through alternating iterations; through the joint optimization of the generator and category discriminator, a small amount of labeled data and a large amount of unlabeled data are used to train the text classification model, and its generalization ability is gradually improved.

[0109] (6) Model deployment and application

[0110] After the overall training of the model is completed, the trained generator, category discriminator, multi-encoder and authenticity discriminator are obtained; in the recognition application stage, the generator and authenticity discriminator parts are not used, and the multi-encoder and category discriminator parts are retained as the final text classification model. The data to be classified is input into the multi-encoding module, and then passes through the feature fusion module and the category discriminator in turn to obtain the predicted category label.

[0111] Embodiment 2:

[0112] A text classification system based on a fusion constraint adversarial mechanism comprises: a preprocessing module for processing the input of a generator and original text data; a feature fusion module for constructing a multi-encoder and performing average pooling on the output representations of multiple encoders; an adversarial constraint module for using a calculation expectation method to perform constraint calculation on the output representation of the generator and the output representation after feature fusion, so that the distribution of the output representations of the two tends to be consistent; a combined discriminator module for constructing a combination of a category discriminator and a true-false discriminator, with the category discriminator as the main method, correctly classifying real labeled samples and identifying samples generated by the generator; the true-false discriminator, as an auxiliary classifier, only distinguishes between real samples and generated samples; after the overall training of the model is completed, the multi-encoder and the category discriminator parts are retained as the final text classification model.

[0113] Embodiment 3:

[0114] A computer-readable storage medium stores a program thereon, which, when executed by a processor, implements the steps in a method for text classification based on a fusion constraint adversarial mechanism as described in Example 1.

[0115] Embodiment 4:

[0116] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in a method for text classification based on a fusion constraint adversarial mechanism as described in Example 1 are implemented.

Claims

1. A method for text classification based on a fusion constraint adversarial mechanism, characterized in that: The steps include: (1) Data preprocessing In the data preprocessing part, the input of the generator and the original text data are processed; among them, for the input of the generator, the input is a 100-dimensional representation vector, and the noise randomly sampled from the normal distribution is expressed as follows: (2) Feature Fusion Module Construct multiple encoders and perform average pooling on the output representations of multiple encoders to capture feature information from different channels; (3) Adversarial Constraint Module Construct an adversarial constraint module and use the expected value calculation method to perform constraint calculation on the output representation of the generator and the output representation after feature fusion, so that the distribution of the output representations of the two tends to be consistent; (4) Construction of combined discriminator Construct a combined module of a category discriminator and a true-false discriminator. The category discriminator is used as the main classifier to correctly classify real labeled samples and identify samples generated by the generator. The true-false discriminator is used as an auxiliary classifier to distinguish between real samples and generated samples. The combined discriminator includes a category discriminator and a true-false discriminator, wherein the category discriminator is a two-layer multilayer perceptron, whose input is a multi-dimensional representation vector of real data and generated data, and whose output is a k+1-dimensional logit representation, wherein k is the number of original categories, and the k+1th dimension is the category of the sample generated by the generator; the true-false discriminator is a one-layer multilayer perceptron, whose input is the representation of all data, including the output representation of real text samples after passing through multiple encoders and feature fusion modules, and the output representation of random noise after passing through the generator, and the output is a 2-dimensional logit representation, which only identifies whether the sample data is true or false; When training the class discriminator in the model, on the one hand, the labeled samples are correctly assigned to one of the kth classes to which they originally belong, and on the other hand, the generated samples are identified as the k+1th class; the loss is calculated as follows: Among them, θ is the parameter of the model, is the computational loss of the class discriminator, is the cross entropy loss calculation on the real labeled samples, p is the probability distribution representation, is the cross entropy loss calculation on the generated samples obtained by the generator, is the predicted category, θ is the parameter of the model; When training the true and false discriminator in the model, only the real samples are correctly identified; for the real samples identified as generated samples, and the generated samples not identified, the loss is calculated as follows: Among them, fake and real represent the categories of generated samples and real samples respectively. is the loss of real samples being identified as generated samples, The computational loss for the true and false discriminator; (5) Adversarial training with integrated constraint mechanism In the overall adversarial game training, the generator loss is also included, which includes the output representation of the generated sample, the discriminator's successful recognition, and the adversarial constraint loss; the total loss calculation formula of the generator is as follows: is the generator loss, P is the distribution G Expectations, P G is the distribution representation of random noise x after passing through the generator, x is the sample generated by the random noise after passing through the generator, and y is the generated sample label; In summary, in the overall training process of the model, calculation and The above three losses are used to gradually update the parameters of the generator, category discriminator, multi-encoder, and authenticity discriminator through alternating iterations; (6) Model deployment and application After the overall training of the model is completed, the trained generator, category discriminator, multi-encoder and authenticity discriminator are obtained; in the recognition application stage, the generator and authenticity discriminator parts are not used, and the multi-encoder and category discriminator parts are retained as the final text classification model. The data to be classified is input into the multi-encoding module, and then passes through the feature fusion module and the category discriminator in turn to obtain the predicted category label.

2. The method for text classification based on fusion constraint adversarial mechanism according to claim 1 is characterized in that: In step (1), in the processing of the original text data, different preprocessing operations are used for different languages. In foreign text sentences, redundant spaces, stop words, and special characters are removed, wherein the special characters include: !, @, #, $, %, &, ~, <, >; in Chinese text sentences, Chinese word segmentation operations are performed, and the word segmentation operation is a known tool using existing software.

3. The method for text classification based on fusion constraint adversarial mechanism according to claim 1 is characterized in that: In step (2), the basis of feature fusion is the construction of a multi-encoder, which is used to mine the semantic information contained in labeled data and unlabeled data; multiple output representations of text samples are obtained through the multi-encoder, that is, different text information is obtained from different encoder channels; wherein the multi-encoder is constructed by stacking multiple Transformer models with the same structure, and its formula is expressed as follows: in, Represents multiple encoders, is the cth encoder, C is the number of encoders; For labeled sample data (x i ,y i ), the output after the multi-encoder is expressed as: For the unlabeled sample data (x j ), the output after the multi-encoder is expressed as: The average pooling method is used for feature fusion. For all text data (x), the output obtained after the feature fusion module is expressed as:

4. The method for text classification based on fusion constraint adversarial mechanism according to claim 1 is characterized in that: In step (3), the data distribution representation of the real sample obtained after feature fusion and the data representation of the generated sample obtained by the generator are constrained by feature matching, that is, the expected value method is used to promote the consistency of the distribution of the two; the formula is as follows: in, represents the expected calculation, To combat constraint loss, is the expectation of the output representation obtained by the generator, is the expected output representation after the feature fusion module, is the expectation of the output obtained by the random noise generator, It is the expected output representation of the real text sample after passing through multiple encoders and feature fusion modules.

5. A text classification system based on a fusion constraint adversarial mechanism, characterized in that: include: The preprocessing module is used to process the generator input and raw text data; Feature fusion module, used to construct multiple encoders and average pool the output representations of multiple encoders; The adversarial constraint module is used to calculate the output representation of the generator and the output representation after feature fusion by using the expected value calculation method, so that the output representation distribution of the two tends to be consistent; The combined discriminator module is used to construct a combination of a category discriminator and a true-false discriminator. The category discriminator is mainly used to correctly classify real labeled samples and identify samples generated by the generator. The true-false discriminator serves as an auxiliary classifier and only distinguishes between real samples and generated samples. After the overall training of the model is completed, the multi-encoder and category discriminator parts are retained as the final text classification model.

6. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the steps in the method for text classification based on a fusion constraint adversarial mechanism as described in any one of claims 1 to 4 are implemented.

7. An electronic device, characterized in that: The invention comprises a memory, a processor and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in the method for text classification based on a fusion constraint adversarial mechanism as claimed in any one of claims 1 to 4 are implemented.

Images