Neutral text generation method based on pre-classifier

Abstract

The application discloses a neutral text generation method based on a pre-classifier. The existing method uses the same text generator to replace or delete the words in the sentence to eliminate subjective bias, however, such a method is affected by different operation distributions, when one operation is less, the model is more inclined to perform another operation. The method is characterized by fine-grained analysis and processing of the debiasing problem, and improves the debiasing effect. The method mainly includes three steps: sequence labeling model marking the operation sequence; classifier judging the operation strategy (extracting the text semantic vector; text syntax feature embedding; feature fusion; MLP realizing classification); and neutral text generation. The method combines the semantic features and syntax features of the sentence to analyze the operation type in a fine-grained manner and mark the operation sequence, and according to the operation type (replacement or deletion), the corresponding generation strategy is adopted to obtain the neutral sentence.

Description

TECHNICAL FIELD

[0001] The present application relates to a text-oriented subjective bias elimination algorithm based on a pre-classifier. BACKGROUND

[0002] In the vast information of today's social media and online platforms, subjective bias is a ubiquitous problem. Subjective bias refers to the influence of one's attitudes, beliefs, and values on the expression of opinions, comments, or evaluations, leading to the non-objectivity and bias of information. This subjective bias not only affects the authenticity and credibility of information, but also exacerbates social division and misleads the public. In order to solve the problem of subjective bias, researchers have proposed various text analysis and natural language processing techniques. Currently, deep learning models such as convolutional neural networks (CNN), recurrent neural networks (RNN), or Transformers are mainly used to train language generation models to achieve subjective bias elimination. However, this method is affected by different operation strategy distributions, and when the replacement operation is less, the model is more inclined to perform deletion operations. This will lead to errors in sentence operations and low accuracy in generating sentences. The subjective bias elimination method based on pre-classifier is an effective method for generating neutral text. This method introduces a pre-classifier before text generation to finely divide the operation strategy of the text and determine whether to perform deletion or replacement operations on the sentence; thereafter, by analyzing the words in the sentence, the words that trigger bias are labeled, and only the word is operated when generating the sentence, without changing other words in the sentence. This not only preserves the semantic of the sentence, but also generates a smooth and neutral sentence after deleting or replacing the word.

[0003] Text generation based on deep learning and other technologies is a popular research direction in the field of natural language processing today. Deep learning models such as generative adversarial networks (GAN), variational autoencoder (VAE), and recurrent neural networks (RNN) are used for text generation tasks. Generative adversarial networks and variational autoencoder can generate text similar to training data, and can generate different results by adjusting the input. Recurrent neural networks can generate text related to input sequences, such as language models and text generation tasks. At the same time, deep learning models can use existing large-scale text data and pre-trained language models through transfer learning. By pre-training on large-scale general text, deep understanding of semantics can be obtained, and these knowledge can be transferred to text classification and generation in specific tasks, improving the performance and generalization ability of the model. Deep reinforcement learning can be used for sequence decision-making processes in text generation problems, such as dialogue systems and machine translation. By introducing a reinforcement learning framework, the model can optimize the generation strategy based on feedback from the generated results, generating more accurate and reasonable text. In summary, the development of these technologies brings higher performance and efficiency to text generation tasks, enriching the application scenarios of natural language processing. However, deep learning technology is an end-to-end architecture to achieve neutral text generation, which lacks domain knowledge and does not process text according to the uniqueness of the task, limiting the ability of the model to generate neutral text. SUMMARY

[0004] The present application is used to solve the problem of lack of fine-grained judgment of text operation strategy in end-to-end model architecture when eliminating text subjective bias, and specifically proposes a text-oriented subjective bias elimination method based on pre-classifier. This method first classifies the sentence in fine-grained before generating text, and judges whether to delete or replace the sentence; then, by analyzing the words in the sentence, the words that cause bias are labeled, and only the word is operated when generating the sentence, without changing other words in the sentence. This way can not only preserve the semantics of the sentence; at the same time, after deleting or replacing the word, a smooth and neutral sentence can be generated, thereby improving the fluency of the generated text and the accuracy of eliminating bias.

[0005] To achieve the above purpose, the technical solution adopted by the present application is a text subjective bias elimination model based on multi-stage processing. The method flow chart is as shown in Figure 1 The implementation steps of the method are as follows:

[0006] Step (1) Obtain the dataset of text subjective bias; Through investigation, we found that Wikipedia mainly follows three principles when verifying entries: neutral point of view (presenting facts in a neutral way, recording various views without taking a stand), verifiability, and non-original research. The principle of neutral point of view requires that for controversial topics, Wikipedia editors should represent all views in proportion and express them without bias to create a neutral entry. If an entry is found to be biased towards a particular view, it should be corrected. This principle is consistent with our view, so we use the edited version of Wikipedia as our dataset. There are more than 50,000 samples in the dataset. A sample is a pair of sentences, containing a sentence with subjective bias and a sentence expressing neutrality, denoted by s and t respectively, and t is obtained by modifying s.

[0007] Step (2) Sequence labeling model; The role of the sequence labeling model is to label each word or character in the input text and associate it with a pre-defined category or label. In this way, various information such as word forms, named entities, emotions, semantic roles, etc. can be captured in the text. In this paper, the model labels the words that trigger bias. First, convert each word in the input sentence s into a feature vector suitable for model input. For example, use a pre-trained word vector model to map each word in the sentence to a corresponding word vector. We use a trained sequence labeling model to label each word in the sentence. In this task, our goal is to label the words that trigger bias in the sentence. We associate each word or subword in the input text sequence with a label, which indicates whether the word or subword contains subjective bias. For each word, the model outputs a label indicating whether the word is a bias-triggering word. A common labeling scheme is based on the BIO (Begin-Inside-Outside) format, where "B" indicates the start of a subjective bias word or phrase, "I" indicates the inside of a subjective bias word or phrase, and "O" indicates no subjective bias. For example, "B" can be used to indicate the start of a biased word, "I" to indicate the middle part of a biased word, and "O" to indicate a non-biased word. The labeling result can be: "O O O O O O B OO", where "B" indicates that this word is a bias-triggering word. Finally, we get the labeled sentence, where the biased words are labeled with specific labels, and the non-biased words remain the same. We use the labeled dataset to train the RoBERTa model (see step 3 for details).

[0008] Step (3) Train the sequence labeling model; the training dataset contains the operation sequence and the corresponding label sequence. We input the operation sequence into the RoBERTa model and use the cross-entropy loss function to calculate the difference between the predicted label and the true label. Through back propagation and gradient descent algorithm, the model can learn to recognize and label subjective bias. In the RoBERTa model, the output of the model is converted into a label probability distribution through the Softmax function. Assuming score i,t represents the score predicted by the model, then the probability of label can be calculated by the Softmax function as follows, where C represents the number of label categories:

[0009]

[0010] We input the sequence into the RoBERTa model and use the cross-entropy loss function to calculate the difference between the predicted label and the true label. Thus, the sequence prediction model is trained for labeling operation sequences.

[0011] Step (4) Determine the operation strategy model; by investigating the operations in the dataset to eliminate the subjective bias of the sentence, we find that it mainly contains two categories: the first category is to delete the adjectives, adverbs, etc. that trigger bias in the sentence to obtain a neutral expression sentence; the second category is to replace the nouns, verbs, etc. that trigger bias in the sentence with words that express more neutrality to generate a debiased sentence. Therefore, we determine the operation strategy for the biased sentence by analyzing the sentence in combination with the semantic and syntactic features of the sentence. It mainly consists of four steps: extracting text semantic features (step 5), extracting text syntactic features (step 6), feature fusion (step 7), and determining the strategy (step 8).

[0012] Step (5) Extract text semantic features. We use text semantic information as sentence features for sentence classification tasks. We use the BERT (Bidirectional Encoder Representation from Transformers) method to extract the hidden vector representation of the sentence. The input of BERT is the representation of each token. In order to complete the specific classification task, in addition to the token of the word, a specific classification token ([CLS]) is also inserted at the beginning of the sequence. The last output of the last Transformer layer corresponding to this token is used to aggregate the entire sequence of feature information. Specifically, the input is represented as (x1, x2,..., x n ), and the corresponding embedding vectors (a1, a2,..., a n). Multiple attention vectors are generated by computing embedding vectors with multiple sets of Q, K, and V using an attention mechanism. These vectors are then merged and dot-multiplied with W o A linear transformation is performed to generate an output corresponding to each token. The specific formula is as follows, where the parameter matrix Q, K, and V represent the query vector, key vector, and value vector, respectively. d represents the dimension.

[0013] MultiHead(Q,K,V)=Concat(head1,...,head h )W O

[0014] where head i =Attention(QW i Q , KW i K , VW i V )

[0015] In the BERT model, each word token outputs a sentence semantic representation by weighing the relevance between words using an attention mechanism. Therefore, after 12 layers of BERT, the embedding of each word has fused the information of all words, which can better represent the semantic of the sentence. Because each word token has its own meaning, the semantic of itself accounts for a large proportion in the final semantic vector. And [CLS] itself has no semantic, so the output of the last layer corresponding to [CLS] can more fairly express the semantic information of the entire sentence. Therefore, we extract the output of the last transformer layer corresponding to [CLS] as the sentence semantic vector by training the BERT model.

[0016] Step (6) extracts the text syntax features; we use the dependency relationship of the text as the sentence feature in the sentence classification task. Since the replacement and deletion operations act on words of different parts of speech, and the dependency relationship can represent the syntactic relationship between words, we embed this feature. We use the spacy natural language processing library and numpy library of python to construct a dependency relationship matrix according to the syntactic relationship of the sentence, and use matrix decomposition method to decompose the matrix into three parts: left singular matrix U, singular value matrix Σ and right singular matrix V. Then select the part corresponding to the first k singular values and reconstruct the sentence representation matrix, and flatten the one-dimensional vector to represent the syntax feature.

[0017] D=UΣV T

[0018] Step (7) Feature fusion; Feature fusion refers to combining different features together to form a more comprehensive and representative feature vector, thereby improving the performance of the model. We splice the semantic features and syntactic features extracted in steps (3) and (4) to represent the feature vector of the sentence, which can better understand and analyze the meaning of the sentence, thereby improving the accuracy of classification.

[0019] Step (8) Determine the strategy. The determination of the strategy is realized by MLP. MLP is composed of an input layer, multiple hidden layers and an output layer, each layer containing multiple neurons. The number of neurons in the input layer is the same as the dimension of the feature vector, and the number of neurons in the output layer is the same as the number of categories of classification. The feature vector fused in step (5) is taken as the input of MLP, and the features are combined through the nonlinear transformation of multiple hidden layers to finally obtain the classification result. Thus, the operation to be performed to eliminate subjective bias is determined.

[0020] Step (9) Neutral text generation model; based on the operation strategy and the marked operation sequence, we generate neutral sentences in different ways. Specifically, according to the operation type (replacement or deletion), we take the corresponding generation strategy to get the neutral sentences. In the replacement operation, we use the BART model to transform the biased words to generate neutral words. BART (Bidirectional and AutoRegressive Transformer) is a pre-trained language model based on the Transformer structure, which has strong generation ability and semantic transformation ability. According to its internal autoregressive generation mechanism, the BART model uses an autoregressive decoder to generate a word sequence. In the generation process, the model uses context information and input biased words to generate neutral words according to the probability distribution. These generated words are usually similar in semantics to biased words, but express neutral views. Fill the generated neutral words into the replacement position of the original sentence to form a neutral sentence. When the operation strategy is determined as deletion, we directly delete the operation sequence to generate a neutral sentence t. The marked operation sequence is removed from the original sentence to achieve it, ensuring that the generated sentence does not contain the operation part. In this way, the generated sentence t will remain neutral and will not contain the subjective bias content in the original sentence.

[0021] Beneficial effects

[0022] The method is based on the neutral text generation method based on deep learning technology, and the fine-grained analysis and processing are carried out according to the characteristics of the debiasing problem, the bias sequence is labeled, the operation strategy is determined, and the text generation task is completed. The problem of low text generation accuracy caused by the lack of fine-grained judgment of the text operation strategy in the end-to-end model architecture is solved. The method is tested on the public data set of Wikipedia, which has more than 50,000 samples. A sample is a sentence pair, which contains a sentence with subjective bias and a sentence expressing neutrality. The accuracy of the end-to-end implementation of eliminating text subjective bias is 45.8%, and the accuracy of the method for generating neutral text is 51.8%, and the performance is significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 Flowchart of the method

[0024] Figure 2 Sequence labeling module framework

[0025] Figure 3 Operation strategy determination module framework

[0026] Figure 4 Neutral text generation module flowchart DETAILED DESCRIPTION

[0027] Firstly, the biased text is input into the sequence labeling model to generate the operation sequence. In the second step, the operation strategy is determined by analyzing the syntax and semantics, that is, whether the operation sequence is deleted or replaced. Finally, the operation sequence is operated. If it is a deletion operation, the sequence is directly deleted, and the corresponding neutral text is output; if it is a replacement operation, the operation sequence is input into the neutral sequence generation model to generate a word similar in semantics but neutral in expression, and the generated word is put into the source sentence, and the neutral text is output. The overall flowchart is as follows Figure 1 .

[0028] To achieve the above object, the present application uses the syntactic features and semantic features of the text to realize the judgment of the operation strategy, and uses the deep learning method to realize the judgment of the operation sequence and the conversion of the biased word, and finally generates the neutral text. The specific implementation steps of the method are as follows:

[0029] Step (1): sequence labeling module: first, convert each word in the input sentence s into a feature vector suitable for model input. We use the trained sequence labeling model to label each word in the sentence. In this task, our goal is to label the words in the sentence that cause bias. The specific framework is as follows Figure 2We use a tagging method based on the BIO (Begin-Inside-Outside) format to correspond each word in the input text sequence to a label, which indicates whether the word or sequence contains subjective bias. For each word, the model outputs a label indicating whether the word is a bias-triggering word. Since the RoBERTa model can better understand the context information of the input text, it improves the accuracy and effectiveness of the labeling. Therefore, we train the RoBERTa model using the labeled dataset. We input the sequence into the RoBERTa model and use the cross-entropy loss function to calculate the difference between the predicted label and the true label. Suppose we have N sequences of labels, each with a length of T. Suppose the predicted label sequence is The true label sequence is y = (y1, y2,..., y T ), then the cross-entropy loss function is defined as:

[0030]

[0031] where, represents the softmax operation on the predicted label, and the result is a probability distribution. y represents the true label. The cross-entropy loss function measures the prediction accuracy and error of the model by calculating the difference between the probability distribution of the predicted label and the true label. Minimizing the cross-entropy loss function can make the model more accurately predict the label sequence. The subjective bias sentence is input into the sequence labeling model, and the output is a sentence with each word labeled. We extract the sequence with a label other than O and define it as the operation sequence. The operation sequence is the bias-triggering sentence, and generating a neutral sentence is to delete or transform this sequence into a sequence expressing neutrality to eliminate bias.

[0032] Step (2) operation strategy determination module: In the subjective bias elimination task, we use the BERT model and matrix decomposition technology to classify the operation sequence. Through the method based on text semantics and dependency analysis, it is determined whether to delete or replace. The specific framework is as follows Figure 3 .

[0033] First, we use the BERT model to learn the semantic representation of the sentence to be operated and obtain the semantic feature representation of the sentence. The sentence to be operated is input into the BERT model, and after the encoding layer and the pooling layer, the semantic vector of the sentence is obtained. Next, we use dependency parsing technology to build the dependency tree of the sentence and generate the corresponding dependency matrix. In order to better utilize the dependency matrix, we use matrix decomposition technology to decompose it. Matrix decomposition can decompose the high-dimensional dependency matrix into low-dimensional sub-matrix to extract more useful information. Based on the semantic feature representation of the sentence and the dependency matrix after matrix decomposition, we perform text semantic classification to determine the operation strategy. By training a classifier, the semantic features of the sentence and the sub-matrix after matrix decomposition are used as input to classify the operation strategy. We can define two categories, such as "delete" and "replace", to decide whether to delete or replace the sentence to be operated. The classifier can be trained through supervised learning using labeled operation strategy samples as training data. In the prediction stage, the semantic classification of the new sentence to be operated is performed to obtain the judgment result of the operation strategy.

[0034] Step (3) Neutral Text Generation Module: According to the operation sequence determined in the previous two steps and its corresponding operation strategy, we generate neutral sentences in different ways. Specifically, according to the operation type (replace or delete), we use the corresponding generation strategy to obtain the neutral sentence. The specific process is as follows Figure 4 .

[0035] When the operation strategy is determined to be delete, we directly delete the operation sequence to generate a neutral sentence. The delete operation can be achieved by removing the marked operation sequence from the original sentence to ensure that the generated sentence does not contain the operation part. In this way, the generated sentence will remain neutral and not contain the subjective bias content of the original sentence.

[0036] When the operation strategy is determined to be replace, bias word conversion is required to generate neutral words. The BART model has learned rich language representations in large-scale unsupervised training, which can fully capture the semantic information of the context and better perform word conversion. Therefore, we realize bias sequence conversion by fine-tuning the BART model, and optimize the parameters of the BART model by minimizing the difference between the generated neutral expression and the neutral expression in the training data. In the generation process, the model generates neutral words according to the probability distribution using context information and input bias words. These generated words are usually similar in semantics to bias words, but express neutral views. Fill the generated neutral words into the replacement position of the original sentence to form a neutral sentence. By replacing the bias words in the original sentence with the generated neutral words, we can generate sentences expressing neutral views.

[0037] The above three modules can realize the elimination of text subjective bias.

Claims

1. A neutral text generation method based on a pre-classifier, characterized in that, The steps are as follows: Step (1) Obtain the dataset of subjective bias in the text; use the edited version of Wikipedia as the dataset; this dataset contains more than 50,000 samples; a sample is a sentence pair, containing a sentence with subjective bias and a sentence expressing neutrality, respectively using and express, By modifying Obtained; Step (2) Sequence labeling model; The role of the sequence labeling model is to label each word or character in the input text and associate it with a predefined category or label; In this paper, the model labels words that cause bias; First, the input sentence Each word in the sentence is converted into a feature vector suitable for model input; a pre-trained word vector model is used to map each word in the sentence to a corresponding word vector; a trained sequence labeling model is used to label each word in the sentence; in this task, the goal is to label words in the sentence that cause bias; each word or subword in the input text sequence is associated with a label indicating whether the word or subword contains subjective bias; for each word, the model outputs a label indicating whether the word is a biased word; Finally, the labeled sentences were obtained, in which biased words were marked with specific labels, while unbiased words remained unchanged; the RoBERTa model was then trained using the labeled dataset; Step (3) Train the sequence labeling model; the training dataset contains the sequence to be operated on and the corresponding labeled sequence; input the sequence to be operated on into the RoBERTa model, and use the cross-entropy loss function to calculate the difference between the predicted label and the true label; through backpropagation and gradient descent algorithms, the model can learn to identify and label subjective biases; in the RoBERTa model, the model output is converted into a label probability distribution through the Softmax function; assuming The score representing the model's prediction is then labeled. probability The Softmax function is used to calculate the number of categories of the label, as shown in the following formula: The sequence is input into the RoBERTa model, and the cross-entropy loss function is used to calculate the difference between the predicted label and the true label; thus, the sequence prediction model is trained to label the sequence to be operated on. Step (4) Determine the operational strategy model; through the survey dataset, it was found that the operations to eliminate subjective bias in sentences mainly include two categories: the first category is to delete adjectives and adverbs in sentences that cause bias, so as to obtain sentences with neutral expression; the second category is to replace nouns and verbs in sentences that cause bias, and replace them with words that express more neutral expression to generate sentences after bias removal. Step (5) Extract semantic features of the text; Textual semantic information is used as sentence features in sentence classification tasks; the hidden vector representation of the sentence is extracted using the BERT (Bidirectional Encoder Representation from Transformers) method; the input of BERT is the representation corresponding to each token; to complete the specific classification task, in addition to word tokens, a specific classification token ([CLS]) needs to be inserted at the beginning of the sequence; the output of the last Transformer layer corresponding to this token is used to aggregate the representation information of the entire sequence; specifically, the input uses ( This indicates that the corresponding embedding vector is generated through the BERT embedding layer. ); Calculate multiple sets of data using an attention mechanism. , and The embedding vectors are used to generate multiple attention vectors; then these vectors are merged and multiplied by a dot product. Perform a linear transformation to generate an output corresponding to each token; The specific formula is as follows, where the parameter matrix... , , , Q, K, and V represent the query vector, key vector, and value vector, respectively; d represents the dimension. In the BERT model, each word token outputs a sentence semantic representation by using an attention mechanism to weigh the relevance between words; by training the BERT model, the output of the last transformer layer corresponding to [CLS] is extracted as the sentence semantic vector; Step (6) Extract text syntactic features; use text dependency relations as sentence features in sentence classification tasks; since replacement and deletion operations operate on words of different parts of speech, and dependency relations can represent the grammatical relationships between words, this feature is embedded; using Python's Spacy natural language processing library and NumPy library, construct a dependency relation matrix based on the grammatical relationships of the sentences, and use matrix factorization to decompose the matrix into three parts: left singular matrix. Singular value matrix And right singular matrix Then, select the parts corresponding to the first k singular values ​​and reconstruct the sentence representation matrix, and flatten the matrix into a one-dimensional vector to represent the syntactic features; Step (7) Feature fusion: The semantic and syntactic features of the sentence extracted in steps (3) and (4) are concatenated to represent the feature vector of the sentence; Step (8) Determine the strategy; the strategy is determined by MLP; MLP consists of an input layer, multiple hidden layers and an output layer, each layer contains multiple neurons; the number of neurons in the input layer is the same as the dimension of the feature vector, and the number of neurons in the output layer is the same as the number of categories. The feature vector fused in step (7) is used as the input of MLP, and the features are combined through nonlinear transformation of multiple hidden layers to finally obtain the classification result. Step (9) Neutral text generation model; Based on the operation strategy and the marked sequence to be operated, generate neutral sentences in different ways; Specifically, according to the operation type, i.e., replacement or deletion, adopt the corresponding generation strategy to obtain sentences that express neutrality; In the replacement operation, use the BART model to convert bias words to generate words that express neutrality; The BART model uses an autoregressive decoder to generate word sequences based on its internal autoregressive generation mechanism. During generation, the model utilizes contextual information and biased input words to generate neutral words according to a probability distribution. These generated words are typically semantically similar to biased words but express neutral viewpoints. The generated neutral words are then used to fill in the replacement positions in the original sentences, forming neutral sentences. When the operation strategy is determined to be deletion, the sequence to be operated on is directly deleted, generating sentences expressing neutrality. This is achieved by removing the marked sequence of operations from the original sentence, ensuring that the generated sentence does not contain the part to be operated on; thus, the generated sentence... We will remain neutral.

Images