Text hidden poison identification method based on large model

Abstract

The invention provides a text hidden poison identification method based on a large model, and relates to the technical field of internet security. The method comprises the steps that a classification head composed of a full connection layer and a softmax layer is added to the top layer of a RoBERTa model, and a text hidden virus detection model is constructed; the method comprises the following steps: crawling comment text data from social software, performing data cleaning and data preprocessing, and classifying the comment text data into a toxic text, a hidden toxic text or a non-toxic text through manual data labeling; inputting the comment text data subjected to manual data annotation into a text hidden poison detection model, and carrying out supervised learning; and inputting a to-be-recognized text into the text toxicity-hidden detection model subjected to supervised learning, and outputting a recognized classification result toxicity-showing text, toxicity-hidden text or non-toxic text. According to the method, the input texts can be classified into the toxic texts, the hidden-toxic texts and the non-toxic texts, namely effective recognition of the hidden-toxic texts is achieved.

Description

Technical Field

[0001] The present invention relates to the field of Internet security technology, and in particular to a method for identifying hidden viruses in text based on a large model. Background Art

[0002] At present, in the field of natural language processing involving Internet security technology, toxicity detection mainly relies on technical means such as manual monitoring, convolutional neural network models, and recurrent neural network models.

[0003] At present, the identification and treatment methods for explicit toxic speech have become mature, but how to identify implicit toxic speech is still a technical problem that needs to be solved urgently. In addition, how to purify toxic speech, eliminate text toxicity, and achieve text detoxification while maintaining the integrity of the text and the correctness of the meaning of implicit toxic speech is also a technical problem that needs to be solved urgently. Summary of the invention

[0004] In order to solve the above technical problems existing in the prior art, the present invention provides a method for identifying hidden viruses in text based on a large model. The specific technical solution includes the following steps:

[0005] Step S1: Build a text hidden virus detection model, specifically by adding a classification head consisting of a fully connected layer and a softmax layer on the top layer of the RoBERTa model;

[0006] Step S2: crawling comment text data from social software and performing data cleaning, data preprocessing and manual data annotation;

[0007] Manual data annotation specifically involves: using classification labels to classify comment text data into explicit toxic text, implicit toxic text, or non-toxic text;

[0008] Step S3: Input the manually annotated comment text data into the text hidden virus detection model for supervised learning;

[0009] Step S4: Input the text to be identified into the supervised learning text hidden poison detection model, and output the identification classification result.

[0010] Preferably, data preprocessing includes word segmentation, sentence segmentation and removal of stop words.

[0011] Preferably, before step S3, the method further includes performing data enhancement on the comment text data by using the GeniusAug model;

[0012] The GeniusAug model includes an input layer, a feature extraction layer, a bidirectional encoding layer, a sketch construction layer, an autoregressive decoding layer, and an output layer connected in sequence;

[0013] The data enhancement process is as follows:

[0014] Input layer: input comment text data d and the corresponding first category label;

[0015] Feature extraction layer: extract the n-grams [w 1 ,w 2 ,...,w m ], n = 1, 2, 3; where m is the number of words in the comment text data d;

[0016] Bidirectional encoding layer: Encode the comment text data d, the first category label and the n-gram to obtain the text embedding e of the comment text data d d 、n-grams[w 1 ,w 2 ,...,w m ]'s grammatical embedding [v 1 ,v 2 ,...,v m ], label embedding e of the first category label t ;

[0017] For the review text data d, calculate each n-gram and fusion embedding e f The similarity S i , the formula is as follows:

[0018] e f =λe d +(1-λ)e t ;

[0019]

[0020] In the formula, λ is the preset fusion weight, i is the word ordinal number;

[0021] Select the top A% of n-grams as keywords;

[0022] Sketch construction layer: Based on the key words, the sketch is constructed by applying the extraction masking projection process;

[0023] Autoregressive decoding layer: Generates new text data based on the sketch using the beam search method through the pre-trained GENIUS model;

[0024] Output layer: outputs new text data.

[0025] Preferably, after constructing the sketch, the process also includes adding a theme or emotional hint to the front end of the sketch.

[0026] Further, the method further comprises the following steps:

[0027] Step C1: construct an implicit toxicity span model, including an input layer, an embedding layer, a bidirectional LSTM layer, a dense layer, a CRF layer, and an output layer connected in sequence;

[0028] Step C2: input the hidden toxic text identified by the text hidden toxicity detection model into the trained hidden toxicity span model, and output the span label;

[0029] Step C3: According to the span label, the seq-to-seq model based on the RoBARTa model rewrites or replaces the toxic content in the hidden toxic text to generate a normalized span label;

[0030] Step C4: Merge the normalized span label with the non-toxic content in the hidden toxic text to generate the detoxified text;

[0031] Step C5: Calculate the toxicity intensity φ of the detoxified text t',

[0032] Rt'=τ-φt';

[0033] L = l + (1-R);

[0034] In the formula, R is the penalty term, τ is the toxicity intensity threshold, L is the comprehensive loss function, and l is the original loss function;

[0035] Step C6: Ask the user whether to publish the detoxified text t';

[0036] If yes, then output the detoxified text t', otherwise output the hidden poison text with * replacing the poisonous content;

[0037] The training process of the implicit toxicity span model includes:

[0038] Input layer: For each first hidden poison text classified by manual data annotation, the range label is manually annotated to generate a text label sequence; the range label includes the start label, continuation label and end label;

[0039] Embedding layer: convert the first hidden toxic text and the corresponding text label sequence into a vector representation and input it into the implicit toxicity span model;

[0040] Bidirectional LSTM layer: The context representation of each first hidden poison text is captured through the bidirectional LSTM to obtain the hidden state representation of each first hidden poison text;

[0041] Dense layer: flattens the hidden state representation of each first hidden text;

[0042] CRF layer: After determining the feature function of the text label sequence based on the flattened hidden state representation, the dependency relationship between labels in each range is modeled in combination with the weight parameter w;

[0043] Map the text label sequence feature function into a high-dimensional feature vector and calculate the conditional probability P(s|x;w) of the text label sequence.

[0044]

[0045] In the formula, x represents the first hidden poisonous text, s represents the text label sequence, and φ(x,s) represents mapping the first hidden poisonous text x and the text label sequence s into a high-dimensional feature vector;

[0046] Output layer: adjust the weight parameter w and use the text label sequence with the highest probability as the span label s of the first hidden poison text * ;

[0047] s * = arg max p(s|x; w * );

[0048] In the formula, w * is the optimal weight parameter, p(s|x; w * ) is the optimal conditional probability of the text label sequence.

[0049] In the technical solution provided by the present invention, the trained text hidden poison detection model can classify the input text into obvious poison text, hidden poison text and non-toxic text, that is, realize the effective identification of hidden poison text. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 The figure is a schematic diagram of the process of identifying hidden viruses in Chinese texts according to the present invention.

[0051] Figure 2 The figure is a schematic diagram of the process of building an enhanced data set in the present invention.

[0052] Figure 3 The figure is a schematic diagram of the process of processing hidden poisonous text in the present invention. DETAILED DESCRIPTION

[0053] Hereinafter, the technical solution provided by the present invention will be further elaborated in detail with reference to the accompanying drawings.

[0054] At present, network text detection mainly relies on technical means such as manual monitoring, convolutional neural network models, and recurrent neural network models. At present, there are good identification methods for explicit toxic language, but the identification of implicit toxic language is still a challenge. In addition, in order to purify the network environment, the field of natural language processing has proposed a text detoxification task, which aims to modify toxic texts to ensure that the toxicity of the text is eliminated while the useful content is intact and the original meaning remains unchanged. In order to solve the problem of the spread of a large amount of toxic information in cyberspace, the present invention proposes a method for identifying hidden toxicity in text based on large models and data enhancement. Specifically, by integrating data enhancement and large model technology, toxic texts in online speech, especially hidden toxicity, are effectively identified, and toxic speech detection is performed from multiple perspectives to help users avoid sending toxic information in a timely manner.

[0055] Specifically, Figure 1 As shown, the technical solution provided by the present invention mainly includes the following five steps.

[0056] Step 1: Text data collection: Collect text data, perform data cleaning and data preprocessing, and data labeling on the collected text data to obtain a data set.

[0057] It includes the following parts:

[0058] Crawling the comment text data of the microblogs with the most likes in Sina Weibo. Performing data cleaning and data preprocessing on the crawled comment data.

[0059] Data cleaning includes deleting duplicate and abnormal data to improve data quality. The main purpose is to ensure the accuracy and completeness of the data and eliminate noise and errors in the data. Data preprocessing includes operations such as word segmentation, sentence segmentation, and removal of stop words.

[0060] Data cleaning is different from data preprocessing: new data problems may be discovered during data preprocessing, which require returning to the data cleaning stage for processing; at the same time, the results of data cleaning may also affect subsequent data preprocessing steps.

[0061] Manual data annotation is performed on the processed data, and the annotation follows the following rules:

[0062] (1) Toxic text: covers a variety of abusive language phenomena, such as hate language, identity attacks, blasphemy or aggression. For example: So-and-so is just uncivilized!

[0063] (2) Hidden toxic text: This type of toxic language may appear harmless on the surface, and usually uses complex language forms such as sarcasm, exaggeration, and rhetorical questions to conceal its true intention. For example: 1, 3, 4, 5, A, C, D, E, but you are missing. This sentence seems harmless, but in fact it links the missing "2" and "B" in "1, 3, 4, 5" and "A, C, D, E" with the object "you", and launches a guessing language attack on "you". This inappropriate statement expression is not only illogical, but also derogatory. This expression implies disrespect and discrimination. Although no offensive words are used directly, its inherent meaning is derogatory.

[0064] (3) Non-toxic text: The content is generally harmless and does not contain any form of abuse or derogatory. Without context or additional information, they may be difficult to understand, but are not inherently offensive or abusive. For example: Huahua is so cute!

[0065] Each text is annotated by multiple people, and the principle of minority obeys majority is adopted for data with different annotation results.

[0066] Step 2: Perform data enhancement on the data annotated in step 1 to obtain a new data set.

[0067] Fine-tuning the data augmentation GeniusAug model, the core module of GeniusAug consists of a bidirectional encoder and an autoregressive decoder. By using perceptual toxicity labels to generate text data in a similar way to the original dataset, the diversity and number of samples in the training set are effectively increased. In this way, the diversity of the data is enhanced, and the classification model can better learn and generalize from a wider range of data.

[0068] Among them, the GeniusAug model is as follows Figure 2 As shown, it includes an input layer, a feature extraction layer, a bidirectional encoding layer, a sketch construction layer, an autoregressive decoding layer and an output layer connected in sequence.

[0069] The data enhancement process is as follows:

[0070] Input layer: Input comment text data d and the corresponding first category label.

[0071] Feature extraction layer: extract the n-grams [w 1 ,w 2 ,...,w m ], n = 1, 2, 3; where m is the number of words in the comment text data d.

[0072] Bidirectional encoding layer: Encode the comment text data d, the first category label and the n-gram to obtain the text embedding e of the comment text data d d 、n-grams[w1 ,w 2 ,...,w m ]'s grammatical embedding [v 1 ,v 2 ,...,v m ], label embedding e of the first category label t .

[0073] Comment text data d , calculate each n-gram and fused embedding e f The similarity S i , the formula is as follows:

[0074] e f =λe d +(1-λ)e t ;

[0075]

[0076] In the formula, λ is the preset fusion weight, and i is the word ordinal number.

[0077] The top 20% of n-grams in similarity are selected as keywords, and this ratio can be fine-tuned according to actual needs.

[0078] Sketch construction layer: Based on the key words, the sketch is constructed by applying an extractive masking projection process.

[0079] Autoregressive decoding layer: Generate new text data based on the sketch using the beam search method through the pre-trained GENIUS model.

[0080] Output layer: outputs new text data.

[0081] During the data augmentation process, the core semantics and outline of the original text are preserved. The data augmentation pre-training task has two significant differences from the previous denoising pre-training objectives: the first is extreme masking, which masks up to 80% of the text; the second is selective masking, which selectively masks the less informative parts of the text based on the sketch extraction pipeline, rather than randomly masking tokens or spans of the original text. After the sketches are completed, new training samples are generated based on these sketches using the pre-trained GENIUS model. In addition, the key parts of the sketch are retained in the generated text to ensure that the text does not deviate significantly from the original text in terms of semantics. To further control the properties of the generated text, thematic or emotional cues can be added before the sketch. The ability to generate content for specific attributes is obtained from the reconstruction process of sketch pre-training.

[0082] Step 3: Build a large model (text hidden virus detection model)

[0083] For text toxicity type detection, the dataset obtained in step 2 is fine-tuned on the RoBERTa pre-trained model. First, a classification head consisting of a fully connected layer and a softmax layer is added to the top layer of the RoBERTa model to output the probability distribution of each category. Then supervised learning is performed on the RoBERTa pre-trained model using the prepared dataset from step 2. Finally, a self-attention layer with a fully connected layer with linear activation is obtained to obtain the toxicity type prediction of the input sample. Usually, the category with the highest probability is selected as the final classification result. After outputting the classification results of toxic speech, the explicit toxic speech is directly shielded, and the original comments are directly output for non-toxic comments. The classification results of implicit toxic speech will be further processed.

[0084] like Figure 3 As shown, further processing of the hidden poison text includes steps 4 and 5.

[0085] Step 4: Build and train an implicit toxicity span model

[0086] The implicit toxicity span model consists of an input layer, an embedding layer, a bidirectional LSTM layer, a dense layer, a CRF layer, and an output layer, which are connected sequentially.

[0087] The training process of the implicit toxicity span model includes:

[0088] Input layer: For each first hidden poison text classified by manual data annotation, the range label (B, I, O) is manually annotated to generate a text label sequence; among which, the range label includes the start label B, the continuation label I and the end label O.

[0089] Embedding layer: Convert the first hidden toxic text and the corresponding text label sequence into a vector representation and input it into the implicit toxic span model.

[0090] Bidirectional LSTM layer: The context representation of each first hidden poison text is captured through a bidirectional LSTM (Long Short-Term Memory) to obtain the hidden state representation of each first hidden poison text.

[0091] Temporally distributed dense layers: Flatten the hidden state representation of each first hidden context.

[0092] CRF (Conditional Random Field) layer: After determining the text label sequence feature function based on the flattened hidden state representation, the dependency relationship between labels in each range is modeled in combination with the weight parameter w.

[0093] Map the text label sequence feature function to a high-dimensional feature vector (d dimension), and calculate the conditional probability P(s|x;w) of the text label sequence.

[0094]

[0095] In the formula, x represents the first hidden poisonous text, s represents the text label sequence, and φ(x,s) represents mapping the first hidden poisonous text x and the text label sequence s into a high-dimensional feature vector;

[0096] The range of the text label sequence s spans different sequences. The most likely label for a sentence can be found as follows: The toxic part of a sentence refers to the part of the sentence that is responsible for conveying "toxicity". If these toxic contents can be normalized (i.e., the text is detoxified), sentence-level normalization can be achieved. A sample can have multiple non-overlapping toxic ranges, and the goal of the implicit toxicity span is to identify all toxic ranges.

[0097] Output layer: adjust the weight parameter w and use the text label sequence with the highest probability as the span label s of the first hidden poison text * ;

[0098] s * = arg max p(s|x; w * );

[0099] In the formula, w * is the optimal weight parameter, p(s|x; w * ) is the optimal conditional probability of the text label sequence.

[0100] Step 5: Build the detox module

[0101] The detoxification model is a Generative Adversarial Networks (GAN) based architecture. The model adopts steps 3 and 4 as auxiliary modules. The detoxification module accepts toxic samples along the span labels identified by the implicit toxic span model. For the detoxification module, the pre-trained RoBARTa model is used and fine-tuned for the detoxification task.

[0102] According to the span label, the seq-to-seq model based on the RoBARTa model rewrites or replaces the toxic content in the hidden toxic text to generate a normalized span label;

[0103] Merge the normalized span labels with the non-toxic content in the hidden toxic text to generate the detoxified text;

[0104] The toxicity intensity φ of the detoxified text t' is calculated through the discriminator model of the text hidden poison detection model.

[0105] Rt'=τ-φt';

[0106] L = l + (1-R);

[0107] In the formula, R is the penalty term, τ is the toxicity intensity threshold, L is the comprehensive loss function, and l is the original loss function;

[0108] Ask the user whether to publish the detoxified text t';

[0109] If yes, then output the detoxified text t', otherwise output the hidden poison text with * replacing the poisonous content.

[0110] If the toxicity intensity φ of the detoxified text t' is still greater than the preset intensity, it can be repeatedly rewritten or replaced, and the comprehensive loss can be minimized through back propagation to generate semantically coherent normalized samples with a hidden toxicity intensity less than or equal to 1. The purpose of detoxification is to generate new samples that retain the original semantics but have lower toxicity. Intuitively, it is first necessary to determine the threshold between strong and weak toxicity. Set the threshold τ=5. For each strongly toxic sample with φt'>τ, the goal is to generate semantically similar samples with the constraint that φt'≤τ.

[0111] It can be seen that in the data enhancement module design of the present invention, the data enhancement method based on the bidirectional encoder and the autoregressive decoder is innovative, which can effectively increase the diversity and quantity of the review text data and save the cost of manual annotation.

[0112] Fine-tuning and building on the RoBERTa pre-trained model and bidirectional LSTM model respectively, we built a text hidden toxicity detection model and an implicit toxicity span model to predict the toxicity intensity of the text and identify toxic spans in the text. Among them, the method of fine-tuning based on the pre-trained model and model combination is innovative and can improve the performance of the model in toxicity detection and span identification tasks.

[0113] The detoxification model based on generative adversarial network is designed to remove the text toxicity module, accept toxic samples along the span labels identified by the implicit toxic span model, and generate new samples that retain the original semantics but have lower toxicity. Among them, the detoxification model design based on GAN is innovative and can effectively generate non-toxic text that meets the semantic requirements.

[0114] In summary, in the technical solution provided by the present invention, the trained text hidden poison detection model can classify the input text into obvious poison text, hidden poison text and non-toxic text, that is, realize the effective identification of hidden poison text.

[0115] Furthermore, on the basis of achieving the above beneficial effects, each preferred scheme also achieves the following beneficial effects: data preprocessing is conducive to improving the accuracy of subsequent data standards; data enhancement can effectively increase the diversity and quantity of comment text data while saving manual annotation costs; generating span tags and detoxifying based on span tags can realize the detoxification of hidden poisonous text and generate non-toxic text that meets semantic requirements.

Claims

1. A method for identifying hidden viruses in text based on a large model, characterized in that: The following steps are involved: Step S1; Build a text hidden poison detection model by adding a classification head consisting of a fully connected layer and a softmax layer on the top layer of the RoBERTa model; Step S2: crawling comment text data from social software and performing data cleaning, data preprocessing and manual data annotation; Manual data annotation specifically involves: using classification labels to classify comment text data into explicit toxic text, implicit toxic text, or non-toxic text; Step S3: Input the manually annotated comment text data into the text hidden virus detection model for supervised learning; Step S4: input the text to be identified into the supervised learning text hidden virus detection model, and output the identification classification result; Among them, the classification results include obviously poisonous text, latently poisonous text and non-toxic text.

2. A method for identifying hidden viruses in text based on a large model as described in claim 1, characterized in that: The data preprocessing includes word segmentation, sentence segmentation and removal of stop words.

3. A method for identifying hidden viruses in text based on a large model as described in claim 1, characterized in that: Before step S3, the review text data is also enhanced by the GeniusAug model; The GeniusAug model includes an input layer, a feature extraction layer, a bidirectional encoding layer, a sketch construction layer, an autoregressive decoding layer, and an output layer connected in sequence; The data enhancement process is as follows: Input layer: input comment text data d and the corresponding first category label; Feature extraction layer: extract the n-grams [w1,w2,...,w m ], n = 1, 2, 3; where m is the number of words in the comment text data d; Bidirectional encoding layer: Encode the comment text data d, the first category label and the n-gram to obtain the text embedding e of the comment text data d d 、n-gram [w1,w2,...,w m ] is a grammatical embedding [v1,v2,...,v m ], label embedding e of the first category label t ; For the review text data d, calculate each n-gram and fusion embedding e f The similarity S i , the formula is as follows: e f =λe d +(1-λ)e t ; In the formula, λ is the preset fusion weight, i is the word ordinal number; Select the top A% of n-grams as keywords; Sketch construction layer: Based on the key words, the sketch is constructed by applying the extraction masking projection process; Autoregressive decoding layer: Generates new text data based on the sketch using the beam search method through the pre-trained GENIUS model; Output layer: outputs new text data.

4. A method for identifying hidden viruses in text based on a large model as described in claim 3, characterized in that: After constructing the sketch, it also includes adding a theme or emotional hint to the front of the sketch.

5. The method for identifying hidden viruses in text based on a large model as described in claim 1, characterized in that: The following steps are also included: Step C1: construct an implicit toxicity span model, including an input layer, an embedding layer, a bidirectional LSTM layer, a dense layer, a CRF layer, and an output layer connected in sequence; Step C2: input the hidden toxic text identified by the text hidden toxicity detection model into the trained hidden toxicity span model, and output the span label; Step C3: According to the span label, the seq-to-seq model based on the RoBARTa model rewrites or replaces the toxic content in the hidden toxic text to generate a normalized span label; Step C4: Merge the normalized span label with the non-toxic content in the hidden toxic text to generate the detoxified text; Step C5: Calculate the toxicity intensity φ of the detoxified text t', Rt'=τ-φt'; L = l + (1-R); In the formula, R is the penalty term, τ is the toxicity intensity threshold, L is the comprehensive loss function, and l is the original loss function; Step C6: Ask the user whether to publish the detoxified text t'; If yes, then output the detoxified text t', otherwise output the hidden poison text with * replacing the poisonous content; The training process of the implicit toxicity span model includes: Input layer: For each first hidden poison text classified by manual data annotation, a range label is manually annotated to generate a text label sequence; the range label includes a start label, a continuation label, and an end label; Embedding layer: convert the first hidden toxic text and the corresponding text label sequence into a vector representation and input it into the implicit toxicity span model; Bidirectional LSTM layer: The context representation of each first hidden poison text is captured through the bidirectional LSTM to obtain the hidden state representation of each first hidden poison text; Dense layer: flattens the hidden state representation of each first hidden text; CRF layer: After determining the feature function of the text label sequence based on the flattened hidden state representation, the dependency relationship between labels in each range is modeled in combination with the weight parameter w; Map the text label sequence feature function to a high-dimensional feature vector and calculate the text label sequence conditional probability P(sx;w). In the formula, x represents the first hidden poisonous text, s represents the text label sequence, and φ(x,s) represents mapping the first hidden poisonous text x and the text label sequence s into a high-dimensional feature vector; Output layer: adjust the weight parameter w and use the text label sequence with the highest probability as the span label s of the first hidden poison text * ; s * =arg max p(s|x;w * ); In the formula, w * is the optimal weight parameter, p(s|x; w * ) is the optimal conditional probability of the text label sequence.

Images