Attack speech detection method based on RoBERTa and multi-granularity context enhancement

Abstract

The invention belongs to the technical field of language detection, and particularly relates to an aggressive speech detection method based on RoBERTa and multi-granularity context enhancement, which comprises the following steps: acquiring original aggressive speech data from a social media platform or a public data set, and performing data cleaning on the original aggressive speech data; performing adversarial data enhancement based on significance on the cleaned data; inputting the enhanced data into a pre-trained RoBERTa model to carry out global semantic coding; performing multi-granularity context feature extraction; and carrying out feature fusion and classification judgment on the extracted features. The method comprises the following steps: capturing global macroscopic semantics of a text by utilizing a powerful self-attention mechanism of RoBERTa; a bidirectional gating loop unit is utilized in parallel to perform secondary modeling on a sequence, a forward and backward time sequence dependency relationship and a local grammar structure between words are specially captured, and the detection precision of a long text and a complex sentence pattern is remarkably improved through'global + local 'feature complementation.

Description

Technical Field

[0001] This invention belongs to the field of language detection technology, specifically relating to a method for detecting offensive speech based on RoBERTa and multi-granularity context enhancement. Background Technology

[0002] With the development of the internet and social media, online platforms have become breeding grounds for offensive rhetoric. Offensive rhetoric typically refers to content expressing prejudice or hatred towards a specific group. The Chinese online environment is complex, and offensive rhetoric often evades moderation through euphemisms, mixed pinyin, and homophone substitution, making manual moderation inefficient and posing a significant challenge.

[0003] Existing technologies struggle to achieve a deep fusion of global semantic features and local sequence features. While existing deep learning models (such as BERT and RoBERTa) possess a certain semantic understanding capability, they often struggle to simultaneously grasp the overall macroscopic control of the entire text and accurately capture fine-grained local sequence information when dealing with complex Chinese linguistic phenomena. The lack of an effective mechanism to deeply fuse the general semantic knowledge of pre-trained models with local features specific to sequence dependencies (such as the word combination rules in Chinese) limits the accuracy and efficiency of models in detecting complex and cryptic offensive language. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a method for detecting offensive speech based on RoBERTa and multi-granularity context enhancement, comprising:

[0005] S1: Obtain raw offensive speech data from social media platforms or public datasets, and perform data cleaning on the raw offensive speech data;

[0006] S2: Perform saliency-based adversarial data augmentation on the cleaned data;

[0007] S3: Input the augmented data into the pre-trained RoBERTa model and perform global semantic encoding to obtain a hidden state sequence containing the semantic information of the entire sequence;

[0008] S4: Extract multi-granularity contextual features from the hidden state sequence containing the semantic information of the entire sequence;

[0009] S5: Perform feature fusion and classification decisions on the extracted features.

[0010] The beneficial effects of this invention are:

[0011] This invention accurately locates the key "sensitive words" that have the greatest impact on the model's judgment by calculating gradient significance, rather than blindly and randomly replacing them; it introduces a specific ratio of homophone replacement and character shape replacement strategies to specifically simulate the censorship avoidance methods in Chinese social media, which greatly improves the robustness of the model when facing spelling errors and variant attacks.

[0012] This invention utilizes RoBERTa's powerful self-attention mechanism to capture the global macro-semantics of text; it also uses bidirectional gated recurrent units to perform secondary modeling of the sequence in parallel, specifically capturing the forward and backward temporal dependencies and local grammatical structures between words. Through the complementary use of "global + local" features, the model can understand the macro-intention of the entire passage and also keenly capture aggressive cues in local phrases, significantly improving the detection accuracy for long texts and complex sentences.

[0013] This invention introduces a multi-head attention mechanism after the Bi-GRU layer. This structure gives the model the ability to "dynamically focus" and can automatically calculate the weights at different time steps, focusing attention on the key segments that contribute the most to the classification. Even if the aggressive expression is very subtle or scattered, the model can highlight it through attention weights, thereby effectively solving the problem of missing implicit aggressive speech. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating an offensive speech detection method based on RoBERTa and multi-granularity context enhancement according to the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] An offensive speech detection method based on RoBERTa and multi-granularity context enhancement is proposed. It primarily runs on computer systems equipped with high-performance GPUs (such as NVIDIA RTX 3090 or Tesla V100). The software environment is based on Python 3.8 and the deep learning framework PyTorch 1.10 or later, and utilizes the Transformers library, such as... Figure 1 As shown, it includes:

[0017] S1: Obtain raw offensive speech data from social media platforms or public datasets, and perform data cleaning on the raw offensive speech data;

[0018] S2: Perform saliency-based adversarial data augmentation on the cleaned data;

[0019] S3: Input the augmented data into the pre-trained RoBERTa model and perform global semantic encoding to obtain a hidden state sequence containing the semantic information of the entire sequence;

[0020] S4: Extract multi-granularity contextual features from the hidden state sequence containing the semantic information of the entire sequence;

[0021] S5: Perform feature fusion and classification decisions on the extracted features.

[0022] Data Acquisition: Obtain raw offensive speech data from social media platforms (such as Weibo and Zhihu) or public datasets (such as the COLD dataset).

[0023] Text cleaning: Input the raw data into the preprocessing module.

[0024] Use regular expressions to remove URL links, HTML tags, and invisible characters from text.

[0025] Remove non-semantic special symbols and emojis, keeping only Chinese characters, English letters, and necessary punctuation marks.

[0026] Preliminary word segmentation was performed using a Chinese word segmentation tool (such as Jieba) for subsequent data augmentation analysis, but the original sentence sequence structure was preserved for RoBERTa input.

[0027] To improve the model's robustness to implicit Chinese expressions and adversarial examples, this embodiment performs augmentation on the preprocessed data. The augmentation process consists of two stages: "vocabulary selection" and "replacement generation."

[0028] Vocabulary selection stage:

[0029] Saliency-based word selection: For 70% of the samples in the dataset, a gradient saliency calculation method is used. Specifically, the text is input into the base BERT model, and the gradient of the loss function with respect to the input word embeddings is calculated. The larger the magnitude of the gradient, the greater the influence (saliency) of the word on the model's judgment result. The system selects the top $k$ words with the highest saliency (e.g., the top 15% of words) as "vulnerable points" for replacement.

[0030] Random Word Selection: For the remaining 30% of the samples, a random strategy is adopted to randomly select non-stop words in the sentence as the words to be replaced, in order to maintain the diversity of data distribution.

[0031] Replacement Generation Phase:

[0032] For the words selected in the above steps, execute three replacement strategies according to the following probability distribution to generate Augmented Data:

[0033] Synonym Substitution (40%): Query the Chinese Thesaurus (such as Cilin), and replace the selected word with a semantically similar word to enhance the model's generalization ability for semantic diversity.

[0034] Homophone Substitution (30%): Use a pinyin conversion library (such as pypinyin) to find Chinese characters with the same or similar pinyin as the selected word for replacement (for example, replace the homophone variant of "尼" with "你") to simulate online homophone attacks.

[0035] Shape-Based Substitution (30%): Based on the Chinese character glyph structure database, find Chinese characters with similar glyphs to the selected word for replacement (for example, "博" and "搏") to simulate near-shape character confusion attacks.

[0036] Input the Augmented Data obtained in step S2 into the pre-trained RoBERTa model for feature encoding:

[0037] Input Embedding: Convert the text into a Token sequence through the Tokenizer, add [CLS] and [SEP] tokens, and map it to the sum of word embeddings, positional encodings, and segment encodings.

[0038] Transformer Encoding: The data enters the 12-layer Transformer Encoder of RoBERTa. Each layer contains a multi-head self-attention mechanism, residual connections and layer normalization (Add & Norm), and a feed-forward neural network.

[0039] Output Obtaining: The model outputs a sequence of hidden states containing the semantic information of the entire sequence , where L is the sequence length and d is the hidden layer dimension (e.g., 768 dimensions).

[0040] To address the shortcomings of existing technologies in understanding local context and implicit semantics, this embodiment divides RoBERTa's output into two parallel processing paths:

[0041] Path 1: Local Context Augmentation (Bi-GRU)

[0042] Bi-GRU modeling: The sequence vector output by RoBERTa is input into a bidirectional gated recurrent unit (Bi-GRU).

[0043] The Forward GRU reads the sequence from left to right, capturing the preceding information;

[0044] The Backward GRU reads the sequence from right to left, capturing the following information.

[0045] The forward hidden state at each time step and backward hidden state Concatenation yields contextual features that include temporal dependencies. = [ ; ].

[0046] Multi-Head Attention: The features from all time steps output by the Bi-GRU are input into a multi-head attention layer. This layer automatically learns the weight (AttentionScore) for each word in determining whether a sentence is aggressive, and then performs a weighted summation of the features from all time steps based on these weights to obtain a locally aggregated feature vector focused on key aggressive segments. (Aggregated Output).

[0047] Path 2: Global Feature Extraction (Pooling Layer)

[0048] The output data from RoBERTa is processed through a pooling layer. This embodiment employs mean pooling or directly extracts the vector corresponding to the [CLS] marker to obtain a global feature vector representing the overall semantics of the sentence. .

[0049] Feature fusion and classification decision:

[0050] Linear concatenation: Combines the local enhanced features obtained from path one. Global features obtained from path two Perform dimensional concatenation to obtain the fused feature vector. = ; This operation allows the model to have both an understanding of the overall macro-context of the sentence and retain sensitivity to local, subtle aggressive cues.

[0051] Classification output: fused features The input is a fully connected layer (Linear), which is mapped to a category space (e.g., two-dimensional: aggressive / non-aggressive).

[0052] Probability calculation: The output is converted into a probability distribution using the Softmax function, and the final offensive detection results are output.

[0053] Model training configuration:

[0054] In practice, to enable those skilled in the relevant technical field to reproduce the results, the specific training parameter settings are explained below:

[0055] Loss function: Cross-entropy loss is used.

[0056] Optimizer: AdamW optimizer is used.

[0057] Learning rate: The RoBERTa layer is set to Bi-GRU and classification layer are set as .

[0058] Batch Size: Set to 32 or 64.

[0059] Dropout: Set Dropout to 0.1 before the fully connected layer to prevent overfitting.

[0060] Training epochs: Set 5-10 epochs and adopt the Early Stopping strategy, stopping training when the F1 score on the validation set no longer improves.

[0061] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for detecting offensive speech based on RoBERTa and multi-granularity context enhancement, characterized in that, include: S1: Obtain raw offensive speech data from social media platforms or public datasets, and perform data cleaning on the raw offensive speech data; S2: Perform saliency-based adversarial data augmentation on the cleaned data; S3: Input the augmented data into the pre-trained RoBERTa model and perform global semantic encoding to obtain a hidden state sequence containing the semantic information of the entire sequence; S4: Extract multi-granularity contextual features from the hidden state sequence containing the semantic information of the entire sequence; S5: Perform feature fusion and classification decisions on the extracted features.

2. The offensive speech detection method based on RoBERTa and multi-granularity context enhancement according to claim 1, characterized in that, Data cleaning of raw offensive rhetoric data, including: S11: Use regular expressions to remove URL links, HTML tags, and invisible characters from text; S12: Remove special symbols and emojis without semantic meaning, and retain only Chinese characters, English letters and necessary punctuation marks; S13: Use a Chinese word segmentation tool for preliminary word segmentation, but retain the original sentence sequence structure.

3. The offensive speech detection method based on RoBERTa and multi-granularity context enhancement according to claim 1, characterized in that, The cleaned data is subjected to saliency-based adversarial data augmentation, including: word selection and replacement generation stages; Vocabulary selection stage: For 70% of the samples in the dataset, the gradient significance calculation method is adopted. The specific operation is as follows: input the text into the base BERT model, calculate the gradient of the loss function with respect to the word embedding. The larger the magnitude of the gradient, the greater the significance of the word to the model's judgment result. The system selects the top n words with the highest significance as "vulnerable points" for replacement. Random word selection: For the remaining 30% of the samples, a random strategy is adopted to randomly select non-stop words in the sentences as words to be replaced, in order to maintain the diversity of data distribution; Replacement generation phase: For the words selected in the word selection phase, three replacement strategies are executed according to the following probability distribution to generate augmented data: Synonym replacement: Query the Chinese thesaurus and replace the selected word with a word that has a similar meaning, thereby enhancing the model's ability to generalize to semantic diversity; Homophone replacement: Using a pinyin conversion library, find Chinese characters with the same or similar pinyin to the selected word and replace them, simulating a network homophone attack; Character replacement: Based on a database of Chinese character structure, find Chinese characters with similar shapes to the selected word and replace them, simulating a similar-shaped character confusion attack.

4. The offensive speech detection method based on RoBERTa and multi-granularity context enhancement according to claim 1, characterized in that, Augmented data is input into a pre-trained RoBERTa model and globally semantically encoded to obtain a hidden state sequence containing semantic information for the entire sequence, including: Input embedding: The text is converted into a token sequence using a tokenizer, [CLS] and [SEP] tags are added, and it is mapped to a weighted sum of word embeddings, positional encodings, and paragraph encodings; Transformer Encoding: Data enters RoBERTa's 12-layer Transformer encoder. Each layer includes a multi-head self-attention mechanism, residual connections and layer normalization, as well as a feedforward neural network. The model outputs a hidden state sequence containing the semantic information of the entire sequence. Where L is the sequence length, For the hidden layer dimension, Represents a real number vector.

5. The offensive speech detection method based on RoBERTa and multi-granularity context enhancement according to claim 1, characterized in that, Multi-granularity contextual feature extraction is performed on the hidden state sequence containing the semantic information of the entire sequence, including: local context enhancement and global feature extraction; Local context enhancement: Bi-GRU modeling: transforming the sequence vector output by RoBERTa The input is a bidirectional gated recurrent unit (GRU). The forward GRU reads the sequence from left to right, capturing the preceding context, while the backward GRU reads the sequence from right to left, capturing the following context. The forward hidden state at each time step is then recorded. and backward hidden state Concatenation yields contextual features that include temporal dependencies. = [ ; ]; Attention Aggregation: The features from all time steps output by the Bi-GRU are input into a multi-head attention layer. This layer automatically learns the weight of each word in determining whether a sentence is aggressive, and then performs a weighted summation of the features from all time steps based on these weights to obtain a locally aggregated feature vector focused on key aggressive segments. ; Global feature extraction: The data output by RoBERTa By extracting the vector corresponding to the [CLS] tag through pooling layers, a global feature vector representing the overall semantics of the sentence is obtained. .

6. The offensive speech detection method based on RoBERTa and multi-granularity context enhancement according to claim 1, characterized in that, Feature fusion and classification decisions are performed on the extracted features, including: Local enhancement feature vectors With global feature vectors Perform dimensional concatenation to obtain the fused feature vector. = ; ; Fusion features Input the fully connected layer and map it to the category space; The output of the fully connected layer is converted into a probability distribution using the Softmax function, which outputs the final attack detection result.

Images