A social network aggressive speech detection method based on multi-task learning

Abstract

The application discloses a social network attackive speech detection method based on multi-task learning, constructs an attackive speech dictionary, collects a large number of tweet data in a microblog platform, carries out strict manual marking on part of the data according to formulated marking rules, and constructs a relatively perfect Chinese attackive speech data set; the collected sample data are analyzed, statistical features including three categories of emotional features, content features and propagation features are extracted; and an attackive speech detection model based on multi-task learning is constructed, an auxiliary task emotional analysis is introduced, and the detection effect of the attackive speech detection model is improved by using the high correlation between the two tasks. The effect of the model provided by the application on the attackive speech detection problem is better than that of other commonly used detection methods, and the model provides a method and a train of thought for future attackive speech detection facing social networks.

Description

Technical Field

[0001] This invention relates to the field of network security technology in computer science and technology, specifically to a method for detecting offensive speech on social networks based on multi-task learning. Background Technology

[0002] With the rapid development of internet and mobile internet technologies, the way people communicate is quietly changing, and the use of social networks is growing exponentially. People are keen to obtain information, express opinions, and share their stances and viewpoints online. Due to the widespread availability of the internet, various social networking platforms are constantly being updated and iterated. For people, the channels for sharing and obtaining information have become more extensive and the cost has become lower. The influence of social networks has also increased accordingly, even rivaling that of traditional media. Many online platforms, both domestically and internationally, boast massive user bases.

[0003] However, while social networks serve as a special space for expressing opinions and feelings, they have also been used to convey hostility and publish other harmful remarks. The rapid dissemination of information provides a breeding ground for the generation and spread of harmful speech on social networks. Social networks enable the widespread, viral spread of harmful speech, with extreme forms of verbal violence proliferating online. Offensive speech, in particular, is rampant and significantly pollutes the online environment. Therefore, the rational use of social media data to research and develop deep learning models for detecting offensive speech and preventing its abuse is crucial for curbing the spread of online chaos, building a clean cyberspace, and maintaining a harmonious internet ecosystem. This is of great significance to individuals, platforms, and the nation.

[0004] Most existing research on offensive speech detection is based on English datasets and focuses primarily on foreign social media platforms, lacking research on publicly available Chinese datasets. Existing research on Chinese is limited to the detection and rewriting of profanity, neglecting the detection of offensive speech. Therefore, this invention proposes a multi-task learning-based method for detecting offensive speech on social networks. Summary of the Invention

[0005] To address the aforementioned problems, the present invention aims to provide a method for detecting aggressive speech on social networks based on multi-task learning. This method introduces external knowledge through multi-task learning, uses relevant auxiliary tasks such as sentiment analysis and its datasets to assist in training the main task of aggressive speech detection, and incorporates statistical features into the model. This approach can effectively detect aggressive speech on social network platforms. The technical solution is as follows:

[0006] A method for detecting offensive speech on social networks based on multi-task learning includes the following steps:

[0007] Step 1: Data Collection and Labeling

[0008] We selected specific social networking platforms as data sources, constructed an offensive dictionary using offensive-related terms, used web crawlers to obtain data based on the keywords in the dictionary, and completed the data annotation.

[0009] Step 2: Feature Extraction

[0010] The features of offensive remarks are analyzed and extracted, and feature vectors are generated for each offensive remark, including semantic feature vectors and statistical feature vectors. The context-related semantic feature vectors of the tweets are generated by training the BERT Chinese pre-trained model. The statistical feature vectors are generated by extracting statistical features from three categories: sentiment features, content features, and dissemination features, and then concatenating them.

[0011] Step 3: Construct the detection model

[0012] The MMoE (Multi-gate Mixture-of-Experts) multi-task learning framework is adopted, with offensive speech detection as the main task and sentiment analysis as the auxiliary task, and offensive speech on the social network platform is detected simultaneously using BERT (Bidirectional Encoder Representation from Transformers), Bi-LSTM (Bi-Directional Long Short-Term Memory) and attention mechanisms.

[0013] Furthermore, step 1 specifically includes:

[0014] Step 1.1: Construction of the offensive dictionary

[0015] Based on the Hurtlex corpus, Chinese sensitive word database, Badwords, and Chinese-English sensitive word database, an offensive dictionary was created.

[0016] Step 1.2: Web crawler implementation

[0017] A web crawler was written in Python, using the Requests and lxml libraries to handle accessing network resources and parsing HTML documents, respectively. The Requests library was used to simulate login and save session data, thereby obtaining tweet data from the social network platform.

[0018] Step 1.3: Data Labeling

[0019] The dataset is labeled according to the definition of offensive speech, thereby constructing an offensive speech dataset.

[0020] Furthermore, the extraction of statistical features in step 2 specifically includes:

[0021] Step 2.1: Extraction of emotional features

[0022] 1) Sentiment Value: The existing application programming interface is used to perform sentiment analysis on the tweets. The sentiment parameter in the result indicates the sentiment polarity classification result, where 0 indicates negative sentiment polarity, 1 indicates neutral sentiment polarity, and 2 indicates positive sentiment polarity.

[0023] 2) Number of positive / negative emojis: The platform's built-in emojis are automatically converted into emoji names, and the third-party Python library emoji is used to convert emoji symbols into words with similar meanings; then the number of positive / negative emojis in the tweet is calculated and used as one of the characteristics to determine whether the tweet is aggressive;

[0024] Step 2.2: Extraction of content features

[0025] 1) Calculate the proportion of question marks and exclamation marks among all punctuation marks in a tweet, and then use this as one of the characteristics to determine whether a tweet is offensive;

[0026] 2) Determine whether a tweet is tagged with a location, using this as one of the characteristics to determine whether a tweet is offensive;

[0027] Step 2.3: Extraction of propagation features

[0028] The number of likes, comments, and retweets are used as one of the characteristics to determine whether a tweet is offensive.

[0029] Furthermore, the detection model in step 3 includes an input layer, an emotion knowledge sharing layer, a statistical feature extraction module, a feature concatenation module, an attention layer, and an output layer;

[0030] 1) Input layer

[0031] The input layer consists of two parts. The first part is designed for offensive speech detection and sentiment analysis tasks, and it takes the pre-processed tweet text... As input, it is tokenized using BERT model tools to obtain a vector. The first part is to make its format conform to the input of BERT; the second part is for the task of detecting offensive speech. First, the relevant information of the target tweet is processed to extract three types of statistical features, including sentiment features, content features and dissemination features. Then, it is fed into the BP neural network for learning and is transformed into a statistical feature vector with the same dimension as the semantic features.

[0032] 2) Emotional Knowledge Sharing Layer

[0033] The sentiment knowledge sharing layer takes offensive speech detection as the main task and sentiment analysis as the auxiliary task. It uses the MMoE multi-task learning framework to fuse BERT and Bi-LSTM to obtain the semantic feature vector of the tweet. The multi-task learning framework adopts a shared underlying structure, and different tasks share the underlying hidden layer.

[0034] The sentiment knowledge sharing layer includes a shared word embedding module and a shared semantic feature extraction module;

[0035] The shared word embedding module uses the BERT model as a tweet word embedding extractor, which is used to train and generate word embeddings that are suitable for both the main task of offensive speech detection data and the auxiliary task of sentiment analysis data.

[0036] The shared semantic feature extraction module uses multiple feature extraction units composed of Bi-LSTM networks and uses a gated attention mechanism to fuse the features to obtain the final semantic feature vector, thereby extracting the same features between the main task and the auxiliary task.

[0037] 3) Statistical Feature Extraction Module

[0038] For the task of detecting offensive speech, the statistical feature extraction module concatenates the sentiment features, content features, and propagation features extracted from the tweet to obtain the statistical features of the target tweet. After normalization and regularization, the statistical features are fed into the backpropagation neural network to learn the implicit relationships between the features. The backpropagation neural network also transforms the statistical features into vectors with the same dimension as the semantic features, thus providing an equal amount of feature information for the detection of offensive speech.

[0039] 4) Feature splicing module

[0040] The feature concatenation module is designed for offensive speech detection tasks. It concatenates the extracted semantic features with statistical characteristics to obtain the feature vector of the tweet.

[0041] 5) Attention layer

[0042] For the offensive speech detection task, the attention layer feeds the feature vector obtained by the feature concatenation module into the attention mechanism.

[0043] 6) Output layer

[0044] The output layer outputs the detection result of whether the target tweet contains offensive language.

[0045] Furthermore, the process of detecting offensive speech in step 3 includes:

[0046] Step 3.1: Preprocessed tweet text The vector is obtained after tokenization. As shown in the following formula:

[0047] (1);

[0048] Step 3.2 Input vector E into the BERT model to obtain word embedding representations. .in, For dimension The vector, For word sequence length, The word vector dimension is shown in the following formula:

[0049] (2);

[0050] Step 3.3: Embed the extracted words The input is fed into a Bi-LSTM-based feature extraction unit to obtain the output of the expert network. , , The number of hidden nodes in the network. The calculation method is as follows:

[0051] (3);

[0052] (4);

[0053] (5);

[0054] in, Indicates the bidirectional LSTM's first... The implicit vector generated by each unit Word embedding representation The elements in It is the hidden vector generated by the previous state of the forward LSTM. It is the hidden vector generated by the previous state of the backward LSTM. This indicates vector concatenation;

[0055] Step 3.4: The gating network uses the input word embeddings and the Softmax activation function to generate different expert combination weights, thereby allowing different tasks to utilize the results of the expert network with different weights, as shown in the following equation:

[0056] (6);

[0057] in, This represents an updatable parameter matrix. This indicates a specific door, corresponding to different tasks. This indicates the weights of the expert network used for a specific task;

[0058] Step 3.5: Apply the generated weights to the output of the expert network, as shown in the following formula:

[0059] (7);

[0060] in, This indicates the number of expert networks, i.e., the number of feature extraction units. This represents the semantic feature vector corresponding to a specific task. This represents the output of a specific task after passing through an expert network. This represents the weights of different feature extraction units in an expert network used for a specific task. This represents the output of different feature extraction units in the expert network;

[0061] Step 3.6: Vectorize the statistical features extracted from the tweets, and then perform regularization and normalization on them. Concatenate them into a one-dimensional vector, as shown in the following formula:

[0062] (8);

[0063] (9);

[0064] in, Indicates emotional characteristics, Indicates content characteristics, Indicates the characteristics of propagation. Indicates splicing characteristics, This represents regularization and normalization. This represents the result after regularization and normalization of the spliced ​​features;

[0065] The processed statistical features are fed into a backpropagation (BP) neural network, and transformed into a statistical feature vector with the same dimension as the semantic features through its fully connected layers, as shown in the following equation:

[0066] (10);

[0067] in, Represents the parameter matrix, Indicates the bias term. S represents the activation function, and S is the statistical feature vector.

[0068] Step 3.7: Concatenate semantic feature vectors With statistical eigenvectors Obtain the feature vector of the target tweet As shown in the following formula:

[0069] (11);

[0070] Step 3.8: Calculate the relevance between the query and the key to obtain the attention score, as shown in the following formula:

[0071] (12);

[0072] in, Represents the parameter matrix; This represents the attention score. Feature vectors representing specific tasks;

[0073] Then, the attention scores are normalized using the Softmax function to obtain the weight coefficients. As shown in the following formula:

[0074] (13);

[0075] in, express The intermediate hidden layer vectors, i.e., the attention scores mentioned above, It is a randomly initialized vector that is automatically updated during training. express Transpose of;

[0076] Finally, the values ​​are weighted and summed according to the weighting coefficients to obtain the final feature representation, as shown in the following formula:

[0077] (14);

[0078] In this case, the key and value are the same, both being [key and value]. ; This represents the feature representation of the final output;

[0079] Step 3.9: For the offensive speech detection task, the semantic features and statistical features are concatenated and weighted using an attention mechanism. The probability that the target tweet is offensive speech is then obtained through the Sigmoid activation function and used as the final classification result, as shown in the following formula:

[0080] (15);

[0081] in, The probability that the targeted tweet is offensive.

[0082] For sentiment analysis tasks, the final classification result is obtained directly through the Sigmoid activation function, as shown in the following formula:

[0083] (16);

[0084] in, This represents the final classification result obtained by applying the Sigmoid activation function to the target tweet in the sentiment analysis task. This represents the semantic feature vector of the target tweet in a sentiment analysis task.

[0085] Furthermore, the text sequence input to the input layer begins with [CLS] and [SEP] is used as the sentence segment. The resulting BERT input representation is obtained by adding three parts: target word embedding, segment embedding, and position embedding. Among them, the target word embedding is a vector that is independent of the context representation; the segment embedding is used to distinguish whether the input text sequence is a single sentence or multiple sentences; and the position embedding is used to represent the position information of the corresponding words in the word sequence.

[0086] Furthermore, the training objective of the detection model is to minimize the loss function, and the cross-entropy function is chosen as the loss function, which is defined as follows:

[0087] (17);

[0088] in, For label values, For predicted values, This represents the total number of categories. This represents the value of the loss function.

[0089] The beneficial effects of this invention are as follows: This invention constructs an offensive speech dictionary, collects a large amount of tweet data from online platforms, and performs rigorous manual annotation on some data according to established annotation rules, thus constructing a relatively complete Chinese offensive speech dataset; it analyzes the collected sample data to extract statistical features including three categories: sentiment features, content features, and dissemination features; and it constructs an offensive speech detection model based on multi-task learning, introduces auxiliary task sentiment analysis, and utilizes the high correlation between the two tasks to improve the detection effect of the offensive speech detection model; the model proposed in this invention outperforms other commonly used detection methods in the problem of offensive speech detection, providing methods and ideas for future offensive speech detection on social networks. Attached Figure Description

[0090] Figure 1 This is a diagram illustrating the overall framework of the social network offensive speech detection method based on multi-task learning according to the present invention.

[0091] Figure 2 This is a structural diagram of the MBBA (MMoE+BERT+Attention) model of this invention.

[0092] Figure 3 This is a comparison chart of ablation results using the MBBA model of this invention.

[0093] Figure 4 A comparison chart showing the performance of different word embedding models.

[0094] Figure 5 A comparison chart showing the performance of different detection models and the MBBA model. Detailed Implementation

[0095] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0096] The framework for inventing a multi-task learning-based method for detecting offensive speech on social networks mainly consists of three parts: data collection and annotation, feature extraction, and detection model, such as... Figure 1 As shown.

[0097] (1) Data Collection and Labeling: In this part, the present invention selects a certain network platform as the data source, constructs an offensive dictionary using offensive-related words, obtains data using web crawlers based on the keywords in the dictionary, and completes the data labeling. The offensive speech dataset constructed in this part provides data support for the present invention.

[0098] (2) Feature Extraction: In this part, the core task is to analyze and extract the features of offensive statements and generate a feature vector for each offensive statement. In terms of semantic features, this invention uses the BERT Chinese pre-trained model to generate context-related semantic feature vectors for the tweets. In addition, in terms of statistical features, this invention extracts statistical features of three categories: sentiment features, content features, and dissemination features, and concatenates them to obtain the statistical feature vector of the tweet.

[0099] (3) Detection Model: In this part, the present invention adopts the MMoE multi-task learning framework proposed by Google, taking offensive speech detection as the main task and sentiment analysis as the auxiliary task for simultaneous learning, so as to provide more feature information for the detection of offensive speech. It also utilizes BERT, Bi-LSTM and attention mechanism to efficiently detect offensive speech from a certain network platform.

[0100] 1. Data Collection and Labeling

[0101] During the data collection process, a specific online platform was used as the data source. The data collection targets were both offensive and normal tweets on this platform. An offensive dictionary was constructed using relevant offensive terms. A web crawler was then built based on the keywords in the dictionary to crawl the online platform from January 1, 2021 to December 31, 2021. The target fields included the text from the online platform, the publisher, and the number of retweets / comments / likes.

[0102] 1.1. Data Acquisition

[0103] (1) Construction of offensive lexicon

[0104] In the task of detecting offensive speech, the dataset consists of two types of data: offensive speech and normal speech. Due to the filtering rules and reporting mechanisms of the platform itself, offensive speech is very sparsely distributed across the platform. Therefore, it is difficult to obtain the required data by directly collecting a large number of tweets from online platforms and filtering them. However, offensive speech usually contains offensive vocabulary, which can aggregate offensive speech to a certain extent. Therefore, this invention constructs an offensive dictionary and crawls online platforms based on keywords to form the dataset of this invention.

[0105] This involved referencing the Hurtlex corpus, Chinese sensitive word database, Badwords, and Chinese-English sensitive word database to select and create an offensive dictionary. After removing keywords that were blocked by online platforms, the final offensive dictionary was formed.

[0106] (2) Web crawling implementation

[0107] In the data acquisition and annotation module, the crawler in this embodiment is written in Python and mainly uses the Requests library and the lxml library to handle accessing network resources and parsing HTML documents, respectively. The Requests third-party library is used to simulate login and save the session, thereby obtaining the tweet data of the network platform.

[0108] The crawler module of this invention needs to parse a large amount of content; almost every field in the acquired data requires a separate code block for parsing. This module uses the lxml library to parse and obtain necessary information, including user ID, tweet content, and time, acquiring a large amount of tweet data from this online platform.

[0109] In addition, due to the restrictions on frequent access by this web platform, this crawler uses the sleep() function to set a sleep time, thereby limiting the frequency of page access.

[0110] 1.2. Data Labeling

[0111] Offensive speech is a complex linguistic phenomenon, and clearly defining it is a crucial step in the task of automatically identifying it. This invention combines annotation suggestions from relevant research results with community guidelines from several major domestic and international online social media platforms, filtering and supplementing them to ultimately arrive at a definition of offensive speech.

[0112] When labeling datasets, use the following criteria; any statement meeting one of these criteria will be classified as offensive speech:

[0113] 1) Making baseless criticisms and hostile remarks.

[0114] 2) Personal attacks targeting individuals or groups.

[0115] 3) Making biased statements about individuals or groups belonging to the following categories:

[0116] a) Nationality and region;

[0117] b) Ethnicity, race, and religious belief;

[0118] c) Gender, age, and body type;

[0119] d) Regional and folk culture;

[0120] e) Gender identity and sexual orientation;

[0121] f) Serious illness or disability;

[0122] g) Other physiological and psychological characteristics.

[0123] 4) Expressing threats, etc.

[0124] If none of the above criteria are met, then it is considered normal speech.

[0125] Finally, an offensive speech dataset was constructed for this invention, and its description is shown in Table 1.

[0126] Table 1. Description of the offensive speech dataset

[0127] .

[0128] 2. Statistical Feature Extraction

[0129] In the statistical feature extraction module, statistical features, including three categories—emotional features, content features, and dissemination features—are extracted by studying and analyzing relevant data on aggressive and normal speech. These features are then generated into statistical feature vectors using a backpropagation (BP) neural network. The statistical features extracted in this invention are shown in Table 2.

[0130] Table 2 Statistical Characteristics List

[0131] .

[0132] 2.1. Emotional Characteristics

[0133] Compared to normal comments, users often express strong negative emotions when posting offensive comments, so the emotional polarity of offensive comments tends to be negative. Furthermore, the emojis included in a tweet are an important indicator of its emotional tone.

[0134] Sentiment Value: When users post offensive comments, it is often accompanied by negative emotions such as "dissatisfaction" and "anger." Sentiment features are of great significance for the detection of offensive comments. We used Baidu's API to perform sentiment analysis on the tweets. The `sentiment` parameter in the result represents the sentiment polarity classification result: 0 indicates negative sentiment polarity, 1 indicates neutral sentiment polarity, and 2 indicates positive sentiment polarity.

[0135] Number of positive / negative emojis: As short texts on social media, tweets from this online platform contain a large number of emojis, which are important indicators of the tweet's sentiment. Emojis in the tweets can be broadly divided into two types: those developed and provided by the online platform, and emojis. For emojis from this platform, they are automatically converted to emoji names during collection, such as [laughing emoji]. For emojis, the Python third-party library `emoji` is used to convert emoji symbols into words with similar meanings. The number of positive / negative emojis in the tweet is then calculated and used as one of the features for judging whether the text is offensive.

[0136] 2.2. Content Characteristics

[0137] The content features of tweets should consider information beyond the text itself that is helpful for detecting offensive speech.

[0138] a) The ratio of "?" to "!": When users post offensive comments, their tone is often quite intense. Specifically, in tweets, question marks and exclamation marks, used to emphasize tone, make up a higher percentage of all punctuation marks than periods or other declarative punctuation. Therefore, calculating the proportion of question marks and exclamation marks among all punctuation marks in a tweet can be used as one of the characteristics for judging whether the text is offensive.

[0139] b. Location tagging: Compared to relatively private social networks, people tend to express offensive statements in communication environments with a degree of anonymity. Therefore, users generally do not want to tag their location when posting offensive tweets to reduce anonymity.

[0140] 2.3. Propagation Characteristics

[0141] The propagation characteristics of tweets consider features related to the spread of offensive rhetoric, such as the number of comments, retweets, and likes. Since the crawler was set to only crawl original tweets, whether a tweet was a retweet or not is not considered here. To some extent, the propagation characteristics of a tweet can reflect its popularity and the way it spreads.

[0142] Likes / Comments / Shares: On online platforms, users can like, comment on, and share tweets that interest them. The statistical count of these actions can, to some extent, reflect the level of attention other users pay to a particular tweet. Aggressive comments targeting a specific group or public figure on trending topics often attract more user attention. Therefore, some aggressive tweets may have higher likes, comments, and shares than normal tweets, and this can be used as a feature to supplement the detection of aggressive comments.

[0143] 3. Detection Model

[0144] This invention designs a deep learning model, MBBA, based on multi-task learning, to detect offensive speech on a certain online platform. The model constructs word embeddings based on a BERT pre-trained model, employs the MMoE multi-task learning framework and simultaneously trains it with a sentiment analysis task to incorporate useful external knowledge, and integrates a Bi-LSTM network and an attention mechanism. It also combines statistical and semantic features, enabling it to effectively detect offensive speech on the online platform. The specific model structure is as follows: Figure 2 As shown.

[0145] 3.1. Input Layer

[0146] The model's input layer consists of two main parts: the preprocessed target tweet text and the statistical features extracted from the target tweet. The model's final output is the probability that the tweet contains offensive language.

[0147] The first part of the input layer is designed for offensive speech detection and sentiment analysis tasks, and it takes the preprocessed tweet text... As input. The preprocessing of the tweets includes removing emojis, web links, special characters, and foreign characters. Setting n to the maximum sequence length (max_seq_len), the tweets are then processed by short-filling-long-cutting, discarding parts exceeding n characters and padding with zeros if insufficient. The resulting preprocessed tweets are then... It contains n characters. Then, the BERT model tool from HuggingFace is used to tokenize it, making its format conform to BERT input. The input text sequence begins with [CLS], uses [SEP] as the sentence segment, and is then transformed into a BERT input representation. The input representation is obtained by adding three parts: target word embeddings, segment embeddings, and position embeddings. Target word embeddings are context-independent vectors; segment embeddings, also known as segmentation embeddings, are used to distinguish whether the input text sequence is a single sentence or multiple sentences; and position embeddings represent the positional information of corresponding words in the word sequence. The vector obtained after tokenizing the text contains word vector information and the word's positional information in the text sequence. This input is then fed into the shared word embedding module in the sentiment knowledge sharing layer to extract context-related word embedding features.

[0148] The second part of the input layer is designed for the task of detecting offensive speech. It processes the relevant information of the target tweet and extracts three types of statistical features, including sentiment features, content features, and dissemination features. These features are then fed into a backpropagation neural network for learning, transforming them into statistical feature vectors with the same dimension as the semantic features.

[0149] 3.2. Emotional Knowledge Sharing Layer

[0150] In the sentiment knowledge-sharing layer, offensive speech detection is taken as the primary task, and sentiment analysis as the auxiliary task. The MMoE multi-task learning framework is used to fuse BERT and Bi-LSTM to obtain the semantic feature vectors of the tweets, including a shared word embedding module and a shared semantic feature extraction module. Specifically, the shared word embedding module is used to train and generate word embeddings applicable to both the primary task (offensive speech detection) and the auxiliary task (sentiment analysis). The shared feature extraction module extracts features common to both the primary and auxiliary tasks. The multi-task learning framework adopts a shared underlying structure, with different tasks sharing the same hidden layers. This structure fundamentally reduces the risk of overfitting.

[0151] The shared word embedding module uses the BERT model as the tweet word embedding extractor. The emergence of large-scale pre-trained language models, especially BERT, has greatly promoted the development of NLP, achieving state-of-the-art performance in many downstream NLP tasks. First, there is a significant correlation between the primary task of offensive speech detection and the auxiliary task of sentiment analysis, both using datasets from the same social media platform. Using the same dataset in the shared word embedding module prevents training failure due to large differences in data domains when extracting word embeddings simultaneously. Furthermore, jointly performing shared word embedding on data from different tasks allows the final word embedding representation to be adapted to multiple tasks. Compared to classic word embedding models such as Word2Vec, BERT generates context-dependent word vectors, resulting in more accurate feature representations. Moreover, the BERT model is pre-trained on a large-scale unsupervised corpus, adding more external information to the training data, thus resulting in richer features in the final generated word vectors.

[0152] Specifically, the pre-processed tweet text After tokenization, the result is... Then, the word embeddings are fed into the BERT model to obtain the word embedding representations. .in, It is a dimension of The vector, For word sequence length, In the BERT model, the word vector dimension is used. Typically, it has 768 dimensions. As shown in formulas (1) and (2):

[0153] (1);

[0154] (2);

[0155] The shared semantic feature extraction module is inspired by MoE layers and Multi-gate Mixture-of-Experts (MMoE) models. It uses multiple feature extraction units and employs a gated attention mechanism to fuse features. MoE layers have multiple identical feature extraction units that share the output of the previous layer as input and output to subsequent layers. The entire model is trained in an end-to-end manner.

[0156] The feature extraction unit here consists of a Bi-LSTM network. The biggest advantage of the Bi-LSTM network is its "memory" function, which allows it to learn sequence features simultaneously from both preceding and following context. Bi-LSTM is chosen as the feature extraction unit for further feature extraction from tweet text because offensive speech detection tasks emphasize the extraction of sequence information, and because it can extract information from limited context, synthesize features, and avoid the gradient vanishing problem caused by RNN models when learning long-term dependencies.

[0157] Then, the features are fused using a gated attention mechanism to obtain the final semantic feature vector. Gated attention can learn to select and use a subset of feature extraction units based on different inputs. Each task corresponds to a gate, and the model's weights differ for different tasks. (Specific gate) The output represents the probability of selecting different feature extraction units, and then the weighted sum is used to obtain the final representation of the sentence, which is then passed to the private layer of each task.

[0158] Specifically, the model embeds the extracted words The input is fed into a Bi-LSTM-based feature extraction unit to obtain the output of the expert network. , , The number of hidden nodes in the network. The calculation method is as shown in formulas (3) to (5):

[0159] (3);

[0160] (4);

[0161] (5);

[0162] in, Indicates the bidirectional LSTM's first... The implicit vector generated by each unit The word embedding representation mentioned above The elements in It is the hidden vector generated by the previous state of the forward LSTM. It is the hidden vector generated by the previous state of the backward LSTM. This indicates vector concatenation.

[0163] The gating network uses the word embeddings of the input and the Softmax activation function to generate different expert combination weights, thereby allowing different tasks to utilize the results of the expert network with different weights, as shown in Equation (6):

[0164] (6);

[0165] in, This represents an updatable parameter matrix. This indicates a specific door, corresponding to different tasks.

[0166] The generated weights are then applied to the output of the expert network, as shown in formula (7):

[0167] (7);

[0168] Among them, among them, This indicates the number of expert networks, i.e., the number of feature extraction units; This represents a semantic feature vector corresponding to a specific task; This represents the output of a specific task after passing through an expert network; This represents the weights of different feature extraction units in an expert network used for a specific task; This represents the output of different feature extraction units in the expert network.

[0169] 3.3. Statistical Feature Extraction Module

[0170] The statistical feature extraction module, designed for offensive speech detection, concatenates the sentiment, content, and propagation features extracted from the tweet to obtain the statistical features of the target tweet. These features are then normalized and regularized before being fed into a backpropagation neural network to learn the implicit relationships between the features. Furthermore, the backpropagation neural network can transform the statistical features into vectors with the same dimension as the semantic features, thus providing an equivalent amount of feature information for offensive speech detection.

[0171] Specifically, the statistical features extracted from the tweets are first vectorized, then regularized, normalized, and other processes are applied to them, and finally concatenated into a one-dimensional vector. As shown in formulas (8) and (9):

[0172] (8);

[0173] (9);

[0174] in, Indicates emotional characteristics, Indicates content characteristics, Indicates the characteristics of propagation. Indicates splicing characteristics, This represents regularization and normalization. This represents the result after regularization and normalization of the spliced ​​features.

[0175] The processed statistical features are then fed into a BP neural network, and transformed into a statistical feature vector with the same dimension as the semantic features through its fully connected layers, as shown in formula (10):

[0176] (10);

[0177] in, Represents the parameter matrix, Indicates the bias term. Let S represent the activation function, and S be the statistical feature vector.

[0178] 3.4. Feature splicing module

[0179] The feature concatenation module, designed for offensive speech detection, concatenates extracted semantic features with statistical properties to obtain the feature vector of the tweet. Statistical features, from a global perspective, obtain global attributes of the text through special symbols, text formatting, etc. Therefore, combining statistical features to obtain global attributes and semantic features to obtain semantic attributes better reflects the distribution of data in the feature space, thereby improving the model's classification performance.

[0180] Specifically, concatenating semantic features Statistical characteristics Obtain the feature vector of the target tweet As shown in formula (11):

[0181] (11);

[0182] 3.5. Attention Layer

[0183] The attention layer, designed for offensive speech detection, feeds the feature vector obtained from the feature concatenation module into the attention mechanism. When users post offensive remarks, they often express strong negative emotions, and the language typically contains insulting and aggressive words. Therefore, capturing offensive words in sentences helps improve the detection rate of offensive speech. The attention mechanism automatically captures words and features that play a crucial role in classifying offensive speech and automatically extracts key semantic information from sentences, thereby enhancing the effectiveness of offensive speech detection.

[0184] Specifically, the attention score is first calculated by determining the relevance between the query and the key, as shown in formula (12). This represents the parameter matrix.

[0185] (12);

[0186] Then, the attention scores are normalized using the Softmax function to obtain the weight coefficients. As shown in formula (13). Wherein, express The intermediate hidden layer vectors, i.e., the attention scores mentioned above, It is a randomly initialized vector that is automatically updated during training. express The transpose of .

[0187] (13);

[0188] Finally, the values ​​are weighted and summed according to the weight coefficients to obtain the final feature representation, as shown in formula (14). Here, the key and value are the same, both being... .

[0189] (14);

[0190] 3.6. Output Layer

[0191] For the task of detecting offensive speech, semantic features and statistical features are concatenated and weighted through an attention mechanism. The probability that the target tweet is offensive speech is obtained through the Sigmoid activation function and used as the final classification result, as shown in formula (15):

[0192] (15);

[0193] For the sentiment analysis task, since it is only used as an auxiliary task in this study, after obtaining semantic features through joint training with the offensive speech detection task, no further processing is performed. The final classification result is obtained directly through the Sigmoid activation function, as shown in formula (16):

[0194] (16);

[0195] The training objective of the model is to minimize the loss function. The cross-entropy loss function measures the difference between the actual output distribution and the expected output distribution, and is suitable for classification models. The cross-entropy function is chosen as the loss function, and its definition is shown in formula (17):

[0196] (17);

[0197] in, For label values, For predicted values, This represents the total number of categories. This represents the value of the loss function.

[0198] 4. Experimental Verification

[0199] This invention evaluates the performance of the MBBA model in detecting offensive speech through three experiments. All experiments were conducted on an Apple M1 Pro. The dataset consisted of offensive speech datasets collected in this project and publicly available sentiment analysis datasets. The offensive speech dataset contained 3,914 offensive tweets and 12,503 non-offensive tweets. In the experiments, 80% of the rumor dataset was used as the training set, 10% as the validation set, and 10% as the test set. Each experiment was repeated 10 times, and the average value was used as the final result.

[0200] 4.1. Evaluate the effectiveness of each component of the multi-task learning framework.

[0201] To verify the effectiveness of each component of the MMoE multi-task learning framework, ablation experiments were designed to analyze the impact of different parts of the model on the experimental results. Accuracy, precision, recall, and Macro-F1 score were selected as comparison metrics. Detailed information about the model involved in this experiment is as follows:

[0202] 1) MBBA\S: It does not introduce sentiment analysis as an auxiliary task, that is, it reduces sentiment knowledge sharing and only uses the offensive speech dataset for training.

[0203] 2) MBBA\G: Reduce the gating attention mechanism in the MMoE framework.

[0204] 3) MBBA: The complete model proposed in this invention.

[0205] Experimental results are as follows Figure 3 As shown in Table 3, the MBBA full model performs best overall, outperforming all evaluation metrics. Comparing MBBA\S with the MBBA model reveals that when sentiment subtraction is used as an auxiliary task, the model loses the advantage of multi-task learning—effectively leveraging the correlation between multiple tasks to improve performance and generalization in each task—due to its inability to learn external sentiment knowledge, resulting in a performance decline. Comparing MBBA\G with the MBBA model shows that gated attention further improves performance. This mechanism generates different expert weight combinations through a gated network, allowing different tasks to utilize the results of the feature extraction unit with different weights. Since offensive speech detection and sentiment analysis are highly correlated, knowledge sharing contributes to improved detection results.

[0206] Table 3 Comparison of ablation results

[0207] .

[0208] 4.2. Evaluate the effectiveness of word embedding.

[0209] BERT, as a pre-trained language model trained on massive corpora, can be used directly as a model for text classification tasks, or simply as a word embedding layer input into other models for training. To demonstrate the effectiveness of using BERT as an embedding layer for training word vectors, we conducted comparative experiments with commonly used deep learning text classification models, including TextCNN, TextRNN, TextRCNN, and TextDPCNN. The experimental results were compared with those of BERT-CNN, BERT-RNN, BERT-RCNN, and BERT-DPCNN, using accuracy, precision, recall, and Macro-F1 score as comparison metrics. Details of the comparative models involved in this experiment are as follows:

[0210] 1) TextCNN: A CNN-based text classification model. It uses Word2Vec to obtain the input word vectors, which are then used as input to a convolutional neural network to capture the semantics of the text for classification tasks.

[0211] 2) TextRNN: A text classification model based on RNN. LSTM is a type of RNN that overcomes the problems of vanishing or exploding gradients in traditional RNNs.

[0212] 3) TextRCNN: Uses Bi-LSTM to extract forward and backward semantic information of words, and then uses max pooling layers to filter useful feature information.

[0213] 4) TextDPCNN: A novel CNN architecture. TextCNN cannot obtain long-range dependencies through convolution operations. DPCNN, by increasing network depth, can effectively extract long-range relationship features from text without increasing computational overhead.

[0214] The experimental results are shown in Table 4. Figure 4As shown, BERT uses the Transformer as a feature extractor, which has a significant advantage in semantic feature extraction. Experimental results show that using BERT as the embedding layer to train word vectors can effectively optimize the vector representation of the input text, thus achieving better training results. Compared to TextCNN, BERT_CNN improves the F1 score by 0.7%; compared to TextRNN, BERT-RNN improves the F1 score by 3.4%; compared to TectRCNN, BERT-RCNN improves the F1 score by 1.9%; and compared to TextDPCNN, BERT-DPCNN improves the F1 score by 3.3%. This is because the BERT model can utilize the contextual information of words to capture the appropriate word meaning, rather than using the same representation for words with different meanings in different contexts. It is a context-dependent word embedding representation, which has a significant advantage in semantic feature extraction.

[0215] Table 4 Performance of different word embedding models

[0216] .

[0217] 4.3. Evaluate the effectiveness of the proposed detection model

[0218] To verify the effectiveness of the offensive speech detection method based on MMoE multi-task learning proposed in this invention, four models that performed relatively well in Experiment 1 and used BERT as the embedding layer for word vector training were selected for comparison. Pure BERT models were also selected for comparison. Accuracy, precision, recall, and Macro-F1 score were chosen as comparison metrics. Detailed information on the comparison models involved in this experiment is as follows:

[0219] 1) BERT: Utilizes the Transformer encoder to extract features from target text, capturing both the features of the words themselves and their contextual features. It achieves state-of-the-art (SOTA) performance in various downstream tasks within the natural language processing field. Here, the output of the BERT model is passed through Dropout and then fed into a fully connected layer to obtain the classification result.

[0220] 2) BERT-CNN: BERT is used to obtain word embeddings, and TextCNN is used as a feature extractor.

[0221] 3) BERT-RNN: Uses BERT to obtain word embeddings and Bi-LSTM as a feature extractor.

[0222] 4) BERT-RCNN: It uses BERT to obtain word embeddings and adopts a bidirectional LSTM connected to a TextCNN model as a feature extractor.

[0223] 5) BERT-DPCNN: BERT is used to obtain word embeddings, and DPCNN is used as a feature extractor.

[0224] The experimental results are shown in Table 5. Figure 5 As shown, MBBA performed best overall among all models, and all models achieved higher precision than recall. Specifically, the MBBA model's accuracy was not significantly different from most of the comparison models, but its precision and Macro-F1 score were both significantly improved.

[0225] Higher precision means a larger proportion of samples correctly predicted as offensive speech out of all samples predicted as offensive speech. Higher recall means a larger proportion of samples correctly predicted as offensive speech out of all samples predicted as offensive speech. This indicates that for all models, very few normal statements are misclassified as offensive, but a relatively large number of offensive statements are misclassified as normal. This also shows that the deceptive nature of offensive speech makes it difficult for general models to effectively filter and identify it from normal speech.

[0226] The MBBA model outperforms other models that use BERT as a word embedding extractor because the data scale for offensive speech detection tasks is small, the samples are sparse, and the context that can be obtained from short texts is limited. MBBA introduces external knowledge through multi-task learning, using relevant auxiliary tasks such as sentiment analysis and its dataset to assist in training for the main task of offensive speech detection. Furthermore, the statistical features incorporated into the MBBA model also contribute to improving the detection model's performance.

[0227] Table 5. Performance of different detection models and MBBA models

[0228] .

[0229] In summary, the MBBA model proposed in this invention constructs word embeddings based on a BERT pre-trained model, employs the MMoE multi-task learning framework and simultaneously trains with a sentiment analysis task to incorporate useful external knowledge, and integrates a Bi-LSTM network and an attention mechanism. Furthermore, it combines statistical and semantic features, effectively detecting offensive speech on online platforms. Therefore, the MBBA offensive speech detection model of this invention has achieved excellent results in the problem of offensive speech detection.

Claims

1. A method for detecting offensive speech on social networks based on multi-task learning, characterized in that, Includes the following steps: Step 1: Data Collection and Labeling We selected specific social networking platforms as data sources, constructed an offensive dictionary using offensive-related terms, used web crawlers to obtain data based on the keywords in the dictionary, and completed the data annotation. Step 2: Feature Extraction The features of offensive remarks are analyzed and extracted, and feature vectors are generated for each offensive remark, including semantic feature vectors and statistical feature vectors; the context-related semantic feature vectors of the tweets are generated using the BERT Chinese pre-trained model. Statistical feature vectors are generated by extracting statistical features from three categories: sentiment features, content features, and dissemination features, and then concatenating them. Step 3: Construct the detection model The MMoE multi-task learning framework is adopted, with offensive speech detection as the main task and sentiment analysis as the auxiliary task, and offensive speech on the social network platform is detected simultaneously using BERT, Bi-LSTM and attention mechanisms. The process of detecting offensive language in step 3 includes: Step 3.1: Preprocessed tweet text The vector is obtained after tokenization. As shown in the following formula: （1）； Step 3.2: Input vector E into the BERT model to obtain word embedding representations. ;in, For dimension The vector, For word sequence length, The word vector dimension is shown in the following formula: （2）； Step 3.3: Embed the extracted words The input is fed into a Bi-LSTM-based feature extraction unit to obtain the output of the expert network. , , The number of hidden nodes in the network. The calculation method is as follows: （3）； （4）； （5）； in, Indicates the bidirectional LSTM's first... The implicit vector generated by each unit Word embedding representation The elements in It is the hidden vector generated by the previous state of the forward LSTM. It is the hidden vector generated by the previous state of the backward LSTM. This indicates vector concatenation; Step 3.4: The gating network uses the input word embeddings and the Softmax activation function to generate different expert combination weights, thereby allowing different tasks to utilize the results of the expert network with different weights, as shown in the following equation: （6）； in, This represents an updatable parameter matrix. This represents a specific gate corresponding to different tasks. This indicates the weights of the expert network used for a specific task; Step 3.5: Apply the generated weights to the output of the expert network, as shown in the following formula: （7）； in, This indicates the number of expert networks, i.e., the number of feature extraction units; This represents a semantic feature vector corresponding to a specific task; This represents the output of a specific task after passing through an expert network; This represents the weights of different feature extraction units in an expert network used for a specific task; This represents the output of different feature extraction units in the expert network; Step 3.6: Vectorize the statistical features extracted from the tweets, and then perform regularization and normalization on them. Concatenate them into a one-dimensional vector, as shown in the following formula: （8）； （9）； in, Indicates emotional characteristics, Indicates content characteristics, Indicates the characteristics of propagation. Indicates splicing characteristics, This represents regularization and normalization. This represents the result after regularization and normalization of the spliced ​​features; The processed statistical features are fed into a backpropagation (BP) neural network, and transformed into a statistical feature vector with the same dimension as the semantic features through its fully connected layers, as shown in the following equation: （10）； in, Represents the parameter matrix, Indicates the bias term. S represents the activation function, and S is the statistical feature vector. Step 3.7: Concatenate semantic feature vectors With statistical eigenvectors Obtain the feature vector of the target tweet As shown in the following formula: （11）； Step 3.8: Calculate the relevance between the query and the key to obtain the attention score, as shown in the following formula: （12）； in, Represents the parameter matrix; This represents the attention score. Feature vectors representing specific tasks; Then, the attention scores are normalized using the Softmax function to obtain the weight coefficients. As shown in the following formula: （13）； in, express The intermediate hidden layer vectors, i.e., the attention scores mentioned above, It is a randomly initialized vector that is automatically updated during training. express Transpose of; Finally, the values ​​are weighted and summed according to the weighting coefficients to obtain the final feature representation, as shown in the following formula: （14）； In this case, the key and value are the same, both being [key and value]. ; This represents the feature representation of the final output; Step 3.9: For the offensive speech detection task, the semantic features and statistical features are concatenated and weighted using an attention mechanism. The probability that the target tweet is offensive speech is then obtained through the Sigmoid activation function and used as the final classification result, as shown in the following formula: （15）； in, The probability that the targeted tweet is offensive. For sentiment analysis tasks, the final classification result is obtained directly through the Sigmoid activation function, as shown in the following formula: （16）； in, This represents the final classification result obtained by applying the Sigmoid activation function to the target tweet in the sentiment analysis task. This represents the semantic feature vector of the target tweet in a sentiment analysis task.

2. The method for detecting offensive speech on social networks based on multi-task learning according to claim 1, characterized in that... Step 1 specifically includes: Step 1.1: Construction of the offensive dictionary Based on the Hurtlex corpus, Chinese sensitive word database, Badwords, and Chinese-English sensitive word database, an offensive dictionary was created. Step 1.2: Web crawler implementation A web crawler was written in Python, using the Requests and lxml libraries to handle accessing network resources and parsing HTML documents, respectively. The Requests library was used to simulate login and save session data, thereby obtaining tweet data from the social network platform. Step 1.3: Data Labeling The dataset is labeled according to the definition of offensive speech, thereby constructing an offensive speech dataset.

3. The method for detecting offensive speech on social networks based on multi-task learning according to claim 1, characterized in that, Step 2, the extraction of statistical features, specifically includes: Step 2.1: Extraction of emotional features 1) Sentiment Value: The existing application programming interface is used to perform sentiment analysis on the tweets. The sentiment parameter in the result indicates the sentiment polarity classification result, where 0 indicates negative sentiment polarity, 1 indicates neutral sentiment polarity, and 2 indicates positive sentiment polarity. 2) Number of positive / negative emojis: The platform's built-in emojis are automatically converted into emoji names, and the third-party Python library emoji is used to convert emoji symbols into words with similar meanings; then the number of positive / negative emojis in the tweet is calculated and used as one of the characteristics to determine whether the tweet is aggressive; Step 2.2: Extraction of content features 1) Calculate the proportion of question marks and exclamation marks among all punctuation marks in a tweet, and then use this as one of the characteristics to determine whether a tweet is offensive; 2) Determine whether a tweet is tagged with a location, using this as one of the characteristics to determine whether a tweet is offensive; Step 2.3: Extraction of propagation features The number of likes, comments, and retweets are used as one of the characteristics to determine whether a tweet is offensive.

4. The method for detecting offensive speech on social networks based on multi-task learning according to claim 1, characterized in that, The detection model in step 3 includes an input layer, an emotion knowledge sharing layer, a statistical feature extraction module, a feature concatenation module, an attention layer, and an output layer. 1) Input layer The input layer consists of two parts. The first part is designed for offensive speech detection and sentiment analysis tasks, and it takes the pre-processed tweet text... As input, it is tokenized using BERT model tools to obtain a vector. The first part is to make its format conform to the input of BERT; the second part is for the task of detecting offensive speech. First, the relevant information of the target tweet is processed to extract three types of statistical features, including sentiment features, content features and dissemination features. Then, it is fed into the BP neural network for learning and is transformed into a statistical feature vector with the same dimension as the semantic features. 2) Emotional Knowledge Sharing Layer The sentiment knowledge sharing layer takes offensive speech detection as the main task and sentiment analysis as the auxiliary task. It uses the MMoE multi-task learning framework to fuse BERT and Bi-LSTM to obtain the semantic feature vector of the tweet. The multi-task learning framework adopts a shared underlying structure, and different tasks share the underlying hidden layer. The sentiment knowledge sharing layer includes a shared word embedding module and a shared semantic feature extraction module; The shared word embedding module uses the BERT model as a tweet word embedding extractor, which is used to train and generate word embeddings that are suitable for both the main task of offensive speech detection data and the auxiliary task of sentiment analysis data. The shared semantic feature extraction module uses multiple feature extraction units composed of Bi-LSTM networks and uses a gated attention mechanism to fuse the features to obtain the final semantic feature vector, thereby extracting the same features between the main task and the auxiliary task. 3) Statistical Feature Extraction Module For the task of detecting offensive speech, the statistical feature extraction module concatenates the sentiment features, content features, and propagation features extracted from the tweet to obtain the statistical features of the target tweet. After normalization and regularization, the statistical features are fed into the backpropagation neural network to learn the implicit relationships between the features. The backpropagation neural network also transforms the statistical features into vectors with the same dimension as the semantic features, thus providing an equal amount of feature information for the detection of offensive speech. 4) Feature splicing module The feature concatenation module is designed for offensive speech detection tasks. It concatenates the extracted semantic features with statistical characteristics to obtain the feature vector of the tweet. 5) Attention layer For the offensive speech detection task, the attention layer feeds the feature vector obtained by the feature concatenation module into the attention mechanism. 6) Output layer The output layer outputs the detection result of whether the target tweet contains offensive language.

5. The method for detecting offensive speech on social networks based on multi-task learning according to claim 4, characterized in that, The text sequence input to the input layer begins with [CLS] and [SEP] is used as the sentence segment. The BERT input representation obtained by the transformation is obtained by adding three parts: target word embedding, segment embedding, and position embedding. Among them, the target word embedding is a vector that is independent of the context representation; the segment embedding is used to distinguish whether the input text sequence is a single sentence or multiple sentences; and the position embedding is used to represent the position information of the corresponding words in the word sequence.

6. The method for detecting offensive speech on social networks based on multi-task learning according to claim 1, characterized in that, The training objective of the detection model is to minimize the loss function. The cross-entropy function is chosen as the loss function, and its definition is as follows: （17）； in, For label values, For predicted values, This represents the total number of categories. This represents the value of the loss function.

Images