A method for detecting hate speech based on hate target features and variant word restoration

Abstract

This invention relates to the fields of network content security and cyberspace security technology, and discloses a hate speech detection method based on hate object features and variant word reconstruction. The method includes fine-tuning the ChatGLM2-6B model using the P-Tuning v2 method to construct a variant word reconstruction module; using BIO annotation to label each sample in the Chinese hate speech dataset with hate object entities, establishing a BERT-CRF model and performing feature extraction and label sequence optimization, while saving the BERT encoder layer parameters after training; initializing the trained BERT encoder layer parameters to the BERT encoder layer of the hate speech detection module, and completing the training of the hate speech detection module; and using the trained hate speech detection model to perform final hate speech detection and classification on any input text to be detected. This method enhances the model's context feature extraction capability, mitigates the negative impact of specific expressions on model decisions, and effectively alleviates the interference of variant words on the semantic extraction of hate speech detection.

Description

Technical Field

[0001] The present invention relates to the fields of network content security, cyberspace security technology, and more specifically, to a method for detecting hate speech based on hate object features and variant word restoration. Background Art

[0002] In the current situation of the full promotion of self-media, online social networks have profoundly changed people's social methods and gradually replaced traditional media as the main channel for obtaining information. As a medium for information dissemination, it facilitates communication and exchanges, enables people to obtain external news in a timely manner, and gives individuals more space to express their opinions. The manual review method used in the early days was difficult to cope with the rapidly growing large-scale network texts. Therefore, the automatic detection technology of hate speech has become a hot research topic.

[0003] According to their development over time, hate speech detection methods can be divided into three categories: feature word matching methods, machine learning methods, and deep learning methods. Detection methods based on feature word matching use a preset hate dictionary to directly find feature words in the text to be detected for judgment. Detection methods based on machine learning identify hate speech by inputting statistical features into a machine learning algorithm classifier. However, the above two types of methods rely heavily on expert knowledge or feature engineering, have limited feature extraction capabilities, and cannot capture deep semantic information in complex contexts, resulting in a high false alarm rate and poor generalization. In contrast, detection methods based on deep learning can automatically learn text features from data, better model semantic information, effectively improve the shortcomings of traditional methods, and gradually become the mainstream method in the field. Although existing work has alleviated the spread of hate speech to a certain extent, the following problems still exist:

[0004] 1. First, hate speech is an offensive expression against an individual or a group. Therefore, paying attention to the object of hate in the text is crucial for the semantic extraction of the hate speech detection model. However, previous studies have ignored the impact of the object of hate on semantic extraction and failed to fully learn the contextual characteristics of the object of hate in a targeted manner, resulting in the model being overly sensitive to certain specific expressions and causing decision errors. For example, in the following situations, the model is prone to misjudgment or omission: (a) detecting non-hate text containing dirty words; (b) detecting hate text without dirty words; (c) detecting non-hate samples containing common attack identities.

[0005] 2. Variant words are a typical phenomenon of non-standard language in online social media. In order to evade regulation or make euphemisms, some hate speech publishers will replace relatively sensitive words with words that are similar to them (such as variants in the form of similar pronunciation, similar shape, and abbreviations) but not easily censored. The appearance of these variant words will seriously interfere with the semantic feature extraction of the model and affect the performance of downstream tasks. However, existing hate speech detection methods lack effective processing of variant words, resulting in a high false negative rate. Summary of the invention

[0006] The purpose of the present invention is to provide a hate speech detection method based on hate object features and variant word restoration, which can enhance the context feature extraction capability of the model, thereby mitigating the negative impact of specific expressions on model decisions, and effectively alleviate the interference of variant words on the semantic extraction of the hate speech detection module by restoring variant words.

[0007] The present invention is implemented by the following technical solution: a method for detecting hate speech based on hate target features and variant word restoration, comprising the following steps:

[0008] 1) Construct and train a hate speech detection model based on hate object features and variant word restoration, which consists of a variant word restoration module, a hate object identification module, and a hate speech detection module:

[0009] 1.1) Fine-tune the ChatGLM2-6B model based on the P-Tuning v2 method and build a variant word restoration module;

[0010] 1.2) Construct and train the hate object recognition module: The hate object recognition module is designed to identify the hate object in the text, so as to provide more accurate hate object feature extraction capabilities for subsequent tasks. The present invention regards hate object recognition as a named entity recognition task, so the BIO annotation method is used to annotate the hate object entity of each sample in the Chinese hate speech dataset, and the BERT-CRF model is established and feature extraction and label sequence optimization are performed to complete the training of the hate object recognition module, thereby achieving accurate hate object recognition. After completing the training of the hate object recognition module, the parameters of the BERT encoder layer after training are saved at the same time;

[0011] 1.3) Training the hate speech detection module based on the BERT model: Initialize the BERT encoder layer parameters obtained in the hate object recognition module to the BERT encoder layer of the hate speech detection module, and carry out and complete the training of the hate speech detection module;

[0012] 2) Use the trained hate speech detection model based on hate object features and variant word restoration to perform final hate speech detection and classification on any input text to be detected.

[0013] In order to better implement the method for detecting hate speech based on hate target features and variant word restoration described in the present invention, the following setting is particularly adopted: the step 1.1) includes the following steps:

[0014] 1.1.1) Establish a variant word restoration dataset containing 43,000 samples;

[0015] That is, through observation and analysis, it is concluded that the common variant forms in Chinese hate speech mainly include: spelling variants (such as similar sounds, similar shapes, full pinyin, pinyin abbreviations and decomposition) and grammatical variants (such as replacement, redundancy, deletion and word order changes). In the field of Chinese spelling detection and grammatical error correction, public datasets such as SIGHAN, CTC2021 and NLPCC2018_GEC are available. For the existing variant forms in these public datasets, consider the balance of the number of different variant types, and randomly extract an appropriate amount of data from them by category to balance the proportion of the number of each type of variant; for variant forms without public datasets, autonomously generate and construct an appropriate amount of samples. Finally, the above samples are integrated to establish a variant word restoration dataset containing 43,000 samples.

[0016] 1.1.2) The samples in the variant word restoration dataset obtained in step 1.1.1) are uniformly processed into a format that can be used for ChatGLM2-6B model training, and the ChatGLM2-6B model is fine-tuned using the P-Tuning v2 method.

[0017] In order to better implement the method for detecting hate speech based on hate target features and variant word restoration described in the present invention, the following setting is particularly adopted: the step 1.1.2) includes the following specific steps:

[0018] 1.1.2.1) Divide the variant word restoration dataset into training set, validation set and test set in a ratio of 8:1:1;

[0019] 1.1.2.2) The samples in the training set, validation set, and test set are unified into JSON format training samples with "query", "response", and "history" keys, where the "query" key and the "response" key represent the user query and model answer of the current round, respectively, and are used to store variant texts and restored texts; the "history" key records the historical rounds of dialogue, including guidance information such as task background and task description;

[0020] 1.1.2.3) Insert a trainable continuous vector as a prefix in the input layer and subsequent network layers of the ChatGLM2-6B model. During training, only these newly added vectors are optimized, while the rest of the parameters of ChatGLM2-6B remain frozen;

[0021] 1.1.2.4) Metaphorize the JSON format training sample into the form of "history+query[gmask][sop]response[eop]", where "[gmask]" indicates that the subsequent part is the autoregressive generation part, "[sop]" indicates the start of the autoregressive generation segment, and "[eop]" indicates the end of the generation segment;

[0022] 1.1.2.5) The ChatGLM2-6B model only calculates the loss for the "response" part, completes the ChatGLM2-6B model fine-tuning, and uses cross entropy as the loss function when calculating the loss. The specific formula is as follows:

[0023] Among them, Loss1 represents the value of the loss function, y res The lemma result of the real “response” part, y pred_res Represents the tokenized result of the "response" part of the model's prediction.

[0024] In order to better realize the hate speech detection method based on hate object features and variant word restoration described in the present invention, the following setting method is particularly adopted: the variant word restoration dataset includes: A. For variant forms without public datasets, an appropriate amount of sample data is independently generated and constructed; B. Data randomly extracted from public datasets such as SIGHAN, CTC2021 and NLPCC2018_GEC.

[0025] In order to better implement the method for detecting hate speech based on hate target features and variant word restoration described in the present invention, the following setting is particularly adopted: the step 1.2) includes the following steps:

[0026] 1.2.1) For each sample in the Chinese hate speech dataset, the present invention uses the BIO (B-begin, I-inside, O-outside) sequence annotation method to annotate the hate objects in the text at the token level;

[0027] 1.2.2) The 12-layer encoder of the BERT model is used to extract the semantic features in the text annotated in step 1.2.1). The final hidden layer state is processed by the fully connected layer, and the output obtained represents the hate object score corresponding to each token in the text annotated in step 1.2.1);

[0028] 1.2.3) A CRF layer is introduced after the BERT model designed in step 1.2.2) to form a BERT-CRF model. The constraint modeling between label sequences is realized through the state transfer matrix, and the output is corrected and optimized to ensure the accuracy of the predicted label sequence, thereby realizing accurate identification of hateful objects in the text and completing the training of the hateful object identification module.

[0029] 1.2.4) After completing the training of the hate object recognition module, save the BERT encoder layer parameters after training.

[0030] In order to better implement the method for detecting hate speech based on hate target features and variant word restoration described in the present invention, the following setting is particularly adopted: the step 1.3) includes the following steps:

[0031] 1.3.1) BERT model initialization for the hate speech detection module: The bottom encoder layer of the BERT model is initialized using the BERT encoder layer parameters trained in the hate object recognition module, while the fully connected layer of the BERT model of the hate speech detection module is randomly initialized;

[0032] 1.3.2) Divide the Chinese hate speech dataset into training set, validation set and test set in a ratio of 8:1:1 as input, and carry out and complete the training of the hate speech detection module.

[0033] In order to better implement the method for detecting hate speech based on hate target features and variant word restoration described in the present invention, the following setting is particularly adopted: the step 2) includes the following specific steps:

[0034] 2.1) Input any text to be detected into the trained hate speech detection model based on hate target features and variant word restoration, and restore the spelling and grammatical variant words in the text to be detected to their original form through the variant word restoration module to generate a restored text;

[0035] 2.2) The hate speech detection module further extracts the deep semantic features of the restored text and outputs the final hate speech classification result through the top fully connected layer and softmax function.

[0036] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0037] (1) The present invention implements a hate speech detection method that integrates hate object context features, which improves the detection model's ability to extract hate object context features. Through two-stage fine-tuning, the model parameters obtained by training the hate object recognition module are used to initialize the hate speech detection module, which enhances the detection model's ability to extract hate object context features, thereby effectively mitigating the negative impact of specific expressions on model decision-making and significantly improving the classification accuracy of the hate speech detection model.

[0038] (2) The present invention proposes a variant word restoration method based on fine-tuning of the ChatGLM2-6B model, which reduces the negative impact of variant words on semantic extraction. By including a data set with multiple variant forms, the detection scenario and application scope are expanded. By designing precise prompts, the variant word restoration task in the hate speech scenario can be more effectively adapted. By utilizing the rich pre-training knowledge of the ChatGLM2-6B model, the model can more accurately understand the Chinese context and cultural background, thereby effectively improving the performance of the variant word restoration task, reducing the negative impact of variant words on the semantic extraction of the hate speech detection model, and significantly reducing the false negative rate of hate speech detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a diagram of the construction framework of the hate speech detection model based on hate object features and variant word restoration according to the present invention.

[0040] Figure 2 Schematic diagram of fine-tuning using the P-Tuning v2 method based on the ChatGLM2-6B model.

[0041] Figure 3 The final hate speech detection and classification flow chart for inputting any text to be detected.

[0042] exist Figure 1 In the text, %%%@@@@ represents a text with variant words, where %%% is the variant word, and ### is the original word formed by restoring the variant word %%%, which are also the hate object labels in the text.

[0043] exist Figure 2 Here, $$&&&&&&&&& represents the language text with variant words, and **&&&&&&&& represents the restored language text, which restores the variant word $$ to **. DETAILED DESCRIPTION

[0044] The present invention is further described in detail below in conjunction with examples, but the embodiments of the present invention are not limited thereto.

[0045] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0046] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0047] Glossary:

[0048] P-Tuning v2 method: is a method based on pre-trained model fine-tuning, which aims to make the model better adapt to the target task during the fine-tuning process by selectively updating the parameters in the pre-trained model.

[0049] ChatGLM2-6B model: ChatGLM-6B is a product jointly developed by Tsinghua University and Zhipu AI. It is an open source conversational language model that supports both Chinese and English. It is based on the General Language Model (GLM) architecture and has 6.2 billion parameters.

[0050] BERT-CRF model: The BERT model can be combined with the CRF model to implement named entity recognition tasks. The BERT model is a pre-trained model for natural language processing (NLP). CRF (Conditional Random Fields) is a basic model in natural language processing and is widely used in scenarios such as word segmentation, entity recognition, and part-of-speech tagging. BERT model: Bidirectional Encoder Representations from Transformers (BERT) is a pre-trained model for natural language processing (NLP).

[0051] SIGHAN: A public dataset for Chinese text error correction tasks. Currently, there are three versions: ①SIGHAN Bake-off 2013; ②SIGHAN Bake-off 2014; ③SIGHAN Bake-off 2015;

[0052] CTC2021: A public dataset for Chinese text error correction tasks, including spelling errors, grammatical errors, linguistic errors, etc.

[0053] NLPCC2018_GEC: A public dataset for Chinese grammar correction tasks.

[0054] Token: A token is a piece of text that is divided into a single minimal unit, such as a word or character, for further processing and analysis.

[0055] CRF: CRF (Conditional Random Fields) is a probabilistic graphical model used to label and segment sequence data, and is commonly used in tasks such as named entity recognition, part-of-speech tagging, and sentence segmentation.

[0056] prompt: prompt refers to the input prompt provided to the language model to guide the generation of a specific type of text or answer a specific question.

[0057] BIO: B-begin, I-inside, O-outside: BIO annotation is a sequence annotation method, B-begin indicates the beginning of an entity, I-inside indicates the inside of an entity, and O-outside indicates the part that does not belong to any entity. This method is often used in named entity recognition tasks to clearly mark the boundaries and positions of each entity in the text.

[0058] Transformer Encoder: Transformer Encoder (encoder layer) is the part of the Transformer model that is used to encode the input sequence into a representation for subsequent processing.

[0059] Fully Connected Layer: A fully connected layer is a type of layer in a neural network in which each neuron is connected to all neurons in the previous layer.

[0060] Pre-trained BERT: The pre-trained BERT model is based on Bidirectional Encoder Representations from Transformers (BERT), which is a pre-trained model for natural language processing.

[0061] Prefix Encoder: It embeds prefix encoding vectors in the model so that the model can better utilize this information to complete training or fine-tuning.

[0062] Embodiment 1:

[0063] The present invention designs a hate speech detection method based on hate target features and variant word restoration, combined with Figure 1 to Figure 3 As shown, the following steps are included:

[0064] 1) Construct and train a hate speech detection model based on hate object features and variant word restoration, which consists of a variant word restoration module, a hate object identification module, and a hate speech detection module:

[0065] 1.1) Fine-tune the ChatGLM2-6B model based on the P-Tuning v2 method and build a variant word restoration module;

[0066] 1.2) Construct and train the hate object recognition module: The hate object recognition module is designed to identify the hate object in the text, so as to provide more accurate hate object feature extraction capabilities for subsequent tasks. The present invention regards hate object recognition as a named entity recognition task, so the BIO annotation method is used to annotate the hate object entity of each sample in the Chinese hate speech dataset, and the BERT-CRF model is established and feature extraction and label sequence optimization are performed to complete the training of the hate object recognition module, thereby achieving accurate hate object recognition. After completing the training of the hate object recognition module, the parameters of the BERT encoder layer after training are saved at the same time;

[0067] 1.3) Training the hate speech detection module based on the BERT model: Initialize the BERT encoder layer parameters obtained in the hate object recognition module to the BERT encoder layer of the hate speech detection module, and carry out and complete the training of the hate speech detection module;

[0068] 2) Use the trained hate speech detection model based on hate object features and variant word restoration to perform final hate speech detection and classification on any input text to be detected.

[0069] Embodiment 2:

[0070] This embodiment is further optimized on the basis of the above embodiment, and the same points as the above technical solution are not repeated here. Figure 1 to Figure 3 As shown, in order to better implement the method for detecting hate speech based on hate target features and variant word restoration described in the present invention, the following setting is particularly adopted: the step 1.1) includes the following steps:

[0071] 1.1.1) Establish a variant word restoration dataset containing 43,000 samples;

[0072] That is, through observation and analysis, it is concluded that the common variant forms in Chinese hate speech mainly include: spelling variants (such as similar sounds, similar shapes, full pinyin, pinyin abbreviations and split characters) and grammatical variants (such as replacement, redundancy, deletion and word order changes). In the field of Chinese spelling detection and grammatical error correction, there are public data sets such as SIGHAN, CTC2021 and NLPCC2018_GEC available for use. For the existing variant forms in these public data sets, the balance of the number of different variant types is considered, and a suitable amount of data is randomly extracted from them by category; for the variant forms of full pinyin, pinyin abbreviations and split characters, there is currently no available public data set. The present invention uses the Python third-party library xpinyin and the open source project hanzi_chaizi to match and replace a suitable amount of samples randomly selected from the hate speech data set to generate variant word data samples. The matching words used are derived from the word frequency statistics of the top 100 hate speech data sets proposed by the present invention. If the match is successful, the vocabulary will be randomly replaced with its full pinyin, pinyin abbreviation or split character type variant form. By integrating all the above samples (data samples randomly selected from public data sets and variant word data samples generated independently), we finally constructed a variant word restoration dataset of multiple variant types containing 43,000 samples.

[0073] 1.1.2) Restore the samples in the variant word data set obtained in step 1.1.1) according to Figure 2 As shown in the training data sample, the data is uniformly processed into a format that can be used for ChatGLM2-6B model training (that is, the training samples are uniformly processed in JSON format), and the ChatGLM2-6B model is fine-tuned using the P-Tuning v2 method.

[0074] Embodiment 3:

[0075] This embodiment is further optimized on the basis of any of the above embodiments, and the same points as the above technical solutions are not repeated here. Figure 1 to Figure 3 As shown, in order to better implement the method for detecting hate speech based on hate target features and variant word restoration described in the present invention, the following setting is particularly adopted: the step 1.1.2) includes the following specific steps:

[0076] 1.1.2.1) Divide the variant word restoration dataset into training set, validation set and test set in a ratio of 8:1:1;

[0077] 1.1.2.2) The samples in the training set, validation set, and test set are unified into JSON format training samples with "query", "response", and "history" keys, where the "query" key and the "response" key represent the user query and model answer of the current round, respectively, and are used to store variant texts and restored texts; the "history" key records the historical rounds of dialogue, including guidance information such as task background and task description;

[0078] 1.1.2.3) The P-Tuning v2 method was used to fine-tune the ChatGLM2-6B model, and the learning rate was set to 1e-2. Specifically, a trainable continuous vector was inserted as a prefix in the input layer and subsequent network layers of the ChatGLM2-6B model, such as Figure 2 As shown in the Prefix Encoder section, during training, only these newly added vectors are optimized, while the rest of the parameters of ChatGLM2-6B remain frozen;

[0079] 1.1.2.4) Metaphorize the JSON format training sample into the form of "history+query[gmask][sop]response[eop]", where "[gmask]" indicates that the subsequent part is the autoregressive generation part, "[sop]" indicates the start of the autoregressive generation segment, and "[eop]" indicates the end of the generation segment;

[0080] 1.1.2.5) The ChatGLM2-6B model only calculates the loss for the "response" part, completes the ChatGLM2-6B model fine-tuning, and uses cross entropy as the loss function when calculating the loss. The specific formula is as follows:

[0081]

[0082] Among them, Loss1 represents the value of the loss function, y res The lemma result of the real “response” part, y pred_res Represents the tokenized result of the "response" part of the model's prediction.

[0083] Embodiment 4:

[0084] This embodiment is further optimized on the basis of any of the above embodiments, and the same points as the above technical solutions are not repeated here. Figure 1 to Figure 3As shown, in order to better realize the hate speech detection method based on hate object features and variant word restoration described in the present invention, the following setting method is particularly adopted: the variant word restoration dataset includes: A. For variant forms without public datasets, an appropriate amount of sample data is independently generated and constructed; B. Data randomly extracted from public datasets such as SIGHAN, CTC2021 and NLPCC2018_GEC.

[0085] Embodiment 5:

[0086] This embodiment is further optimized on the basis of any of the above embodiments, and the same points as the above technical solutions are not repeated here. Figure 1 to Figure 3 As shown, in order to better implement the method for detecting hate speech based on hate target features and variant word restoration described in the present invention, the following setting is particularly adopted: the step 1.2) includes the following steps:

[0087] 1.2.1) The present invention regards hate object recognition as a task similar to named entity recognition. In the hate object recognition module, a Chinese hate speech dataset containing 57,626 samples is used, and it is divided into a training set, a validation set, and a test set in a ratio of 8:1:1. For each sample in the Chinese hate speech dataset, the present invention uses the BIO (B-begin, I-inside, O-outside) sequence labeling method to perform token-level labeling on the hate objects in the text, using "B-HO" to indicate the beginning of the hate object, "I-HO" to indicate the middle or end of the hate object, and "O" to indicate a token that does not belong to any hate object. All tokens of non-hate samples are marked with the "O" label;

[0088] 1.2.2) The 12-layer encoder of the BERT model is used to extract the semantic features in the text annotated in step 1.2.1). The final hidden layer state is processed by the fully connected layer, and the output obtained represents the hate object score corresponding to each token in the text annotated in step 1.2.1); that is, the present invention uses BERT to extract semantic features. In the input stage, the text t2 is converted into a vector representation Where n and d represent the text length and embedding dimension respectively, and special tags "[CLS]" and "[SEP]" are added at the beginning and end of the text sequence respectively. Subsequently, the word vector sequence is processed by the 12-layer Transformer Encoder structure (encoder layer) of the BERT model, and the learning rate of the BERT model is set to 2e-5. Each layer of Transformer Encoder contains a multi-head attention mechanism (Multi-HeadAttention), a feedforward layer (Feed Forward), and two residual connections and layer normalization layers (Add&Norm). The mapping process from input to output can be expressed as:

[0089] M0=X t2 ;

[0090] M h+1 = Transformer Encoder(M h );

[0091] P=FC(M H );

[0092] Among them, X t2 represents the input of the BERT model, h represents the hth hidden state, and P is the last hidden state M H The output after processing by the fully connected layer represents the hate object score corresponding to each token in the text t2.

[0093] 1.2.3) A CRF layer is introduced after the BERT model designed in step 1.2.2) to form a BERT-CRF model. The constraint modeling between label sequences is realized through the state transfer matrix, and the output is corrected and optimized to ensure the accuracy of the predicted label sequence, thereby realizing accurate identification of hateful objects in the text and completing the training of the hateful object identification module.

[0094] That is, the CRF layer is introduced after the BERT model, and the state transfer matrix is ​​used to model the constraint relationship of the label sequence to ensure that the predicted label sequence is reasonable and effective, and the learning rate of the CRF layer is set to 2e-3. Specifically, the calculation formula of the emission score is as follows:

[0095]

[0096] Where p represents the input sequence, and p∈(p1, p2, ..., p n ), v represents the corresponding label sequence (v1, v2, ..., vn), p i v i It represents the score of the i-th character corresponding to its label, which comes from BERT and the fully connected layer. The calculation formula of the transfer score is as follows:

[0097]

[0098] Where T represents the transfer score matrix, Indicates that from label v i To label v i+1 The transfer scores of , at the beginning of training, these scores will be randomly initialized. The total score of the label sequence is calculated as follows:

[0099] Score(p, v)=EmissionScore(p, v)+TransitionScore(v);

[0100] The learning goal of the CRF layer is to maximize the probability score of the true label sequence among all possible label sequences, so there is the following formula:

[0101]

[0102] Among them, V p Represents the set of all possible label sequences for the input sequence p, and the numerator represents the fraction of the true label sequence. The denominator is the sum of the fractions of all possible label sequences. The hate object recognition module uses negative log-likelihood as the loss function, and the formula is as follows:

[0103]

[0104] Embodiment 6:

[0105] This embodiment is further optimized on the basis of any of the above embodiments, and the same points as the above technical solutions are not repeated here. Figure 1 to Figure 3 As shown, in order to better implement the method for detecting hate speech based on hate target features and variant word restoration described in the present invention, the following setting is particularly adopted: the step 1.3) includes the following steps:

[0106] In the hate speech detection stage, the present invention uses a Chinese hate speech dataset containing 57,626 samples, and divides it into a training set, a validation set, and a test set in a ratio of 8:1:1. To prevent the mixing of training data during testing, the dataset division results of the two stages of hate object identification and hate speech detection are kept consistent. In addition, the homologous texts generated by text enhancement are divided into the same set to avoid data leakage that interferes with the test results.

[0107] 1.3.1) BERT model initialization for the hate speech detection module: The bottom encoder layer of the BERT model (including the bottom encoder layer for feature extraction and the top fully connected layer for classification) is initialized using the BERT encoder layer parameters trained in the hate object recognition module. Given the differences in the top structures of the two modules (the hate object recognition module and the hate speech detection module), the top fully connected layer of the BERT model for the hate speech detection module is randomly initialized.

[0108] 1.3.2) Divide the Chinese hate speech dataset into training set, validation set and test set in a ratio of 8:1:1 as input, and carry out and complete the training of the hate speech detection module.

[0109] That is, the text t2 is converted into X after embedding. t2 , and feature extraction is performed by the encoder layer of the BERT model, and the learning rate is set to 2e-5. The encoder layer uses the initialization parameters inherited from the hate object recognition module to provide a basis for the hate speech detection module to understand the identity word context in the text, enhance the context feature extraction and semantic modeling capabilities, and further extract deeper semantic features f HSD Then, the classification result is output through the top fully connected layer and softmax function. The hate speech detection module uses the cross entropy loss function, which is expressed as follows:

[0110]

[0111] Among them, y is the true label result, The label result predicted by the model.

[0112] Embodiment 7:

[0113] This embodiment is further optimized on the basis of any of the above embodiments, and the same points as the above technical solutions are not repeated here. Figure 1 to Figure 3 As shown, in order to better implement the method for detecting hate speech based on hate target features and variant word restoration described in the present invention, the following setting is particularly adopted: the step 2) includes the following specific steps:

[0114] 2.1) Input any text to be detected into the trained hate speech detection model based on hate target features and variant word restoration, and restore the spelling and grammatical variant words in the text to be detected to their original form through the variant word restoration module to generate a restored text;

[0115] 2.2) The hate speech detection module further extracts the deep semantic features of the restored text and outputs the final hate speech classification result through the top fully connected layer and softmax function.

[0116] The variant word restoration module, hate object identification module, and hate speech detection module will be placed in a centralized location after training. Figure 1 In the framework shown, for an input text, the goal of the hate speech detection model based on hate object features and variant word restoration is to give a prediction result y∈L={l normal , l hate}, the specific detection process is as follows:

[0117] First, the variant word restoration module restores the spelling and grammatical variants in the input text t1 to their original form to generate text t2. Then, text t2 is input to the hate speech detection module and is embedded as n and d represent the text length and embedding dimension respectively. Then, the hate speech detection module after double-stage training will further extract the deep semantic features f of the word embedding vector. HSD , and output the final classification results

[0118] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for detecting hate speech based on hate target features and variant word restoration, characterized in that it comprises the following steps: 1) Construct and train a hate speech detection model based on hate object features and variant word restoration, which consists of a variant word restoration module, a hate object identification module, and a hate speech detection module: 1.1) Fine-tune the ChatGLM2-6B model based on the P-Tuning v2 method and build a variant word restoration module; 1.2) Build and train the hate object recognition module: Use the BIO annotation method to annotate the hate object entity for each sample in the Chinese hate speech dataset, build a BERT-CRF model and perform feature extraction and label sequence optimization to complete the training of the hate object recognition module, and save the BERT encoder layer parameters after training; 1.3) Training the hate speech detection module based on the BERT model: Initialize the BERT encoder layer parameters obtained in the hate object recognition module to the BERT encoder layer of the hate speech detection module, and carry out and complete the training of the hate speech detection module; 2) Using the trained hate speech detection model based on hate object features and variant word restoration to perform final hate speech detection and classification on any input text to be detected; including the following specific steps: 2.1) Input any text to be detected into the trained hate speech detection model based on hate target features and variant word restoration, and restore the spelling and grammatical variant words in the text to be detected to their original form through the variant word restoration module to generate a restored text; 2.2) The hate speech detection module further extracts the deep semantic features of the restored text and outputs the final hate speech classification result through the top fully connected layer and softmax function.

2. The method for detecting hate speech based on hate target features and variant word restoration according to claim 1, characterized in that: The step 1.1) comprises the following steps: 1.1.1) Establish a variant word restoration dataset containing 43,000 samples; 1.1.2) The samples in the variant word restoration dataset obtained in step 1.1.1) are uniformly processed into a format that can be used for ChatGLM2-6B model training, and the ChatGLM2-6B model is fine-tuned using the P-Tuning v2 method.

3. The method for detecting hate speech based on hate target features and variant word restoration according to claim 2, characterized in that: The step 1.1.2) includes the following specific steps: 1.1.2.1) Divide the variant word restoration dataset into training set, validation set and test set in a ratio of 8:1:1; 1.1.2.2) The samples in the training set, validation set, and test set are unified into JSON format training samples with "query", "response", and "history" keys, where the "query" key and the "response" key represent the user query and model answer of the current round, respectively, and are used to store variant texts and restored texts; the "history" key records the historical rounds of dialogue, including task background and task description; 1.1.2.3) Insert a trainable continuous vector as a prefix in the input layer and subsequent network layers of the ChatGLM2-6B model. During training, only these newly added vectors are optimized, while the rest of the parameters of ChatGLM2-6B remain frozen; 1.1.2.4) Metaphorize the JSON format training sample into the form of "history+query[gmask][sop]response[eop]", where "[gmask]" indicates that the subsequent part is the autoregressive generation part, "[sop]" indicates the start of the autoregressive generation segment, and "[eop]" indicates the end of the generation segment; 1.1.2.5) The ChatGLM2-6B model only calculates the loss for the "response" part, completes the ChatGLM2-6B model fine-tuning, and uses cross entropy as the loss function when calculating the loss. The specific formula is as follows: Among them, Loss1 represents the value of the loss function, y res Indicates the lemma result of the real "response" part, y pred_res Represents the tokenized result of the "response" part of the model's prediction.

4. The method for detecting hate speech based on hate target features and variant word restoration according to claim 2 or 3, characterized in that: The variant word restoration dataset includes: A. For variant forms without public datasets, an appropriate amount of sample data is independently generated and constructed; B. Data randomly extracted from SIGHAN, CTC2021 and NLPCC2018_GEC.

5. The method for detecting hate speech based on hate target features and variant word restoration according to claim 1, 2 or 3, characterized in that: The step 1.2) comprises the following steps: 1.2.1) For each sample in the Chinese hate speech dataset, the BIO sequence annotation method is used to annotate the hate objects in the text at the token level; 1.2.2) The 12-layer encoder of the BERT model is used to extract the semantic features in the text annotated in step 1.2.1). The final hidden layer state is processed by the fully connected layer, and the output obtained represents the hate object score corresponding to each token in the text annotated in step 1.2.1); 1.2.3) After the BERT model designed in step 1.2.2), a CRF layer is introduced to form a BERT-CRF model. The constraint modeling between label sequences is implemented through the state transfer matrix, and the output is corrected and optimized to complete the training of the hate object recognition module; 1.2.4) Save the BERT encoder layer parameters after training.

6. The method for detecting hate speech based on hate target features and variant word restoration according to claim 1, 2 or 3, characterized in that: The step 1.3) comprises the following steps: 1.3.1) BERT model initialization for the hate speech detection module: The bottom encoder layer of the BERT model is initialized using the encoder layer parameters of the BERT model trained in the hate object recognition module, while the fully connected layer of the BERT model of the hate speech detection module is randomly initialized; 1.3.2) Divide the Chinese hate speech dataset into training set, validation set and test set in a ratio of 8:1:1, and carry out and complete the training of the hate speech detection module.

Images