Knowledge-enhanced Chinese multi-mode hatred speech detection method

Abstract

The invention discloses a knowledge-enhanced Chinese multi-mode hatred speech detection method, and belongs to the technical field of natural language processing. The method comprises the following steps: acquiring an image-text pair; extracting visual, scene text, text and hatred knowledge features in parallel through a feature representation module; inputting the text and the prompt template into a large language model to generate a background knowledge text, and splicing the background knowledge text with the text features and hatred knowledge features after BERT coding to form final text features; by means of a knowledge-guided cross-modal attention mechanism, an image-text affinity matrix generated by the CLIP is introduced into an attention map, multi-modal feature fusion is achieved, and the hatred category is predicted. According to the method, large language model background knowledge and a Chinese hatred vocabulary are introduced, the problems of Chinese context understanding, modal difference and hatred word recognition are remarkably relieved, the higher accuracy rate, precision rate, recall rate and F1 score are obtained on a CMMHS data set, and the method is suitable for Chinese multi-modal hatred content auditing on a social platform.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology, specifically relating to a knowledge-enhanced method for detecting Chinese multimodal hate speech. Background Technology

[0002] Hate speech, a form of discriminatory, offensive, and hostile rhetoric targeting specific groups or individuals, has emerged with the development of social media platforms. This phenomenon has evolved into a serious social hazard, potentially misleading the public and causing panic. To address this issue, hate speech detection tasks have emerged. Multimodal hate speech detection aims to identify whether content contains hate speech and to identify the specific hate categories by integrating image and text descriptions. The main challenges of this task are threefold: 1) how to mitigate the difficulty of understanding the Chinese language context; 2) how to mitigate the difficulties caused by differences in multimodal expression; and 3) how to detect hate words in the Chinese context. In recent years, thanks to the development of large multimodal models, some researchers have proposed multimodal hate speech detection methods and datasets. However, existing work mainly focuses on the English domain, while Chinese hate speech detection primarily concentrates on the text domain, and research on Chinese multimodal hate speech detection remains insufficient. Therefore, researching a knowledge-enhanced Chinese multimodal hate speech detection framework is of great significance, as it will help improve the performance of large multimodal models in related tasks. Summary of the Invention

[0003] To address the problems existing in the prior art, this invention proposes a knowledge-enhanced Chinese multimodal hate speech detection method. The aim is to integrate image and text descriptions to identify whether the content contains hate speech and to identify the types of hate speech present, thereby improving the performance of the multimodal large model in Chinese multimodal hate speech detection tasks.

[0004] To achieve the above objectives, the specific solution of the present invention is as follows:

[0005] This invention provides a knowledge-enhanced Chinese multimodal hate speech detection method, comprising the following steps:

[0006] S1. Acquisition Step, used to acquire image-text pairs;

[0007] S2. Feature representation step: The image-text pair is processed in parallel by the feature representation module to obtain:

[0008] Visual features The The appearance features of M targets are obtained by pre-training Mask R-CNN. and 4-dimensional relative position features Learnable weight matrix , After mapping and layer normalization, element-wise summation yields: = ,in and These are learnable parameters. Representation layer normalization;

[0009] Scene text features The Obtain the FastText 300-dimensional semantic features of N scene texts using Baidu AI Cloud OCR API. Normalized bounding box location features and confidence characteristics Learnable weight matrix , , After mapping and layer normalization, element-wise summation yields: = ,in , and These are learnable parameters. Representation layer normalization;

[0010] Text features The 768-dimensional sentence-level features were extracted using BERT.

[0011] Characteristics of hatred knowledge The The words in the text are matched with a preset Chinese hate vocabulary list by a lookup table and then embedded.

[0012] S3. Background Supplementation Step: Concatenate the text with the preset prompt template and input it into the large language model to generate background knowledge text. This background knowledge text is then encoded using BERT to obtain external knowledge features. and text features and characteristics of hatred knowledge The features are concatenated along the feature dimensions to form the final text features. =cat( , , );

[0013] S4. The multimodal reconstruction step utilizes a knowledge-guided cross-modal attention mechanism to calculate a correction matrix to bridge the semantic gap between images and text, fuse multimodal features, and predict hate categories.

[0014] Furthermore, the text in step S1 includes one or more of the following: Chinese Weibo text, short video captions, comments, or bullet comments.

[0015] Furthermore, the multimodal reconstruction step specifically includes:

[0016] 1) Affinity matrix generation steps: Visual features... Image representation obtained by CLIP image encoder , and joint text features Aggregated text representation obtained by pooling Cosine similarity is calculated, and then weighted using a learnable Gaussian kernel function to generate an affinity matrix. This is used to quantify the semantic association strength between text and image modalities;

[0017] 2) KGCA Network Processing Steps: Each encoder layer contains a KGCA network that integrates the affinity matrix into the attention map. The initial input to the encoder layer consists of visual features. and joint text features The KGCA layer is composed of a self-attention layer and a multilayer perceptron (MLP) for cross-modal interaction. The output is then used as an intermediate state input to the decoder.

[0018] 3) Decoder output steps: Using the initial features of the input text as the query and the intermediate state as the key, the decoder output is passed through the MLP and softmax layers to obtain the final prediction result.

[0019] Furthermore, the KGCA network processing steps specifically include: 1) mapping the input to a high-dimensional space through a learnable weight matrix, and then performing matrix multiplication to generate the corresponding weight map; 2) using external knowledge to perform Hadamard product operation on the similarity map and the weight map to effectively bridge the differences between modes; 3) normalizing the attention map through the L1 norm to obtain the optimized attention map.

[0020] Furthermore, in the affinity matrix generation step, the affinity matrix... The calculation formula is as follows:

[0021] =ɑ· ,

[0022] Where α and σ are learnable parameters, and L2 represents normalization.

[0023] Furthermore, the joint text features From the final text features Scene text features The result obtained by piecing together:

[0024] = .

[0025] Furthermore, the optimized attention graph is calculated as follows:

[0026] = ,

[0027] = ,

[0028] = ,

[0029] Where Q and K represent the query vector and key vector, respectively. and It is a learnable query matrix and key matrix, ⊙ represents the Hadamard product operator, represents the input channel dimension, and L1 is the L1 normalization operation.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] This invention proposes a knowledge-enhanced Chinese multimodal hate speech detection method. This method utilizes a large language model to extract background information for hate speech detection from text, bridges the gap between visual and textual modalities through CLIP technology, and introduces a Chinese hate speech vocabulary as external knowledge to embed into text features, enhancing the model's ability to identify Chinese hate speech terms. This invention addresses the difficulties in understanding the Chinese language background, the challenges arising from differences in multimodal expression, and how to detect hate speech terms in Chinese multimodal hate speech detection. It improves the model's understanding of the Chinese context by extracting background information from a large language model and significantly enhances the accuracy, recall, and F1 score of the model in Chinese multimodal hate speech detection by using CLIP technology to narrow the distance between images and text. Attached Figure Description

[0032] Figure 1 This is a flowchart of the Chinese multimodal hate speech detection method for knowledge enhancement as described in this invention.

[0033] Figure 2 This is the overall design diagram of the knowledge-enhanced Chinese multimodal hate speech detection method of this invention. Detailed Implementation

[0034] The present invention will be further explained and described below with reference to the accompanying drawings and specific embodiments. It should be noted that the specific embodiments are not intended to limit the scope of the present invention.

[0035] The method of this invention is implemented on the CMMHS public dataset, which covers 8,122 labeled samples. This dataset contains more granular annotation information, namely, multi-category labels such as race, gender, region, and LGBTQ. It not only includes image data containing OCR text, but also integrates the corresponding post text.

[0036] like Figure 1 As shown in the figure, this specific embodiment provides a knowledge-enhanced Chinese multimodal hate speech detection method, the steps of which are as follows:

[0037] S1. Acquisition Steps: This step is used to obtain image-text pairs; the text is Chinese microblog text.

[0038] S2. Feature representation step: The image-text pair is processed in parallel by the feature representation module, and the steps are as follows:

[0039] 1) Text features of Weibo For a given Weibo text containing a sequence of K words, BERT is used to extract 768-dimensional text features. To leverage the inherent characteristics of hate speech in Chinese texts, we introduce the characteristics of hate knowledge. The words in the Weibo text are matched with a preset Chinese hate vocabulary list by looking up a table and then embedded to obtain the vocabulary.

[0040] 2) Visual characteristics For a given image, a pre-trained Mask R-CNN is used as the object detector. The appearance features of the m-th object extracted from the region proposal network are selected. To utilize the target's location information, a 4D location feature is introduced for each target. , containing the relative position bounding box of each target [ , , , ];

[0041] 3) Scene text features For a given image, scene text information is first extracted using the Baidu AI Cloud OCR API, including the content, bounding box location, and confidence score of each scene text. To accurately represent scene text features, the following three types of features are extracted for each text:

[0042] ① Extracting 300-dimensional semantic features using the FastText model ;

[0043] ② The location features are obtained by normalizing the bounding box of the scene text. =[ / , / , / , / ];

[0044] ③ Introduce scene text confidence features To control the degree of reliance on textual information.

[0045] The visual features The final image features are extracted as follows:

[0046] = ,

[0047] in and These are learnable parameters. Representation layer normalization.

[0048] The final scene text extraction features are as follows:

[0049] = ,

[0050] in , and These are learnable parameters. Representation layer normalization.

[0051] S3. Background Supplementation Step: The designed question is posed to the large language model. The model's output answer set is used as the external text. The Weibo text is concatenated with a preset prompt template and input into the large language model to generate background knowledge text. This background knowledge text is then encoded using BERT to obtain external knowledge features. and with Weibo text features and characteristics of hatred knowledge The features are concatenated along the feature dimensions to form the final text features. ;

[0052] The background supplementation step uses Represent the connection of the answer, and then , and The concatenations form the final text feature, as shown in the following formula:

[0053] =cat( , , ),

[0054] in For the final text features, Features of Weibo text Characterized by hatred of knowledge. Features of external knowledge.

[0055] S4. The multimodal reconstruction step utilizes a knowledge-guided cross-modal attention mechanism to calculate a correction matrix to bridge the semantic gap between images and text, fuse multimodal features, and predict hate categories.

[0056] The multimodal reconstruction steps are as follows:

[0057] 1) Affinity matrix generation steps: Visual features... Image representation obtained by CLIP image encoder , and joint text features Aggregated text representation obtained by pooling Cosine similarity is calculated, and then weighted using a learnable Gaussian kernel function to generate an affinity matrix. This is used to quantify the semantic association strength between text and image modalities;

[0058] 2) KGCA Network Processing Steps: Each encoder layer contains a KGCA network that integrates the affinity matrix into the attention map. The initial input to the encoder layer consists of visual features. and joint text features The KGCA layer is composed of a self-attention layer and a multilayer perceptron (MLP) for cross-modal interaction. The representation is further fused, and the output is used as an intermediate state input to the decoder.

[0059] 3) Decoder output steps: Using the initial features of the input text as the query and the intermediate state as the key, the decoder output is passed through the MLP and softmax layers to obtain the final prediction result.

[0060] The KGCA network processing steps are as follows: 1) The input is mapped to a high-dimensional space through a learnable weight matrix, and then matrix multiplication is performed to generate the corresponding weight map; 2) The similarity map and the weight map are subjected to Hadamard product operation using external knowledge to effectively bridge the differences between modes; 3) The attention map is normalized by L1 norm to obtain the optimized attention map.

[0061] The affinity matrix generation step, affinity matrix The calculation formula is as follows:

[0062] =ɑ· ,

[0063] Where α and σ are learnable parameters, and L2 represents normalization.

[0064] The joint text features From the final text features Scene text features The result obtained by piecing together:

[0065] = ,

[0066] in Joint features of scene text and text features For the final text features, The scene text features output by the text encoder.

[0067] The optimized attention graph calculation is as follows:

[0068] = ,

[0069] = ,

[0070] = ,

[0071] Where Q and K represent the query vector and key vector, respectively. and It is a learnable query matrix and key matrix, ⊙ represents the Hadamard product operator, represents the input channel dimension, and L1 is the L1 normalization operation.

[0072] The following section evaluates the knowledge-enhanced Chinese multimodal hate speech detection method of this embodiment on the CMMHS public dataset. The CMMHS public dataset contains 8122 samples, which are labeled not only for whether the samples contain hate speech, but also for the types of hate speech.

[0073] This embodiment compares the model with several other models, including unimodal models and pre-trained multimodal models, specifically: 1) unimodal models; 2) multimodal models; and 3) multimodal large language models (MLLMs). Common evaluation metrics are used to measure model performance in classification tasks, primarily including accuracy, precision, recall, and F1 score, as detailed below:

[0074] (1) Accuracy (Acc (%)): refers to the proportion of correctly predicted samples to the total number of samples. It is used to reflect the overall accuracy of the model's predictions. The higher the value of this indicator, the better.

[0075] (2) Precision (P (%)): refers to the proportion of samples that are predicted to be positive but are actually positive. It is used to measure the accuracy of the prediction results. The higher the value of this indicator, the better.

[0076] (3) Recall (R (%)): refers to the proportion of samples that are actually positive that are correctly predicted. The higher the value of this indicator, the better.

[0077] (4) F1 score (F1 (%)): is the harmonic mean of precision and recall, which can comprehensively reflect the performance of both and avoid the bias caused by a single indicator. The higher the value of this indicator, the better.

[0078] The final performance of the model evaluation metrics in this embodiment is shown in Table 1.

[0079]

[0080] The final test set results for each category in this embodiment are shown in Table 2.

[0081]

[0082] The results from the above specific embodiments show that, combined with the method described in this embodiment, all evaluation metrics of the model are significantly improved, indicating that this method can effectively enhance the model's performance in hate speech detection tasks. Furthermore, different text representation models were used to encode the Weibo text, including BERT, RoBERTa, and BGE. The last three rows of Table 1 show that the BGE model achieves superior performance; in the results of each category of test sets, all evaluation metrics reached a high level.

[0083] The above embodiments are merely preferred examples of the present invention and are not intended to limit the invention. Those skilled in the art can make various modifications and variations based on the present invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are within the scope of protection of the present invention.

Claims

1. A knowledge-enhanced method for detecting Chinese multimodal hate speech, characterized in that, Includes the following steps: S1. Acquisition Step, used to acquire image-text pairs; S2. Feature representation step: The image-text pair is processed in parallel by the feature representation module to obtain: Visual features The The appearance features of M targets are obtained by pre-training Mask R-CNN. and 4-dimensional relative position features Learnable weight matrix , After mapping and layer normalization, element-wise summation yields: = ,in and These are learnable parameters. Representation layer normalization; Scene text features The Obtain FastText 300-dimensional semantic features of N scene texts using Baidu AI Cloud OCR API. Normalized bounding box location features and confidence characteristics Learnable weight matrix , , After mapping and layer normalization, element-wise summation yields: = ,in , and These are learnable parameters. Representation layer normalization; Text features The 768-dimensional sentence-level features were extracted using BERT. Characteristics of hatred knowledge The The words in the text are matched with a preset Chinese hate vocabulary list by a lookup table and then embedded. S3. Background Supplementation Step: Concatenate the text with the preset prompt template and input it into the large language model to generate background knowledge text. This background knowledge text is then encoded using BERT to obtain external knowledge features. and text features and characteristics of hatred knowledge The features are concatenated along the feature dimensions to form the final text features. =cat( , , ); S4. The multimodal reconstruction step utilizes a knowledge-guided cross-modal attention mechanism to calculate a correction matrix to bridge the semantic gap between images and text, fuse multimodal features, and predict hate categories.

2. The method according to claim 1, characterized in that, The text in step S1 includes one or more of the following: Chinese Weibo text, short video captions, comments, or bullet comments.

3. The method according to claim 1, characterized in that, The multimodal reconstruction step specifically includes: 1) Affinity matrix generation steps: Visual features... Image representation obtained by CLIP image encoder , and joint text features Aggregated text representation obtained by pooling Cosine similarity is calculated, and then weighted using a learnable Gaussian kernel function to generate an affinity matrix. This is used to quantify the semantic association strength between text and image modalities; 2) KGCA Network Processing Steps: Each encoder layer contains a KGCA network that integrates the affinity matrix into the attention map. The initial input to the encoder layer consists of visual features. and joint text features The KGCA layer is composed of a self-attention layer and a multilayer perceptron (MLP) for further fusion representation after cross-modal interaction. The output is then used as an intermediate state input to the decoder. 3) Decoder output steps: Using the initial features of the input text as the query and the intermediate state as the key, the decoder output is passed through the MLP and softmax layers to obtain the final prediction result.

4. The method according to claim 3, characterized in that, The KGCA network processing steps specifically include: 1) mapping the input to a high-dimensional space through a learnable weight matrix, and then performing matrix multiplication to generate the corresponding weight map; 2) using external knowledge to perform Hadamard product operation on the similarity map and the weight map to effectively bridge the differences between modes; 3) normalizing the attention map through the L1 norm to obtain the optimized attention map.

5. The method according to claim 3, characterized in that, In the affinity matrix generation step, the affinity matrix... The calculation formula is as follows: =ɑ· , Where α and σ are learnable parameters, and L2 represents normalization.

6. The method according to claim 3, characterized in that, The joint text features From the final text features Scene text features The result obtained by piecing together: = 。 7. The method according to claim 4, characterized in that, The optimized attention graph calculation is as follows: = , = , = ， Where Q and K represent the query vector and key vector, respectively. and It is a learnable query matrix and key matrix, ⊙ represents the Hadamard product operator, represents the input channel dimension, and L1 is the L1 normalization operation.

Images