Semantic enhancement-based hotspot speech detection method, program, equipment and storage medium

Abstract

The invention provides a hotspot speech detection method based on semantic enhancement, a program, equipment and a storage medium. The invention provides a hot speech detection model HSIBSE based on semantic enhancement. The system comprises an information embedding layer device, a text embedding layer device, a coding layer device and an output layer device. The method comprises the following steps of: classifying texts by using a hot speech dictionary, and dividing the texts into two categories containing hot words and not containing the hot words; for each word, the invention matches the word with a word in a hot-spot word dictionary to determine whether the word belongs to a hot-spot word. The hotspot speech can be identified and analyzed more accurately, and important clues are provided for hotspot speech detection. By enhancing context understanding, analyzing emotional tendency in the hotspot speech and enhancing information association, the method is beneficial to identifying implicit information in some texts, and can be widely applied to mutual hotspot speech detection in social networks.

Description

Technical Field

[0001] The present invention belongs to the field of social networks, and specifically relates to a hot speech detection method, program, device and storage medium based on semantic enhancement. Background Art

[0002] In social networks, the changes that social media has brought to human beings cannot be ignored. Through social media, people can easily communicate and interact with friends, family, colleagues and strangers. Social media also makes information dissemination faster and more extensive, bringing many new business opportunities. However, social media also brings some hot effects, such as time waste, health problems and privacy issues caused by addiction to social media. At the same time, social media may also lead to information flooding and deterioration of information quality, and may also be used for malicious purposes, such as hot speech, rumors, etc. Therefore, people need to think seriously about how to make better use of social media, while paying attention to the risks and challenges it may bring.

[0003] The harmfulness of hot comments is particularly serious in the problems caused by these social media. Natural language has semantic complexity, and understanding the semantics of natural language requires a certain degree of reasoning ability and the ability to combine context. For example, a sentence may have multiple different interpretations and has semantic complexity, such as "This is great!". This sentence seems to express positive emotions, but if the specific context is not known, it may also lead to deviations in understanding. For example, if the context is "This is great! It makes us realize that our company has serious loopholes in data protection", then the semantics of this sentence becomes more complex because it contains a negative message. The semantic complexity of hot comments lies in that they often contain multiple emotions and meanings, and these emotions and meanings often depend on the context and cultural background. For example, some words or expressions that seem harmless on the surface may be regarded as hot comments in a specific context.

[0004] In addition, the semantic complexity of hot topics is also reflected in the fact that they often use various rhetorical devices such as metaphors, similes and allusions, making it difficult to express their true intentions directly.

[0005] These comments are often highly emotional and aggressive, and spread quickly on social media, affecting a large number of people in a short period of time. Therefore, it is very important to detect and classify these hot comments so that they can be identified and prevented from spreading in a timely manner, and maintain the security and fairness of social media platforms. Summary of the invention

[0006] The purpose of the present invention is to provide a method, program, device and storage medium for detecting hot speech based on semantic enhancement.

[0007] The purpose of the present invention is achieved through the following technical solutions:

[0008] A hot topic speech detection method based on semantic enhancement includes the following steps:

[0009] Step 1: Information embedding layer, embedding hot information in the hot speech dictionary into the model;

[0010] First, it is necessary to integrate hot information into sentences based on the constructed hot speech dictionary; in sentences, hot speeches usually express concern about a certain event, and the content of tweets is identified based on the hot words of hot speeches; effectively improve the coverage and feature dimensions of feature words, thereby improving the detection performance of the model; through the processing of the information embedding layer, the model can identify hot speeches and provide comprehensive analysis results to provide support for combating hot speeches.

[0011] Step 2: Text embedding layer to perform language model prediction tasks;

[0012] The model will add some special symbols "[MASK]" to the sequence. These symbols indicate that the words at the corresponding positions need to be predicted, that is, they are replaced by appropriate words by the model; that is, the missing words are predicted given the context, and the "[MASK]" symbol indicates the position of the word that needs to be predicted;

[0013] Step 3: The encoding layer includes multiple Transformer modules, each of which includes: a multi-head attention layer, a feedforward neural network sublayer, and a normalization layer;

[0014] Each sub-layer has a residual connection and a layer normalization step. After the text is embedded, the representation vector is sent to the multi-head attention layer, and the input is mapped into three vectors: query vector Q, key vector K and value vector V. These three vectors are used to calculate the attention score, thereby determining the weight of the input vector, and then the input vector is weighted averaged according to the weight.

[0015] Step 4: Output layer: The output layer predicts whether the sentence contains hot topics.

[0016] Through a fully connected layer, all hidden states of the last layer are linearly transformed and a bias term is added, and then an activation function is used to obtain the final output vector.

[0017] Furthermore, the step 1 specifically includes the following steps:

[0018] Step 1.1. It is necessary to integrate hot information into sentences based on the constructed hot speech dictionary; in sentences, hot speech usually expresses concern about a certain event;

[0019] Step 1.2. Use the N-gram algorithm to determine whether the sentence contains hot speech words, where n ranges from 1 to 3;

[0020] Step 1.3. Determine the word w through the N-gram method i When it is a hot speech word, the corresponding hot information contained is embedded into the sentence.

[0021] Furthermore, the step 2 specifically includes the following steps:

[0022] Step 2.1. For part of the input text sequence, the model will add some special symbols "[MASK]" to the sequence. These symbols indicate that the words at the corresponding positions need to be predicted, that is, they are replaced by appropriate words by the model;

[0023] Step 2.2. Use Token Embedding to map each word in the sentence that embeds the hot speech information in this study to a vector representation of a fixed size; use Segment Embedding to distinguish the two sentences so that the model knows which words belong to which sentence;

[0024] Step 2.3. The model tokenizes the sentences and adds special tags [CLS] and [SEP] to indicate the start and end of the sentences.

[0025] Step 2.4. Use Position Embedding to encode the position information of each word into the embedding vector. The model recognizes the pattern in the sequence based on the position of the word in the sequence.

[0026] Furthermore, in step 3, the input vector is weighted averaged according to the weight, and the formula is as follows:

[0027]

[0028] Among them, Q, K, and V are equal; each word in the sentence must calculate attention with other words in the same sentence to learn the relevance between words; is a scaling factor, where dk is the dimension of the K vector, which is used to control the range of the attention distribution.

[0029] Furthermore, in step 4, the final output vector is obtained by an activation function, and the formula is as follows:

[0030] P(y|x)=σ(W·h i +b)

[0031] Among them, hi represents the output vector of the last layer of the model, W and b are the weight matrix and bias vector, and σ is the probability of the sigmoid function used to output the prediction result.

[0032] Furthermore, in step 1.3, in order to better utilize the information in the hot speech vocabulary, the hot category is attached to the embedding of the sentence.

[0033] A hot speech detection system based on semantic enhancement, characterized by: comprising an information embedding layer device, a text embedding layer device, a coding layer device and an output layer device;

[0034] The information embedding layer device: the main task is to embed the hot information in the hot speech dictionary into the model;

[0035] Text embedding layer device: Use the BERT model to perform deep learning and embedding on text. In the BERT model, a technology called mask mechanism is used to enable the model to process single-sentence and double-sentence inputs at the same time.

[0036] Encoding layer device: including Transformer modules; each module contains: multi-head attention layer, feedforward neural network sublayer and normalization layer;

[0037] Output layer device: predicts whether the sentence contains hot words, through a fully connected layer, it linearly transforms all the hidden states of the last layer and adds a bias term, and then obtains the final output vector through an activation function.

[0038] A computer device / equipment / system includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-mentioned hot speech detection method based on semantic enhancement.

[0039] A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: when the computer program / instruction is executed by a processor, the steps of the above-mentioned hot speech detection method based on semantic enhancement are implemented.

[0040] A computer program product, including a computer program / instruction, characterized in that: when the computer program / instruction is executed by a processor, the steps of the above-mentioned hot speech detection method based on semantic enhancement are implemented.

[0041] The beneficial effects of the present invention are:

[0042] The present invention detects and classifies hot comments so as to timely identify and prevent the spread of such comments and maintain the security and fairness of social media platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Schematic diagram of the system of the present invention;

[0044] Figure 2 A structural block diagram of the system of the present invention;

[0045] Figure 3 Multi-head attention mechanism in the system of the present invention. DETAILED DESCRIPTION

[0046] The present invention is further described below in conjunction with the accompanying drawings.

[0047] The overall structure of the system is as follows Figure 1 As shown, specifically:

[0048] The information embedding layer plays an important role in hot speech detection. Its main task is to embed the hot information in the hot speech dictionary into the model. To achieve this goal, it is first necessary to integrate the hot information into the sentence based on the constructed hot speech dictionary. In the sentence, hot speech usually expresses concern about a certain event, so the content of the tweet can be identified based on the hot words of the hot speech. This approach can effectively improve the coverage and feature dimension of the feature words, thereby improving the detection performance of the model. Through the processing of the information embedding layer, the model can better identify hot speech and provide more comprehensive analysis results, providing strong support for combating hot speech.

[0049] The N-gram algorithm is used to determine whether a sentence contains a hot speech word, where n ranges from 1 to 3. The hot speech dictionary divides tweets into two categories, either containing derogatory words or not, and then assigns these two categories to each sentence in the tweet.

[0050] Assume that a sentence S consists of n words, S = [w1, w2, …, wn], where w represents a word and wi represents the i-th word in the sentence.

[0051] When the word wi is determined to be a hot speech word by the N-gram method, the corresponding hot information contained therein is embedded into the sentence. In order to better utilize the information in the hot speech words, the present invention attaches the hot category to the embedding of the sentence. For example, in the sentence "Go fucking kill yourself and die already useless ugly pile of shitscumbag.", "fucking" and "shit scumbag" are obviously hot language tendencies, so the present invention adds the hot speech information of "offensive content" to the sentence to form a sentence representation rich in hot speech information.

[0052] The text embedding layer device uses the BERT (Bidirectional Encoder Representations from Transformers) model, which can perform deep learning and embedding of text. In the BERT model, in order to enable the model to process single-sentence and double-sentence inputs at the same time, a technology called the mask mechanism is used. Specifically, for a part of the input text sequence, the model will add some special symbols "[MASK]" to the sequence. These symbols indicate that the words at the corresponding positions need to be predicted, that is, replaced by appropriate words by the model. This process can be understood as a prediction task of the language model, that is, predicting missing words given a context, and the "[MASK]" symbol indicates the position of the word that needs to be predicted.

[0053] In the present invention, the sentence with information layer embedding is first converted into a vector representation through three embedding layers:

[0054] Token Embedding is used to map each word in the sentence that completes the embedding of hot speech information in the present invention to a vector representation of a fixed size. Segment Embedding is used to distinguish the two sentences so that the model knows which words belong to which sentence. At the input layer, the model segmented the sentences used in this experiment and added special tags [CLS] and [SEP] to indicate the beginning and end of the sentence. Position Embedding is used to encode the position information of each word into the embedding vector so that the model can recognize the pattern in the sequence based on the position of the word in the sequence. The three embedding layers together constitute the input layer of this experiment. They can convert an input sequence into a multidimensional tensor for processing in subsequent layers of the model.

[0055] Figure 2 The structure block diagram of the system of the present invention is shown, comprising:

[0056] Information embedding layer device: The main task is to embed the hot information in the hot speech dictionary into the model.

[0057] Text embedding layer device: Using the BERT (Bidirectional Encoder Representations from Transformers) model, text can be deeply learned and embedded. In the BERT model, a technology called mask mechanism is used to enable the model to process single-sentence and double-sentence inputs at the same time.

[0058] Encoding layer device: It is composed of multiple Transformer modules. Each module contains: multi-head attention layer, feedforward neural network sublayer and normalization layer.

[0059] Output layer device: predicts whether the sentence contains hot words, through a fully connected layer, which linearly transforms all hidden states of the last layer and adds a bias term, and then obtains the final output vector through an activation function.

[0060] like Figure 3 , the encoding layer device in the model is composed of multiple Transformer modules. Each module contains: a multi-head attention layer, a feedforward neural network sublayer, and a normalization layer. Specifically, each sublayer has a residual connection and a layer normalization step. In this experiment, after the text is embedded, the representation vector is sent to the multi-head attention layer, and the input is mapped into three vectors: the query vector Q, the key vector K, and the numerical vector V. These three vectors are used to calculate the attention score, thereby determining the weight of the input vector, and then the input vector is weighted averaged according to the weight. The formula is as follows:

[0061]

[0062] In the task of the present invention, Q, K, and V are equal. Each word in a sentence must calculate attention with other words in the same sentence to learn the relevance between words. is the scaling factor, where d k is the dimension of the K vector, which is used to control the range of attention distribution. When calculating the relevance, for each word vector, it is necessary to perform a dot product with other word vectors, then divide it by the square root of the scaling factor, and finally obtain the attention weight through the normalization function. The normalization method used here is RELU:

[0063]

[0064] The model contains multiple Transformer encoding layers, each of which repeats the calculations of the above-mentioned multi-head attention layer and feedforward neural network sublayer to capture different levels of semantic information in the input sequence.

[0065] The output layer device predicts whether the sentence contains hot words, passes through a fully connected layer, which linearly transforms all hidden states of the last layer and adds a bias term, and then obtains the final output vector through an activation function. The formula is as follows:

[0066] P(y|x)=σ(W·h i +b)

[0067] Among them, h iRepresents the output vector of the last layer of the model, W and b are the weight matrix and bias vector, and σ is the probability of the sigmoid function used to output the prediction result. When the output result P>0.5, the prediction is positive, otherwise it is negative.

[0068] The loss function is calculated using cross entropy:

[0069]

[0070] Where N is the size of the data set; Q(i,j) indicates whether the prediction result is correct, 1 for correct and 0 for wrong; P(ij) is the probability of the predicted sample; λ||θ|| 2 is the regularization term.

[0071] The following describes in detail the experimental example scenarios of the present invention, and analyzes the implementation results in combination with the advantages of the present invention.

[0072] Experiments were conducted on three datasets: HASOC-21

[49] , HASOC-20

[48] , and HASOC-19

[47] , all of which were provided by the international conference Forum for Information Retrieval Evaluation (FIRE) in the Hate Speech and Offensive Content Identification in Indo-European Languages ​​task. HASOC is a research project funded by the Indian government and the European Union that aims to improve the detection and identification capabilities of hot speech and offensive content on social media platforms. The project provides datasets for evaluating and improving existing hot speech and offensive content detection algorithms. The goal of the project is to promote research on hot speech and offensive content on social media and to improve the accuracy and efficiency of related technologies. These data are mainly derived from posts, comments, and tweets on multiple social media platforms, and are used to evaluate the hot speech detection task on social media. In this experiment, the present invention uses the English dataset in the dataset for experiments.

[0073] HASOC-21: A dataset for detecting hot speech and offensive content proposed by the FIRE conference in 2021. It is used for hot speech detection tasks. It has a total of 3843 data, of which 1342 data are labeled NOT and 2501 data are labeled HOF. NOT means that the sample does not contain hot speech, and HOF means that the sample contains hot speech.

[0074] HASOC-20: A dataset for detecting hot speech and offensive content proposed by the FIRE conference in 2020. It contains 3708 pieces of data, of which 1852 pieces are labeled as NOT and 1856 pieces are labeled as HOF.

[0075] HASOC-19: A dataset for detecting hot topics and offensive content proposed by the FIRE conference in 2019. It contains 5,852 pieces of data, of which 3,591 pieces are labeled NOT and 2,261 pieces are labeled HOF.

[0076] The distribution of each data set is shown in Table 1. Table 2 is an example of data in the HASOC data set. It shows the format of the data in the HASOC-21, HASOC-20, and HASOC-19 data sets, including text content and text labels. In the experiment, the data set is divided into training set, validation set, and test set in a ratio of 3:1:1, and the data set is randomly shuffled.

[0077] Table 1

[0078]

[0079] Table 2

[0080]

[0081] This experiment uses Google Colab to build the model based on the Pytorch framework and fine-tune the BERT model. The vector dimension of BERT-base is 768, the maximum text input size is set to 128 characters, the model size is 394M, and the initial learning rate is set to 2e-5. If the training speed is slow or the accuracy is poor during training, the learning rate can be gradually increased. The GPU is NVIDIA Tesla T4, and other parameters are shown in Table 3.

[0082] Table 3

[0083]

[0084] Batch_size is initially set to 16. It is a hyperparameter in model training. It defines how many samples the model takes out from the data set for training each time. Using a larger batch size can improve training efficiency, but it will take up more memory and may cause the model to not fit the data well enough. Using a smaller batch_size can improve the model's fit to the data, but it will reduce training efficiency. In practice, it is necessary to choose a suitable batch_size based on many factors such as the model architecture, the size of the data set, and the hardware of the training environment. Epoch refers to the number of times the complete data set is trained by the model. When the model completes a training of the entire data set, an epoch ends. Usually, the model needs multiple epochs to get the best results, and each epoch will update the model's parameters. In this experiment, epoch is set to 5.

[0085] In order to verify the effect of the hot speech detection system and method based on semantic enhancement proposed by the present invention, the present invention adopts a comparative experiment method as shown below:

[0086] (1) GCN: Graph Convolutional Network (GCN) is a version of graph neural network that contains additional convolutional layers. et al. proposed a hot speech detection system and method based on graph convolutional network in the HASOC2021 competition. They used GCN to vectorize the text and induced the embedding vector based on the neighbor characteristics of the node. This method won the first place in the competition.

[0087] (2) HatabaseTwitter: This method is the one used by Glazkova et al. in the HASOC 2021 competition. Additional datasets were used for training and the RoBERTa model was fine-tuned. This method mainly addresses the problem of less training data and unbalanced data in different categories.

[0088] (3) TextCNN: TextCNN is a deep learning model for text classification. TextCNN represents the input text as a matrix, where each row represents the word vector of a word, and then performs convolution operations through multiple convolution kernels of different sizes to extract local features of the text. The result after convolution is pooled through a pooling layer and finally classified through a fully connected layer. The advantage of TextCNN is that it can effectively extract local features in the text, has fewer parameters, and is faster to train. The disadvantage is that it cannot handle long texts well and needs to be truncated. At the same time, for some complex text classification tasks, the effect may not be as good as other more complex deep learning models.

[0089] (4) BERT: In the BERT model, the input layer processes each input text sequence, including word segmentation, adding special tags (such as CLS and SEP), and word vector embedding. During the embedding process, BERT uses a subword embedding method based on WordPiece. For words that are not in the dictionary, character-level decomposition is performed to generate corresponding subwords. In the encoding layer, BERT uses a Transformer structure to encode the input text through a multi-layer self-attention mechanism and a feedforward neural network. Unlike traditional convolutional neural networks and recurrent neural networks, the Transformer structure can process all position information in the input sequence at the same time, allowing the model to capture more global contextual information, thereby achieving better results. The present invention fine-tuned the BERT model using the HASOC dataset and applied it to this experiment.

[0090] (5) Nlplab_isi: This method is a model for detecting hot speech proposed by Singh et al. in 2022. It integrates the Google-MuRIL, XLM-Roberta, and BERT models. After training the dataset with three Transformer models, the model output category probabilities are summed up, and the category with the highest sum of probabilities is selected as the output.

[0091] Experiment 1:

[0092] It can be seen from Table 4 that the semantically enhanced hot speech detection model HSIBSE that integrates the hot speech dictionary has achieved good results on the HASOC-21 dataset. Compared with the classic deep learning model TextCNN model, it has been significantly improved. Compared with the detection systems and methods that rank high in the competition website, the method proposed in the present invention also has similar effects and a certain degree of improvement.

[0093] Table 4

[0094]

[0095] The convolution operation of TextCNN only considers the local word relationship and ignores the position information of the word in the entire text. In some texts, the position information may be very important for correct classification, but TextCNN cannot effectively capture this information. In addition, TextCNN has limited semantic understanding ability. TextCNN is mainly based on the convolution operation of local windows, and it may not be able to capture the global semantic information of the text.

[0096] The GCN method mainly relies on the graph structure and the connection relationship between nodes for information propagation and feature update. However, for complex natural language texts, only considering the connection between nodes may not fully capture the semantic information of the sentence. This may lead to certain limitations in the model's judgment of hot topics.

[0097] Both the HatabaseTwitter method and the Nlplab_isi method fine-tuned the pre-trained language model of the Transformer structure. The Transformer model can better model the contextual information in the text, especially for tasks such as hot speech detection that need to consider the overall semantics of the text. Its context modeling ability is very advantageous. The Transformer model can capture the global relationship in the text, so as to better understand the meaning of the text. The self-attention mechanism is used to model the correlation between words in different positions. This is especially important for the hot speech detection task, because the key words may be distributed in different positions of the sentence, and the attention mechanism can help the model better focus on important words and understand their contextual meaning.

[0098] The present invention not only makes full use of the advantages of the Transformer model mentioned above, but also introduces the embedding of external information, and embeds external information into the input sequence to perform semantic enhancement, enriching the semantic understanding ability of the model. The proposed hot speech detection system and method based on semantic enhancement, the HSIBSE model, achieved the highest accuracy and Macro F1 value in the experiment, thus proving that semantic enhancement can provide the model with more semantic information and contextual relationships, thereby better capturing the semantic information hidden in the text.

[0099] Experiment 2:

[0100] As shown in Table 5, in the experiment of HASOC-20 dataset, the semantically enhanced hot speech detection model proposed in the present invention achieved the highest accuracy and macro-average index, and the accuracy was improved by 2.79%, 6.57%, 0.78% and 0.38% respectively compared with GCN, TextCNN, BERT and Nlplab_isi methods.

[0101] Table 5

[0102]

[0103] The training effect of the model built based on the self-attention mechanism is significantly better than that of the TextCNN method. This is because the self-attention mechanism can better capture the internal correlation of data or features, solve the long-distance dependency problem, and enable the model to understand the contextual meaning of words in the text.

[0104] The method proposed in the present invention not only utilizes the advantages of the above-mentioned model, but also introduces external information to enhance the text semantics, so that the model can better capture semantics and extract features. In the experiment of the HASOC-20 dataset, the experimental results of these methods are better than the experimental results on the HASOC-21 dataset mentioned above. This is because the data distribution of the HASOC-20 dataset is more balanced, and the number of data in each category is close, while the data in the HASOC-21 dataset has a large difference in the number of data in the two categories. The uneven data distribution leads to the gap in experimental results.

[0105] In the experiment of HASOC-19 dataset, the method proposed in this paper achieved the highest accuracy. Although it was insufficient in the macro-average indicator, it was close to the baseline method, which also demonstrated the effectiveness of this experimental method.

[0106] Experiment 3:

[0107] It can be seen from Table 6 that in the experiment of the HASOC-19 dataset, the method based on the attention mechanism is also better than the experimental method based on the CNN network. The experimental method proposed in the present invention has improved the accuracy by 5.93%, 1.00%, 0.43% and 1.45% respectively compared with the TextCNN, GCN, BERT and Nlplab_isi methods. This once again proves that the hot speech detection system and method based on semantic enhancement proposed in the present invention have achieved good results in the experiment. Compared with other models based on the attention mechanism without adding semantic enhancement, the method of the present invention has been improved to a certain extent.

[0108] Table 6

[0109]

[0110] The experimental results of the HASOC-19 dataset are similar to those of the HASOC-21 dataset. This is because the data distribution of these two datasets also has the problem of uneven distribution. Therefore, the experimental results are significantly lower than those of the HASOC-20 dataset, where the data distribution is more even.

[0111] In order to verify whether the method proposed in the present invention to embed the information in the hot speech dictionary into sentences is effective, the present invention also feeds the sentences embedded with the hot speech dictionary information into ALBERT, RoBERTa and DeBERTa, a variety of pre-trained language models based on the BERT model for experiments to observe whether the accuracy of hot speech detection can be improved. ALBERT is an improvement on BERT, which uses a series of techniques to reduce the number of parameters and improve model efficiency and generalization ability. RoBERTa is an improvement on BERT, which uses more training data and training techniques to improve the model effect. DeBERTa is a further improvement on BERT and RoBERTa, and uses an adaptive attention mechanism and a dynamic mask mechanism to further improve the model effect. The development of these models can be said to be constantly promoting the development of natural language processing technology, so that the effect of the model on various tasks has been significantly improved.

[0112] Experiment 4:

[0113] From the experimental results in Table 7, it can be seen that the ALBERT model that integrates external information has the highest accuracy in hot speech detection and the best model performance. The RoBERTa and DeBERTa models are slightly worse, but both are higher than the model that does not integrate hot speech dictionary information. In addition, the performance of all models in terms of Accuracy and Macro F1 is very close, and both are above 0.82, indicating that their performance in the hot speech detection task is similar. This shows that the method of introducing external information proposed in the present invention can enrich the semantic information of the text and improve the semantic understanding ability of the model.

[0114] Table 7

[0115]

[0116] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A hot speech detection method based on semantic enhancement, characterized by: The following steps are involved: Step 1: Information embedding layer, embedding hot information in the hot speech dictionary into the model; First, it is necessary to integrate hot information into sentences based on the constructed hot speech dictionary; in sentences, hot speeches usually express concern about a certain event, and the content of tweets is identified based on the hot words in the hot information; effectively improve the coverage and feature dimensions of feature words, thereby improving the detection performance of the model; through the processing of the information embedding layer, the model can identify hot speeches and provide comprehensive analysis results to provide support for combating hot speeches. Step 2: Text embedding layer to perform language model prediction tasks; The model will add some special symbols "[MASK]" to the sequence. These symbols indicate that the words at the corresponding positions need to be predicted, that is, they are replaced by appropriate words by the model; that is, the missing words are predicted given the context, and the "[MASK]" symbol indicates the position of the word that needs to be predicted; Step 3: The encoding layer includes multiple Transformer modules, each of which includes: a multi-head attention layer, a feedforward neural network sublayer, and a normalization layer; Each sub-layer has a residual connection and a layer normalization step. After the text is embedded, the representation vector is sent to the multi-head attention layer, and the input is mapped into three vectors: query vector Q, key vector K and value vector V. These three vectors are used to calculate the attention score, thereby determining the weight of the input vector, and then the input vector is weighted averaged according to the weight. Step 4: Output layer: The output layer predicts whether the sentence contains hot topics. Through a fully connected layer, all hidden states of the last layer are linearly transformed and a bias term is added, and then an activation function is used to obtain the final output vector.

2. The method for detecting hot topics based on semantic enhancement according to claim 1, characterized in that: The step 1 specifically comprises the following steps: Step 1.

1. It is necessary to integrate hot information into sentences based on the constructed hot speech dictionary; in sentences, hot speech usually expresses concern about a certain event; Step 1.

2. Use the N-gram algorithm to determine whether the sentence contains hot speech words, where n ranges from 1 to 3; Step 1.

3. Determine the word w through the N-gram method i When it is a hot speech word, the corresponding hot information contained is embedded into the sentence.

3. The method for detecting hot topics based on semantic enhancement according to claim 1, characterized in that: The step 2 specifically includes the following steps: Step 2.

1. For part of the input text sequence, the model will add some special symbols "[MASK]" to the sequence. These symbols indicate that the words at the corresponding positions need to be predicted, that is, they are replaced by appropriate words by the model; Step 2.

2. Use Token Embedding to map each word in the sentence that embeds the hot speech information in this study to a vector representation of a fixed size; use Segment Embedding to distinguish the two sentences so that the model knows which words belong to which sentence; Step 2.

3. The model tokenizes the sentences and adds special tags [CLS] and [SEP] to indicate the start and end of the sentences. Step 2.

4. Use Position Embedding to encode the position information of each word into the embedding vector. The model recognizes the pattern in the sequence based on the position of the word in the sequence.

4. The method for detecting hot topics based on semantic enhancement according to claim 1, characterized in that: In step 3, the input vector is weighted averaged according to the weight, and the formula is as follows: Among them, Q, K, and V are equal; each word in the sentence must calculate attention with other words in the same sentence to learn the relevance between words; is the scaling factor, where d k is the dimension of the K vector, which is used to control the range of attention distribution.

5. The method for detecting hot topics based on semantic enhancement according to claim 1, characterized in that: In step 4, the final output vector is obtained through an activation function, and the formula is as follows: P(y|x)=σ(W·h i +b) Among them, h i represents the output vector of the last layer of the model, W and b are the weight matrix and bias vector, and σ is the probability of the sigmoid function used to output the prediction result.

6. The method for detecting hot topics based on semantic enhancement according to claim 2, characterized in that: In step 1.3, in order to better utilize the information in the hot speech vocabulary, the hot category is attached to the embedding of the sentence.

7. A hot speech detection system based on semantic enhancement according to any one of claims 1 to 6, characterized in that: It includes an information embedding layer device, a text embedding layer device, a coding layer device and an output layer device; The information embedding layer device: the main task is to embed the hot information in the hot speech dictionary into the model; Text embedding layer device: Use the BERT model to perform deep learning and embedding on text. In the BERT model, a technology called mask mechanism is used to enable the model to process single-sentence and double-sentence inputs at the same time. Encoding layer device: including Transformer modules; each module contains: multi-head attention layer, feedforward neural network sublayer and normalization layer; Output layer device: predicts whether the sentence contains hot words, through a fully connected layer, it linearly transforms all the hidden states of the last layer and adds a bias term, and then obtains the final output vector through an activation function.

8. A computer device / equipment / system comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images