A method for generating an emotion-controllable news comment

Abstract

The application provides a controllable sentiment news comment generation method, which comprises the following steps: firstly, cleaning and preprocessing a news data set; training an entity-level sentiment analysis model based on deep learning; further improving the performance of the entity-level sentiment analysis model through contrast learning; combining the pre-trained entity-level sentiment analysis model with a comment generation model to train an end-to-end controllable sentiment news comment generation model; inputting news data into the trained end-to-end controllable sentiment news comment generation model to obtain news comments with appropriate sentiment polarity and high relevance between content and news; and the comment content generated by the method is concise, clear and easy to understand, and there is no inconsistent information between news and comments.

Description

Technical Field

[0001] This invention relates to the field of deep learning, and in particular to a method for generating emotion-controlled news comments. Background Technology

[0002] Under trending news headlines, we often see people expressing their opinions on the events, which plays a significant role in shaping people's understanding of the world. However, in today's information age, the sheer volume of news can cause some important stories to be overlooked. If a technology could generate and publish commentary on these less-known but important news stories, it would encourage public participation and interaction, bringing these crucial stories into the public eye.

[0003] However, existing automated comment generation methods suffer from numerous problems, such as a severe lack of diversity and a deficiency in expressing the emotions of the news content. Such comments often lack distinctiveness and are difficult to classify as genuine human-written comments. Therefore, how to generate more authentic comments based on news is an urgent problem that needs to be solved.

[0004] Automatic comment generation is an effective technological tool that automatically generates comments by modeling news texts, playing an important role in controlling public opinion and improving human interaction. Summary of the Invention

[0005] The main objective of this invention is to overcome the aforementioned deficiencies in the prior art and propose an emotion-controlled news comment generation method. The comments generated by this method are concise, clear, and easy to understand, and there is no inconsistency between the news and the comments.

[0006] The present invention adopts the following technical solution:

[0007] A method for generating emotion-controlled news comments includes the following steps:

[0008] Collect news datasets, preprocess the news datasets, and obtain preprocessed news datasets;

[0009] An entity-level sentiment analysis model was trained using a preprocessed news dataset, resulting in a pre-trained entity-level sentiment analysis model. This model comprises five parts: a sentence encoding module that encodes the context of each sentence in the news body; a heterogeneous graph-based news encoding module that constructs relationships between sentences and words using a heterogeneous graph neural network; a sequential entity encoding module that encodes the news story sequence of entities; an attention module that captures the association between news and entities; and a prediction module that classifies sentiment based on various factors.

[0010] The pre-trained entity-level sentiment analysis model was retrained using both unsupervised and supervised methods to obtain a fully trained entity-level sentiment analysis model.

[0011] By combining a trained entity-level sentiment analysis model with a comment generation model, an end-to-end sentiment-controlled news comment generation model is trained. This model consists of three parts: the first part is a news content encoding module, which obtains a representation of the news content; the second part is a sentiment encoding module, which uses the trained entity-level sentiment analysis model to obtain a sentiment tendency representation of entities in the news, and then merges this sentiment tendency representation with the news content representation obtained from the content encoding module; the third part is a comment generation module, which uses a bidirectional decoder to generate news comments before and after the entities, and optimizes the comment generation effect based on word frequency and the position of the generated words.

[0012] By inputting news data into an end-to-end emotion-controlled news comment generation model, news comments with emotional polarity and high relevance to the content and news are obtained.

[0013] Specifically, the news dataset is preprocessed to obtain the preprocessed news dataset, as follows:

[0014] The process involves collecting news data from online news platforms, first dividing the news content into sentences using periods, extracting news entities from the content, and then removing emoticons and illegal symbols.

[0015] Specifically, an entity-level sentiment analysis model is trained using the preprocessed news dataset to obtain the pre-trained entity-level sentiment analysis model. The specific training steps are as follows:

[0016] Construct a training set and use an unsupervised method to label the sentiment tendency of each news entity in the news, and label the sentiment score f of the news entity;

[0017] A news article contains a sequence of sentences. Where s iFor words in the sentence, The length of the i-th sentence is obtained after the embedding layer. Where d w This represents the dimension of the word embedding vector. That is, the word s in the sentence sequence. i The embedding vectors are then processed by a CNN layer using convolution to capture the n-gram features of each sentence. The output of the CNN is then processed by a BiLSTM layer to capture the dependencies between sentences, finally yielding the output of the i-th sentence. Where d s This indicates the dimension of the sentence embedding vector;

[0018] Construct a heterogeneous graph data structure by creating word nodes and sentence nodes, initializing them, and then constructing edges between them with weight w. i,j = tfi,j / df(j), where tf(i,j) is the number of times word j appears in sentence i, and df(j) is the number of sentences containing word j;

[0019] The state of word nodes is updated layer by layer using an attention graph neural network.

[0020]

[0021]

[0022]

[0023] LeakyReLU is a commonly used activation function in deep learning, providing a non-linear transformation; || is the connection operation; σ is the sigmoid function; and W... a , All of these are learnable parameter matrices. It is the set of word nodes adjacent to the i-th sentence node. and This is an intermediate value in the attention graph neural network, where l represents the current layer number. This represents the updated state of the i-th sentence node at layer l+1. Sentence nodes are updated in the same way, and finally, the states of all sentence nodes obtained from the last layer of the attention graph neural network are passed through a feedforward neural network (FFN) to obtain the representation h of the news. n :

[0024]

[0025] in The state of the i-th sentence node in the last layer L of the graph neural network;

[0026] For each news item n, obtain its entity set ε(n) from the dataset. For each entity E in the entity set ε, construct a sequence q for the entity based on the news publication time. E = < n E,1 ,…,n E,t ,…,n E,N(E) > represents a time-ordered news sequence containing entity E, N(E) represents the number of news items containing entity E, and t represents the sequence number in which the news item appears in this sequence.

[0027] like Figure 1 As shown, the constructed sequence will be used as the input to the GRU. The input to the GRU at each time step is the corresponding news representation obtained in step three. For example, for q... E News in sequence n E,t Our input to the GRU is... GRU uses a reset gate and an update gate to update the hidden state. The specific process is as follows:

[0028]

[0029]

[0030]

[0031]

[0032] Among them W z W r W o Both are learnable weight matrices, z o,t and r o,t The outputs for updating the doors and resetting the doors are respectively. For candidate output states, o t This represents the actual output state, where tanh(·) and σ(·) are the tanh and sigmoid activation functions, respectively.

[0033] Here, an attention mechanism is used to capture the relationship between entities and news items, let E be an example. o E p All are entities of news n, and E o It is the entity with the longest sequence, i.e. We will E o E p The news sequence constructed from entities can be processed through the GRU network described above to obtain E. o t and E p t The calculation process for the attention mechanism is as follows:

[0034]

[0035]

[0036]

[0037] Among them W M , All are learnable parameter matrices;

[0038] The obtained relation is represented as f n and the embedding vector e of entity E∈ε(n) E By piecing these together, an MLP layer is used to predict the sentiment score of news for entities.

[0039] h = Sigmoid((f n ||e E W h +b h ),

[0040]

[0041] Among them, W h W o b is a learnable parameter matrix. h b o These are the bias vectors for the hidden layer and the output layer, respectively;

[0042] Finally, the predicted sentiment score will be... The entity-level sentiment analysis model is trained by calculating the loss between the labeled sentiment score f and the loss using the binary cross-entropy loss function.

[0043]

[0044] Specifically, the pre-trained entity-level sentiment analysis model is retrained using both unsupervised and supervised methods to obtain a fully trained entity-level sentiment analysis model, as follows:

[0045] Construct training sets, including datasets for unsupervised contrastive learning and training sets for supervised contrastive learning; the unsupervised contrastive learning dataset is... For news n i Positive examples, where m is the length of the contrastive learning training set; the training set dataset for supervised contrastive learning. For news n j , For news items with the same, non-repeating emotional polarity, News that is non-repeating and has opposite emotional polarity Datasets constituting supervised contrastive learning

[0046] The pre-trained entity-level sentiment analysis model is trained using an unsupervised method. First, the data from the unsupervised contrastive learning dataset is compared... The results were obtained by inputting the data into the pre-trained entity-level sentiment analysis model. The comparative loss is calculated using the following formula:

[0047]

[0048] Where N is the number of samples in a batch, h k Then, it represents the hidden state of other news items within a batch obtained through the entity-level sentiment analysis model. sim is the similarity calculation function, and the calculation method used here is cosine similarity. τ is the temperature coefficient, set to 0.05, and e is the natural constant.

[0049] The entity-level sentiment analysis model is trained using a supervised method. First, datasets from the supervised contrastive learning dataset are input into the unsupervised entity-level sentiment analysis model, resulting in the hidden state representation. The comparative loss is calculated using the following formula:

[0050]

[0051] Where N is the number of samples in a batch. and These are the hidden states obtained from the entity-level sentiment analysis model for positive and negative sample news items in a batch.

[0052] Specifically, a pre-trained entity-level sentiment analysis model is combined with a comment generation model to train an end-to-end sentiment-controlled news comment generation model. The training steps are as follows:

[0053] Construct a training set, filter comments containing entities, with one news item n corresponding to one comment y containing entity E, and one news title t, and filter the dataset according to 2-gram word frequency;

[0054] The news content encoding module encodes the news content. The news title `t` is input into the Transformer's encoding end. The news content encoding module includes a multi-head attention network and an FFN (Free-Forwarding Network). The formula for the multi-head attention network is as follows:

[0055]

[0056] MultiHead(Q,K,V)=[head1;…;head h W m ,

[0057] Where Attention is the calculation formula for the self-attention mechanism, as shown below:

[0058]

[0059] Where d k Let K be the length. W m All are learnable parameter matrices of a multi-head network, h=4, Q, K, and V are inputs to the news title t at the encoding end, and then passed through an FNN, with the following formula:

[0060] FFN(x) = max(0, xW1+b1)W2+b2

[0061] Where x is the input to the FFN network, W1 and W2 are learnable parameter matrices, b1 and b2 are bias vectors, and the news title t is mapped to a matrix representation Z of the content through the content encoding module. t .

[0062] The sentiment encoding module, through a trained entity-level sentiment analysis model, takes news item n and entity E as input, obtains the hidden state h of the sentiment representation, and concatenates it with the output of the content encoding module, i.e., [h; Z t ], [;] represent splicing operations;

[0063] The comment generation module mainly consists of a forward Transformer decoder and a backward Transformer decoder. Using entities in the comment as delimiters, a decoder is used before and after each entity to generate comment sequences. During training, I locate entities in the target comment sequence y, extract the sequence before the entity and reverse it, denoted as y′, which serves as the target sequence for the backward decoder. The target comment sequence y serves as the target sequence for the forward decoder. After inputting into the decoder, the final output o is obtained. V =Decoder(y), and finally use the cross-entropy loss function to calculate the loss:

[0064]

[0065] Where |V| represents the size of the vocabulary. This represents the value of the output probability distribution at the target word position. This represents the output probability distribution in dimension k;

[0066] Improvements to cross-entropy loss based on word frequency:

[0067]

[0068] where freq(token) t ) represents the word frequency of the target word in the total corpus, and λ is an adjustable hyperparameter.

[0069] Specifically, the process of inputting news data into an end-to-end emotion-controlled news comment generation model to obtain news comments with emotional polarity and high relevance to the news content involves:

[0070] Scores are calculated based on word frequency, and a beam search generation method is used. The probability calculation formula in the beam search is as follows:

[0071] p nextword =w c p nextword ,

[0072]

[0073] token t The word generated by the decoder in the current bundle is λ, which is an adjustable hyperparameter that controls the word frequency's influence on the bundle search probability.

[0074] The final beam search score is determined by the following formula:

[0075]

[0076] Where α is the length penalty factor, and different length penalty factors are used in the forward decoder and the backward decoder.

[0077] As can be seen from the above description of the present invention, compared with the prior art, the present invention has the following beneficial effects:

[0078] This invention provides a method for generating news comments with controllable sentiment. First, the news dataset is cleaned and preprocessed. Then, a deep learning-based entity-level sentiment analysis model is trained. The performance of the entity-level sentiment analysis model is further improved through contrastive learning. The pre-trained entity-level sentiment analysis model is combined with a comment generation model to train an end-to-end sentiment-controlled news comment generation model. News data is input into the trained end-to-end sentiment-controlled news comment generation model to obtain news comments with appropriate sentiment polarity and high relevance to the news content. The comments generated by this method are concise, clear, and easy to understand, with no inconsistencies between the news and the comments. Attached Figure Description

[0079] Figure 1 This is a flowchart of an emotion-controlled news comment generation method proposed in an embodiment of the present invention.

[0080] Figure 2 This is the entity-level sentiment analysis model proposed in the embodiments of the present invention.

[0081] Figure 3 This is an emotion-controlled news comment generation model proposed in this embodiment of the invention.

[0082] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Detailed Implementation

[0083] like Figure 1 This invention provides a method for generating emotion-controlled news comments. The core idea of ​​this invention is to propose an emotion-controlled news comment generation model. This model combines a news comment generation model and an entity-level sentiment analysis model to generate emotion-controlled news comments. Step 101 collects news data; steps 102 and 103 respectively use graph neural networks and contrastive learning to train the entity-level sentiment analysis model; step 104 trains an end-to-end model combining the entity-level sentiment analysis model and the news comment generation model. Finally, in step 105, the trained parameters are used for actual output.

[0084] Specifically, the following steps are included:

[0085] S101: Collect news data and clean and preprocess the news dataset.

[0086] News data was collected from online news platforms and then cleaned and preprocessed. First, the news body was segmented into sentences using periods, entities were extracted, and emoticons and other illegal symbols were removed. Simultaneously, the news text was analyzed and converted into a list format. The final sample of collected news data after simple preprocessing is shown below:

[0087] S102: Using the processed news dataset, train an entity-level sentiment analysis model to detect the sentiment towards entities appearing in the news. This sentiment model ( Figure 2The algorithm mainly consists of five parts. The first part is the sentence encoding module, which encodes the context of each sentence in the news body to represent local factors for entity-level sentiment classification. The second part is the heterogeneous graph-based news encoding module, which uses a heterogeneous graph neural network to construct relationships between sentences and words to capture short-term factors for entity-level sentiment classification. The third part is the sequential entity encoding module, which encodes the news story sequence of entities to represent long-term factors for entity-level sentiment classification. The fourth part is the attention module, which captures the associations between news and entities to represent global factors for entity-level sentiment classification. The fifth part is the prediction module, which integrates various factors to perform a final sentiment classification.

[0088] The preprocessed news dataset is used to train an entity-level sentiment analysis model to detect the sentiment towards entities appearing in the news. The training is performed according to the following steps:

[0089] S1021: Construct a training set and use an unsupervised method to label the sentiment of each entity in the news. The news body has been divided into multiple sentences. If a sentence contains the entity, its sentiment score is labeled using an open-source sentiment analysis tool. After labeling, these scores are averaged to obtain the sentiment score of the news for that entity.

[0090] S1022: First, for a news article, which contains several sentence sequences... Where s i For words in the sentence, Let be the length of the i-th sentence. After passing it through the embedding layer, we can obtain... Where d w This represents the dimension of the word embedding vector. That is, the word s in the sentence sequence. i The embedding vector is then used. Next, a CNN layer employs convolution to capture the n-gram features of each sentence. The CNN output is then passed through a BiLSTM layer to capture the dependencies between sentences, finally yielding the output of the i-th sentence. Where d s This indicates the dimension of the sentence embedding vector.

[0091] S1023: Construct the data structure for the heterogeneous graph, building nodes for word and sentence types respectively. The initial state of sentence nodes is obtained using the method from the previous step. Initialization: the initial state of the word nodes is determined using the word embedding vectors. initialization, Next, we construct the edges between word nodes and sentence nodes. If the i-th sentence contains the j-th word, then there is an edge between them. The weight of the edge is calculated using TF-IDF, which is equivalent to TF-IDF. i,j = tfi,j / df(j), where tf(i,j) is the number of times word j appears in sentence i, df(j) is the number of sentences containing word j, and the resulting edge weight is denoted as w. i,j Next, we use an attention graph neural network and update the state of the sentence nodes layer by layer:

[0092]

[0093]

[0094]

[0095] LeakyReLU is a commonly used activation function in deep learning, providing a non-linear transformation; || is the connection operation; σ is the sigmoid function; and W... a , All of these are learnable parameter matrices. It is the set of word nodes adjacent to the i-th sentence node. and This is an intermediate value in the attention graph neural network, where l represents the current layer number. This represents the updated state of the i-th sentence node at layer l+1. Sentence nodes are updated in a similar way. Finally, we pass all the sentence node states obtained from the last layer of the attention graph neural network through a feedforward neural network (FFN) to obtain the representation h of the news. n :

[0096]

[0097] in The state of the i-th sentence node in the last layer L of the graph neural network.

[0098] S1024: For each news item n, we can obtain its entity set ε(n) from the dataset. For each entity E in the entity set ε, a sequence q will be constructed for the entity based on the news publication time. E = < n E,1 ,…,n E,t ,…,n E,N(E) > represents a time-ordered news sequence containing entity E, N(E) represents the number of news items containing entity E, and t represents the sequence number of the news item in this sequence. For example... Figure 1As shown, the constructed sequence will be used as the input to the GRU. The input to the GRU at each time step is the corresponding news representation obtained in step three. For example, for q... E News in sequence n E,t Our input to the GRU is... GRU uses a reset gate and an update gate to update the hidden state. The specific process is as follows:

[0099]

[0100]

[0101]

[0102]

[0103] Among them W z W r W o Both are learnable weight matrices, z o,t and r o,t The outputs for updating the doors and resetting the doors are respectively. For candidate output states, o t This represents the actual output state, where tanh(·) and σ(·) are the tanh and sigmoid activation functions, respectively.

[0104] S1025: Because each news item contains multiple entities, and each entity is related to multiple news items, an attention mechanism is used here to capture the relationship between entities and news items. Let E... o E p All are entities of news n, and E o It is the entity with the longest sequence, that is... We will E o E p The news sequence constructed from entities can be processed through the GRU network described above to obtain E. o t and E p t The calculation process for the attention mechanism is as follows:

[0105]

[0106]

[0107]

[0108] Among them W M , All of these are learnable parameter matrices.

[0109] S1026: Finally, we will use the representation f obtained in the previous step. n and the embedding vector e of entity E∈ε(n) E By piecing these together, and using an MLP layer, the sentiment of the news towards that entity can be predicted.

[0110] h = Sigmoid((f n ||e E W h +b h ),

[0111]

[0112] W h W o b is a learnable parameter matrix. h b o These are the bias vectors for the hidden layer and the output layer, respectively. Finally, the predicted sentiment score is... The entity-level sentiment analysis model is trained by calculating the loss between the pre-labeled sentiment score f and the pre-labeled sentiment score f using a binary cross-entropy loss function.

[0113]

[0114] Model diagram as follows Figure 2 As shown, before training, the dataset first needs to be labeled. If a sentence contains the entity, its sentiment score is labeled using an open-source sentiment analysis tool. After labeling, these scores are averaged to obtain the sentiment score of the news article for that entity. The format of the labeled dataset is as follows:

[0115] S103: Continue training the entity-level sentiment analysis model after it has been trained. Enhance the performance of the entity-level sentiment analysis model through comparative learning. This method is mainly divided into supervised and unsupervised methods.

[0116] The performance of the entity-level sentiment analysis model is further improved through contrastive learning. First, the dataset is processed, and then trained according to the following steps:

[0117] S1031: Construct training sets, including training sets for unsupervised and supervised contrastive learning. First, for each news item n, discard some words with probability P=0.1 and use these discarded words as positive examples for contrastive learning, forming a data pair with the original news items. m is the length of the contrastive learning training set. Next, we construct the supervised contrastive learning training set. Given the same entity E, we select news items with the same and opposite sentiment polarities as a single news item. Specifically, we first select all news items containing entity E and then evaluate the sentiment score of each news item for entity E. The score ranges from 0 to 10. If the score is greater than or equal to 5, we consider the news item to have a positive sentiment towards entity E; if it is less than 5, we consider the news item to have a negative sentiment towards entity E. After evaluation, we assign a value to any news item n. j Choose news items with the same, non-repeating emotional polarity. News that is opposite to emotional polarity Datasets constituting supervised contrastive learning

[0118] S1032: The entity-level sentiment analysis model is further trained using an unsupervised approach. First, we use the unsupervised contrastive learning data to... The values ​​are input into the entity-level sentiment analysis model initially trained in step 2). Instead of using the final sentiment score, we take the hidden state h of the MLP layer as the target to be approximated in the contrastive learning. This allows us to obtain the values ​​respectively. The comparative loss is calculated using the following formula:

[0119]

[0120] Where N is the number of samples in a batch, h k Then, it represents the hidden state of other news items within a batch obtained through the entity-level sentiment analysis model. sim is the similarity calculation function, and the calculation method used here is cosine similarity. τ is the temperature coefficient, set to 0.05, and e is the natural constant.

[0121] S1033: The entity-level sentiment analysis model is further trained using a supervised method. First, the supervised contrastive learning data sets are input into the entity-level sentiment analysis model preliminarily trained in step 2), resulting in the hidden state representation as follows. The comparative loss is calculated using the following formula:

[0122]

[0123] Where N is the number of samples in a batch. and These are the hidden states obtained from the entity-level sentiment analysis model for positive and negative sample news items in a batch.

[0124] Unsupervised methods directly mask some words in the original news with a probability of P=0.1, using these as data pairs for unsupervised contrastive learning. For supervised contrastive learning, taking the data format of step 102 as an example, suppose the currently selected news entity is "Nazha," and the labeled score is 9.9. Since the score range is 0-10, 9.9 is greater than 5, so we can consider the sentiment of this news towards the entity "Nazha" to be positive. To construct supervised contrastive learning data pairs, we need to find other news articles that all contain the entity "Nazha." If any of these news articles has a score of 5 or greater for the entity "Nazha," we use this news article as a positive sample for contrastive learning; if any of these news articles has a score less than 5 for the entity "Nazha," we use this news article as a negative sample for contrastive learning. Then, we can further train the entity-level sentiment analysis model through supervised contrastive learning.

[0125] This invention compares the entity-level sentiment analysis model trained in steps 102 and 103 with various existing entity-level sentiment analysis models. The comparison methods are: (1) SVM. Support Vector Machine, a traditional supervised machine learning method. (2) TD-LSTM. An improved model based on LSTM, which uses the aspect of the sentence as a keyword, passes the sequences before and after it through LSTM networks, and uses the hidden state of the last time step to determine the category of the text through a classification layer. (3) TC-LSTM. Based on TD-LSTM, the information of the keyword is added to the LSTM networks before and after it, and combined with the hidden state of the time step. (4) AF-LSTM. Fast Fourier Transform is used to combine the aspect keyword with other words in the sentence sequence. An attention network is used to make the model pay attention to the relationship between the aspect keyword and each word in the sentence, so as to perform better aspect-level text classification. (5) ASGCN. A syntax tree is used to model the original sentence, and a graph neural network is constructed to capture the relationship between sentence words and aspects. The results of graph neural network modeling are used to classify the text. (6) BERT-single. The results at [CLS] in BERT were used for classification. Table 1 shows a comparison between the entity-level sentiment analysis model and other models. Experimental results demonstrate that the entity-level sentiment analysis model can better capture the sentiment tendency of news for a given entity.

[0126] Table 1: Comparison of Entity-Level Sentiment Analysis Model with Other Models

[0127]

[0128]

[0129] S104: By combining a pre-trained entity-level sentiment analysis model with a comment generation model, an end-to-end sentiment-controlled news comment generation model is trained. This model consists of three main parts: the first is a news content encoding module, which takes the news title as input to obtain a representation of the news content; the second is a sentiment encoding module, which uses the pre-trained entity-level sentiment analysis model to obtain the sentiment tendency towards entities in the news and merges the sentiment tendency representation with the content representation obtained from the content encoding module; the third is a comment generation module, which uses a bidirectional decoder to generate news comments before and after the entity, and optimizes the comment generation effect based on word frequency and the position of the generated words.

[0130] By combining a pre-trained entity-level sentiment analysis model with a comment generation model, an end-to-end sentiment-controlled news comment generation model can be trained, such as... Figure 3 As shown, the training should be performed in the following steps:

[0131] S1041: Construct a training set, filter comments containing entities, one news item n corresponds to one comment y containing entity E, and a news title t, and filter the dataset according to 2-gram word frequency, with the goal that the 2-gram word frequency of the filtered dataset will not show a large difference.

[0132] S1042: The news content encoding module encodes the news content. The news title t is input into the Transformer's encoding end. It includes a multi-head attention network and an FFN, where the multi-head attention network formula is as follows:

[0133]

[0134] MultiHead(Q,K,V)=[head1;…;head h W m ,

[0135] Where Attention is the calculation formula for the self-attention mechanism, as shown below:

[0136]

[0137] Where d k Let K be the length of K. W m All are learnable parameter matrices of a multi-head network. We use 4 heads, i.e., h=4. Q, K, and V are all inputs to the news title t at the encoding end. Next, we pass it through an FNN, the formula of which is as follows:

[0138] FFN(x) = max(0, xW1+b1)W2+b2,

[0139] Where x is the input to the FFN network, W1 and W2 are learnable parameter matrices, and b1 and b2 are bias vectors. Finally, the title t is mapped to a matrix representation Z of the content representation through the content encoding module. t .

[0140] S1043: The sentiment encoding module, after inputting news n and entity E into a pre-trained entity-level sentiment analysis model, obtains the hidden state h of the sentiment representation, which is then concatenated with the output of the content encoding module, i.e., [h||Z t The symbol ] guides the output of the comment generation module, while || represents the concatenation operation. The comment generation module mainly consists of a forward Transformer decoder and a backward Transformer decoder. Here, forward represents the direction of generation in the forward order of the sentences, and backward represents the direction of generation in the reverse order. Our comment generation module needs to generate comments based on entities appearing in the news. Therefore, we use the entities in the comments as dividing points, using a decoder before and after each entity to generate comment sequences. The advantage of this method is that it ensures that the final comment always contains an entity, without imposing a qualitative requirement on the entity's position. Specifically, during training, we need to locate the entities in the target comment sequence, extract the sequence before the entity (including the entity itself), and arrange it in reverse order, denoted as y′. This will serve as the target sequence for the backward decoder, while the original target comment sequence y will serve as the target sequence for the forward decoder. After inputting into the decoder, the final output o is obtained. V =Decoder(y), and finally use the cross-entropy loss function to calculate the loss:

[0141]

[0142] Where |V| represents the size of the vocabulary. This represents the value of the output probability distribution at the target word position. This represents the output probability distribution in dimension k. The cross-entropy loss formula is also the most commonly used loss function in sequence-to-sequence models. However, it assigns weights fairly to each word. This means that if some words appear frequently in the dataset, calculating the loss with the same weights would be unfair, potentially causing the model to tend to generate more frequent words, thus affecting the diversity of the final output sequence. Therefore, this work adds improvements to the cross-entropy loss based on word frequency, as shown in Equation 3-13:

[0143]

[0144] We added a term related to word frequency to the denominator of the cross-entropy loss function, where freq(token) t ) represents the word frequency of the target word in the total corpus, and λ is an adjustable hyperparameter. The purpose is to reduce the loss when some high-frequency words appear, so that the optimization of model parameters is less. Conversely, for some low-frequency words, the loss when they appear can be increased, expanding the optimization range of model parameters, thereby weakening the impact of word frequency in the original corpus on the diversity of model output sequences.

[0145] S105: In the application phase, news data is processed through an end-to-end sentiment-controlled news comment generation model to obtain news comments with appropriate sentiment polarity and high relevance to the news content.

[0146] News data is input into a pre-trained end-to-end sentiment-controlled news comment generation model. Since the generation process uses an autoregressive approach, we calculate scores based on word frequency and employ a beam search generation method. The probability calculation formula in the beam search is as follows:

[0147] p nextword =w c p nextword ,

[0148]

[0149] token t For the words generated by the decoder in the current bundle, the parameter λ controls the influence of word frequency on the bundle search probability. Here, the parameter λ is the same as the word frequency control parameter λ used when training the comment generation model in step 4). The final bundle search score is determined by the following formula:

[0150]

[0151] Here, α is the length penalty factor, with different length penalty factors used in the forward and backward decoders. Since entities in news comments are usually positioned near the beginning, the backward sequence is generally shorter than the forward sequence. We first generate comments preceding the entities based on the forward decoder, then input these comments as prompts into the backward decoder to generate comments following the entities. This allows us to obtain news comments with appropriate sentiment polarity and high relevance to the news content.

[0152] This invention compares a sentiment-controlled news comment generation system with various existing comment generation systems using the following comparison methods: (1) LSTM-T. The traditional LSTM seq2seq model, where both the Encoder and Decoder are composed of LSTM, and the Encoder's input sequence is a sequence of news headlines. (2) LSTM-B. Similarly, the Encoder's input sequence is a sequence of news text. (3) LSTM+Attention-T. An Attention mechanism is added to the LSTM seq2seq model, and the Encoder's input sequence is a sequence of news headlines. (4) LSTM+Attention-B. Similarly, the Encoder's input sequence is a sequence of news text. (5) ControlVAE. A comment generator is constructed using VAE, and then an adversarial training is performed using a GAN network to control the sentiment of the generated text. (6) Self-Attention. The Encoder uses a Transformer, and the Decoder uses a regular LSTM. (7) Graph2seq. The Encoder transforms the news text and title into a graph data structure according to certain rules, and uses a graph neural network to model and capture information from the news. The Decoder uses LSTM and Attention mechanisms to notice information from previous news and generate appropriate news comments. Table 2 shows that the proposed model of this system has the highest ROUGE and BLEU scores.

[0153] Table 2: Comparison of ROUGE and BLEU scores among various comment generation systems

[0154]

[0155]

[0156] Furthermore, automatic readability assessments revealed that the models excluded by this patent exhibited better readability compared to others. Table 3 compares the automatic readability assessment metrics of various news comment generation systems. A higher Flesch Reading score indicates better readability, while lower scores for Linsearwrite, Gunningfog, and Textstandard are better. For example, a score of 8.2 indicates that an eighth-grade student could understand the text.

[0157] Table 3: Readability Automatic Assessment Metrics

[0158] method Flesch Reading Linsearwrite Gunningfog Textstandard LSTM-T 112.51 3.803 3.824 2.904 LSTM-B 112.02 3.835 3.924 3.043 LSTM-AT 114.01 3.05 3.24 2.56 LSTM-AB 113.53 3.09 3.24 2.67 ControlVAE 113.94 3.05 3.24 2.52 Self-Attention 112.1 3.98 4.01 3.11 Graph2seq 113.68 3.225 3.37 2.292 Ours 115.96 2.093 2.47 2.154

[0159] The most important feature of the comment generation system of this invention is that it can generate comments with more appropriate sentiment polarity than other systems. Therefore, in Table 4, we compare the sentiment control accuracy of the sentiment-controllable news comment generation system of this invention with other comment generation systems:

[0160] Table 4: Comparison Results of Comment Generation Accuracy

[0161]

[0162]

[0163] As the results in the table show, our model generates news comments that are no less readable and have lower co-occurrence metrics than other models, while achieving excellent control over comment sentiment. This is because sentiment-controlled comment generation systems are relatively rare; most of the benchmark works compared above directly generate comments without further sentiment control. Therefore, it can be said that the sentiment-controlled news comment generation system proposed in this work can effectively generate sentiment-controlled comments based on news and outperforms the vast majority of existing known systems.

[0164] Finally, an example of emotion-controlled news comment generation is given in the application stage. Table 5 contains a comparison between comments generated by the system of this invention and comments generated by other models:

[0165] Table 5 Comparison of comments output by different comment generation systems

[0166]

[0167] The above are merely specific embodiments of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial modifications made to the present invention using this concept shall be considered as infringing upon the protection scope of the present invention.

Claims

1. A method for generating news comments with controllable emotion, characterized in that, Includes the following steps: Collect news data and generate a news dataset. Preprocess the news dataset to obtain a preprocessed news dataset. An entity-level sentiment analysis model was trained using a preprocessed news dataset, resulting in a pre-trained entity-level sentiment analysis model comprising five parts: The first part is a sentence encoding module, which encodes each sentence in the news body to represent local factors for entity-level sentiment classification; the second part is a heterogeneous graph-based news encoding module, which constructs relationships between sentences and words based on a heterogeneous graph neural network to capture short-term factors for entity-level sentiment classification; the third part is a sequential entity encoding module, which encodes the sequence of news entities to represent long-term factors for entity-level sentiment classification; the fourth part is an attention module, which captures the associations between news items and entities to represent global factors for entity-level sentiment classification; and the fifth part is a prediction module, which integrates various factors to classify entity-level sentiment. The pre-trained entity-level sentiment analysis model was retrained using both unsupervised and supervised methods to obtain a fully trained entity-level sentiment analysis model. By combining a trained entity-level sentiment analysis model with a comment generation model, an end-to-end sentiment-controlled news comment generation model is trained. This model consists of three parts: the first part is a news content encoding module, which inputs the news title into the encoding module to obtain a representation of the news content; the second part is a sentiment encoding module, which uses the trained entity-level sentiment analysis model to obtain a sentiment tendency representation of the entities in the news, and then merges this sentiment tendency representation with the news content representation obtained from the news content encoding module; the third part is a comment generation module, which uses a bidirectional decoder consisting of a forward Transformer decoder and a backward Transformer decoder to generate news comments before and after the entities, and optimizes the comment generation effect based on word frequency and the position of the generated words. By inputting news data into an end-to-end emotion-controlled news comment generation model, news comments with emotional polarity and content relevance to the news are obtained.

2. The method for generating emotion-controlled news comments as described in claim 1, characterized in that, News data is collected and a news dataset is generated. This news dataset is then preprocessed to obtain a preprocessed news dataset, specifically: To collect news data from online news platforms, the news content is first divided into sentences using periods, news entities are extracted, and emoticons and illegal symbols are removed.

3. The method for generating emotion-controlled news comments as described in claim 1, characterized in that, The pre-processed news dataset is used to train an entity-level sentiment analysis model, resulting in a pre-trained entity-level sentiment analysis model. The specific training steps are as follows: A training set is constructed, and the sentiment tendency of each news entity in the news is labeled in an unsupervised manner, and the sentiment score of the news entity is labeled. ; A news article contains a sequence of sentences. ,in For the first One sentence. The length of the i-th sentence is obtained after the embedding layer. , ,in This represents the dimension of the word embedding vector. That is, words in the sentence sequence The embedding vectors are then processed by a CNN layer using convolution to capture the n-gram features of each sentence. The output of the CNN is then processed by a BiLSTM layer to capture the dependencies between sentences, finally outputting the embedding vector of the i-th sentence. ,in This indicates the dimension of the sentence embedding vector; Construct a heterogeneous graph data structure by creating word nodes and sentence nodes, initializing them, and then constructing edges between them with weights of 1 / 2 and 2 / 3. , It is a word Appears in sentence The number of times in, It contains words The number of sentences; The state of word nodes is updated layer by layer using an attention graph neural network. ， ， ， in, , , It is an intermediate quantity in attention graph neural networks. Represents the current floor number. It is a commonly used activation function in deep learning, providing non-linear transformations. It's a splicing operation. It is the embedding vector of the i-th sentence in the l-th layer, initialized as follows: , It is the embedding vector of word j. It is the Sigmoid function. , , , All of these are learnable parameter matrices. It is the set of word nodes adjacent to the i-th sentence node. Finally, the embedding vectors of all sentence nodes obtained from the last layer of the attention graph neural network are passed through a feedforward neural network (FFN) to obtain the news representation of the news. : ； in The state of the i-th sentence node in the last layer L of the graph neural network; For each news item n, obtain its entity set from the dataset. For entity sets Each entity in They will construct a news sequence for each entity based on the time of its news release. , representing entities All news in an orderly fashion The number of news items containing entity E is represented by t, where t represents the sequence number of the news item in this sequence. The constructed sequence will be used as input to the GRU. The input to the GRU at each time step is the corresponding news representation. News in sequence The input to the GRU is... GRU uses a reset gate to update the hidden state. The specific process is as follows: ， ， ， ， in , , All are learnable weight matrices. and The outputs for updating the door and resetting the door are respectively. This is a candidate output state. This is the actual output state. and These are the tanh and sigmoid activation functions, respectively. Employing an attention mechanism to capture the relationship between entities and news items, let... , All are news articles. The entities contained therein, and It is the entity with the longest sequence, i.e. ,Will , The news sequence constructed from entities is obtained after passing through a GRU network. and The computational process of the attention mechanism is as follows: ， ， ， in , , , All are learnable parameter matrices; The obtained relation representation and entity Embedded vector By piecing these together, an MLP layer is used to predict the sentiment score of news for entities. : ， ， in, , For a learnable parameter matrix, , These are the bias vectors for the learnable hidden layer and the output layer, respectively. Finally, the predicted sentiment score will be... With the labeled sentiment score The loss is calculated using the binary cross-entropy loss function, and this loss is used to train the entity-level sentiment analysis model. 。 4. The method for generating emotion-controlled news comments as described in claim 3, characterized in that, The pre-trained entity-level sentiment analysis model is retrained using both unsupervised and supervised methods to obtain a fully trained entity-level sentiment analysis model, specifically as follows: Construct training sets, including datasets for unsupervised contrastive learning and training sets for supervised contrastive learning; the unsupervised contrastive learning dataset is... , For News Positive examples, where m is the length of the contrastive learning training set; the training set dataset for supervised contrastive learning. Regarding the news , For non-repeating and with News with the same emotional polarity News with opposite emotional polarities that are not repeated; The pre-trained entity-level sentiment analysis model is trained using an unsupervised method. First, the data from the unsupervised contrastive learning dataset is compared... The results were obtained by inputting the data into the pre-trained entity-level sentiment analysis model. The comparative loss is calculated using the following formula: ， Where N is the number of samples in a batch. This represents the hidden states of other news items within a batch, obtained through an entity-level sentiment analysis model. `sim` is the similarity calculation function, and the method used here is cosine similarity. It's the temperature coefficient, set to 0.

05. It is a natural constant; The entity-level sentiment analysis model is trained using a supervised method. First, datasets from the supervised contrastive learning dataset are input into the unsupervised entity-level sentiment analysis model, resulting in the hidden state representation. The comparative loss is calculated using the following formula: ， Where N is the number of samples in a batch. and These are the hidden states obtained from the entity-level sentiment analysis model for positive and negative sample news items in a batch.

5. The method for generating emotion-controlled news comments as described in claim 1, characterized in that, By combining a pre-trained entity-level sentiment analysis model with a comment generation model, an end-to-end sentiment-controlled news comment generation model is trained. The training steps are as follows: Build a training set, filter comments containing entities, and select a news article. Corresponding to a line containing entities Comments and a news headline The dataset is filtered based on the frequency of bigrams in the comments, prioritizing the deletion of data samples containing bigrams with high frequency. The news content encoding module encodes the content of the news and the news title. The news content encoding module, fed into the Transformer's encoding end, includes a multi-head attention network and an FFN, with the multi-head attention network formula as follows: ， ， in The calculation formula for the self-attention mechanism is as follows: ， Where d_k is Length, , , , Both are learnable parameter matrices for multi-head networks, h=4. , , All are news headlines from the encoding end. The input is then processed through an FNN, and the formula is as follows: ； in For the input of the FFN network, , For a learnable parameter matrix, , For the bias vector, the news title The matrix representation is mapped to content representation through the content encoding module. ; The sentiment encoding module uses a trained entity-level sentiment analysis model to input news. and entity The hidden state of emotional representation is then obtained. And concatenate it with the output of the content encoding module, that is , For splicing operations; The comment generation module mainly consists of a forward Transformer decoder and a backward Transformer decoder. It uses entities in the comment as delimiters, employing a decoder before and after each entity to generate the comment sequence. During training, the target comment sequence is located. The entity in the sequence is extracted and reversed, and represented as: And as the target sequence for the backward Transformer decoder, the target comment sequence The target sequence for the forward Transformer decoder; for the p-th position, the outputs of the forward and backward decoders are represented as follows: Where TransformerF represents the forward Transformer decoder and TransformerB represents the backward Transformer decoder. It is the probability distribution of |V| words in the vocabulary, and finally the cross-entropy loss function is used to calculate the loss: ； in Indicates the size of the dictionary. This represents the value of the word at position p in the target sequence y of the forward decoder. This represents the value of the word at position p in the target sequence y' of the backward decoder. This represents the probability value of the output probability of the forward Transformer decoder in word k. This is the probability value of the output probability of the backward Transformer decoder in word k; Improvements to cross-entropy loss based on word frequency: ； in This represents the word frequency of the target word in the overall corpus. These are adjustable hyperparameters.

6. The method for generating emotion-controlled news comments as described in claim 5, characterized in that, The process of inputting news data into an end-to-end emotion-controlled news comment generation model to obtain news comments with emotional polarity and high relevance to the news content is as follows: Scores are calculated based on word frequency, and a beam search generation method is used. The probability calculation formula in the beam search is as follows: ， ， in For the words generated by the decoder in the current bundle, the parameters These are adjustable hyperparameters that control the impact of word frequency on the beam search probability. The final beam search score is determined by the following formula: ； in The length penalty factor is used in the forward decoder and the backward decoder, respectively.

Images